Runtime architecture

vllm-mlx adapts vLLM-style serving concepts to MLX. The server owns protocol compatibility and request policy, engines own generation, schedulers own concurrent decode state, and model wrappers bridge to the MLX ecosystem.

Layer map

OpenAI SDKs        Anthropic SDKs        curl / custom clients
      |                  |                        |
      +------------------+------------------------+
                         |
                 FastAPI server.py
       auth, limits, validation, protocol adapters
                         |
                BaseEngine contract
                  /             \
           SimpleEngine      BatchedEngine
             direct          AsyncEngineCore
                                  |
                              Scheduler
                 queues, BatchGenerator, KV caches
                                  |
             mlx-lm | mlx-vlm | mlx-audio | mlx-embeddings
                                  |
                        MLX and Metal

API and protocol layer

vllm_mlx.server creates the FastAPI application and implements the OpenAI-compatible, Anthropic-compatible, audio, embedding, reranking, cache, status, and MCP routes. Its main responsibilities are:

1. Authenticate and rate-limit requests when configured.
2. Validate the requested model and endpoint-specific limits.
3. Normalize OpenAI, Anthropic, Responses API, multimodal, and tool inputs.
4. Resolve the active model and acquire a model lease when registry serving is enabled.
5. Construct sampling, reasoning, structured-output, and tool-parser state.
6. Invoke the selected engine and translate GenerationOutput values into protocol responses.
7. Preserve terminal reasons, usage, cancellation, and model release across normal and exceptional exits.

The Pydantic wire models live in vllm_mlx.api. Protocol conversion is deliberately separate from model execution so the same engines can support multiple client contracts.

Engine layer

BaseEngine defines the common async contract for loading, stopping, text generation, chat generation, streaming, cache management, and tokenizer access.

Two primary implementations serve different workloads:

Engine	Execution model	Best fit	Main tradeoff
SimpleEngine	Direct calls into model wrappers	One active user, lowest orchestration overhead	Serialized generation paths do not continuously batch independent requests
BatchedEngine	Delegates to AsyncEngineCore and a scheduler	Concurrent serving and aggregate throughput	More queue, cache, and lifecycle state

Both return GenerationOutput, which carries text, token IDs, token counts, finish reason, incremental text, completion state, and speculative decoding counters.

Continuous-batching core

EngineCore owns a model, tokenizer, scheduler, output collectors, and request completion events. AsyncEngineCore wraps it with the async interface used by BatchedEngine.

The core runs scheduler steps on one dedicated worker thread. MLX streams are thread-local, so generation streams are rebound inside that worker before decode. This thread-affinity requirement is a correctness invariant, not only a performance choice.

Scheduler layer

Scheduler turns waiting requests into a running batch, advances mlx-lm's BatchGenerator, collects deltas, and finalizes completed or failed requests. Its configuration controls maximum sequences, batch sizes, prefill step size, scheduling policy, cache strategy, memory limits, and optional SSD tiering.

Multimodal continuous batching uses MLLMScheduler, MLLMBatchGenerator, and MultimodalProcessor. These components preserve image and video preprocessing state while applying the same request lifecycle concepts.

Model layer

• MLXLanguageModel wraps text-only mlx-lm loading and generation.
• MLXMultimodalLM wraps mlx-vlm models and multimodal preprocessing.
• EmbeddingEngine serves vector embeddings.
• RerankEngine serves cross-encoder scores.
• STTEngine and TTSEngine provide optional audio routes.

Runtime patches under vllm_mlx.patches adapt specific upstream architectures. They should stay model-specific and must not silently alter unrelated model families.

Parser and constraint layer

Reasoning parsers split thinking content from final content. Tool parsers translate model-specific call syntax into OpenAI-compatible tool deltas. Constrained processors modify logits to enforce JSON Schema or to control a reasoning model's transition into final content.

These components can suppress or buffer individual deltas. Streaming code must therefore treat parser finalization and the terminal finish-reason chunk as independent obligations.

Ownership boundaries

Several resources have explicit owners:

Resource	Owner	Release point
Model and tokenizer	Engine or registry-managed loaded model	Engine stop, registry eviction, or residency unload
Request state	Scheduler and engine core	Completion, cancellation, or failure
Model lease	Request model context	Response cleanup, including streaming disconnects
Prefix and paged KV state	Scheduler cache managers	Cache clear, reset, eviction, or engine close
Parser state	Individual request stream	Terminal flush or request cleanup
MCP processes	MCP manager	Server lifespan shutdown

Keeping these boundaries intact prevents model eviction during generation, stale parser state after reload, leaked request futures, and cache state surviving an incompatible model.

Static documentation on Linux

The documentation pipeline parses source with the Python AST and Griffe. It does not import vllm_mlx, so GitHub Pages can build on Linux without MLX. Runtime validation remains an Apple Silicon responsibility.