Skip to content

Architecture

vllm-mlx is a layered inference server for Apple Silicon. Protocol code is separated from model execution, concurrent scheduling, caching, model ownership, and model-specific compatibility patches.

System overview

OpenAI and Anthropic clients
            |
      FastAPI server
            |
  protocol normalization
            |
       BaseEngine
       /        \
SimpleEngine  BatchedEngine
                   |
             AsyncEngineCore
                   |
               Scheduler
                   |
        model wrappers and caches
                   |
 mlx-lm | mlx-vlm | mlx-audio | mlx-embeddings
                   |
              MLX / Metal

Primary layers

API layer

server.py owns the FastAPI process, route handlers, authentication, rate limits, endpoint policy, model acquisition, streaming protocol output, and shutdown integration. Pydantic requests, responses, and protocol adapters live in api/.

Engine layer

BaseEngine is the common contract used by the server. SimpleEngine calls model wrappers directly. BatchedEngine delegates concurrent work to AsyncEngineCore.

Scheduler layer

Scheduler manages waiting and running requests, mlx-lm BatchGenerator state, prefill and decode steps, cache attachment, cancellation, recovery, and terminal outputs. The engine core routes scheduler results into per-request output collectors.

Multimodal batching uses separate scheduler, batch generator, processor, and cache components because vision inputs and cache shapes differ from text-only inference.

Model layer

Text generation uses MLXLanguageModel. Vision-language generation uses MLXMultimodalLM. Embedding, reranking, STT, and TTS engines remain separate optional services with endpoint-specific compatibility policy.

Cache layer

The scheduler can use legacy prefix entries, memory-aware entries, block-aware prefix reuse, or paged KV blocks, with optional SSD persistence. Multimodal and vision-embedding caches cover different preprocessing state. See Caching for invariants and selection guidance.

Parser and constraint layer

Reasoning parsers separate hidden thinking from final content. Tool parsers convert model-family syntax into protocol tool calls. Constrained processors enforce JSON Schema and reasoning-budget transitions at the logits level. See Parsing and Structured Output.

Model ownership layer

ModelManager provides registry-backed multi-model loading, memory budgets, eviction, and request leases. ResidencyManager provides lazy loading and idle unload for the default model.

Request flow

  1. Middleware authenticates, meters, and validates transport policy.
  2. The endpoint validates the request model and protocol schema.
  3. Protocol input is normalized into a prompt or internal message list.
  4. The server builds sampling, reasoning, tool, and structured-output state.
  5. The request acquires its model or resident engine.
  6. The simple engine executes directly, or the batched engine submits to the scheduler.
  7. Deltas pass through reasoning and tool parsers before protocol encoding.
  8. Terminal reason and usage are emitted even when the final textual delta was suppressed.
  9. Completion, error, timeout, cancellation, and disconnect paths release request and model state.

See Request Lifecycle for the complete path.

Concurrency invariants

  • MLX generation streams are thread-local. Scheduler steps stay on one bound worker thread.
  • One model cannot be owned by incompatible active engines unless ownership is transferred through the registry contract.
  • A request-scoped model lease prevents eviction during generation.
  • Request collectors receive one terminal result.
  • Streaming cleanup runs in finally paths because clients can disconnect between yields.

Cache invariants

  • Cache identity includes model compatibility, not only token equality.
  • Expired entries cannot satisfy exact or partial prefix lookups.
  • Memory accounting changes once for each inserted or removed entry.
  • Paged block reference counts agree with block-table ownership and free-list membership.
  • Non-trimmable layers are never shortened to manufacture a prefix hit.
  • Timers, threads, executors, and disk resources have explicit shutdown paths.

Extension points

Extension Primary location Required validation
New HTTP field or event api/ and server.py Protocol model and server tests
New tool format tool_parsers/ Complete, streaming, split-marker, and server tests
New reasoning format reasoning/ Complete, streaming, finalization, and server tests
New model family model detection, wrapper, optional patch Loader, dispatch, generation, and cache tests
New cache policy cache module and scheduler Hit, miss, eviction, concurrency, reset, and recovery tests
New endpoint model engine plus endpoint policy Compatibility, lazy load, limit, and error tests

Further reading