# vllm-mlx complete documentation > Complete human-authored documentation and static Python API inventory. Use `/llms.txt` for a compact index. # Documentation page: `benchmarks/README.md` # Benchmarks Performance benchmarks for vllm-mlx on Apple Silicon. ## Benchmark Types - [LLM Benchmarks](llm.md) - Text generation performance - [Image Benchmarks](image.md) - Image understanding performance - [Video Benchmarks](video.md) - Video understanding performance ## Quick Commands ```bash # LLM benchmark vllm-mlx-bench --model mlx-community/Qwen3-0.6B-8bit # Image benchmark vllm-mlx-bench --model mlx-community/Qwen3-VL-8B-Instruct-4bit # Video benchmark vllm-mlx-bench --model mlx-community/Qwen3-VL-8B-Instruct-4bit --video # Running-server prompt sweep with Prometheus metric deltas vllm-mlx bench-serve --url http://localhost:8000 --prompts short,long \ --concurrency 1,4 --output bench.json --format json # Running-server product-style workload with quality checks vllm-mlx bench-serve --url http://localhost:8000 \ --workload ./workload.json --output workload-results.json ``` ## Contract Workloads `vllm-mlx bench-serve --workload` runs declarative cases against an already running OpenAI-compatible server. This is intended for model and feature-stack qualification, where raw speed is not enough and every run needs provenance, quality checks, Prometheus metric deltas, and policy-timeout evidence. Use `--repetitions` to measure variance; workload summaries report per-case sample counts, failure rates, and min/median/max latency and throughput. `required_regex` and `forbidden_regex` entries are Python regular expressions; plain literal strings are valid regex patterns. Workload `cache_policy` accepts `preserve`, `before-run`, and `before-case`; JSON/YAML-style underscore spellings such as `before_case` are normalized to the same values. Example workload: ```json { "name": "writing-contract", "description": "Representative long-form writing requests", "defaults": { "max_tokens": 32768, "enable_thinking": true, "policy_timeout_ms": 180000, "checks": { "finish_reason": "stop", "forbidden_regex": ["", "prompt leakage"], "min_chars": 500 } }, "cases": [ { "id": "resume-golden-1", "messages": [ {"role": "user", "content": "Write the requested artifact..."} ], "tags": ["resume", "quality-floor"] } ] } ``` Cases can also reference an existing OpenAI-compatible request JSON instead of duplicating a large prompt body: ```json { "name": "writing-contract", "cases": [ { "id": "resume-golden-1", "request_path": "./fixtures/job543_resume_precise_request.json", "checks": { "finish_reason": "stop", "forbidden_regex": [""] } } ] } ``` When `request_path` is used, `messages`, `max_tokens`, `enable_thinking`, and extra request-body fields such as `thinking_token_budget` are read from that file. Case-level `extra_body` values override request-file values. `policy_timeout_ms` is recorded as comparison evidence. It is not treated as a hardware capability claim. Use it to answer "would this run fit my product policy?" after first measuring what the model and serving stack can actually do. Workload output defaults to JSON for full provenance. Use `--format csv` for flat per-case rows, `--format sql` to emit importable SQL, or `--format sqlite --output bench.db` to append rows directly into a local benchmark database. `--request-timeout-s` is the HTTP transport ceiling for each request in workload mode. Product policy timeouts belong in the workload as `policy_timeout_ms` and are recorded as comparison evidence. ```bash vllm-mlx bench-serve --url http://localhost:8000 \ --workload ./workload.json --repetitions 5 --output workload-results.json vllm-mlx bench-serve --url http://localhost:8000 \ --workload ./workload.json --repetitions 5 --format sqlite --output bench.db ``` ## Standalone Test Defaults Standalone benchmark test scripts have built-in default models, so you can run: ```bash python tests/test_continuous_batching.py python tests/test_prefix_cache.py ``` Defaults: - `tests/test_continuous_batching.py` → `mlx-community/Qwen3-8B-6bit` - `tests/test_prefix_cache.py` → `mlx-community/Qwen3-0.6B-8bit` To test different models, use the optional `--model` flag: ```bash python tests/test_continuous_batching.py --model mlx-community/Qwen3-0.6B-8bit python tests/test_prefix_cache.py --model mlx-community/Qwen3-8B-6bit ``` ## Hardware Benchmarks have been collected on the following Apple Silicon configurations: | Chip | Memory | Python | |------|--------|--------| | Apple M4 Max | 128 GB unified | 3.13 | | Apple M1 Max | 64 GB unified | 3.12 | Results will vary on different Apple Silicon chips. ## Contributing Benchmarks If you have a different Apple Silicon chip, please share your results: ```bash vllm-mlx-bench --model mlx-community/Qwen3-0.6B-8bit --output results.json ``` Open an issue with your results at [GitHub Issues](https://github.com/waybarrios/vllm-mlx/issues). # Documentation page: `benchmarks/audio.md` # Audio Benchmarks ## Speech-to-Text (STT) Benchmarks ### Running STT Benchmarks ```bash # Run with default test audio python examples/benchmark_audio.py --stt # Run with your own audio file python examples/benchmark_audio.py --stt --audio path/to/audio.wav ``` ### Results (M4 Max, 128GB) **Test audio:** 46.7 seconds of synthesized speech | Model | Parameters | Load Time | Transcribe Time | RTF* | |-------|------------|-----------|-----------------|------| | whisper-tiny | 39M | 0.34s | 0.24s | **197x** | | whisper-small | 244M | 0.18s | 0.47s | **98x** | | whisper-medium | 769M | 0.35s | 1.15s | **41x** | | whisper-large-v3 | 1.5B | 0.50s | 1.96s | **24x** | | whisper-large-v3-turbo | 809M | 0.12s | 0.86s | **55x** | *RTF = Real-Time Factor (higher is faster). RTF of 100x means 1 minute of audio transcribes in ~0.6 seconds.* ### Results (M1 Max, 64GB) STT with Parakeet (default environment, Whisper unavailable due to numpy dependency mismatch): | Model | Load Time | Transcribe Time | RTF | |-------|-----------|-----------------|-----| | parakeet-tdt-0.6b-v2 | 0.28s | 1.01s | **9.9x** | | parakeet-tdt-0.6b-v3 | 0.30s | 0.19s | **52.7x** | STT with Whisper (explicit `numpy==2.3.5` + `uv run --no-sync`): | Model | Load Time | Transcribe Time | RTF | |-------|-----------|-----------------|-----| | whisper-tiny | 4.02s | 1.05s | **9.5x** | | whisper-small | 10.15s | 1.03s | **9.7x** | | whisper-medium | 22.96s | 2.20s | **4.6x** | | whisper-large-v3 | 38.34s | 0.96s | **10.5x** | | whisper-large-v3-turbo | 21.79s | 0.70s | **14.3x** | | parakeet-tdt-0.6b-v2 | 0.47s | 0.18s | **54.4x** | | parakeet-tdt-0.6b-v3 | 1.13s | 0.18s | **54.6x** | ### Model Recommendations | Use Case | Recommended Model | Why | |----------|-------------------|-----| | **Real-time transcription** | whisper-tiny | Fastest (197x RTF), low latency | | **General use** | whisper-large-v3-turbo | Best balance of speed (55x) and quality | | **Highest accuracy** | whisper-large-v3 | Most accurate, supports 99+ languages | | **Low memory** | whisper-small | Good quality at 244M params | ### Transcription Quality All models correctly transcribed the test audio. Example output: ``` Input text: "Welcome to this comprehensive speech to text demonstration. This audio sample is designed to test the accuracy and speed of various speech recognition models. The quick brown fox jumps over the lazy dog..." Whisper-large-v3 output: "Welcome to this comprehensive speech to text demonstration. This audio sample is designed to test the accuracy and speed of various speech recognition models. The quick brown fox jumps over the lazy dog..." (identical) ``` ### Supported Languages Whisper models support 99+ languages including: - English, Spanish, French, German, Italian, Portuguese - Chinese (Mandarin, Cantonese), Japanese, Korean - Arabic, Hindi, Russian, Turkish, Ukrainian - And many more ## Text-to-Speech (TTS) Benchmarks ### Running TTS Benchmarks ```bash python examples/benchmark_audio.py --tts ``` ### Results (M4 Max, 128GB) **Test:** Generate audio for 3 text samples (short, medium, long) | Model | Load Time | Chars/sec | RTF* | |-------|-----------|-----------|------| | Kokoro-82M-bf16 | 0.8s | 350+ | **22x** | | Kokoro-82M-4bit | 0.4s | 320+ | **20x** | *RTF = Real-Time Factor. RTF of 22x means 1 second of audio generates in ~0.045 seconds.* ### TTS Results (M1 Max, 64GB) | Model | Load Time | Avg Chars/s | Avg RTF | |-------|-----------|-------------|---------| | Kokoro-82M-bf16 | 2.81s | 176.0 | **11.9x** | | Kokoro-82M-4bit | 0.22s | 225.6 | **15.5x** | ### TTS Quality Kokoro produces natural-sounding speech with: - 11 built-in voices (male and female) - Support for 8 languages (English, Spanish, French, Japanese, Chinese, Italian, Portuguese, Hindi) - 82M parameters, fast and lightweight ## Audio Processing Benchmarks ### SAM-Audio (Source Separation) **Test:** Separate drums from 30-second rock song | Metric | Value | |--------|-------| | Model | sam-audio-large-fp16 | | Processing time | ~20s | | Peak memory | ~27 GB | | Output sample rate | 48000 Hz | ## Running All Audio Benchmarks ```bash # Run all benchmarks python examples/benchmark_audio.py --all # Or run individually python examples/benchmark_audio.py --stt python examples/benchmark_audio.py --tts ``` ## Available Models on mlx-community ### STT Models - `mlx-community/whisper-tiny-mlx` - `mlx-community/whisper-small-mlx` - `mlx-community/whisper-medium-mlx` - `mlx-community/whisper-large-v3-mlx` - `mlx-community/whisper-large-v3-turbo` - `mlx-community/parakeet-tdt-0.6b-v2` - `mlx-community/parakeet-tdt-0.6b-v3` ### TTS Models - `mlx-community/Kokoro-82M-bf16` (recommended) - `mlx-community/Kokoro-82M-4bit` - `mlx-community/chatterbox-turbo-fp16` - `mlx-community/VibeVoice-Realtime-0.5B-4bit` ### Audio Processing - `mlx-community/sam-audio-large-fp16` # Documentation page: `benchmarks/image.md` # Image Benchmarks ## Running Image Benchmarks ```bash # Full benchmark (10 resolutions) vllm-mlx-bench --model mlx-community/Qwen3-VL-8B-Instruct-4bit # Quick benchmark (4 resolutions) vllm-mlx-bench --model mlx-community/Qwen3-VL-8B-Instruct-4bit --quick ``` ## Results - Qwen3-VL-8B-Instruct-4bit (M4 Max, 128GB) | Resolution | Pixels | Time | Tokens | Speed | |------------|--------|------|--------|-------| | 224x224 | 50K | 1.04s | 78 | 74.8 tok/s | | 336x336 | 113K | 0.94s | 64 | 68.3 tok/s | | 448x448 | 201K | 1.45s | 70 | 48.1 tok/s | | 512x512 | 262K | 1.58s | 99 | 62.8 tok/s | | 672x672 | 452K | 1.83s | 83 | 45.3 tok/s | | 768x768 | 590K | 2.05s | 91 | 44.3 tok/s | | 896x896 | 803K | 2.61s | 90 | 34.5 tok/s | | 1024x1024 | 1.0M | 2.79s | 76 | 27.2 tok/s | | 1280x720 | 922K | 2.97s | 96 | 32.4 tok/s | | 1920x1080 | 2.1M | 6.30s | 89 | 14.1 tok/s | **Summary:** Average 45.2 tok/s across all resolutions. Fastest at 224x224 (74.8 tok/s), slowest at 1920x1080 (14.1 tok/s) ## Results - Qwen3-VL-8B-Instruct-4bit (M1 Max, 64GB) Local MLLM benchmark: | Resolution | Pixels | Time | Tokens | Speed | |------------|--------|------|--------|-------| | 224x224 | 50K | 1.84s | 78 | 42.5 tok/s | | 448x448 | 201K | 2.28s | 70 | 30.7 tok/s | | 768x768 | 590K | 4.39s | 91 | 20.7 tok/s | | 1024x1024 | 1.0M | 6.41s | 76 | 11.9 tok/s | | Configs | Total Time (s) | Total Tokens | Aggregate Tok/s | |---------|-----------------|--------------|-----------------| | 4 | 14.92 | 315 | 21.1 | ## Results - Qwen3-VL-4B-Instruct-3bit Server (M1 Max, 64GB) | Resolution | Pixels | Time | Tokens | Speed | |------------|--------|------|--------|-------| | 224x224 | 50K | 1.65s | 113 | 68.4 tok/s | | 448x448 | 201K | 2.09s | 120 | 57.5 tok/s | | 768x768 | 590K | 2.93s | 106 | 36.2 tok/s | | 1024x1024 | 1.0M | 4.12s | 100 | 24.3 tok/s | | Configs | Total Time (s) | Total Tokens | Aggregate Tok/s | |---------|-----------------|--------------|-----------------| | 4 | 10.79 | 439 | 40.7 | ## MLLM Prefix Cache Results ``` ====================================================================== MLLM PREFIX CACHE TEST ====================================================================== Model: mlx-community/Qwen3-VL-4B-Instruct-3bit Test: Verify KV cache reuse for repeated image/video + prompt combinations Expected behavior: - Same image + same prompt → cache HIT - Same image + different prompt → cache MISS - Different image + same prompt → cache MISS ---------------------------------------------------------------------- SETUP: Loading Model ---------------------------------------------------------------------- Model loaded in 0.11s ---------------------------------------------------------------------- SETUP: Creating Test Images ---------------------------------------------------------------------- Resized: 224x224, 336x336, 512x512, 768x768 ---------------------------------------------------------------------- TEST 1: Image Cache - Basic Hit/Miss ---------------------------------------------------------------------- Results: Step | Description | Expected | Actual | Time | Status -------+---------------------------+----------+--------+--------+------- 1a | First image+prompt | MISS | MISS | 0.10ms | ✓ 1b | Same image+prompt | HIT | HIT | 0.18ms | ✓ 1c | Different prompt | MISS | MISS | 0.01ms | ✓ 1d | Return to original | HIT | HIT | 0.18ms | ✓ ---------------------------------------------------------------------- TEST 2: Different Images ---------------------------------------------------------------------- Results: Step | Description | Expected | Actual | Time | Status -------+---------------------------+----------+--------+--------+------- 2a | Image A first request | MISS | MISS | 0.01ms | ✓ 2b | Image B first request | MISS | MISS | 0.01ms | ✓ 2c | Image A cached | HIT | HIT | 0.13ms | ✓ ---------------------------------------------------------------------- TEST 3: Image Resolutions ---------------------------------------------------------------------- Results: Step | Description | Expected | Actual | Time | Status -------+-----------------------+----------+--------+--------+------- 3.1a | 224x224 first | MISS | MISS | 0.01ms | ✓ 3.1b | 224x224 cached | HIT | HIT | 0.20ms | ✓ 3.2a | 336x336 first | MISS | MISS | 0.01ms | ✓ 3.2b | 336x336 cached | HIT | HIT | 0.21ms | ✓ 3.3a | 512x512 first | MISS | MISS | 0.12ms | ✓ 3.3b | 512x512 cached | HIT | HIT | 0.20ms | ✓ 3.4a | 768x768 first | MISS | MISS | 0.12ms | ✓ 3.4b | 768x768 cached | HIT | HIT | 0.24ms | ✓ ====================================================================== ``` ## Cache Key Strategy - **Images**: `hash(image_content) + hash(prompt)` Same image with same prompt will always hit cache. Different image or different prompt will miss. ## Performance Tips - Smaller resolutions process faster (224x224 vs 1920x1080) - Use appropriate resolution for your task - Batch similar-sized images for consistent throughput ## Metrics Reference | Metric | Description | |--------|-------------| | Resolution | Image dimensions (width x height) | | Pixels | Total pixel count | | Time | Generation time | | Tokens | Output tokens generated | | Speed | Tokens per second (tok/s) | # Documentation page: `benchmarks/llm.md` # LLM Benchmarks ## Running LLM Benchmarks ```bash vllm-mlx-bench --model mlx-community/Llama-3.2-1B-Instruct-4bit --prompts 5 --max-tokens 256 ``` ## Results (M4 Max, 128GB) | Model | Gen Speed | TTFT* | Memory | |-------|-----------|-------|--------| | Qwen3-0.6B-8bit | 402.3 tok/s | 58.6 ms | 0.68 GB | | Llama-3.2-1B-Instruct-4bit | 463.6 tok/s | 49.2 ms | 0.69 GB | | Qwen2.5-1.5B-Instruct-4bit | 308.5 tok/s | 86.2 ms | 0.84 GB | | Llama-3.2-3B-Instruct-4bit | 200.1 tok/s | 81.4 ms | 1.79 GB | | Qwen3-30B-A3B-4bit | 123.9 tok/s | 126.9 ms | 16.05 GB | | NVIDIA-Nemotron-3-Nano-30B-A3B-MLX-6Bit | 122.9 tok/s | 72.3 ms | 23.98 GB | *TTFT = Time to First Token (latency until the model starts generating) ## Results (M1 Max, 64GB) | Model | Runs | Prompt Tok | Gen Tok | Total Time (s) | TTFT Mean (ms) | TPOT Mean (ms) | Gen Speed (tok/s) | Total Throughput (tok/s) | |-------|------|------------|---------|-----------------|-----------------|-----------------|-------------------|--------------------------| | Qwen3-0.6B-8bit | 5 | 56 | 1280 | 5.66 | 119.0 | 3.97 | 251.9 | 236.1 | ## Continuous Batching Results | Model | Single Request | Batch (5 req) | Speedup | |-------|----------------|---------------|---------| | Llama-3.2-1B-Instruct-4bit | 299.1 tok/s | 613.0 tok/s | **2.05x** | | Llama-3.2-3B-Instruct-4bit | 137.6 tok/s | 208.1 tok/s | **1.51x** | | Qwen3-0.6B-8bit | 328.1 tok/s | 1111.8 tok/s | **3.39x** | | Qwen3-30B-A3B-4bit | 98.1 tok/s | 233.3 tok/s | **2.38x** | | Qwen2.5-1.5B-Instruct-4bit | 196.9 tok/s | 322.2 tok/s | **1.64x** | *Batching 5 concurrent requests shows 1.5-3x throughput improvement.* ### Continuous Batching (M1 Max, 64GB) | Requests | Total Tokens | Total Time (s) | Throughput (tok/s) | Requests/sec | |----------|--------------|-----------------|--------------------|--------------| | 5 | 315 | 0.64 | 492.5 | 7.82 | ## Streaming Performance | Model | TTFT | Generation Speed | |-------|------|------------------| | Llama-3.2-1B-Instruct-4bit | ~4.6ms | 218.9 tok/s | | Llama-3.2-3B-Instruct-4bit | ~10.7ms | 93.6 tok/s | | Qwen3-0.6B-8bit | ~3.0ms | 328.5 tok/s | | Qwen3-30B-A3B-4bit | ~10.2ms | 98.4 tok/s | | Qwen2.5-1.5B-Instruct-4bit | ~7.1ms | 140.3 tok/s | ### Streaming Detokenizer (M1 Max, 64GB) `vllm-mlx bench-detok`: | Tokens | Iterations | Naive Time | Streaming Time | Speedup | |--------|------------|------------|----------------|---------| | 742 | 5 | 1.69ms | 0.71ms | 2.39x | `examples/benchmark_detokenizer.py`: | Sequence | Tokens | decode() | Streaming | Speedup | |----------|--------|----------|-----------|---------| | Short | 8 | 0.029ms | 0.028ms | 1.04x | | Medium | 103 | 0.206ms | 0.129ms | 1.59x | | Long | 511 | 1.040ms | 0.502ms | 2.07x | | 1K | 1191 | 2.446ms | 1.178ms | 2.08x | | 2K | 2381 | 4.949ms | 2.356ms | 2.10x | | 4K | 4761 | 9.887ms | 5.398ms | 1.83x | Average speedup: 1.79x ## Prefix Cache Results ### Prefix Cache (M4 Max, 128GB) ``` ====================================================================== LLM PREFIX CACHE TEST ====================================================================== Model: mlx-community/Qwen3-0.6B-8bit Expected behavior: - Same prompt → cache HIT - Different prompt → cache MISS ---------------------------------------------------------------------- Results: Step | Description | Expected | Actual | Status -------+---------------------+----------+--------+------- 1a | First request | MISS | MISS | ✓ 1b | Same prompt | HIT | HIT | ✓ 1c | Different prompt | MISS | MISS | ✓ 1d | Return to prompt 1 | HIT | HIT | ✓ ====================================================================== ``` ### Prefix Cache (M1 Max, 64GB) | Test | Expected | Actual | Time | Status | |------|----------|--------|------|--------| | First request | MISS | MISS | 203.5ms | PASS | | Same prompt | HIT | HIT | 131.6ms | PASS | | Different prompt | MISS or PREFIX_HIT | PREFIX_HIT (5 tok) | 135.3ms | PASS | Final cache stats: | Cache Hits | Cache Misses | Hit Rate | Tokens Saved | Cached Speedup | |------------|--------------|----------|--------------|----------------| | 2 | 1 | 66.7% | 20 | 1.55x | ## Paged Cache Results *Test: 20 real inference requests in 2 rounds with ~286 token shared system prompt* ``` ====================================================================== PAGED KV CACHE - REAL INFERENCE TEST ====================================================================== -------------------------------------------------- Test 1: WITHOUT Paged Cache (2 rounds of 10) -------------------------------------------------- Time: 1.47s Throughput: 681.2 tok/s Cache hits: 0 Tokens saved: 0 -------------------------------------------------- Test 2: WITH Paged Cache (2 rounds of 10) -------------------------------------------------- Time: 1.31s Throughput: 765.8 tok/s Paged Cache Stats: Blocks allocated: 25 Shared blocks: 4 Cache hits: 10 Tokens saved: 2560 ================================================== SUMMARY ================================================== Without paged cache: 681.2 tok/s With paged cache: 765.8 tok/s Speedup: 1.12x Cache hits: 10 (all Round 2 requests) Tokens saved: 2,560 (~256 tokens × 10 requests) ================================================== ``` ### Paged KV Cache (M1 Max, 64GB) Inference benchmark (20 requests): | Mode | Time (s) | Throughput (tok/s) | |------|----------|--------------------| | Without paged cache | 3.43 | 291.8 | | With paged cache | 3.42 | 292.2 | | Speedup | Blocks Allocated | Shared Blocks | Cache Hits | Tokens Saved | |---------|------------------|---------------|------------|--------------| | 1.00x | 45 | 4 | 10 | 2560 | Real concurrent inference (20 requests): | Mode | Time (s) | Throughput (tok/s) | |------|----------|--------------------| | Without paged cache | 4.32 | 231.7 | | With paged cache | 4.35 | 229.7 | | Speedup | Blocks Allocated | Shared Blocks | Cache Hits | Tokens Saved | |---------|------------------|---------------|------------|--------------| | 0.99x | 49 | 8 | 10 | 5120 | Memory savings demo: | Scenario | Memory Savings | |----------|----------------| | Shared system prompts | 70.8% | | Concurrent memory efficiency | 83.5% | | Prefix sharing branches | 38.5% | ## Streaming Detokenizer Analysis *Phase 9.1 Investigation: mlx-lm's `BPEStreamingDetokenizer` vs naive `tokenizer.decode()`* ### Background The naive approach calls `decode([token])` for each token. In theory, streaming detokenizers provide O(T) complexity vs O(T²) for naive decode. ### Isolated Benchmark Results ```bash vllm-mlx bench-detok ``` When reusing the same detokenizer instance (with `reset()` between uses): | Sequence | Tokens | Naive decode() | Streaming | Speedup | |----------|--------|----------------|-----------|---------| | Short | 8 | 0.020ms | 0.019ms | 1.05x | | Medium | 103 | 0.155ms | 0.097ms | 1.59x | | Long | 511 | 0.752ms | 0.371ms | **2.03x** | | 1K tokens | 1191 | 1.743ms | 0.833ms | **2.09x** | | 2K tokens | 2381 | 3.493ms | 1.737ms | **2.01x** | ### Critical Finding: Instance Creation Overhead Creating a new `BPEStreamingDetokenizer` instance is **extremely expensive**: ``` 100 tokenizer.detokenizer calls: 5.266s (52.7ms each!) ``` This means creating a new detokenizer per request adds **~52ms overhead**, negating any benefits. ### Real-World Impact When integrated into the scheduler (one detokenizer per request): | Metric | Naive decode() | Streaming (new instance) | |--------|----------------|--------------------------| | Throughput (20 req) | 681 tok/s | 275 tok/s | | Impact | - | **-60% slower** | ### Conclusion The streaming detokenizer is **not currently viable** for per-request usage due to instance creation cost. The naive `decode([token])` approach remains faster in practice. **Future optimization**: Pre-create a pool of detokenizer instances at startup and reuse them across requests. ## Metrics Reference | Metric | Description | |--------|-------------| | **TTFT** | Time to First Token - latency until model starts responding (ms) | | **TPOT** | Time Per Output Token - time between each generated token (ms/token) | | **Generation TPS** | Output tokens per second (tok/s) | | **Processing TPS** | Input/prompt tokens processed per second (tok/s) | | **End-to-End Latency** | Total time from request to complete response | | **Total Throughput** | Overall tokens (input + output) per second | ## Running Benchmarks ```bash # Basic benchmark vllm-mlx-bench --model mlx-community/Qwen3-0.6B-8bit # With more prompts vllm-mlx-bench --model mlx-community/Qwen3-0.6B-8bit --prompts 10 # Save results vllm-mlx-bench --model mlx-community/Qwen3-0.6B-8bit --output results.json # Continuous batching test python tests/test_continuous_batching.py # Prefix cache test python tests/test_prefix_cache.py # Paged cache test python tests/test_paged_cache_real_inference.py # Streaming detokenizer benchmark vllm-mlx bench-detok vllm-mlx bench-detok mlx-community/Llama-3.2-1B-Instruct-4bit --iterations 5 ``` # Documentation page: `benchmarks/video.md` # Video Benchmarks ## Running Video Benchmarks ```bash # Full benchmark (10 configurations, 2-64 frames) vllm-mlx-bench --model mlx-community/Qwen3-VL-8B-Instruct-4bit --video # Quick benchmark (3 frame counts) vllm-mlx-bench --model mlx-community/Qwen3-VL-8B-Instruct-4bit --video --quick # Custom video vllm-mlx-bench --model mlx-community/Qwen3-VL-8B-Instruct-4bit --video --video-url https://example.com/video.mp4 ``` ## Results - Qwen3-VL-8B-Instruct-4bit (M4 Max, 128GB) | Configuration | Frames | Time | Tokens | Speed | Memory | |---------------|--------|------|--------|-------|--------| | 2 frames @ 0.5fps | 2 | 4.48s | 256 | 57.1 tok/s | 6.4 GB | | 4 frames @ 1fps | 4 | 4.65s | 256 | 55.0 tok/s | 6.4 GB | | 6 frames @ 1fps | 6 | 5.15s | 197 | 38.2 tok/s | 6.6 GB | | 8 frames @ 2fps | 8 | 6.45s | 240 | 37.2 tok/s | 6.8 GB | | 12 frames @ 2fps | 12 | 8.73s | 256 | 29.3 tok/s | 7.1 GB | | 16 frames @ 2fps | 16 | 10.96s | 256 | 23.4 tok/s | 7.6 GB | | 24 frames @ 4fps | 24 | 14.95s | 226 | 15.1 tok/s | 8.4 GB | | 32 frames @ 4fps | 32 | 20.00s | 256 | 12.8 tok/s | 9.2 GB | | 48 frames @ 8fps | 48 | 31.11s | 246 | 7.9 tok/s | 11.1 GB | | 64 frames @ 8fps | 64 | 59.81s | 256 | 4.3 tok/s | 12.9 GB | **Summary:** Fastest at 2 frames (57.1 tok/s), slowest at 64 frames (4.3 tok/s). Memory scales from 6.4 GB to 12.9 GB. > **Note:** 96+ frames causes GPU timeout on most hardware due to memory/compute limits ## Results - Qwen3-VL-8B-Instruct-4bit (M1 Max, 64GB) | Configuration | Frames | FPS | Time | Tokens | Speed | |---------------|--------|-----|------|--------|-------| | 4 frames @ 1fps | 4 | 1.0 | 8.84s | 256 | 29.0 tok/s | | 8 frames @ 2fps | 8 | 2.0 | 13.05s | 256 | 19.6 tok/s | | 16 frames @ 2fps | 16 | 2.0 | 21.60s | 256 | 11.9 tok/s | | Configs | Total Time (s) | Total Tokens | Aggregate Tok/s | |---------|-----------------|--------------|-----------------| | 3 | 43.48 | 768 | 17.7 | ## Results - Qwen3-VL-4B-Instruct-3bit (M1 Max, 64GB) | Configuration | Frames | FPS | Time | Tokens | Speed | |---------------|--------|-----|------|--------|-------| | 4 frames @ 1fps | 4 | 1.0 | 5.09s | 150 | 29.5 tok/s | | 8 frames @ 2fps | 8 | 2.0 | 8.36s | 150 | 17.9 tok/s | | 16 frames @ 2fps | 16 | 2.0 | 15.21s | 150 | 9.9 tok/s | | Configs | Total Time (s) | Total Tokens | Aggregate Tok/s | |---------|-----------------|--------------|-----------------| | 3 | 28.66 | 450 | 15.7 | ## Video Cache Results ``` ---------------------------------------------------------------------- TEST 4: Video Cache - fps/max_frames in Cache Key ---------------------------------------------------------------------- Config: fps=2.0, max_frames=16 Results: Step | Description | Expected | Actual | Time | Status -------+---------------------------+----------+--------+--------+------- 4a | Video first request | MISS | MISS | 0.03ms | ✓ 4b | Same video+params | HIT | HIT | 0.14ms | ✓ 4c | Different fps (4.0) | MISS | MISS | 0.01ms | ✓ 4d | Different max_frames (32) | MISS | MISS | 0.01ms | ✓ 4.0.5a | fps=0.5 first | MISS | MISS | 0.01ms | ✓ 4.0.5b | fps=0.5 cached | HIT | HIT | 0.14ms | ✓ 4.1.0a | fps=1.0 first | MISS | MISS | 0.01ms | ✓ 4.1.0b | fps=1.0 cached | HIT | HIT | 0.14ms | ✓ 4.2.0a | fps=2.0 first | MISS | MISS | 0.01ms | ✓ 4.2.0b | fps=2.0 cached | HIT | HIT | 0.14ms | ✓ 4.4.0a | fps=4.0 first | MISS | MISS | 0.01ms | ✓ 4.4.0b | fps=4.0 cached | HIT | HIT | 0.14ms | ✓ ---------------------------------------------------------------------- TEST 5: Additional Videos ---------------------------------------------------------------------- Results: Step | Description | Expected | Actual | Time | Status -------+---------------------------+----------+--------+--------+------- 5a | Video 1 first | MISS | MISS | 0.01ms | ✓ 5b | Video 2 first | MISS | MISS | 0.01ms | ✓ 5c | Video 1 cached | HIT | HIT | 0.13ms | ✓ 5d | Video 2 cached | HIT | HIT | 0.13ms | ✓ ``` ## Cache Key Strategy - **Videos**: `hash(video_path) + hash(fps) + hash(max_frames) + hash(prompt)` Same video with same fps, max_frames, and prompt will hit cache. Changing any parameter causes a miss. ## Performance Tips - Lower FPS = faster processing - Fewer frames = less memory usage - 64 frames is practical maximum - 96+ frames causes GPU timeout ## Frame Extraction | FPS | 10s Video | 30s Video | 60s Video | |-----|-----------|-----------|-----------| | 0.5 | 5 frames | 15 frames | 30 frames | | 1.0 | 10 frames | 30 frames | 60 frames | | 2.0 | 20 frames | 60 frames | 120 frames* | | 4.0 | 40 frames | 120 frames* | 240 frames* | *May hit `max_frames` limit ## Metrics Reference | Metric | Description | |--------|-------------| | Configuration | FPS and max frames settings | | Frames | Actual frames extracted | | Time | Total generation time | | Tokens | Output tokens generated | | Speed | Tokens per second (tok/s) | | Memory | GPU memory usage | # Documentation page: `concepts/caching.md` # Caching vllm-mlx contains several caches because text KV state, paged blocks, multimodal embeddings, and disk persistence have different ownership and reuse rules. ## Cache selection | Component | Unit of reuse | Primary purpose | | --- | --- | --- | | [`PrefixCacheManager`](../reference/api/vllm_mlx/prefix_cache.md) | Whole token prefix entry | Simple in-memory prefix reuse | | [`BlockAwarePrefixCache`](../reference/api/vllm_mlx/prefix_cache.md) | Token blocks | Prefix reuse aligned with block boundaries | | [`MemoryAwarePrefixCache`](../reference/api/vllm_mlx/memory_cache.md) | KV cache entry with measured bytes | Enforce a byte or memory-percentage budget | | [`PagedCacheManager`](../reference/api/vllm_mlx/paged_cache.md) | Reference-counted KV blocks | Share blocks across active and cached sequences | | [`SSDCacheTier`](../reference/api/vllm_mlx/ssd_cache.md) | Serialized cache entry | Extend prefix reuse beyond RAM | | [`MLLMPrefixCacheManager`](../reference/api/vllm_mlx/mllm_cache.md) | Multimodal prompt state | Reuse text and multimodal prefill work | | [`VisionEmbeddingCache`](../reference/api/vllm_mlx/vision_embedding_cache.md) | Encoded image or pixel input | Avoid repeated vision encoding | ## Cache identity Reusable KV state depends on more than matching token IDs. The memory-aware cache associates entries with a model fingerprint so cache data is not reused across incompatible models. Multimodal caches additionally account for media-derived state. Quantization format and cache layout must remain compatible with the consumer. When changing model loading, tokenizer selection, or model residency, verify that every cache either receives a distinct identity or is cleared before the new model becomes active. ## Prefix matching Prefix lookup can be exact or partial: - Exact match reuses the complete cached token sequence. - Prefix match reuses a cached sequence that is a prefix of the new prompt. - Supersequence handling can trim a longer reusable entry when its cache layers support trimming. - Longest-common-prefix matching can reuse the compatible portion of a related prompt. The cache must not trim a layer that declares itself non-trimmable. In that case it should choose a safe reusable boundary or decline the hit. ## Memory-aware cache `MemoryAwarePrefixCache` estimates the bytes held by nested MLX arrays and evicts entries to stay within its configured budget. Its statistics expose entry counts, hit and miss data, eviction reasons, current bytes, and utilization. Important controls include: - An explicit memory limit in MiB. - A percentage of available memory. - Minimum prefix length. - Optional cache quantization. - Optional persistence integration. All lookup paths must apply the same model-identity, prefix-length, and layer-trimming rules. Cache mutation and accounting must stay coordinated so an entry is charged or subtracted exactly once. ## Paged cache Paged caching divides tokens into fixed-size blocks. Each block tracks an ID, reference count, parent-dependent hash, free-list links, and per-layer cache data. A chain hash makes the same token block under different prefixes distinguishable. The free-block queue supports constant-time removal and insertion. The hash map finds reusable blocks. Block tables map each request to the blocks it currently references. Copy-on-write preserves isolation when a request needs to modify a shared block. Reference counts are the main invariant: - A block with active owners cannot be returned to the free queue. - A shared block is released only after the final owner leaves. - Hash mappings cannot point at a block that has been reassigned. - Reset and error recovery must rebuild both ownership and lookup structures consistently. ## SSD tier The SSD tier serializes eligible prefix entries and restores them when memory lookup misses. Storage includes enough metadata to reject incompatible or corrupt entries. Disk I/O should remain outside latency-sensitive event-loop work, and shutdown must close resources after pending persistence completes. Treat the SSD directory as disposable generated state. Do not share it between model configurations unless their cache identity contract explicitly permits it. ## Cache lifecycle Cache state may be cleared through the API, reset by the scheduler, persisted during engine unload, or evicted due to memory pressure. A component that starts a thread, file handle, or executor needs a separate close operation. Clearing entries alone does not release background resources. ## Observing cache behavior Use `GET /v1/cache/stats` for server-visible cache statistics and `DELETE /v1/cache` or `DELETE /v1/cache/prefix` for explicit invalidation. Benchmark both cold and warm runs. A high hit rate is useful only when entries are compatible, memory remains within budget, and tail latency improves. # Documentation page: `concepts/index.md` # Core concepts These pages explain how vllm-mlx works beneath the command line and HTTP APIs. Use them when choosing an engine mode, debugging latency or memory behavior, or changing runtime code. ## Runtime and requests - [Runtime architecture](runtime-architecture.md) describes the API, engine, scheduler, model, parser, and hardware layers. - [Request lifecycle](request-lifecycle.md) traces one request from validation through terminal output and cleanup. - [Scheduling and batching](scheduling-and-batching.md) explains queues, continuous batching, worker-thread affinity, and output collection. ## Models and memory - [Caching](caching.md) compares the legacy prefix cache, memory-aware cache, paged cache, multimodal cache, and SSD tier. - [Models and modalities](models-and-modalities.md) explains text, vision, audio, embeddings, reranking, model registration, and residency. ## Output interpretation - [Parsing and structured output](parsing-and-structured-output.md) covers reasoning extraction, tool-call parsers, JSON Schema enforcement, streaming, and terminal events. For individual Python objects, continue to the [generated API reference](../reference/api/index.md). Every source definition is indexed there with an exact line link. # Documentation page: `concepts/models-and-modalities.md` # Models and modalities vllm-mlx routes several model families through one server while keeping their loading and preprocessing contracts separate. ## Text generation [`MLXLanguageModel`](../reference/api/vllm_mlx/models/llm.md) wraps mlx-lm for text-only generation. It owns the model, tokenizer, chat template behavior, and generation parameters used by the simple engine. Continuous batching uses the model and tokenizer through the engine core and scheduler. ## Vision-language generation [`MLXMultimodalLM`](../reference/api/vllm_mlx/models/mllm.md) wraps mlx-vlm. [`MultimodalProcessor`](../reference/api/vllm_mlx/multimodal_processor.md) extracts and prepares images or videos while retaining the textual conversation required by the chat template. Model-specific patches adapt attention or multi-token prediction implementations that need different mask, cache, or hidden-state handling in a batch. Patch activation must be narrow and reversible because upstream mlx-vlm behavior can change independently. ## Audio Audio endpoints are optional because mlx-audio and its model families have separate dependencies. [`STTEngine`](../reference/api/vllm_mlx/audio/stt.md) handles transcription models. [`TTSEngine`](../reference/api/vllm_mlx/audio/tts.md) handles speech synthesis and voice selection. [`audio_limits`](../reference/api/vllm_mlx/audio_limits.md) enforces upload and text limits before expensive processing. The API layer should reject oversized content before loading an optional model or allocating a large buffer. ## Embeddings and reranking [`EmbeddingEngine`](../reference/api/vllm_mlx/embedding.md) uses mlx-embeddings to produce vectors. [`RerankEngine`](../reference/api/vllm_mlx/rerank.md) uses a BERT-family sequence-classification forward pass implemented in [`rerank_forward`](../reference/api/vllm_mlx/rerank_forward.md). These endpoints apply model compatibility policies independently from generation. A chat model name should not silently select an incompatible embedding or reranking model. ## Model detection The registry and API utilities inspect model identifiers and configuration metadata to classify text, vision, embedding, reranking, STT, and TTS workloads. Name heuristics are fallbacks. Configuration metadata and successful loader selection are stronger signals. When adding a model family: 1. Confirm which upstream loader owns it. 2. Add the narrowest detection rule. 3. Keep text and multimodal classification mutually coherent. 4. Add loader and endpoint-policy tests. 5. Document required optional dependencies and a known model identifier. ## Registry-backed serving [`ModelManager`](../reference/api/vllm_mlx/model_registry.md) loads registered models under a memory budget. A registry entry describes the source model and serving defaults. A loaded model owns its engine and accounting metadata. A request obtains a [`ModelLease`](../reference/api/vllm_mlx/model_registry.md) so eviction cannot remove a model while it is in use. The manager estimates whether a candidate fits, chooses an eviction candidate according to policy, loads the model, and updates budget state. Loading and eviction must be serialized because device memory and engine ownership are shared resources. ## Single-model residency [`ResidencyManager`](../reference/api/vllm_mlx/lifecycle.md) controls lazy loading and automatic unload of the default model. Its state machine distinguishes unloaded, loading, loaded, and unloading behavior while tracking activity. Key invariants: - Concurrent first requests share one load operation. - An active request prevents idle unload. - Unload persists eligible state before stopping the engine. - Reload invalidates tokenizer-derived parser instances. - Server shutdown closes the current engine even if background lifecycle work is active. ## Model workflow [`model_workflow`](../reference/api/vllm_mlx/model_workflow.md) implements inspect, acquire, convert, register, and qualify operations. Its manifests make source revision, file inventory, conversion recipe, and output artifacts auditable. Use inspection before download or conversion. Use acquisition when a complete local artifact is required. Use conversion for upstream weights that are not already in MLX format. Use qualification to exercise the resulting artifact against the intended server behavior. # Documentation page: `concepts/parsing-and-structured-output.md` # Parsing, tools, and structured output Model output may contain final text, hidden reasoning, tool calls, or syntax constrained by a schema. vllm-mlx keeps token constraints and text interpretation separate so each concern can be tested independently. ## Reasoning parsers [`ReasoningParser`](../reference/api/vllm_mlx/reasoning/base.md) defines complete-output and streaming extraction. A parser returns reasoning content, final content, or no delta when the current text is only a control marker. Implementations cover tagged formats such as Qwen3, DeepSeek-R1, GLM, Gemma, Mistral, Poolside, and channel-based Harmony output. Stateful parsers reset before each request and may flush buffered content at stream finalization. Streaming implementations receive previous text, current text, and the new delta. They must handle markers split across chunks, repeated markers, missing closing markers, and final markers that contain no user-visible text. ## Thinking-aware token constraints [`ThinkingAwareLogitsProcessor`](../reference/api/vllm_mlx/constrained/thinking_processor.md) tracks four phases: 1. `IDLE` waits for the reasoning start sequence. 2. `THINKING` counts reasoning tokens. 3. `TRANSITIONING` forces the reasoning end sequence when the budget is exhausted. 4. `CONTENT` delegates to the inner structured-output processor and prevents reasoning control tokens from reappearing. The processor keeps snapshots because speculative generation can roll token history back. After entering content, state no longer changes and redundant snapshots are avoided. ## Tool parsers [`ToolParser`](../reference/api/vllm_mlx/tool_parsers/abstract_tool_parser.md) is the common interface for complete and streaming extraction. [`ToolParserManager`](../reference/api/vllm_mlx/tool_parsers/abstract_tool_parser.md) registers parser names and resolves aliases. Parsers under `vllm_mlx.tool_parsers` cover model-specific JSON, XML, bracketed, token-delimited, and Harmony formats. Auto detection is convenient, but an explicit parser is more predictable in production. A streaming tool parser can buffer partial markup until it knows whether text is ordinary assistant content or a tool call. The server must not leak half of a tool marker as content, and it must flush valid ordinary text if a suspected marker never completes. ## MCP execution MCP expands tool calling from parsing into external execution: - Configuration defines allowed server processes and environment. - Clients connect to individual MCP servers. - The manager aggregates tool discovery across servers. - Tool schemas are converted into the OpenAI representation sent to models. - The executor applies concurrency limits and dispatches calls. - Security validation rejects unsafe commands, arguments, paths, or environment values. MCP crosses an execution trust boundary. Review configuration parsing, command validation, subprocess environment, timeouts, and output size limits when changing this area. ## JSON Schema enforcement [`JSONSchemaLogitsProcessor`](../reference/api/vllm_mlx/constrained/json_schema_processor.md) adapts lm-format-enforcer to mlx-lm logits. Tokenizer-specific enforcement data is cached by tokenizer identity in [`constrained.cache`](../reference/api/vllm_mlx/constrained/cache.md). The API accepts JSON object or JSON Schema response formats. The server builds the processor before generation and validates or cleans the final result according to the selected contract. When reasoning is active, schema enforcement applies to final content rather than hidden thinking. ## Streaming terminal contract OpenAI-style streams end with a chunk carrying the finish reason and final usage, followed by `data: [DONE]`. Anthropic streams use typed content, message-delta, and message-stop events. Responses API streams use typed response lifecycle events. Parser output and generation completion are independent signals. A terminal model delta can be consumed entirely by a reasoning or tool parser, so the server tracks whether a finish reason has actually been emitted and synthesizes the terminal protocol event when necessary. ## Adding a parser 1. Implement both complete and streaming extraction. 2. Register a stable parser name and any compatibility aliases. 3. Add tests for ordinary content, one tool or reasoning block, multiple blocks, split markers, malformed output, and finalization. 4. Test the server integration for both streaming and non-streaming responses. 5. Document the required CLI flag and a known-compatible model family. # Documentation page: `concepts/request-lifecycle.md` # Request lifecycle This page follows a generation request through the HTTP server. Exact helper signatures and source are available in the [`vllm_mlx.server` reference](../reference/api/vllm_mlx/server.md). ## 1. Transport and middleware FastAPI accepts the request and runs the HTTP middleware. Depending on configuration, the request may be subject to bearer authentication, per-client rate limiting, request timing metrics, remote-media safety validation, and endpoint-specific size limits. Errors raised here are protocol errors. Model generation has not started, so no scheduler request or model lease needs cleanup. ## 2. Request model validation The server resolves the user-facing `model` field against either the single active engine or the registry-backed model set. Registry serving returns a request-scoped context that holds a lease while generation is active. A missing or incompatible model is rejected before expensive work begins. Optional endpoints such as embeddings, reranking, and audio apply their own model policy. They may use a preloaded endpoint model, a compatible requested model, or reject the request when the model type does not match. ## 3. Protocol normalization Each public protocol is converted into the internal chat or prompt representation: - Chat Completions normalizes messages, media, tools, tool choice, and chat-template keyword overrides. - Completions accepts an already textual prompt. - Responses converts input items and prior persisted response items into chat messages, then converts generated output back into Responses API events or objects. - Anthropic Messages moves system content into the leading system prompt, converts content blocks and tools, and maps stop reasons back to Anthropic values. - Multimodal requests separate text from image, video, or audio inputs before model preprocessing. This boundary is also where server defaults are merged with request-level overrides. ## 4. Generation policy construction The server derives a generation invocation containing token limits, sampling values, stop sequences, chat-template arguments, and optional processors. Processors may include: - Logit bias. - JSON object or JSON Schema enforcement. - A thinking-aware wrapper that controls reasoning budget and the transition to final content. - Forced-tool instructions or a model-native tool format. Tool and reasoning parsers are selected separately from logit processors. Parsers interpret generated text, while processors constrain token selection. ## 5. Engine acquisition For a resident single model, the lifecycle manager may load the engine lazily and increments activity before returning it. For registry serving, the request acquires a model lease. The server then invokes the common `BaseEngine` contract. Cleanup callbacks are prepared before generation begins. This ensures cancellation, timeouts, disconnects, and ordinary exceptions all release the same resources. ## 6. Scheduling and generation `SimpleEngine` performs the model call directly. `BatchedEngine` submits a request to `AsyncEngineCore`, which creates internal request state and places it in the scheduler waiting queue. The scheduler admits work according to capacity, creates or reuses KV cache state, and advances the active batch. Output collectors turn scheduler outputs into complete results or async deltas for each caller. ## 7. Streaming transformation Streaming endpoints process every generation delta in this order: 1. Track accumulated and incremental model text. 2. Extract reasoning content if a reasoning parser is active. 3. Extract or buffer tool-call markup if a tool parser is active. 4. Apply response-format cleanup where required. 5. Encode the protocol-specific event or Server-Sent Event payload. 6. Update usage and finish-reason state. A parser may consume a marker and return no user-visible delta. The stream still has to flush buffered parser content, emit a terminal finish reason and usage when the protocol requires them, and finally emit `[DONE]` for OpenAI-style streams. ## 8. Completion, timeout, or disconnect Normal completion records the model finish reason, usually `stop` or `length`. A timeout cancels the internal request and returns the server's timeout response. A client disconnect follows the same cancellation path without continuing unnecessary generation. The request ID routes make cancellation explicit: - `POST /v1/requests/{request_id}/cancel` asks the active engine to cancel. - `DELETE /v1/requests/{request_id}` is the deletion alias. Cancellation is idempotent at the HTTP boundary, but internal cleanup must still distinguish an unknown request from a request that has already reached a terminal state. ## 9. Final cleanup The response cleanup path releases model activity or the model lease, detaches request-local parser state, records metrics, and lets the scheduler reclaim request state. Streaming generators must perform this work in `finally` blocks because the client can disconnect between any two yielded events. ## Lifecycle invariants - Validate before scheduling whenever possible. - Acquire the model before touching tokenizer-derived request state. - Release the same model context that was acquired, even if the global active engine changes. - Never emit content after the terminal protocol event. - Never omit a terminal reason because a parser consumed the final textual delta. - Keep final usage tied to the last generation output, not to a parser-only delta. - Treat disconnect, timeout, cancellation, and exception cleanup as first-class paths. # Documentation page: `concepts/runtime-architecture.md` # Runtime architecture vllm-mlx adapts vLLM-style serving concepts to MLX. The server owns protocol compatibility and request policy, engines own generation, schedulers own concurrent decode state, and model wrappers bridge to the MLX ecosystem. ## Layer map ```text OpenAI SDKs Anthropic SDKs curl / custom clients | | | +------------------+------------------------+ | FastAPI server.py auth, limits, validation, protocol adapters | BaseEngine contract / \ SimpleEngine BatchedEngine direct AsyncEngineCore | Scheduler queues, BatchGenerator, KV caches | mlx-lm | mlx-vlm | mlx-audio | mlx-embeddings | MLX and Metal ``` ## API and protocol layer [`vllm_mlx.server`](../reference/api/vllm_mlx/server.md) creates the FastAPI application and implements the OpenAI-compatible, Anthropic-compatible, audio, embedding, reranking, cache, status, and MCP routes. Its main responsibilities are: 1. Authenticate and rate-limit requests when configured. 2. Validate the requested model and endpoint-specific limits. 3. Normalize OpenAI, Anthropic, Responses API, multimodal, and tool inputs. 4. Resolve the active model and acquire a model lease when registry serving is enabled. 5. Construct sampling, reasoning, structured-output, and tool-parser state. 6. Invoke the selected engine and translate `GenerationOutput` values into protocol responses. 7. Preserve terminal reasons, usage, cancellation, and model release across normal and exceptional exits. The Pydantic wire models live in [`vllm_mlx.api`](../reference/api/vllm_mlx/api/index.md). Protocol conversion is deliberately separate from model execution so the same engines can support multiple client contracts. ## Engine layer [`BaseEngine`](../reference/api/vllm_mlx/engine/base.md) defines the common async contract for loading, stopping, text generation, chat generation, streaming, cache management, and tokenizer access. Two primary implementations serve different workloads: | Engine | Execution model | Best fit | Main tradeoff | | --- | --- | --- | --- | | `SimpleEngine` | Direct calls into model wrappers | One active user, lowest orchestration overhead | Serialized generation paths do not continuously batch independent requests | | `BatchedEngine` | Delegates to `AsyncEngineCore` and a scheduler | Concurrent serving and aggregate throughput | More queue, cache, and lifecycle state | Both return [`GenerationOutput`](../reference/api/vllm_mlx/engine/base.md), which carries text, token IDs, token counts, finish reason, incremental text, completion state, and speculative decoding counters. ## Continuous-batching core [`EngineCore`](../reference/api/vllm_mlx/engine_core.md) owns a model, tokenizer, scheduler, output collectors, and request completion events. [`AsyncEngineCore`](../reference/api/vllm_mlx/engine_core.md) wraps it with the async interface used by `BatchedEngine`. The core runs scheduler steps on one dedicated worker thread. MLX streams are thread-local, so generation streams are rebound inside that worker before decode. This thread-affinity requirement is a correctness invariant, not only a performance choice. ## Scheduler layer [`Scheduler`](../reference/api/vllm_mlx/scheduler.md) turns waiting requests into a running batch, advances mlx-lm's `BatchGenerator`, collects deltas, and finalizes completed or failed requests. Its configuration controls maximum sequences, batch sizes, prefill step size, scheduling policy, cache strategy, memory limits, and optional SSD tiering. Multimodal continuous batching uses [`MLLMScheduler`](../reference/api/vllm_mlx/mllm_scheduler.md), [`MLLMBatchGenerator`](../reference/api/vllm_mlx/mllm_batch_generator.md), and [`MultimodalProcessor`](../reference/api/vllm_mlx/multimodal_processor.md). These components preserve image and video preprocessing state while applying the same request lifecycle concepts. ## Model layer - [`MLXLanguageModel`](../reference/api/vllm_mlx/models/llm.md) wraps text-only mlx-lm loading and generation. - [`MLXMultimodalLM`](../reference/api/vllm_mlx/models/mllm.md) wraps mlx-vlm models and multimodal preprocessing. - [`EmbeddingEngine`](../reference/api/vllm_mlx/embedding.md) serves vector embeddings. - [`RerankEngine`](../reference/api/vllm_mlx/rerank.md) serves cross-encoder scores. - [`STTEngine`](../reference/api/vllm_mlx/audio/stt.md) and [`TTSEngine`](../reference/api/vllm_mlx/audio/tts.md) provide optional audio routes. Runtime patches under [`vllm_mlx.patches`](../reference/api/vllm_mlx/patches/index.md) adapt specific upstream architectures. They should stay model-specific and must not silently alter unrelated model families. ## Parser and constraint layer Reasoning parsers split thinking content from final content. Tool parsers translate model-specific call syntax into OpenAI-compatible tool deltas. Constrained processors modify logits to enforce JSON Schema or to control a reasoning model's transition into final content. These components can suppress or buffer individual deltas. Streaming code must therefore treat parser finalization and the terminal finish-reason chunk as independent obligations. ## Ownership boundaries Several resources have explicit owners: | Resource | Owner | Release point | | --- | --- | --- | | Model and tokenizer | Engine or registry-managed loaded model | Engine stop, registry eviction, or residency unload | | Request state | Scheduler and engine core | Completion, cancellation, or failure | | Model lease | Request model context | Response cleanup, including streaming disconnects | | Prefix and paged KV state | Scheduler cache managers | Cache clear, reset, eviction, or engine close | | Parser state | Individual request stream | Terminal flush or request cleanup | | MCP processes | MCP manager | Server lifespan shutdown | Keeping these boundaries intact prevents model eviction during generation, stale parser state after reload, leaked request futures, and cache state surviving an incompatible model. ## Static documentation on Linux The documentation pipeline parses source with the Python AST and Griffe. It does not import `vllm_mlx`, so GitHub Pages can build on Linux without MLX. Runtime validation remains an Apple Silicon responsibility. # Documentation page: `concepts/scheduling-and-batching.md` # Scheduling and continuous batching Continuous batching keeps a changing set of requests in one decode loop. New requests can join as capacity becomes available, and completed requests leave without waiting for the original batch to finish. ## Core objects | Object | Responsibility | | --- | --- | | [`Request`](../reference/api/vllm_mlx/request.md) | Prompt tokens, sampling parameters, status, generated tokens, cache state, and timing | | [`SchedulerConfig`](../reference/api/vllm_mlx/scheduler.md) | Capacity, batching, prefill, cache, and scheduling controls | | [`Scheduler`](../reference/api/vllm_mlx/scheduler.md) | Waiting and running sets, cache attachment, `BatchGenerator` steps, completion, and recovery | | [`EngineCore`](../reference/api/vllm_mlx/engine_core.md) | Background loop, request submission, output collectors, and model ownership | | [`RequestOutputCollector`](../reference/api/vllm_mlx/output_collector.md) | Low-latency aggregation of scheduler outputs for one request | | [`AsyncEngineCore`](../reference/api/vllm_mlx/engine_core.md) | Async facade used by the batched engine | ## Admission and queues New requests begin in a waiting queue. A scheduler step admits requests while respecting `max_num_seqs`, prefill capacity, and the selected scheduling policy. First-come-first-served is the default; priority scheduling uses the request priority when enabled. Admission also resolves reusable prefix state. A cache hit reduces the number of prompt tokens that require prefill. Cache state must match the current model and remain valid for the cache implementation in use. ## Prefill and decode Prefill processes prompt tokens and creates KV state. Decode advances active sequences one or more tokens at a time. `prefill_batch_size`, `completion_batch_size`, and `prefill_step_size` control the work submitted to mlx-lm's `BatchGenerator`. Large prefill steps can improve throughput but increase latency for other queued work and raise peak memory. Smaller steps improve interleaving at the cost of more scheduler overhead. ## Worker-thread affinity The engine core uses one dedicated worker thread for scheduler steps. MLX generation streams are thread-local, so the worker binds its streams before touching the model. Moving scheduler work between arbitrary executors can produce stream ownership errors even when Python state appears thread-safe. The engine includes a narrow recovery path for recognized stream-thread failures. It rebuilds affected cache or batch state and reschedules active requests. This is a fallback, not permission to ignore thread affinity. ## Output delivery Each scheduler step can produce deltas for several request IDs. The engine core routes each output to its request collector. Streaming callers consume incremental text, while non-streaming callers wait for the collector to assemble a terminal `RequestOutput` or `GenerationOutput`. `stream_interval` controls how often token deltas cross the engine boundary. A value of one minimizes token latency. Larger values reduce Python and serialization overhead at the cost of chunk latency. ## Completion and cancellation A request leaves the running set when it reaches a stop token, token limit, explicit cancellation, or error. The scheduler must finalize cache state, detach the sequence from `BatchGenerator`, notify the matching collector, and make capacity available to waiting requests. Cancellation can race a scheduler step. Code that changes request state should preserve these properties: - A request has one terminal outcome. - A collector is notified exactly once. - Removed sequences cannot reappear in a later batch step. - Cache state is stored only when it is safe and complete enough to reuse. ## Multimodal batching Multimodal scheduling has additional preprocessing and cache inputs. Image or video embeddings can be reused separately from text KV state. The multimodal batch generator tracks prompt and generation throughput while adapting model-specific cache behavior. Use the multimodal scheduler only for wrappers that support its cache and batch contracts. Model-specific runtime patches under `vllm_mlx.patches` extend support for architectures whose upstream attention or MTP implementations do not accept batched cache objects directly. ## Tuning sequence When tuning a server, change one limit at a time: 1. Establish a representative workload with `bench-serve`. 2. Set a safe memory budget and cache mode. 3. Increase `max_num_seqs` until throughput stops improving or tail latency becomes unacceptable. 4. Tune prefill and completion batch sizes. 5. Tune `stream_interval` for the client latency target. 6. Re-run with cache-warm and cache-cold cases. Throughput, time to first token, inter-token latency, memory pressure, and fairness should be evaluated together. # Documentation page: `development/agent-guide.md` # Guide for LLMs and coding agents This page is a stable entry point for tools that need to understand or modify vllm-mlx. It describes where authoritative information lives and how to avoid common mistakes. ## Preferred context order 1. Read [`/llms.txt`](../llms.txt) for the documentation map. 2. Read the relevant concept or user guide. 3. Find the owning module in the [codebase map](codebase-map.md). 4. Query [`/api-inventory.json`](https://vllm-mlx.is-a.dev/api-inventory.json) for exact symbols, signatures, docstrings, visibility, and source lines. 5. Open the generated module page for inline source and related members. 6. Read the nearest tests before proposing a change. Use [`/llms-full.txt`](https://vllm-mlx.is-a.dev/llms-full.txt) only when a large context window can hold the complete corpus. Use [`/source-inventory.json`](https://vllm-mlx.is-a.dev/source-inventory.json) when work also touches maintenance scripts or runnable examples. ## Machine-readable API inventory The JSON inventory has this top-level shape: ```json { "schema_version": "1.0", "repository": "waybarrios/vllm-mlx", "source_branch": "gh-pages", "module_count": 117, "symbol_count": 2003, "modules": [] } ``` Counts change as code is added. Each module includes its path, generated page, docstring, members, and symbols. Each symbol includes: - `full_name` and `qualname` - `kind` - `signature` - complete `docstring` and compact `summary` - conservative `implementation` facts plus calls, state access, returns, raises, decorators, await, and yield metadata - `public`, `addressable`, and `documented` flags - `line`, `end_line`, and a GitHub `#Lx-Ly` source URL Do not infer a source line from a rendered HTML page. Use the inventory URL so review comments stay attached to the exact definition. ## Platform boundary The runtime is designed for macOS on Apple Silicon. Importing many package modules on Linux fails because MLX is unavailable. Documentation tools use the Python AST and Griffe to avoid imports. On Linux, prefer: ```bash python scripts/check_docs_coverage.py mkdocs build --strict ruff check vllm_mlx/ tests/ --select E,F,W --ignore E402,E501,E731,F811,F841 black --check vllm_mlx/ tests/ ``` Run MLX-dependent tests on Apple Silicon. Do not treat a Linux skip as proof of runtime correctness. ## Runtime invariants - MLX generation streams are thread-local. Scheduler steps must stay on their bound worker thread. - A model lease must outlive every request operation that touches the model or tokenizer. - Parser state is request-local and must reset before a new stream. - A parser-suppressed terminal delta still requires a terminal protocol event and finish reason. - Cache reuse requires compatible model identity, token prefix, layout, and layer trimming behavior. - Paged block reference counts and free-list membership must agree. - Model reload must invalidate tokenizer-derived parser caches. - Streaming cleanup must run after disconnect, timeout, cancellation, and ordinary failure. ## Change map | Change | Start here | Focused tests to find | | --- | --- | --- | | OpenAI or Anthropic request fields | `api/models.py`, `api/anthropic_models.py`, `server.py` | API model, adapter, and server tests | | Streaming terminal behavior | `server.py`, `api/streaming.py`, parser implementation | server and streaming regression tests | | Continuous batching | `engine_core.py`, `scheduler.py`, `request.py` | batching, deterministic, stream-safety tests | | Prefix or paged cache | cache module plus `scheduler.py` | memory, prefix, paged, and untrimmable cache tests | | Model loading or eviction | `model_registry.py`, `lifecycle.py`, engine wrappers | registry and lifecycle tests | | Tool calling | parser, `api/tool_calling.py`, `server.py` | parser-specific and promotion tests | | Reasoning | parser, thinking processor, `server.py` | reasoning, thinking-aware, and streaming tests | | MCP | `mcp/` and MCP endpoints | MCP security and execution tests | | Multimodal model | `models/mllm.py`, processor, MLLM scheduler | MLLM and continuous-batching tests | ## Public API documentation rule Every public module, class, function, and method needs a source docstring. New modules are added to the reference automatically. Private and nested helpers still appear in the line-precise source map even when they are not importable API objects. Write docstrings that answer: 1. What contract does this object provide? 2. What do non-obvious parameters mean? 3. What is returned or yielded? 4. Which exceptions or lifecycle constraints matter? 5. Which side effects, locks, threads, caches, or external processes are involved? Avoid duplicating the implementation line by line. Document the behavioral contract and invariants that a caller or maintainer cannot safely infer from the signature. ## Generated files Do not edit generated files under `site/`. They are recreated by MkDocs and are not committed. Edit source Markdown, Python docstrings, or the documentation scripts instead. ## Security-sensitive areas Treat MCP execution, remote media fetching, API authentication, subprocess launch, model download, archive or manifest handling, and filesystem cache paths as trust boundaries. Changes in these areas need focused security review in addition to ordinary correctness tests. # Documentation page: `development/architecture.md` # Architecture vllm-mlx is a layered inference server for Apple Silicon. Protocol code is separated from model execution, concurrent scheduling, caching, model ownership, and model-specific compatibility patches. ## System overview ```text OpenAI and Anthropic clients | FastAPI server | protocol normalization | BaseEngine / \ SimpleEngine BatchedEngine | AsyncEngineCore | Scheduler | model wrappers and caches | mlx-lm | mlx-vlm | mlx-audio | mlx-embeddings | MLX / Metal ``` ## Primary layers ### API layer [`server.py`](../reference/api/vllm_mlx/server.md) owns the FastAPI process, route handlers, authentication, rate limits, endpoint policy, model acquisition, streaming protocol output, and shutdown integration. Pydantic requests, responses, and protocol adapters live in [`api/`](../reference/api/vllm_mlx/api/index.md). ### Engine layer [`BaseEngine`](../reference/api/vllm_mlx/engine/base.md) is the common contract used by the server. [`SimpleEngine`](../reference/api/vllm_mlx/engine/simple.md) calls model wrappers directly. [`BatchedEngine`](../reference/api/vllm_mlx/engine/batched.md) delegates concurrent work to [`AsyncEngineCore`](../reference/api/vllm_mlx/engine_core.md). ### Scheduler layer [`Scheduler`](../reference/api/vllm_mlx/scheduler.md) manages waiting and running requests, mlx-lm `BatchGenerator` state, prefill and decode steps, cache attachment, cancellation, recovery, and terminal outputs. The engine core routes scheduler results into per-request output collectors. Multimodal batching uses separate scheduler, batch generator, processor, and cache components because vision inputs and cache shapes differ from text-only inference. ### Model layer Text generation uses [`MLXLanguageModel`](../reference/api/vllm_mlx/models/llm.md). Vision-language generation uses [`MLXMultimodalLM`](../reference/api/vllm_mlx/models/mllm.md). Embedding, reranking, STT, and TTS engines remain separate optional services with endpoint-specific compatibility policy. ### Cache layer The scheduler can use legacy prefix entries, memory-aware entries, block-aware prefix reuse, or paged KV blocks, with optional SSD persistence. Multimodal and vision-embedding caches cover different preprocessing state. See [Caching](../concepts/caching.md) for invariants and selection guidance. ### Parser and constraint layer Reasoning parsers separate hidden thinking from final content. Tool parsers convert model-family syntax into protocol tool calls. Constrained processors enforce JSON Schema and reasoning-budget transitions at the logits level. See [Parsing and Structured Output](../concepts/parsing-and-structured-output.md). ### Model ownership layer [`ModelManager`](../reference/api/vllm_mlx/model_registry.md) provides registry-backed multi-model loading, memory budgets, eviction, and request leases. [`ResidencyManager`](../reference/api/vllm_mlx/lifecycle.md) provides lazy loading and idle unload for the default model. ## Request flow 1. Middleware authenticates, meters, and validates transport policy. 2. The endpoint validates the request model and protocol schema. 3. Protocol input is normalized into a prompt or internal message list. 4. The server builds sampling, reasoning, tool, and structured-output state. 5. The request acquires its model or resident engine. 6. The simple engine executes directly, or the batched engine submits to the scheduler. 7. Deltas pass through reasoning and tool parsers before protocol encoding. 8. Terminal reason and usage are emitted even when the final textual delta was suppressed. 9. Completion, error, timeout, cancellation, and disconnect paths release request and model state. See [Request Lifecycle](../concepts/request-lifecycle.md) for the complete path. ## Concurrency invariants - MLX generation streams are thread-local. Scheduler steps stay on one bound worker thread. - One model cannot be owned by incompatible active engines unless ownership is transferred through the registry contract. - A request-scoped model lease prevents eviction during generation. - Request collectors receive one terminal result. - Streaming cleanup runs in `finally` paths because clients can disconnect between yields. ## Cache invariants - Cache identity includes model compatibility, not only token equality. - Expired entries cannot satisfy exact or partial prefix lookups. - Memory accounting changes once for each inserted or removed entry. - Paged block reference counts agree with block-table ownership and free-list membership. - Non-trimmable layers are never shortened to manufacture a prefix hit. - Timers, threads, executors, and disk resources have explicit shutdown paths. ## Extension points | Extension | Primary location | Required validation | | --- | --- | --- | | New HTTP field or event | `api/` and `server.py` | Protocol model and server tests | | New tool format | `tool_parsers/` | Complete, streaming, split-marker, and server tests | | New reasoning format | `reasoning/` | Complete, streaming, finalization, and server tests | | New model family | model detection, wrapper, optional patch | Loader, dispatch, generation, and cache tests | | New cache policy | cache module and scheduler | Hit, miss, eviction, concurrency, reset, and recovery tests | | New endpoint model | engine plus endpoint policy | Compatibility, lazy load, limit, and error tests | ## Further reading - [Runtime Architecture](../concepts/runtime-architecture.md) - [Scheduling and Batching](../concepts/scheduling-and-batching.md) - [Models and Modalities](../concepts/models-and-modalities.md) - [Codebase Map](codebase-map.md) - [Complete Python API](../reference/api/index.md) # Documentation page: `development/codebase-map.md` # Codebase map This map identifies the primary owner of each runtime concern. Use the generated [Python API reference](../reference/api/index.md) for every object and line-precise source links. ## Entry points | Path | Responsibility | | --- | --- | | `vllm_mlx/cli.py` | `vllm-mlx` command tree, server configuration, model workflow, and benchmark dispatch | | `vllm_mlx/server.py` | FastAPI application, route handlers, protocol streaming, lifecycle integration, and process entry point | | `vllm_mlx/benchmark.py` | Local model benchmark entry point | | `vllm_mlx/bench_serve.py` | HTTP serving benchmark and workload contract runner | | `vllm_mlx/gradio_app.py` | Multimodal Gradio chat application | | `vllm_mlx/gradio_text_app.py` | Text-only Gradio chat application | | `vllm_mlx/plugin.py` | vLLM out-of-tree MLX platform registration | ## API contracts `vllm_mlx/api/` contains Pydantic wire models and conversion helpers: - `models.py` defines OpenAI-compatible requests and responses. - `responses_models.py` defines Responses API items and streaming events. - `anthropic_models.py` defines Anthropic Messages types. - `anthropic_adapter.py` converts Anthropic content and tools to internal OpenAI-style messages and converts results back. - `prompt_canonicalize.py` normalizes system prompts. - `streaming.py` provides a low-overhead SSE JSON encoder. - `tool_calling.py` contains protocol-level tool utilities. - `harmony_tools.py` renders Harmony tool definitions. - `utils.py` contains shared content and model-detection helpers. ## Engines and request state - `engine/base.py` is the stable engine interface and shared generation output. - `engine/simple.py` handles direct text and multimodal generation. - `engine/batched.py` adapts the continuous-batching core to `BaseEngine`. - `engine/chat_template_safety.py` normalizes messages before Jinja templates. - `engine_core.py` owns the background scheduler loop, request collectors, and model ownership. - `request.py` defines request status, sampling parameters, request state, and scheduler output. - `output_collector.py` maps scheduler deltas and terminal results back to individual async callers. ## Scheduling and inference - `scheduler.py` runs text continuous batching with mlx-lm `BatchGenerator`. - `mllm_scheduler.py` schedules multimodal requests. - `mllm_batch_generator.py` advances multimodal batches and reports throughput. - `model_runner.py` exposes the vLLM-facing MLX model runner. - `mlx_streams.py` owns MLX thread-stream binding helpers. - `multimodal_processor.py` prepares text, image, and video model inputs. ## Model ownership and workflow - `models/llm.py` wraps text models. - `models/mllm.py` wraps vision-language models. - `model_registry.py` provides registry-backed loading, leases, memory budgets, and eviction. - `lifecycle.py` provides lazy load and automatic idle unload for the default model. - `model_workflow.py` implements inspect, acquire, convert, register, and qualify operations. - `text_model_from_vlm.py` reconstructs an mlx-lm text model from mlx-vlm-loaded weights. - `endpoint_model_policies.py` resolves compatible optional-endpoint models. ## Caches - `prefix_cache.py` implements entry and block-aware prefix reuse. - `memory_cache.py` implements memory-budgeted prefix reuse and optional quantization. - `paged_cache.py` implements reference-counted block storage and sharing. - `ssd_cache.py` implements serialized disk tiering. - `mllm_cache.py` stores multimodal prompt state. - `vision_embedding_cache.py` stores reusable vision preprocessing results. - `utils/mamba_cache.py` adapts state-space model caches to batching. ## Output interpretation - `reasoning/` contains complete and streaming reasoning parsers. - `tool_parsers/` contains model-family-specific tool-call parsers and the parser registry. - `constrained/` contains tokenizer enforcement caches, JSON Schema logits processing, and the thinking state machine. - `utils/harmony_render.py` renders GPT-OSS Harmony prompts. - `api/harmony_tools.py` converts tool definitions for Harmony. ## Optional model services - `audio/` contains preprocessing, STT, and TTS engines. - `audio_limits.py` validates optional audio route inputs. - `embedding.py` loads and serves embedding models. - `rerank.py` loads and serves reranking models. - `rerank_forward.py` implements the MLX sequence-classification forward pass. ## MCP - `mcp/config.py` loads and validates server definitions. - `mcp/client.py` manages one MCP connection. - `mcp/manager.py` coordinates all configured servers. - `mcp/tools.py` converts tool schemas. - `mcp/executor.py` applies concurrency and invokes tools. - `mcp/security.py` validates commands, paths, arguments, and environment. - `mcp/types.py` defines MCP-facing data structures. ## Model-specific compatibility `patches/` contains narrow runtime adaptations for Gemma 4, GLM-4V MoE, Qwen3.5, and Qwen3-Next MTP. `specprefill.py` contains sparse-prefill logic. `optimizations.py` reports hardware and selects safe optimization values. ## Tests Tests are organized by behavior rather than mirroring every module. Search for the public object or endpoint first, then inspect the nearest regression file. Linux CI covers static checks and non-MLX behavior. Apple Silicon CI covers model, scheduler, cache, server, and streaming paths that require MLX. ## Documentation tools - `scripts/docs_inventory.py` parses tracked Python source without importing it. - `scripts/gen_api_reference.py` creates module pages and source maps. - `scripts/check_docs_coverage.py` enforces module, symbol, and public-docstring coverage. - `scripts/mkdocs_hooks.py` creates Markdown mirrors, `llms-full.txt`, and `api-inventory.json`. - `.github/workflows/docs.yml` validates pull requests targeting `gh-pages` and deploys pushes from that branch. # Documentation page: `development/contributing.md` # Contributing We welcome contributions to vllm-mlx! ## Getting Started ```bash # Clone the repository git clone https://github.com/waybarrios/vllm-mlx.git cd vllm-mlx # Install with dev dependencies pip install -e ".[dev]" ``` For documentation work, also install the documentation extra: ```bash pip install -r docs/requirements.txt ``` ## Development Workflow ### Running Tests ```bash # Run the full suite on Apple Silicon pytest tests/ # Run a focused test first pytest tests/test_paged_cache.py -v ``` MLX-dependent tests require an Apple Silicon environment. Other platforms can still run supported static checks and non-MLX tests. ### Code Style ```bash # Lint ruff check vllm_mlx/ tests/ --select E,F,W --ignore E402,E501,E731,F811,F841 # Check formatting black --check vllm_mlx/ tests/ # Type-check relevant changes mypy vllm_mlx/ --ignore-missing-imports --no-error-summary ``` ### Documentation checks ```bash python scripts/check_docs_coverage.py mkdocs build --strict ``` Every public module, class, function, and method must have a docstring. The API reference and line-precise source maps are generated automatically from tracked Python files. See [Documentation Development](documentation.md). ### Running Benchmarks ```bash # LLM benchmark vllm-mlx-bench --model mlx-community/Qwen3-0.6B-8bit # Image benchmark vllm-mlx-bench --model mlx-community/Qwen3-VL-8B-Instruct-4bit # Video benchmark vllm-mlx-bench --model mlx-community/Qwen3-VL-8B-Instruct-4bit --video ``` ## Areas for Contribution - **Bug fixes** - Fix issues and improve stability - **Performance optimizations** - Improve inference speed - **New features** - Add functionality - **Documentation** - Improve docs and examples - **Benchmarks** - Test on different Apple Silicon chips - **Model support** - Test and add new models ## Pull Request Process 1. Fork the repository. 2. Create a focused feature branch. 3. Make the smallest coherent change that resolves the problem. 4. Add regression coverage when practical. 5. Run the relevant tests and code-quality checks. 6. Submit a pull request describing the impact and verification performed. ## Code Structure See [Architecture](architecture.md) for the runtime overview and the [Codebase Map](codebase-map.md) for module ownership. ## Testing on Different Hardware If you have access to different Apple Silicon chips (M1, M2, M3, M4, M5), benchmark results are valuable: ```bash vllm-mlx-bench --model mlx-community/Qwen3-0.6B-8bit --output results_m4.json ``` ## Questions? Open an issue at [GitHub Issues](https://github.com/waybarrios/vllm-mlx/issues). # Documentation page: `development/documentation.md` # Documentation development The documentation site uses MkDocs Material, mkdocstrings, and static AST inventory tools. It publishes all existing Markdown pages, an exhaustive Python reference, and machine-readable artifacts for LLMs and coding agents. ## Install documentation dependencies ```bash python -m pip install -r docs/requirements.txt ``` This installs only the static documentation toolchain, so Linux builders do not need MLX. Apple Silicon developers may instead use `python -m pip install -e ".[docs]"` when they also want an editable runtime installation. ## Preview locally ```bash python scripts/check_docs_coverage.py mkdocs serve ``` Open `http://127.0.0.1:8000/`. Checked-in API pages are refreshed with `python scripts/gen_api_reference.py`. ## Run the release-equivalent build ```bash python scripts/check_docs_coverage.py mkdocs build --strict ``` Strict mode turns configuration, navigation, cross-reference, and link warnings into failures. The generated site is written to `site/`. ## Coverage contract `scripts/check_docs_coverage.py` enforces: - One generated API page for every tracked `vllm_mlx/**/*.py` module. - One source-map entry for every class, function, method, nested class, and nested function. - One searchable symbol-index entry and callable contract for every definition. - An explicit parameter record with kind, type, requirement, default, and description for every callable input. - A module docstring for every module. - A docstring for every public, addressable class, function, and method. - Conservative implementation facts for every private or nested definition, generated from its own AST body. - An H1 heading in every hand-written Markdown page. - Presence of the Pages workflow and agent-facing artifacts. The check prints module, symbol, public explanation, and Markdown page totals. New code cannot silently reduce documentation coverage. ## Add or change Python code New modules are discovered from Git-tracked Python files. No navigation file needs updating. Add a module docstring and document every public object in the source: ```python def resolve_model(name: str, *, allow_remote: bool = True) -> ModelSpec: """Resolve a user-facing model name into a validated model specification. Args: name: Registry name, local path, or supported remote identifier. allow_remote: Whether remote model identifiers may be resolved. Returns: The validated model specification used by the loader. Raises: ValueError: If the name is empty or incompatible with policy. """ ``` Private and nested definitions are included in the source map automatically. Add a docstring when their contract or invariants are not obvious. ## Add a guide Place English pages in the appropriate `docs/` section. Use a descriptive H1 and relative links between documentation pages. Add translated pages under `docs/es`, `docs/fr`, or `docs/zh` when a translation is available. Every guide should distinguish: - Supported behavior from examples or recommendations. - Defaults from optional configuration. - Linux-compatible checks from Apple Silicon runtime checks. - Public contracts from implementation details. ## Generated API pages `scripts/gen_api_reference.py` writes deterministic, checked-in pages for every Python module. The workflow runs the generator with `--check` so stale pages fail CI. Each module page contains: - Module summary and complete source link. - Full mkdocstrings rendering with signatures, parsed parameter sections, and inline source. - Expandable contracts for every public, private, and nested definition. - Explicit inputs, defaults, return annotations, direct exceptions, and source-grounded behavior. - Exact `#Lx-Ly` links for every definition. `docs/reference/python-symbols.md` adds a filterable index over every runtime symbol and signature. Source links are stored against `gh-pages` for readable Markdown, then rewritten during the build to the exact commit in `VLLM_MLX_DOCS_SOURCE_REVISION`. This keeps `#Lx-Ly` permalinks correct after either branch changes. ## LLM and agent artifacts `docs/llms.txt` is the compact, curated index defined by the emerging llms.txt convention. `scripts/mkdocs_hooks.py` creates these build artifacts: - `llms-full.txt`: every hand-written page plus a complete record for every Python symbol. - `api-inventory.json`: structured module and symbol metadata. - Markdown mirrors of hand-written and generated API pages. The compact index is hand-maintained because page priority and descriptions require editorial judgment. The complete corpus and inventory are generated so they cannot drift from source. ## GitHub Pages deployment `.github/workflows/docs.yml` builds documentation on pull requests targeting `gh-pages`. A successful push to `gh-pages` uploads the site artifact and deploys it through the protected `github-pages` environment. Documentation changes stay on that dedicated branch and do not need to enter `main`. Repository administrators must select **GitHub Actions** as the Pages publishing source once in repository settings. The workflow uses least-privilege permissions: the build reads contents, and only the deploy job receives `pages: write` and `id-token: write`. ## Review checklist - Run the coverage check and strict build. - Open the changed page in the local preview. - Verify commands and request examples against current code. - Follow every new relative link. - Confirm source links use the correct module, immutable commit, and line range. - Inspect `site/llms.txt`, `site/llms-full.txt`, and `site/api-inventory.json`. - Check that no generated `site/` content is staged. - Run focused tests when documentation tools or source docstrings change. # Documentation page: `es/benchmarks/README.md` # Benchmarks Benchmarks de rendimiento para vllm-mlx en Apple Silicon. ## Tipos de benchmark - [Benchmarks de LLM](llm.md) - Rendimiento de generación de texto - [Benchmarks de imagen](image.md) - Rendimiento de comprension de imagenes - [Benchmarks de video](video.md) - Rendimiento de comprension de video ## Comandos rápidos ```bash # LLM benchmark vllm-mlx-bench --model mlx-community/Qwen3-0.6B-8bit # Image benchmark vllm-mlx-bench --model mlx-community/Qwen3-VL-8B-Instruct-4bit # Video benchmark vllm-mlx-bench --model mlx-community/Qwen3-VL-8B-Instruct-4bit --video ``` ## Valores predeterminados de los scripts de prueba Los scripts de benchmark independientes tienen modelos predeterminados integrados, por lo que puedes ejecutar: ```bash python tests/test_continuous_batching.py python tests/test_prefix_cache.py ``` Valores predeterminados: - `tests/test_continuous_batching.py` → `mlx-community/Qwen3-8B-6bit` - `tests/test_prefix_cache.py` → `mlx-community/Qwen3-0.6B-8bit` Para probar con otros modelos, usa el parámetro opcional `--model`: ```bash python tests/test_continuous_batching.py --model mlx-community/Qwen3-0.6B-8bit python tests/test_prefix_cache.py --model mlx-community/Qwen3-8B-6bit ``` ## Hardware Los benchmarks se recopilaron en las siguientes configuraciones de Apple Silicon: | Chip | Memoria | Python | |------|---------|--------| | Apple M4 Max | 128 GB unificada | 3.13 | | Apple M1 Max | 64 GB unificada | 3.12 | Los resultados pueden variar en distintos chips de Apple Silicon. ## Contribuir benchmarks Si tienes un chip de Apple Silicon diferente, comparte tus resultados: ```bash vllm-mlx-bench --model mlx-community/Qwen3-0.6B-8bit --output results.json ``` Abre un issue con tus resultados en [GitHub Issues](https://github.com/waybarrios/vllm-mlx/issues). # Documentation page: `es/benchmarks/audio.md` # Benchmarks de Audio ## Benchmarks de Speech-to-Text (STT) ### Ejecutar benchmarks de STT ```bash # Run with default test audio python examples/benchmark_audio.py --stt # Run with your own audio file python examples/benchmark_audio.py --stt --audio path/to/audio.wav ``` ### Resultados (M4 Max, 128GB) **Audio de prueba:** 46.7 segundos de voz sintetizada | Model | Parameters | Load Time | Transcribe Time | RTF* | |-------|------------|-----------|-----------------|------| | whisper-tiny | 39M | 0.34s | 0.24s | **197x** | | whisper-small | 244M | 0.18s | 0.47s | **98x** | | whisper-medium | 769M | 0.35s | 1.15s | **41x** | | whisper-large-v3 | 1.5B | 0.50s | 1.96s | **24x** | | whisper-large-v3-turbo | 809M | 0.12s | 0.86s | **55x** | *RTF = Real-Time Factor (mayor es más rápido). Un RTF de 100x significa que 1 minuto de audio se transcribe en aprox. 0.6 segundos.* ### Resultados (M1 Max, 64GB) STT con Parakeet (entorno predeterminado, Whisper no disponible por incompatibilidad de dependencia con numpy): | Model | Load Time | Transcribe Time | RTF | |-------|-----------|-----------------|-----| | parakeet-tdt-0.6b-v2 | 0.28s | 1.01s | **9.9x** | | parakeet-tdt-0.6b-v3 | 0.30s | 0.19s | **52.7x** | STT con Whisper (`numpy==2.3.5` explícito + `uv run --no-sync`): | Model | Load Time | Transcribe Time | RTF | |-------|-----------|-----------------|-----| | whisper-tiny | 4.02s | 1.05s | **9.5x** | | whisper-small | 10.15s | 1.03s | **9.7x** | | whisper-medium | 22.96s | 2.20s | **4.6x** | | whisper-large-v3 | 38.34s | 0.96s | **10.5x** | | whisper-large-v3-turbo | 21.79s | 0.70s | **14.3x** | | parakeet-tdt-0.6b-v2 | 0.47s | 0.18s | **54.4x** | | parakeet-tdt-0.6b-v3 | 1.13s | 0.18s | **54.6x** | ### Recomendaciones de modelos | Use Case | Recommended Model | Why | |----------|-------------------|-----| | **Transcripcion en tiempo real** | whisper-tiny | El más rápido (197x RTF), baja latencia | | **Uso general** | whisper-large-v3-turbo | Mejor equilibrio entre velocidad (55x) y calidad | | **Mayor precision** | whisper-large-v3 | El más preciso, soporta más de 99 idiomas | | **Memoria reducida** | whisper-small | Buena calidad con 244M parámetros | ### Calidad de transcripción Todos los modelos transcribieron correctamente el audio de prueba. Ejemplo de salida: ``` Input text: "Welcome to this comprehensive speech to text demonstration. This audio sample is designed to test the accuracy and speed of various speech recognition models. The quick brown fox jumps over the lazy dog..." Whisper-large-v3 output: "Welcome to this comprehensive speech to text demonstration. This audio sample is designed to test the accuracy and speed of various speech recognition models. The quick brown fox jumps over the lazy dog..." (identical) ``` ### Idiomas soportados Los modelos Whisper soportan más de 99 idiomas, entre ellos: - Inglés, español, francés, alemán, italiano, portugués - Chino (mandarín, cantonés), japonés, coreano - Árabe, hindi, ruso, turco, ucraniano - Y muchos más ## Benchmarks de Text-to-Speech (TTS) ### Ejecutar benchmarks de TTS ```bash python examples/benchmark_audio.py --tts ``` ### Resultados (M4 Max, 128GB) **Prueba:** Generar audio para 3 muestras de texto (corta, media, larga) | Model | Load Time | Chars/sec | RTF* | |-------|-----------|-----------|------| | Kokoro-82M-bf16 | 0.8s | 350+ | **22x** | | Kokoro-82M-4bit | 0.4s | 320+ | **20x** | *RTF = Real-Time Factor. Un RTF de 22x significa que 1 segundo de audio se genera en aprox. 0.045 segundos.* ### Resultados de TTS (M1 Max, 64GB) | Model | Load Time | Avg Chars/s | Avg RTF | |-------|-----------|-------------|---------| | Kokoro-82M-bf16 | 2.81s | 176.0 | **11.9x** | | Kokoro-82M-4bit | 0.22s | 225.6 | **15.5x** | ### Calidad de TTS Kokoro produce voz con sonido natural, con: - 11 voces integradas (masculinas y femeninas) - Soporte para 8 idiomas (inglés, español, francés, japonés, chino, italiano, portugués, hindi) - 82M parámetros, rápido y liviano ## Benchmarks de procesamiento de audio ### SAM-Audio (separacion de fuentes) **Prueba:** Separar la bateria de una cancion de rock de 30 segundos | Metric | Value | |--------|-------| | Model | sam-audio-large-fp16 | | Processing time | ~20s | | Peak memory | ~27 GB | | Output sample rate | 48000 Hz | ## Ejecutar todos los benchmarks de audio ```bash # Run all benchmarks python examples/benchmark_audio.py --all # Or run individually python examples/benchmark_audio.py --stt python examples/benchmark_audio.py --tts ``` ## Modelos disponibles en mlx-community ### Modelos STT - `mlx-community/whisper-tiny-mlx` - `mlx-community/whisper-small-mlx` - `mlx-community/whisper-medium-mlx` - `mlx-community/whisper-large-v3-mlx` - `mlx-community/whisper-large-v3-turbo` - `mlx-community/parakeet-tdt-0.6b-v2` - `mlx-community/parakeet-tdt-0.6b-v3` ### Modelos TTS - `mlx-community/Kokoro-82M-bf16` (recommended) - `mlx-community/Kokoro-82M-4bit` - `mlx-community/chatterbox-turbo-fp16` - `mlx-community/VibeVoice-Realtime-0.5B-4bit` ### Procesamiento de audio - `mlx-community/sam-audio-large-fp16` # Documentation page: `es/benchmarks/image.md` # Benchmarks de Imágenes ## Ejecutar Benchmarks de Imágenes ```bash # Benchmark completo (10 resoluciones) vllm-mlx-bench --model mlx-community/Qwen3-VL-8B-Instruct-4bit # Benchmark rápido (4 resoluciones) vllm-mlx-bench --model mlx-community/Qwen3-VL-8B-Instruct-4bit --quick ``` ## Resultados - Qwen3-VL-8B-Instruct-4bit (M4 Max, 128GB) | Resolution | Pixels | Time | Tokens | Speed | |------------|--------|------|--------|-------| | 224x224 | 50K | 1.04s | 78 | 74.8 tok/s | | 336x336 | 113K | 0.94s | 64 | 68.3 tok/s | | 448x448 | 201K | 1.45s | 70 | 48.1 tok/s | | 512x512 | 262K | 1.58s | 99 | 62.8 tok/s | | 672x672 | 452K | 1.83s | 83 | 45.3 tok/s | | 768x768 | 590K | 2.05s | 91 | 44.3 tok/s | | 896x896 | 803K | 2.61s | 90 | 34.5 tok/s | | 1024x1024 | 1.0M | 2.79s | 76 | 27.2 tok/s | | 1280x720 | 922K | 2.97s | 96 | 32.4 tok/s | | 1920x1080 | 2.1M | 6.30s | 89 | 14.1 tok/s | **Resumen:** Promedio de 45.2 tok/s en todas las resoluciones. Más rápido en 224x224 (74.8 tok/s), más lento en 1920x1080 (14.1 tok/s) ## Resultados - Qwen3-VL-8B-Instruct-4bit (M1 Max, 64GB) Benchmark MLLM local: | Resolution | Pixels | Time | Tokens | Speed | |------------|--------|------|--------|-------| | 224x224 | 50K | 1.84s | 78 | 42.5 tok/s | | 448x448 | 201K | 2.28s | 70 | 30.7 tok/s | | 768x768 | 590K | 4.39s | 91 | 20.7 tok/s | | 1024x1024 | 1.0M | 6.41s | 76 | 11.9 tok/s | | Configs | Total Time (s) | Total Tokens | Aggregate Tok/s | |---------|-----------------|--------------|-----------------| | 4 | 14.92 | 315 | 21.1 | ## Resultados - Qwen3-VL-4B-Instruct-3bit Server (M1 Max, 64GB) | Resolution | Pixels | Time | Tokens | Speed | |------------|--------|------|--------|-------| | 224x224 | 50K | 1.65s | 113 | 68.4 tok/s | | 448x448 | 201K | 2.09s | 120 | 57.5 tok/s | | 768x768 | 590K | 2.93s | 106 | 36.2 tok/s | | 1024x1024 | 1.0M | 4.12s | 100 | 24.3 tok/s | | Configs | Total Time (s) | Total Tokens | Aggregate Tok/s | |---------|-----------------|--------------|-----------------| | 4 | 10.79 | 439 | 40.7 | ## Resultados del Prefix Cache MLLM ``` ====================================================================== MLLM PREFIX CACHE TEST ====================================================================== Model: mlx-community/Qwen3-VL-4B-Instruct-3bit Test: Verify KV cache reuse for repeated image/video + prompt combinations Expected behavior: - Same image + same prompt → cache HIT - Same image + different prompt → cache MISS - Different image + same prompt → cache MISS ---------------------------------------------------------------------- SETUP: Loading Model ---------------------------------------------------------------------- Model loaded in 0.11s ---------------------------------------------------------------------- SETUP: Creating Test Images ---------------------------------------------------------------------- Resized: 224x224, 336x336, 512x512, 768x768 ---------------------------------------------------------------------- TEST 1: Image Cache - Basic Hit/Miss ---------------------------------------------------------------------- Results: Step | Description | Expected | Actual | Time | Status -------+---------------------------+----------+--------+--------+------- 1a | First image+prompt | MISS | MISS | 0.10ms | ✓ 1b | Same image+prompt | HIT | HIT | 0.18ms | ✓ 1c | Different prompt | MISS | MISS | 0.01ms | ✓ 1d | Return to original | HIT | HIT | 0.18ms | ✓ ---------------------------------------------------------------------- TEST 2: Different Images ---------------------------------------------------------------------- Results: Step | Description | Expected | Actual | Time | Status -------+---------------------------+----------+--------+--------+------- 2a | Image A first request | MISS | MISS | 0.01ms | ✓ 2b | Image B first request | MISS | MISS | 0.01ms | ✓ 2c | Image A cached | HIT | HIT | 0.13ms | ✓ ---------------------------------------------------------------------- TEST 3: Image Resolutions ---------------------------------------------------------------------- Results: Step | Description | Expected | Actual | Time | Status -------+-----------------------+----------+--------+--------+------- 3.1a | 224x224 first | MISS | MISS | 0.01ms | ✓ 3.1b | 224x224 cached | HIT | HIT | 0.20ms | ✓ 3.2a | 336x336 first | MISS | MISS | 0.01ms | ✓ 3.2b | 336x336 cached | HIT | HIT | 0.21ms | ✓ 3.3a | 512x512 first | MISS | MISS | 0.12ms | ✓ 3.3b | 512x512 cached | HIT | HIT | 0.20ms | ✓ 3.4a | 768x768 first | MISS | MISS | 0.12ms | ✓ 3.4b | 768x768 cached | HIT | HIT | 0.24ms | ✓ ====================================================================== ``` ## Estrategia de Clave de Cache - **Images**: `hash(image_content) + hash(prompt)` La misma imagen con el mismo prompt siempre generara un acierto en el cache. Una imagen diferente o un prompt diferente generara un fallo. ## Consejos de Rendimiento - Las resoluciones menores se procesan más rápido (224x224 vs 1920x1080) - Usa la resolucion adecuada para tu tarea - Agrupa imagenes de tamanio similar para un rendimiento consistente ## Referencia de Métricas | Metric | Description | |--------|-------------| | Resolution | Dimensiones de la imagen (ancho x alto) | | Pixels | Total pixel count | | Time | Tiempo de generación | | Tokens | Tokens de salida generados | | Speed | Tokens por segundo (tok/s) | # Documentation page: `es/benchmarks/llm.md` # Benchmarks de LLM ## Ejecutar benchmarks de LLM ```bash vllm-mlx-bench --model mlx-community/Llama-3.2-1B-Instruct-4bit --prompts 5 --max-tokens 256 ``` ## Resultados (M4 Max, 128GB) | Model | Gen Speed | TTFT* | Memory | |-------|-----------|-------|--------| | Qwen3-0.6B-8bit | 402.3 tok/s | 58.6 ms | 0.68 GB | | Llama-3.2-1B-Instruct-4bit | 463.6 tok/s | 49.2 ms | 0.69 GB | | Qwen2.5-1.5B-Instruct-4bit | 308.5 tok/s | 86.2 ms | 0.84 GB | | Llama-3.2-3B-Instruct-4bit | 200.1 tok/s | 81.4 ms | 1.79 GB | | Qwen3-30B-A3B-4bit | 123.9 tok/s | 126.9 ms | 16.05 GB | | NVIDIA-Nemotron-3-Nano-30B-A3B-MLX-6Bit | 122.9 tok/s | 72.3 ms | 23.98 GB | *TTFT = Time to First Token (latencia hasta que el modelo comienza a generar) ## Resultados (M1 Max, 64GB) | Model | Runs | Prompt Tok | Gen Tok | Total Time (s) | TTFT Mean (ms) | TPOT Mean (ms) | Gen Speed (tok/s) | Total Throughput (tok/s) | |-------|------|------------|---------|-----------------|-----------------|-----------------|-------------------|--------------------------| | Qwen3-0.6B-8bit | 5 | 56 | 1280 | 5.66 | 119.0 | 3.97 | 251.9 | 236.1 | ## Resultados de continuous batching | Model | Single Request | Batch (5 req) | Speedup | |-------|----------------|---------------|---------| | Llama-3.2-1B-Instruct-4bit | 299.1 tok/s | 613.0 tok/s | **2.05x** | | Llama-3.2-3B-Instruct-4bit | 137.6 tok/s | 208.1 tok/s | **1.51x** | | Qwen3-0.6B-8bit | 328.1 tok/s | 1111.8 tok/s | **3.39x** | | Qwen3-30B-A3B-4bit | 98.1 tok/s | 233.3 tok/s | **2.38x** | | Qwen2.5-1.5B-Instruct-4bit | 196.9 tok/s | 322.2 tok/s | **1.64x** | *Con 5 solicitudes concurrentes se observa una mejora de throughput de 1.5x a 3x.* ### Continuous batching (M1 Max, 64GB) | Requests | Total Tokens | Total Time (s) | Throughput (tok/s) | Requests/sec | |----------|--------------|-----------------|--------------------|--------------| | 5 | 315 | 0.64 | 492.5 | 7.82 | ## Rendimiento de streaming | Model | TTFT | Generation Speed | |-------|------|------------------| | Llama-3.2-1B-Instruct-4bit | ~4.6ms | 218.9 tok/s | | Llama-3.2-3B-Instruct-4bit | ~10.7ms | 93.6 tok/s | | Qwen3-0.6B-8bit | ~3.0ms | 328.5 tok/s | | Qwen3-30B-A3B-4bit | ~10.2ms | 98.4 tok/s | | Qwen2.5-1.5B-Instruct-4bit | ~7.1ms | 140.3 tok/s | ### Detokenizador en streaming (M1 Max, 64GB) `vllm-mlx bench-detok`: | Tokens | Iterations | Naive Time | Streaming Time | Speedup | |--------|------------|------------|----------------|---------| | 742 | 5 | 1.69ms | 0.71ms | 2.39x | `examples/benchmark_detokenizer.py`: | Sequence | Tokens | decode() | Streaming | Speedup | |----------|--------|----------|-----------|---------| | Short | 8 | 0.029ms | 0.028ms | 1.04x | | Medium | 103 | 0.206ms | 0.129ms | 1.59x | | Long | 511 | 1.040ms | 0.502ms | 2.07x | | 1K | 1191 | 2.446ms | 1.178ms | 2.08x | | 2K | 2381 | 4.949ms | 2.356ms | 2.10x | | 4K | 4761 | 9.887ms | 5.398ms | 1.83x | Speedup promedio: 1.79x ## Resultados del prefix cache ### Prefix cache (M4 Max, 128GB) ``` ====================================================================== LLM PREFIX CACHE TEST ====================================================================== Model: mlx-community/Qwen3-0.6B-8bit Expected behavior: - Same prompt → cache HIT - Different prompt → cache MISS ---------------------------------------------------------------------- Results: Step | Description | Expected | Actual | Status -------+---------------------+----------+--------+------- 1a | First request | MISS | MISS | ✓ 1b | Same prompt | HIT | HIT | ✓ 1c | Different prompt | MISS | MISS | ✓ 1d | Return to prompt 1 | HIT | HIT | ✓ ====================================================================== ``` ### Prefix cache (M1 Max, 64GB) | Test | Expected | Actual | Time | Status | |------|----------|--------|------|--------| | First request | MISS | MISS | 203.5ms | PASS | | Same prompt | HIT | HIT | 131.6ms | PASS | | Different prompt | MISS or PREFIX_HIT | PREFIX_HIT (5 tok) | 135.3ms | PASS | Estadísticas finales del cache: | Cache Hits | Cache Misses | Hit Rate | Tokens Saved | Cached Speedup | |------------|--------------|----------|--------------|----------------| | 2 | 1 | 66.7% | 20 | 1.55x | ## Resultados del paged cache *Prueba: 20 solicitudes de inferencia reales en 2 rondas con un system prompt compartido de aproximadamente 286 tokens* ``` ====================================================================== PAGED KV CACHE - REAL INFERENCE TEST ====================================================================== -------------------------------------------------- Test 1: WITHOUT Paged Cache (2 rounds of 10) -------------------------------------------------- Time: 1.47s Throughput: 681.2 tok/s Cache hits: 0 Tokens saved: 0 -------------------------------------------------- Test 2: WITH Paged Cache (2 rounds of 10) -------------------------------------------------- Time: 1.31s Throughput: 765.8 tok/s Paged Cache Stats: Blocks allocated: 25 Shared blocks: 4 Cache hits: 10 Tokens saved: 2560 ================================================== SUMMARY ================================================== Without paged cache: 681.2 tok/s With paged cache: 765.8 tok/s Speedup: 1.12x Cache hits: 10 (all Round 2 requests) Tokens saved: 2,560 (~256 tokens × 10 requests) ================================================== ``` ### Paged KV cache (M1 Max, 64GB) Benchmark de inferencia (20 solicitudes): | Mode | Time (s) | Throughput (tok/s) | |------|----------|--------------------| | Without paged cache | 3.43 | 291.8 | | With paged cache | 3.42 | 292.2 | | Speedup | Blocks Allocated | Shared Blocks | Cache Hits | Tokens Saved | |---------|------------------|---------------|------------|--------------| | 1.00x | 45 | 4 | 10 | 2560 | Inferencia concurrente real (20 solicitudes): | Mode | Time (s) | Throughput (tok/s) | |------|----------|--------------------| | Without paged cache | 4.32 | 231.7 | | With paged cache | 4.35 | 229.7 | | Speedup | Blocks Allocated | Shared Blocks | Cache Hits | Tokens Saved | |---------|------------------|---------------|------------|--------------| | 0.99x | 49 | 8 | 10 | 5120 | Demostración de ahorro de memoria: | Scenario | Memory Savings | |----------|----------------| | Shared system prompts | 70.8% | | Concurrent memory efficiency | 83.5% | | Prefix sharing branches | 38.5% | ## Análisis del detokenizador en streaming *Investigación Fase 9.1: `BPEStreamingDetokenizer` de mlx-lm vs `tokenizer.decode()` naive* ### Contexto El enfoque naive llama a `decode([token])` por cada token. En teoria, los detokenizadores en streaming ofrecen complejidad O(T) frente a O(T²) del decode naive. ### Resultados del benchmark aislado ```bash vllm-mlx bench-detok ``` Al reutilizar la misma instancia del detokenizador (con `reset()` entre usos): | Sequence | Tokens | Naive decode() | Streaming | Speedup | |----------|--------|----------------|-----------|---------| | Short | 8 | 0.020ms | 0.019ms | 1.05x | | Medium | 103 | 0.155ms | 0.097ms | 1.59x | | Long | 511 | 0.752ms | 0.371ms | **2.03x** | | 1K tokens | 1191 | 1.743ms | 0.833ms | **2.09x** | | 2K tokens | 2381 | 3.493ms | 1.737ms | **2.01x** | ### Hallazgo clave: costo de creación de instancias Crear una nueva instancia de `BPEStreamingDetokenizer` es **extremadamente costoso**: ``` 100 tokenizer.detokenizer calls: 5.266s (52.7ms each!) ``` Esto significa que crear un nuevo detokenizador por solicitud agrega **aproximadamente 52ms de sobrecarga**, anulando cualquier beneficio. ### Impacto en uso real Al integrarlo en el scheduler (un detokenizador por solicitud): | Metric | Naive decode() | Streaming (new instance) | |--------|----------------|--------------------------| | Throughput (20 req) | 681 tok/s | 275 tok/s | | Impact | - | **-60% slower** | ### Conclusión El detokenizador en streaming **no es viable actualmente** para uso por solicitud, debido al costo de creación de instancias. El enfoque naive con `decode([token])` sigue siendo más rápido en la práctica. **Optimizacion futura**: crear un pool de instancias de detokenizador al inicio y reutilizarlas entre solicitudes. ## Referencia de métricas | Metric | Description | |--------|-------------| | **TTFT** | Time to First Token: latencia hasta que el modelo comienza a responder (ms) | | **TPOT** | Time Per Output Token: tiempo entre cada token generado (ms/token) | | **Generation TPS** | Tokens de salida por segundo (tok/s) | | **Processing TPS** | Tokens de entrada/prompt procesados por segundo (tok/s) | | **End-to-End Latency** | Tiempo total desde la solicitud hasta la respuesta completa | | **Total Throughput** | Tokens totales (entrada + salida) por segundo | ## Ejecutar benchmarks ```bash # Basic benchmark vllm-mlx-bench --model mlx-community/Qwen3-0.6B-8bit # With more prompts vllm-mlx-bench --model mlx-community/Qwen3-0.6B-8bit --prompts 10 # Save results vllm-mlx-bench --model mlx-community/Qwen3-0.6B-8bit --output results.json # Continuous batching test python tests/test_continuous_batching.py # Prefix cache test python tests/test_prefix_cache.py # Paged cache test python tests/test_paged_cache_real_inference.py # Streaming detokenizer benchmark vllm-mlx bench-detok vllm-mlx bench-detok mlx-community/Llama-3.2-1B-Instruct-4bit --iterations 5 ``` # Documentation page: `es/benchmarks/video.md` # Benchmarks de Video ## Ejecutar Benchmarks de Video ```bash # Benchmark completo (10 configuraciones, 2-64 fotogramas) vllm-mlx-bench --model mlx-community/Qwen3-VL-8B-Instruct-4bit --video # Benchmark rápido (3 conteos de fotogramas) vllm-mlx-bench --model mlx-community/Qwen3-VL-8B-Instruct-4bit --video --quick # Video personalizado vllm-mlx-bench --model mlx-community/Qwen3-VL-8B-Instruct-4bit --video --video-url https://example.com/video.mp4 ``` ## Resultados - Qwen3-VL-8B-Instruct-4bit (M4 Max, 128GB) | Configuration | Frames | Time | Tokens | Speed | Memory | |---------------|--------|------|--------|-------|--------| | 2 frames @ 0.5fps | 2 | 4.48s | 256 | 57.1 tok/s | 6.4 GB | | 4 frames @ 1fps | 4 | 4.65s | 256 | 55.0 tok/s | 6.4 GB | | 6 frames @ 1fps | 6 | 5.15s | 197 | 38.2 tok/s | 6.6 GB | | 8 frames @ 2fps | 8 | 6.45s | 240 | 37.2 tok/s | 6.8 GB | | 12 frames @ 2fps | 12 | 8.73s | 256 | 29.3 tok/s | 7.1 GB | | 16 frames @ 2fps | 16 | 10.96s | 256 | 23.4 tok/s | 7.6 GB | | 24 frames @ 4fps | 24 | 14.95s | 226 | 15.1 tok/s | 8.4 GB | | 32 frames @ 4fps | 32 | 20.00s | 256 | 12.8 tok/s | 9.2 GB | | 48 frames @ 8fps | 48 | 31.11s | 246 | 7.9 tok/s | 11.1 GB | | 64 frames @ 8fps | 64 | 59.81s | 256 | 4.3 tok/s | 12.9 GB | **Resumen:** Más rápido con 2 fotogramas (57.1 tok/s), más lento con 64 fotogramas (4.3 tok/s). La memoria escala de 6.4 GB a 12.9 GB. > **Nota:** 96 fotogramas o más provoca un timeout de GPU en la mayoría del hardware debido a los límites de memoria y cómputo. ## Resultados - Qwen3-VL-8B-Instruct-4bit (M1 Max, 64GB) | Configuration | Frames | FPS | Time | Tokens | Speed | |---------------|--------|-----|------|--------|-------| | 4 frames @ 1fps | 4 | 1.0 | 8.84s | 256 | 29.0 tok/s | | 8 frames @ 2fps | 8 | 2.0 | 13.05s | 256 | 19.6 tok/s | | 16 frames @ 2fps | 16 | 2.0 | 21.60s | 256 | 11.9 tok/s | | Configs | Total Time (s) | Total Tokens | Aggregate Tok/s | |---------|-----------------|--------------|-----------------| | 3 | 43.48 | 768 | 17.7 | ## Resultados - Qwen3-VL-4B-Instruct-3bit (M1 Max, 64GB) | Configuration | Frames | FPS | Time | Tokens | Speed | |---------------|--------|-----|------|--------|-------| | 4 frames @ 1fps | 4 | 1.0 | 5.09s | 150 | 29.5 tok/s | | 8 frames @ 2fps | 8 | 2.0 | 8.36s | 150 | 17.9 tok/s | | 16 frames @ 2fps | 16 | 2.0 | 15.21s | 150 | 9.9 tok/s | | Configs | Total Time (s) | Total Tokens | Aggregate Tok/s | |---------|-----------------|--------------|-----------------| | 3 | 28.66 | 450 | 15.7 | ## Resultados de Caché de Video ``` ---------------------------------------------------------------------- TEST 4: Video Cache - fps/max_frames in Cache Key ---------------------------------------------------------------------- Config: fps=2.0, max_frames=16 Results: Step | Description | Expected | Actual | Time | Status -------+---------------------------+----------+--------+--------+------- 4a | Video first request | MISS | MISS | 0.03ms | ✓ 4b | Same video+params | HIT | HIT | 0.14ms | ✓ 4c | Different fps (4.0) | MISS | MISS | 0.01ms | ✓ 4d | Different max_frames (32) | MISS | MISS | 0.01ms | ✓ 4.0.5a | fps=0.5 first | MISS | MISS | 0.01ms | ✓ 4.0.5b | fps=0.5 cached | HIT | HIT | 0.14ms | ✓ 4.1.0a | fps=1.0 first | MISS | MISS | 0.01ms | ✓ 4.1.0b | fps=1.0 cached | HIT | HIT | 0.14ms | ✓ 4.2.0a | fps=2.0 first | MISS | MISS | 0.01ms | ✓ 4.2.0b | fps=2.0 cached | HIT | HIT | 0.14ms | ✓ 4.4.0a | fps=4.0 first | MISS | MISS | 0.01ms | ✓ 4.4.0b | fps=4.0 cached | HIT | HIT | 0.14ms | ✓ ---------------------------------------------------------------------- TEST 5: Additional Videos ---------------------------------------------------------------------- Results: Step | Description | Expected | Actual | Time | Status -------+---------------------------+----------+--------+--------+------- 5a | Video 1 first | MISS | MISS | 0.01ms | ✓ 5b | Video 2 first | MISS | MISS | 0.01ms | ✓ 5c | Video 1 cached | HIT | HIT | 0.13ms | ✓ 5d | Video 2 cached | HIT | HIT | 0.13ms | ✓ ``` ## Estrategia de Clave de Caché - **Videos**: `hash(video_path) + hash(fps) + hash(max_frames) + hash(prompt)` El mismo video con los mismos valores de fps, max_frames y prompt utilizará la caché. Cambiar cualquier parámetro genera un miss. ## Consejos de Rendimiento - Menor FPS = procesamiento más rápido - Menos fotogramas = menor uso de memoria - 64 fotogramas es el máximo práctico - 96 fotogramas o más provoca un timeout de GPU ## Extracción de Fotogramas | FPS | 10s Video | 30s Video | 60s Video | |-----|-----------|-----------|-----------| | 0.5 | 5 frames | 15 frames | 30 frames | | 1.0 | 10 frames | 30 frames | 60 frames | | 2.0 | 20 frames | 60 frames | 120 frames* | | 4.0 | 40 frames | 120 frames* | 240 frames* | *Puede alcanzar el límite de `max_frames` ## Referencia de Métricas | Metric | Description | |--------|-------------| | Configuration | Configuración de FPS y fotogramas máximos | | Frames | Fotogramas extraídos realmente | | Time | Tiempo total de generación | | Tokens | Tokens de salida generados | | Speed | Tokens por segundo (tok/s) | | Memory | Uso de memoria GPU | # Documentation page: `es/getting-started/installation.md` # Instalacion ## Requisitos - macOS en Apple Silicon (M1/M2/M3/M4/M5) - Python 3.10+ ## Instalar con uv (Recomendado) ```bash git clone https://github.com/waybarrios/vllm-mlx.git cd vllm-mlx uv pip install -e . ``` ## Instalar con pip ```bash git clone https://github.com/waybarrios/vllm-mlx.git cd vllm-mlx pip install -e . ``` ### Opcional: Soporte para vision Para procesamiento de video con transformers: ```bash pip install -e ".[vision]" ``` ### Opcional: Soporte de audio (STT/TTS) ```bash pip install mlx-audio ``` ### Opcional: Embeddings ```bash pip install mlx-embeddings ``` ## Que se instala - `mlx`, `mlx-lm`, `mlx-vlm` - Framework MLX y bibliotecas de modelos - `transformers`, `tokenizers` - Bibliotecas de HuggingFace - `opencv-python` - Procesamiento de video - `gradio` - Interfaz de chat - `psutil` - Monitoreo de recursos - `mlx-audio` (opcional) - Speech-to-Text y Text-to-Speech - `mlx-embeddings` (opcional) - Text embeddings ## Verificar la instalacion ```bash # Verificar comandos CLI vllm-mlx --help vllm-mlx-bench --help vllm-mlx-chat --help # Probar con un modelo pequeño vllm-mlx-bench --model mlx-community/Llama-3.2-1B-Instruct-4bit --prompts 1 ``` ## Solucion de problemas ### MLX no encontrado Asegurate de estar en Apple Silicon: ```bash uname -m # Should output "arm64" ``` ### Fallo en la descarga del modelo Verifica tu conexion a internet y el acceso a HuggingFace. Algunos modelos requieren autenticacion: ```bash huggingface-cli login ``` ### Sin memoria Usa un modelo cuantizado más pequeno: ```bash vllm-mlx serve mlx-community/Llama-3.2-1B-Instruct-4bit ``` ### Interrupciones del servidor en ejecuciones largas (suspensión de macOS) Tu máquina macOS puede entrar en suspensión durante ejecuciones largas como servidor. Prueba usar `caffeinate` para evitar la suspensión: ```bash caffeinate -dimsu ``` # Documentation page: `es/getting-started/quickstart.md` # Inicio rápido ## Opción 1: Servidor compatible con OpenAI Inicia el servidor: ```bash # Simple mode - maximum throughput for single user vllm-mlx serve mlx-community/Llama-3.2-3B-Instruct-4bit --port 8000 # Continuous batching - for multiple concurrent users vllm-mlx serve mlx-community/Llama-3.2-3B-Instruct-4bit --port 8000 --continuous-batching ``` Úsalo con el SDK de Python de OpenAI: ```python from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed") response = client.chat.completions.create( model="mlx-community/Llama-3.2-3B-Instruct-4bit", messages=[{"role": "user", "content": "Hello!"}], ) print(response.choices[0].message.content) ``` O con curl: ```bash curl http://localhost:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{"model": "default", "messages": [{"role": "user", "content": "Hello!"}]}' ``` ## Opción 2: API de Python directa ```python from vllm_mlx.models import MLXLanguageModel model = MLXLanguageModel("mlx-community/Llama-3.2-3B-Instruct-4bit") model.load() # Generate text output = model.generate("What is the capital of France?", max_tokens=100) print(output.text) # Streaming for chunk in model.stream_generate("Tell me a story"): print(chunk.text, end="", flush=True) ``` ## Opción 3: Interfaz de chat con Gradio ```bash vllm-mlx-chat --served-model-name mlx-community/Llama-3.2-3B-Instruct-4bit ``` Abre una interfaz web en http://localhost:7860 ## Modelos multimodales Para comprensión de imágenes y video, usa un modelo VLM: ```bash vllm-mlx serve mlx-community/Qwen3-VL-4B-Instruct-3bit --port 8000 ``` ```python response = client.chat.completions.create( model="default", messages=[{ "role": "user", "content": [ {"type": "text", "text": "What's in this image?"}, {"type": "image_url", "image_url": {"url": "https://example.com/image.jpg"}} ] }], max_tokens=256 ) ``` ## Modelos de razonamiento Separa el proceso de pensamiento del modelo de la respuesta final: ```bash vllm-mlx serve mlx-community/Qwen3-8B-4bit --reasoning-parser qwen3 ``` ```python response = client.chat.completions.create( model="default", messages=[{"role": "user", "content": "What is 17 × 23?"}] ) print(response.choices[0].message.content) # Final answer ``` ## Embeddings Genera embeddings de texto para búsqueda semántica y RAG: ```bash vllm-mlx serve mlx-community/Qwen3-4B-4bit --embedding-model mlx-community/multilingual-e5-small-mlx ``` ```python response = client.embeddings.create( model="mlx-community/multilingual-e5-small-mlx", input="Hello world" ) ``` ## Tool Calling Habilita la llamada a funciones con cualquier modelo compatible: ```bash vllm-mlx serve mlx-community/Devstral-Small-2507-4bit \ --enable-auto-tool-choice --tool-call-parser mistral ``` ## Próximos pasos - [Guía del servidor](../guides/server.md) - Configuración completa del servidor - [API de Python](../guides/python-api.md) - Uso directo de la API - [Guía multimodal](../guides/multimodal.md) - Imágenes y video - [Guía de audio](../guides/audio.md) - Speech-to-Text y Text-to-Speech - [Guía de embeddings](../guides/embeddings.md) - Embeddings de texto - [Modelos de razonamiento](../guides/reasoning.md) - Modelos con pensamiento - [Tool Calling](../guides/tool-calling.md) - Llamada a funciones - [Modelos compatibles](../reference/models.md) - Modelos disponibles # Documentation page: `es/guides/audio.md` # Soporte de Audio vllm-mlx soporta el procesamiento de audio mediante [mlx-audio](https://github.com/Blaizzy/mlx-audio), y ofrece: - **STT (Speech-to-Text)**: Whisper, Parakeet - **TTS (Text-to-Speech)**: Kokoro, Chatterbox, VibeVoice, VoxCPM - **Procesamiento de audio**: SAM-Audio (separación de voz) ## Instalación ```bash # Soporte de audio principal pip install mlx-audio>=0.2.9 # Dependencias requeridas para TTS pip install sounddevice soundfile scipy numba tiktoken misaki spacy num2words loguru phonemizer # Descargar el modelo de inglés de spacy python -m spacy download en_core_web_sm # Para TTS en idiomas distintos al inglés (español, francés, etc.), instalar espeak-ng: # macOS brew install espeak-ng # Ubuntu/Debian # sudo apt-get install espeak-ng ``` O instalar todas las dependencias de audio de una sola vez: ```bash pip install vllm-mlx[audio] python -m spacy download en_core_web_sm brew install espeak-ng # macOS, para idiomas distintos al inglés ``` ## Inicio Rápido ### Speech-to-Text (Transcripción) ```python from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed") # Transcribir un archivo de audio with open("audio.mp3", "rb") as f: transcript = client.audio.transcriptions.create( model="whisper-large-v3", file=f, language="en" # opcional ) print(transcript.text) ``` ### Text-to-Speech (Generación) ```python # Generar voz audio = client.audio.speech.create( model="kokoro", input="Hello, how are you?", voice="af_heart", speed=1.0 ) # Guardar en archivo with open("output.wav", "wb") as f: f.write(audio.content) ``` ### Separación de Voz (SAM-Audio) Aislar la voz del ruido de fondo, música u otros sonidos: ```python from vllm_mlx.audio import AudioProcessor # Cargar el modelo SAM-Audio processor = AudioProcessor("mlx-community/sam-audio-large-fp16") processor.load() # Separar el habla del audio result = processor.separate("meeting_with_music.mp3", description="speech") # Guardar la voz aislada y el fondo processor.save(result.target, "voice_only.wav") processor.save(result.residual, "background_only.wav") ``` **Ejemplo de CLI:** ```bash python examples/audio_separation_example.py meeting.mp3 --play python examples/audio_separation_example.py song.mp3 --description music -o music.wav ``` ### Demo de Separación de Batería Aislar la batería de una canción de rock usando SAM-Audio: | Audio | Descripción | Escuchar | |-------|-------------|----------| | Original | "Get Ready" de David Fesliyan (30s, libre de regalías) | [🎵 rock_get_ready.mp3](https://github.com/waybarrios/vllm-mlx/blob/main/examples/rock_get_ready.mp3?raw=1) | | Batería aislada | Batería extraída por SAM-Audio | [🥁 drums_isolated.wav](https://github.com/waybarrios/vllm-mlx/blob/main/examples/drums_isolated.wav?raw=1) | | Sin batería | Pista con la batería eliminada | [🎸 rock_no_drums.wav](https://github.com/waybarrios/vllm-mlx/blob/main/examples/rock_no_drums.wav?raw=1) | ```bash # Aislar la batería de una canción de rock python examples/audio_separation_example.py examples/rock_get_ready.mp3 \ --description "drums" \ --output drums_isolated.wav \ --background rock_no_drums.wav ``` **Rendimiento:** 30 segundos de audio procesados en ~20 segundos en M4 Max. ## Modelos Soportados ### Modelos STT (Speech-to-Text) | Modelo | Alias | Idiomas | Velocidad | Calidad | |--------|-------|---------|-----------|---------| | `mlx-community/whisper-large-v3-mlx` | `whisper-large-v3` | 99+ | Media | Mejor | | `mlx-community/whisper-large-v3-turbo` | `whisper-large-v3-turbo` | 99+ | Rápida | Muy buena | | `mlx-community/whisper-medium-mlx` | `whisper-medium` | 99+ | Rápida | Buena | | `mlx-community/whisper-small-mlx` | `whisper-small` | 99+ | Muy rápida | Aceptable | | `mlx-community/parakeet-tdt-0.6b-v2` | `parakeet` | Inglés | La más rápida | Muy buena | | `mlx-community/parakeet-tdt-0.6b-v3` | `parakeet-v3` | Inglés | La más rápida | Mejor | **Recomendación:** - Multilingüe: `whisper-large-v3` - Solo inglés: `parakeet` (3x más rápido) ### Modelos TTS (Text-to-Speech) #### Kokoro (Rápido y ligero) - Recomendado | Modelo | Alias | Tamaño | Idiomas | |--------|-------|--------|---------| | `mlx-community/Kokoro-82M-bf16` | `kokoro` | 82M | EN, ES, FR, JA, ZH, HI, IT, PT | | `mlx-community/Kokoro-82M-4bit` | `kokoro-4bit` | 82M | EN, ES, FR, JA, ZH, HI, IT, PT | **Voces (11):** - Femenino estadounidense: `af_heart`, `af_bella`, `af_nicole`, `af_sarah`, `af_sky` - Masculino estadounidense: `am_adam`, `am_michael` - Femenino británico: `bf_emma`, `bf_isabella` - Masculino británico: `bm_george`, `bm_lewis` **Códigos de idioma:** | Código | Idioma | Código | Idioma | |--------|--------|--------|--------| | `a` / `en` | English (US) | `e` / `es` | Español | | `b` / `en-gb` | English (UK) | `f` / `fr` | Français | | `j` / `ja` | 日本語 | `z` / `zh` | 中文 | | `i` / `it` | Italiano | `p` / `pt` | Português | | `h` / `hi` | हिन्दी | | | #### Chatterbox (Multilingüe y expresivo) | Modelo | Alias | Tamaño | Idiomas | |--------|-------|--------|---------| | `mlx-community/chatterbox-turbo-fp16` | `chatterbox` | 134M | 15+ idiomas | | `mlx-community/chatterbox-turbo-4bit` | `chatterbox-4bit` | 134M | 15+ idiomas | **Idiomas soportados:** EN, ES, FR, DE, IT, PT, RU, JA, ZH, KO, AR, HI, NL, PL, TR #### VibeVoice (Tiempo real) | Modelo | Alias | Tamaño | Caso de uso | |--------|-------|--------|-------------| | `mlx-community/VibeVoice-Realtime-0.5B-4bit` | `vibevoice` | 200M | Baja latencia, inglés | #### VoxCPM (Chino/Inglés) | Modelo | Alias | Tamaño | Idiomas | |--------|-------|--------|---------| | `mlx-community/VoxCPM1.5` | `voxcpm` | 0.9B | ZH, EN | | `mlx-community/VoxCPM1.5-4bit` | `voxcpm-4bit` | 200M | ZH, EN | ### Modelos de Procesamiento de Audio #### SAM-Audio (Separación de Voz) | Modelo | Tamaño | Caso de uso | |--------|--------|-------------| | `mlx-community/sam-audio-large-fp16` | 3B | Mejor calidad | | `mlx-community/sam-audio-large` | 3B | Estándar | | `mlx-community/sam-audio-small-fp16` | 0.6B | Rápido | | `mlx-community/sam-audio-small` | 0.6B | Ligero | ## Referencia de API ### POST /v1/audio/transcriptions Transcribir audio a texto (compatible con la API OpenAI Whisper). **Parámetros:** - `file`: Archivo de audio (mp3, wav, m4a, webm) - `model`: Nombre o alias del modelo - `language`: Código de idioma (opcional, se detecta automáticamente) - `response_format`: `json` o `text` **Límites:** - Tamaño máximo de carga por defecto: 25 MiB - Se puede ajustar con `--max-audio-upload-mb` **Ejemplo:** ```bash curl http://localhost:8000/v1/audio/transcriptions \ -F file=@audio.mp3 \ -F model=whisper-large-v3 ``` ### POST /v1/audio/speech Generar voz a partir de texto (compatible con la API OpenAI TTS). **Parámetros:** - `model`: Nombre o alias del modelo - `input`: Texto a sintetizar - `voice`: ID de la voz - `speed`: Velocidad del habla (0.5 a 2.0) - `response_format`: `wav`, `mp3` **Límites:** - Límite de entrada por defecto: 4096 caracteres - Se puede ajustar con `--max-tts-input-chars` **Ejemplo:** ```bash curl http://localhost:8000/v1/audio/speech \ -d '{"model": "kokoro", "input": "Hello world", "voice": "af_heart"}' \ -H "Content-Type: application/json" \ --output speech.wav ``` ### GET /v1/audio/voices Listar las voces disponibles para un modelo. **Ejemplo:** ```bash curl http://localhost:8000/v1/audio/voices?model=kokoro ``` ## Ejemplos de CLI ### Transcripción en Vivo / Subtítulos Transcripción de voz a texto en tiempo real desde el micrófono: ```bash # Subtítulos con whisper-large-v3 (mejor calidad) python examples/closed_captions.py --language es --chunk 5 # Modelo más rápido para menor latencia python examples/closed_captions.py --language en --model whisper-turbo --chunk 3 # Transcripción básica por micrófono (grabar y luego transcribir) python examples/mic_transcribe.py --language es # Transcripción en fragmentos en tiempo real python examples/mic_realtime.py --language es --chunk 3 # Transcripción en vivo con detección de actividad de voz python examples/mic_live.py --language es ``` **Requisitos:** ```bash pip install sounddevice soundfile numpy ``` ### TTS Básico ```bash # Ejemplo simple de TTS python examples/tts_example.py "Hello, how are you?" --play # Con una voz diferente python examples/tts_example.py "Hello!" --voice am_michael --play # Guardar en archivo python examples/tts_example.py "Welcome to the demo" -o greeting.wav # Listar las voces disponibles python examples/tts_example.py --list-voices ``` ### TTS Multilingüe ```bash # Inglés (selecciona automáticamente el mejor modelo) python examples/tts_multilingual.py "Hello world" --play # Español python examples/tts_multilingual.py "Hola mundo" --lang es --play # Francés python examples/tts_multilingual.py "Bonjour le monde" --lang fr --play # Japonés python examples/tts_multilingual.py "こんにちは" --lang ja --play # Chino python examples/tts_multilingual.py "你好世界" --lang zh --play # Usar un modelo específico python examples/tts_multilingual.py "Hello" --model chatterbox --play # Listar todos los modelos python examples/tts_multilingual.py --list-models # Listar todos los idiomas python examples/tts_multilingual.py --list-languages ``` ### Ejemplos de Asistente de Voz para Negocios Muestras de voz pregeneradas con **voces nativas** para casos de uso empresariales comunes: | Idioma | Voz | Mensaje | Escuchar | |--------|-----|---------|----------| | 🇺🇸 Inglés | af_heart | "Welcome to First National Bank. How may I assist you today?" | [▶️ assistant_bank_en.wav](https://github.com/waybarrios/vllm-mlx/blob/main/examples/assistant_bank_en.wav?raw=1) | | 🇪🇸 Español | ef_dora | "Gracias por llamar a servicio al cliente. Un agente le atenderá pronto." | [▶️ assistant_service_es.wav](https://github.com/waybarrios/vllm-mlx/blob/main/examples/assistant_service_es.wav?raw=1) | | 🇫🇷 Francés | ff_siwis | "Bienvenue. Votre appel est important pour nous." | [▶️ assistant_callcenter_fr.wav](https://github.com/waybarrios/vllm-mlx/blob/main/examples/assistant_callcenter_fr.wav?raw=1) | | 🇨🇳 Chino | zf_xiaobei | "欢迎致电技术支持中心。我们将竭诚为您服务。" | [▶️ assistant_support_zh.wav](https://github.com/waybarrios/vllm-mlx/blob/main/examples/assistant_support_zh.wav?raw=1) | **Genera tus propias muestras con voces nativas:** ```bash # Inglés - Asistente bancario (voz nativa: af_heart) python -m mlx_audio.tts.generate --model mlx-community/Kokoro-82M-bf16 \ --text "Welcome to First National Bank. How may I assist you today?" \ --voice af_heart --lang_code a --file_prefix assistant_bank_en # Español - Atención al cliente (voz nativa: ef_dora) python -m mlx_audio.tts.generate --model mlx-community/Kokoro-82M-bf16 \ --text "Gracias por llamar a servicio al cliente. Un agente le atendera pronto." \ --voice ef_dora --lang_code e --file_prefix assistant_service_es # Francés - Centro de llamadas (voz nativa: ff_siwis) python -m mlx_audio.tts.generate --model mlx-community/Kokoro-82M-bf16 \ --text "Bienvenue. Votre appel est important pour nous." \ --voice ff_siwis --lang_code f --file_prefix assistant_callcenter_fr # Chino - Soporte técnico (voz nativa: zf_xiaobei) python -m mlx_audio.tts.generate --model mlx-community/Kokoro-82M-bf16 \ --text "欢迎致电技术支持中心。我们将竭诚为您服务。" \ --voice zf_xiaobei --lang_code z --file_prefix assistant_support_zh ``` ### Referencia de Voces Nativas | Idioma | Código | Voces | |--------|--------|-------| | English (US) | `a` | af_heart, af_bella, af_nicole, am_adam, am_michael | | English (UK) | `b` | bf_emma, bf_isabella, bm_george, bm_lewis | | Español | `e` | ef_dora, em_alex, em_santa | | Français | `f` | ff_siwis | | 中文 | `z` | zf_xiaobei, zf_xiaoni, zf_xiaoxiao, zm_yunjian, zm_yunxi | | 日本語 | `j` | jf_alpha, jf_gongitsune, jm_kumo | | Italiano | `i` | if_sara, im_nicola | | Português | `p` | pf_dora, pm_alex | | हिन्दी | `h` | hf_alpha, hf_beta, hm_omega | ## API de Python ### Uso Directo (sin servidor) ```python from vllm_mlx.audio import STTEngine, TTSEngine, AudioProcessor # Speech-to-Text stt = STTEngine("mlx-community/whisper-large-v3-mlx") stt.load() result = stt.transcribe("audio.mp3") print(result.text) # Text-to-Speech tts = TTSEngine("mlx-community/Kokoro-82M-bf16") tts.load() audio = tts.generate("Hello world", voice="af_heart") tts.save(audio, "output.wav") # Separación de voz processor = AudioProcessor("mlx-community/sam-audio-large-fp16") processor.load() result = processor.separate("mixed_audio.mp3", description="speech") processor.save(result.target, "voice_only.wav") processor.save(result.residual, "background.wav") ``` ### Funciones de Conveniencia ```python from vllm_mlx.audio import transcribe_audio, generate_speech, separate_voice # Transcripción rápida result = transcribe_audio("audio.mp3") print(result.text) # TTS rápido audio = generate_speech("Hello world", voice="af_heart") # Separación de voz rápida voice, background = separate_voice("mixed.mp3") ``` ## Audio en el Chat Incluir audio en mensajes de chat (se transcribe automáticamente): ```python response = client.chat.completions.create( model="default", messages=[{ "role": "user", "content": [ {"type": "text", "text": "Summarize this audio"}, {"type": "audio_url", "audio_url": {"url": "file://meeting.mp3"}} ] }] ) ``` ## Benchmarks Probado en Apple M2 Max (32GB). ### Benchmarks de TTS (Kokoro-82M-bf16) | Longitud del texto | Duración del audio | Tiempo de generación | RTF | Chars/seg | |--------------------|--------------------|----------------------|-----|-----------| | 25 chars | 1.95s | 0.43s | 4.6x | 58.5 | | 88 chars | 6.00s | 0.32s | 18.6x | 272.4 | | 117 chars | 7.92s | 0.27s | 29.0x | 427.4 | **Resumen:** - Tiempo de carga del modelo: ~1.0s - RTF promedio: **17.4x** (17 veces más rápido que en tiempo real) - Chars/seg promedio: **252.8** ### Benchmarks de STT | Modelo | Tiempo de carga | Transcripción (audio de 6s) | RTF | |--------|-----------------|------------------------------|-----| | whisper-small | 0.25s | 0.20s | 30.2x | | whisper-medium | 18.1s | 0.38s | 15.5x | | whisper-large-v3 | ~30s | ~0.6s | ~10x | | parakeet | ~0.5s | ~0.15s | ~40x | **Notas:** - RTF (Real-Time Factor) indica cuántas veces más rápido que en tiempo real es el procesamiento - La primera carga incluye la descarga del modelo desde HuggingFace - Las cargas siguientes usan los modelos en caché ### Recomendaciones por Caso de Uso | Caso de uso | Modelo recomendado | Motivo | |-------------|-------------------|--------| | STT en inglés rápido | `parakeet` | RTF de 40x, bajo consumo de memoria | | STT multilingüe | `whisper-large-v3` | 99+ idiomas | | STT de baja latencia | `whisper-small` | RTF de 30x, carga rápida | | TTS general | `kokoro` | RTF de 17x, buena calidad | | TTS con poca memoria | `kokoro-4bit` | Cuantizado a 4 bits | ## Consejos de Rendimiento 1. **Usa Parakeet para inglés**: 40x más rápido que en tiempo real 2. **Usa modelos de 4 bits** para menor uso de memoria 3. **Usa SAM-Audio small** para una separación de voz más rápida 4. **Guarda los modelos en caché**: los motores se cargan de forma diferida y quedan en caché 5. **Descarga los modelos previamente** para evitar la latencia en la primera ejecución ## Solución de Problemas ### mlx-audio no está instalado ``` pip install mlx-audio>=0.2.9 ``` ### La descarga del modelo es lenta Los modelos se descargan desde HuggingFace en el primer uso. Usa `huggingface-cli download` para descargarlos previamente: ```bash huggingface-cli download mlx-community/whisper-large-v3-mlx huggingface-cli download mlx-community/Kokoro-82M-bf16 ``` ### Sin memoria suficiente Usa modelos más pequeños o versiones cuantizadas a 4 bits: - `whisper-small-mlx` en lugar de `whisper-large-v3-mlx` - `Kokoro-82M-4bit` en lugar de `Kokoro-82M-bf16` - `sam-audio-small` en lugar de `sam-audio-large` ### Error multilingüe de Kokoro (mlx-audio 0.2.9) Si obtienes `ValueError: too many values to unpack` al usar idiomas distintos al inglés (español, chino, japonés, etc.) con Kokoro, aplica esta corrección: ```python # Corrección para mlx_audio/tts/models/kokoro/pipeline.py línea 443 # Cambia: # ps, _ = self.g2p(chunk) # Por: g2p_result = self.g2p(chunk) ps = g2p_result[0] if isinstance(g2p_result, tuple) else g2p_result ``` **Corrección en una sola línea:** ```bash python -c " import os path = os.path.join(os.path.dirname(__import__('mlx_audio').__file__), 'tts/models/kokoro/pipeline.py') with open(path, 'r') as f: content = f.read() old = ' ps, _ = self.g2p(chunk)' new = ''' # Fix: handle both tuple (en) and string (zh/ja/es) returns from g2p g2p_result = self.g2p(chunk) ps = g2p_result[0] if isinstance(g2p_result, tuple) else g2p_result''' if old in content: with open(path, 'w') as f: f.write(content.replace(old, new)) print('Fix applied!') " ``` Este error ocurre porque el g2p para inglés devuelve una tupla `(phonemes, tokens)` mientras que otros idiomas devuelven solo una cadena de texto. # Documentation page: `es/guides/continuous-batching.md` # Continuous Batching El continuous batching permite mayor throughput al servir múltiples usuarios concurrentes. ## Activar Continuous Batching ```bash vllm-mlx serve mlx-community/Qwen3-0.6B-8bit --continuous-batching ``` ## Con Paged Cache Para compartir prefijos de forma eficiente en memoria: ```bash vllm-mlx serve mlx-community/Qwen3-0.6B-8bit --continuous-batching --use-paged-cache ``` ## Cómo Funciona ### Modo Simple (Predeterminado) - Una solicitud a la vez - Máximo throughput para un solo usuario - Sin sobrecarga por batching ### Modo Continuous Batching - Múltiples solicitudes procesadas en conjunto - Mejor throughput para usuarios concurrentes - Pequeña sobrecarga por solicitud ### Paged Cache - KV cache almacenado en bloques de tamaño fijo - Los system prompts compartidos usan los mismos bloques - Ahorro de memoria: 80% o más con 10 o más usuarios concurrentes ## Resultados de Rendimiento **Resultados de Continuous Batching (M4 Max, 128GB):** | Modelo | Solicitud Individual | Batch (5 req) | Mejora | |--------|----------------------|---------------|--------| | Llama-3.2-1B-Instruct-4bit | 299.1 tok/s | 613.0 tok/s | **2.05x** | | Llama-3.2-3B-Instruct-4bit | 137.6 tok/s | 208.1 tok/s | **1.51x** | | Qwen3-0.6B-8bit | 328.1 tok/s | 1111.8 tok/s | **3.39x** | | Qwen3-30B-A3B-4bit | 98.1 tok/s | 233.3 tok/s | **2.38x** | | Qwen2.5-1.5B-Instruct-4bit | 196.9 tok/s | 322.2 tok/s | **1.64x** | *El batching de 5 solicitudes concurrentes muestra una mejora de throughput de 1.5 a 3 veces.* ## Rendimiento en Streaming **Rendimiento de Streaming (M4 Max, 128GB):** | Modelo | TTFT | Velocidad de Generación | |--------|------|-------------------------| | Llama-3.2-1B-Instruct-4bit | ~4.6ms | 218.9 tok/s | | Llama-3.2-3B-Instruct-4bit | ~10.7ms | 93.6 tok/s | | Qwen3-0.6B-8bit | ~3.0ms | 328.5 tok/s | | Qwen3-30B-A3B-4bit | ~10.2ms | 98.4 tok/s | | Qwen2.5-1.5B-Instruct-4bit | ~7.1ms | 140.3 tok/s | *TTFT = Time to First Token* ## Configuración de Streaming Controla la entrega de tokens con `--stream-interval`: ```bash # Cada token (más fluido) vllm-mlx serve model --continuous-batching --stream-interval 1 # Tokens en batch (mejor para alta latencia) vllm-mlx serve model --continuous-batching --stream-interval 5 ``` | Valor | Comportamiento | |-------|----------------| | `1` | Envía cada token de inmediato | | `2-5` | Agrupa tokens antes de enviar | | `10+` | Máximo throughput, salida en fragmentos más grandes | ## Gestión de Memoria En modelos grandes, el prefix cache puede consumir una cantidad significativa de memoria. El cache con gestión automática de memoria administra esto de forma transparente: ```bash # Detección automática (usa el 20% de la RAM disponible) vllm-mlx serve model --continuous-batching # Límite explícito vllm-mlx serve model --continuous-batching --cache-memory-mb 2048 # Porcentaje personalizado vllm-mlx serve model --continuous-batching --cache-memory-percent 0.10 ``` | Opción | Descripción | |--------|-------------| | `--cache-memory-mb` | Establece un límite explícito en MB | | `--cache-memory-percent` | Fracción de la RAM disponible (predeterminado: 0.20) | | `--no-memory-aware-cache` | Usa el cache heredado basado en conteo de entradas | ## Prefix Cache El prefix caching reutiliza el KV cache para prompts repetidos. ### Cómo Funciona ``` User 1: System prompt (500 tokens) → Creates 8 blocks User 2: Same system prompt → Shares 8 blocks (ref_count++) User N: Same system prompt → Shares 8 blocks (ref_count++) Memory savings: 80%+ for 10+ concurrent users ``` ### Estrategia de Clave de Cache - **LLM**: `hash(prompt)` - **Images**: `hash(image_content) + hash(prompt)` - **Videos**: `hash(video_path) + hash(fps) + hash(max_frames) + hash(prompt)` ### Probar el Prefix Cache ```bash python tests/test_prefix_cache.py ``` ``` ====================================================================== LLM PREFIX CACHE TEST ====================================================================== Model: mlx-community/Qwen3-0.6B-8bit Expected behavior: - Same prompt → cache HIT - Different prompt → cache MISS or PREFIX_HIT (shared template tokens) ---------------------------------------------------------------------- Results: Step | Description | Expected | Actual | Status -------+---------------------+----------+--------+------- 1a | First request | MISS | MISS | PASS 1b | Same prompt | HIT | HIT | PASS 1c | Different prompt | MISS | MISS | PASS 1d | Return to prompt 1 | HIT | HIT | PASS ====================================================================== ``` ## Ejecutar Benchmarks ```bash # Benchmark de continuous batching python tests/test_continuous_batching.py # Prueba de prefix cache python tests/test_prefix_cache.py ``` ## Cuándo Usarlo | Escenario | Modo | |-----------|------| | Usuario individual, máxima velocidad | Simple (predeterminado) | | Múltiples usuarios concurrentes | `--continuous-batching` | | Modelos grandes (7B+) | `--continuous-batching --cache-memory-mb 2048` | | Producción con prompts compartidos | `--continuous-batching --use-paged-cache` | ## Configuración para Producción ```bash vllm-mlx serve mlx-community/Qwen3-0.6B-8bit \ --continuous-batching \ --use-paged-cache \ --port 8000 ``` # Documentation page: `es/guides/embeddings.md` # Embeddings vllm-mlx soporta embeddings de texto usando [mlx-embeddings](https://github.com/Blaizzy/mlx-embeddings), y expone un endpoint `/v1/embeddings` compatible con OpenAI. ## Instalacion ```bash pip install mlx-embeddings>=0.0.5 ``` ## Inicio rápido ### Iniciar el servidor con un modelo de embeddings ```bash # Precarga un modelo de embeddings especifico al inicio vllm-mlx serve my-llm-model --embedding-model mlx-community/all-MiniLM-L6-v2-4bit ``` Si no se usa `--embedding-model`, el modelo de embeddings se carga de forma diferida en la primera solicitud, pero solo desde la lista de modelos permitidos en tiempo de solicitud. ### Generar embeddings con el SDK de OpenAI ```python from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed") # Texto individual response = client.embeddings.create( model="mlx-community/all-MiniLM-L6-v2-4bit", input="Hello world" ) print(response.data[0].embedding[:5]) # First 5 dimensions # Lote de textos response = client.embeddings.create( model="mlx-community/all-MiniLM-L6-v2-4bit", input=[ "I love machine learning", "Deep learning is fascinating", "Natural language processing rocks" ] ) for item in response.data: print(f"Text {item.index}: {len(item.embedding)} dimensions") ``` ### Usando curl ```bash curl http://localhost:8000/v1/embeddings \ -H "Content-Type: application/json" \ -d '{ "model": "mlx-community/all-MiniLM-L6-v2-4bit", "input": ["Hello world", "How are you?"] }' ``` ## Modelos soportados Modelos disponibles en tiempo de solicitud: | Model | Use Case | Size | |-------|----------|------| | `mlx-community/all-MiniLM-L6-v2-4bit` | Fast, compact | Small | | `mlx-community/embeddinggemma-300m-6bit` | High quality | 300M | | `mlx-community/bge-large-en-v1.5-4bit` | Best for English | Large | | `mlx-community/multilingual-e5-small-mlx` | Multilingual retrieval | Small | | `mlx-community/multilingual-e5-large-mlx` | Multilingual retrieval | Large | | `mlx-community/bert-base-uncased-mlx` | General BERT baseline | Base | | `mlx-community/ModernBERT-base-mlx` | ModernBERT baseline | Base | Otros modelos de embeddings requieren `--embedding-model` al iniciar el servidor. ## Gestion de modelos ### Carga diferida Por defecto, el modelo de embeddings se carga en la primera solicitud a `/v1/embeddings`. Es posible cambiar entre los modelos permitidos en tiempo de solicitud, y el modelo anterior se descarga automaticamente. ### Precarga al inicio Usa `--embedding-model` para cargar un modelo al iniciar el servidor. Cuando se establece esta opcion, solo ese modelo puede usarse para embeddings: ```bash vllm-mlx serve my-llm-model --embedding-model mlx-community/all-MiniLM-L6-v2-4bit ``` Solicitar un modelo diferente devolvera un error 400. ## Referencia de la API ### POST /v1/embeddings Crea embeddings para los textos de entrada proporcionados. **Cuerpo de la solicitud:** | Field | Type | Required | Description | |-------|------|----------|-------------| | `model` | string | Yes | Supported embedding model ID, or the startup-pinned model when `--embedding-model` is used | | `input` | string or list[string] | Yes | Text(s) to embed | **Respuesta:** ```json { "object": "list", "data": [ {"object": "embedding", "index": 0, "embedding": [0.023, -0.982, ...]}, {"object": "embedding", "index": 1, "embedding": [0.112, -0.543, ...]} ], "model": "mlx-community/all-MiniLM-L6-v2-4bit", "usage": {"prompt_tokens": 12, "total_tokens": 12} } ``` ## API de Python ### Uso directo sin servidor ```python from vllm_mlx.embedding import EmbeddingEngine engine = EmbeddingEngine("mlx-community/all-MiniLM-L6-v2-4bit") engine.load() vectors = engine.embed(["Hello world", "How are you?"]) print(f"Dimensions: {len(vectors[0])}") tokens = engine.count_tokens(["Hello world"]) print(f"Token count: {tokens}") ``` ## Solucion de problemas ### mlx-embeddings no esta instalado ``` pip install mlx-embeddings>=0.0.5 ``` ### Modelo no encontrado Asegurate de que el nombre del modelo coincida con alguno de los IDs permitidos en tiempo de solicitud, o inicia el servidor con `--embedding-model` para fijar un modelo personalizado. Puedes descargar los modelos soportados con anticipacion: ```bash huggingface-cli download mlx-community/all-MiniLM-L6-v2-4bit ``` # Documentation page: `es/guides/mcp-tools.md` # MCP y Tool Calling vllm-mlx soporta el Model Context Protocol (MCP) para integrar herramientas externas con LLMs. ## Como funciona el tool calling ``` ┌─────────────────────────────────────────────────────────────────────┐ │ Tool Calling Flow │ ├─────────────────────────────────────────────────────────────────────┤ │ │ │ 1. User Request │ │ ─────────────────► "List files in /tmp" │ │ │ │ 2. LLM Generates Tool Call │ │ ─────────────────► tool_calls: [{ │ │ name: "list_directory", │ │ arguments: {path: "/tmp"} │ │ }] │ │ │ │ 3. App Executes Tool via MCP │ │ ─────────────────► MCP Server executes list_directory │ │ Returns: ["file1.txt", "file2.txt"] │ │ │ │ 4. Tool Result Sent Back to LLM │ │ ─────────────────► role: "tool", content: [...] │ │ │ │ 5. LLM Generates Final Response │ │ ─────────────────► "The /tmp directory contains 2 files..." │ │ │ └─────────────────────────────────────────────────────────────────────┘ ``` ## Inicio rápido ### 1. Crear la configuración de MCP Crea el archivo `mcp.json`: ```json { "mcpServers": { "filesystem": { "command": "npx", "args": ["-y", "@modelcontextprotocol/server-filesystem", "/tmp"] } } } ``` ### 2. Iniciar el servidor con MCP ```bash # Modo simple vllm-mlx serve mlx-community/Qwen3-4B-4bit --mcp-config mcp.json # Continuous batching vllm-mlx serve mlx-community/Qwen3-4B-4bit --mcp-config mcp.json --continuous-batching ``` ### 3. Verificar el estado de MCP ```bash # Verificar estado de MCP curl http://localhost:8000/v1/mcp/status # Listar las herramientas disponibles curl http://localhost:8000/v1/mcp/tools ``` ## Ejemplo de tool calling ```python import json import httpx BASE_URL = "http://localhost:8000" # 1. Get available tools tools_response = httpx.get(f"{BASE_URL}/v1/mcp/tools") tools = tools_response.json()["tools"] # 2. Send request with tools response = httpx.post( f"{BASE_URL}/v1/chat/completions", json={ "model": "default", "messages": [{"role": "user", "content": "List files in /tmp"}], "tools": tools, "max_tokens": 1024 } ) result = response.json() message = result["choices"][0]["message"] # 3. Check for tool calls if message.get("tool_calls"): tool_call = message["tool_calls"][0] # 4. Execute tool via MCP exec_response = httpx.post( f"{BASE_URL}/v1/mcp/execute", json={ "server": "filesystem", "tool": tool_call["function"]["name"], "arguments": json.loads(tool_call["function"]["arguments"]) } ) tool_result = exec_response.json() # 5. Send result back to LLM messages = [ {"role": "user", "content": "List files in /tmp"}, message, { "role": "tool", "tool_call_id": tool_call["id"], "content": json.dumps(tool_result["result"]) } ] final_response = httpx.post( f"{BASE_URL}/v1/chat/completions", json={"model": "default", "messages": messages} ) print(final_response.json()["choices"][0]["message"]["content"]) ``` ## Endpoints de MCP | Endpoint | Metodo | Descripcion | |----------|--------|-------------| | `/v1/mcp/status` | GET | Verificar el estado de MCP | | `/v1/mcp/tools` | GET | Listar las herramientas disponibles | | `/v1/mcp/execute` | POST | Ejecutar una herramienta | ## Ejemplos de servidores MCP ### Sistema de archivos ```json { "mcpServers": { "filesystem": { "command": "npx", "args": ["-y", "@modelcontextprotocol/server-filesystem", "/tmp"] } } } ``` ### GitHub ```json { "mcpServers": { "github": { "command": "npx", "args": ["-y", "@modelcontextprotocol/server-github"], "env": { "GITHUB_TOKEN": "your-token" } } } } ``` ### PostgreSQL ```json { "mcpServers": { "postgres": { "command": "npx", "args": ["-y", "@modelcontextprotocol/server-postgres"], "env": { "DATABASE_URL": "postgresql://user:pass@localhost/db" } } } } ``` ### Brave Search ```json { "mcpServers": { "brave-search": { "command": "npx", "args": ["-y", "@modelcontextprotocol/server-brave-search"], "env": { "BRAVE_API_KEY": "your-key" } } } } ``` ## Multiples servidores MCP ```json { "mcpServers": { "filesystem": { "command": "npx", "args": ["-y", "@modelcontextprotocol/server-filesystem", "/tmp"] }, "github": { "command": "npx", "args": ["-y", "@modelcontextprotocol/server-github"], "env": { "GITHUB_TOKEN": "your-token" } } } } ``` ## Chat MCP interactivo Para probar MCP de forma interactiva: ```bash python examples/mcp_chat.py ``` ## Formatos de herramientas soportados vllm-mlx soporta 12 tool call parsers que cubren todas las familias de modelos principales. Consulta [Tool Calling](tool-calling.md) para ver la lista completa de parsers, alias y ejemplos. ## Seguridad vllm-mlx incluye medidas de seguridad para prevenir ataques de inyeccion de comandos a traves de servidores MCP. ### Lista blanca de comandos Solo se permiten comandos de confianza por defecto: | Categoria | Comandos permitidos | |----------|-----------------| | Node.js | `npx`, `npm`, `node` | | Python | `uvx`, `uv`, `python`, `python3`, `pip`, `pipx` | | Docker | `docker` | | Servidores MCP | `mcp-server-*` (servidores oficiales) | ### Patrones bloqueados Los siguientes patrones estan bloqueados para prevenir ataques de inyeccion: - Encadenamiento de comandos: `;`, `&&`, `||`, `|` - Sustitucion de comandos: `` ` ``, `$()` - Recorrido de rutas: `../` - Variables de entorno peligrosas: `LD_PRELOAD`, `PATH`, `PYTHONPATH` ### Ejemplo: ataque bloqueado ```json { "mcpServers": { "malicious": { "command": "bash", "args": ["-c", "rm -rf /"] } } } ``` Esta configuración sera rechazada: ``` ValueError: MCP server 'malicious': Command 'bash' is not in the allowed commands whitelist. ``` ### Modo de desarrollo (inseguro) Solo para desarrollo, es posible omitir la validación de seguridad: ```json { "mcpServers": { "custom": { "command": "my-custom-server", "skip_security_validation": true } } } ``` **ADVERTENCIA**: nunca uses `skip_security_validation` en produccion. ### Lista blanca personalizada Para agregar comandos personalizados a la lista blanca mediante código: ```python from vllm_mlx.mcp import MCPCommandValidator, set_validator # Add custom commands validator = MCPCommandValidator( custom_whitelist={"my-trusted-server", "another-server"} ) set_validator(validator) ``` ## Sandboxing de ejecución de herramientas Ademas de la validación de comandos, vllm-mlx ofrece sandboxing en tiempo de ejecución para las herramientas: ### Caracteristicas del sandbox | Caracteristica | Descripcion | |---------|-------------| | Lista blanca de herramientas | Permite ejecutar solo herramientas especificas | | Lista negra de herramientas | Bloquea herramientas peligrosas especificas | | Validacion de argumentos | Bloquea patrones peligrosos en los argumentos de las herramientas | | Limite de frecuencia | Limita las llamadas a herramientas por minuto | | Registro de auditoria | Registra todas las ejecuciones de herramientas | ### Patrones de argumentos bloqueados Los argumentos de las herramientas son validados para detectar patrones peligrosos: - Recorrido de rutas: `../` - Directorios del sistema: `/etc/`, `/proc/`, `/sys/` - Acceso root: `/root/`, `~root` ### Deteccion de herramientas de alto riesgo Las herramientas que coincidan con estos patrones generan advertencias de seguridad: - `execute`, `run_command`, `shell`, `eval`, `exec`, `system`, `subprocess` ### Configuracion personalizada del sandbox ```python from vllm_mlx.mcp import ToolSandbox, set_sandbox # Create sandbox with custom settings sandbox = ToolSandbox( # Only allow specific tools (whitelist mode) allowed_tools={"read_file", "list_directory"}, # Block specific tools (blacklist mode) blocked_tools={"execute_command", "run_shell"}, # Rate limit: max 30 calls per minute max_calls_per_minute=30, # Optional audit callback audit_callback=lambda audit: print(f"Tool: {audit.tool_name}, Success: {audit.success}"), ) set_sandbox(sandbox) ``` ### Acceso a los registros de auditoria ```python from vllm_mlx.mcp import get_sandbox sandbox = get_sandbox() # Get recent audit entries entries = sandbox.get_audit_log(limit=50) # Filter by tool name file_ops = sandbox.get_audit_log(tool_filter="file") # Get only errors errors = sandbox.get_audit_log(errors_only=True) # Clear audit log sandbox.clear_audit_log() ``` ### Redaccion de datos sensibles Los registros de auditoria redactan automaticamente los campos sensibles (password, token, secret, key, credential, auth) y truncan los valores de gran tamano. ## Solucion de problemas ### El servidor MCP no se conecta Verifica que el comando del servidor MCP sea correcto: ```bash npx -y @modelcontextprotocol/server-filesystem /tmp ``` ### La herramienta no se ejecuta Verifica que la herramienta este disponible: ```bash curl http://localhost:8000/v1/mcp/tools | jq '.tools[].name' ``` ### La llamada a la herramienta no se analiza Asegurate de usar un modelo que soporte llamadas a funciones (Qwen3, Llama-3.2-Instruct). ### El comando no esta en la lista blanca Si ves "Command X is not in the allowed commands whitelist", puedes: 1. Usar un comando permitido (ver lista blanca arriba) 2. Agregar el comando a una lista blanca personalizada 3. Usar `skip_security_validation: true` (solo para desarrollo) # Documentation page: `es/guides/moe-top-k.md` # MoE top_k override (`--moe-top-k`) Reduce el numero de experts activados por token en modelos Mixture of Experts como Qwen3-30B-A3B, intercambiando una pequeña cantidad de calidad por un aumento significativo en el throughput de decodificacion. > **Estado:** flag opt-in. El comportamiento por defecto no cambia. Los numeros > de calidad que se muestran son para Qwen3-30B-A3B-4bit en M4 Max 128 GB. > Verifica con tu modelo antes de usarlo en cargas de produccion. ## Que hace Qwen3-30B-A3B se entrena con `top_k=8`. cada token selecciona 8 de 128 experts. En Apple Silicon con batch=1 durante la decodificacion, la multiplicacion de matrices de experts (`SwitchGLU`) es la parte más costosa del computo por capa, y ese costo escala de forma aproximadamente lineal con `top_k`. Reducir `top_k` en tiempo de inferencia ha demostrado (LExI 2025, Lynx 2024) preservar la mayor parte de la calidad entrenada mientras reduce materialmente el tiempo de decodificacion. `--moe-top-k N` itera cada capa del modelo cargado y, en cada capa que tenga `.mlp.switch_mlp` (es decir, un bloque sparse-MoE), establece `top_k = N`. Las capas densas y los modelos densos no se modifican: el flag es un no-op para ellos. ## Uso ```bash # Server vllm-mlx serve mlx-community/Qwen3-30B-A3B-4bit \ --continuous-batching \ --moe-top-k 4 # Bench vllm-mlx bench mlx-community/Qwen3-30B-A3B-4bit --moe-top-k 4 ``` El flag se rechaza si `N` es mayor que el `top_k` entrenado del modelo (solo tiene sentido reducirlo, nunca aumentarlo). ## Impacto medido ### Throughput de decodificacion (M4 Max 128 GB, batch=1, greedy) | top_k | tok/s | vs baseline | |---:|---:|---:| | 8 (baseline) | 126.5 | - | | 6 | 136.1 | +7.6% | | 5 | 140.3 | +10.9% | | 4 | 147.3 | +16.5% | ### Calidad (Qwen3-30B-A3B-4bit, lm-evaluation-harness, MLX backend) | top_k | MMLU (acc) | GSM8K (exact match) | Delta vs baseline | |---:|---:|---:|---:| | 8 | TBD | TBD | - | | 6 | TBD | TBD | TBD | | 5 | TBD | TBD | TBD | | 4 | TBD | TBD | TBD | MMLU: 200 muestras seleccionadas aleatoriamente, 0-shot. GSM8K: 100 muestras seleccionadas aleatoriamente, 0-shot, exact-match estricto. Estos numeros son **indicativos**: los conjuntos de evaluacion completos son más grandes y desplazarian la precision absoluta, pero no el delta relativo entre configuraciones de forma significativa. ### Paridad de salida greedy Con `top_k=4` en el checkpoint de 4 bits observamos **los primeros 16 tokens generados identicos** al baseline en todos los prompts de prueba que usamos. Esto sugiere que top_k=4 no cambia el argmax en los pasos iniciales de decodificacion: el modelo es internamente robusto a eliminar la mitad de sus experts activados. Con `top_k=3` o menor, la calidad comenzaria a degradarse de forma visible (no medido aquí; inferido del paper LExI), por lo que el flag no permite bajar por debajo de 1 en la capa de validación de configuración. Sin embargo, el piso recomendado para produccion es `top_k=4`. ## Cuando usarlo y cuando no Usalo cuando: - Ejecutas un Qwen3 MoE (o compatible: Qwen3.5 MoE, Gemma-MoE) y el throughput de decodificacion con un solo usuario es tu cuello de botella. - Tienes una carga de trabajo donde una pequeña perdida de calidad es aceptable a cambio de una mejora visible en latencia. - Despliegas en hardware limitado por ancho de banda de memoria (Apple Silicon serie M) donde el gather de experts domina el tiempo de decodificacion por paso. No lo uses cuando: - Sirves modelos densos: el flag es un no-op y no aporta nada. - Te importa la precision en el top-1% de suites de evaluacion de leaderboard. - Ejecutas generaciones largas de chain-of-thought o "modo thinking", donde el acantilado de calidad puede ser más pronunciado que lo que sugiere MMLU en 0-shot. ## Combinacion con otras optimizaciones Este flag se compone con la cuantizacion. En Qwen3-30B-A3B-4bit nuestra combinacion medida es: - 4-bit + top_k=8: 126.5 tok/s (baseline) - 4-bit + top_k=4: 147.3 tok/s (+16.5%) - 3-bit + top_k=8: 138.6 tok/s (+9.6%) - 3-bit + top_k=6: 147.1 tok/s (+16.3%) . divergencia de calidad medible - 3-bit + top_k=4: 157.3 tok/s (+24%) . **la calidad de salida se rompe** (el modelo respondió una pregunta diferente en nuestra prueba de humo) 3-bit + top_k=4 acumulo el error numérico más alla del punto donde el argmax es estable. Usa a lo sumo un parámetro agresivo: ya sea 4-bit + top_k=4 o 3-bit + top_k=6. Ambos dan aproximadamente el mismo tok/s (~147) con perfiles de calidad muy distintos. ## Internos - Helper de parcheo: `vllm_mlx.scheduler.apply_moe_top_k_override(model, k)` - Se aplica en `Scheduler.__init__` despues de cargar el modelo. - Tests: `tests/test_moe_top_k.py`. cubre modelos densos, arquitecturas mixtas y rutas de validación. ## Referencias - LExI: Layer-Adaptive Active Experts, [arXiv 2509.02753](https://arxiv.org/html/2509.02753) - Not All Experts are Equal (NAEE), [ACL 2024](https://aclanthology.org/2024.acl-long.334.pdf) - SwiftLM (`SWIFTLM_TOP_K` env knob prior art), [github.com/SharpAI/SwiftLM](https://github.com/SharpAI/SwiftLM) # Documentation page: `es/guides/multimodal.md` # Modelos Multimodales (Imágenes y Video) vllm-mlx soporta modelos de visión y lenguaje (VLM) para el análisis de imágenes y video. ## Modelos Soportados - Qwen3-VL (recomendado) - Qwen2-VL - Gemma 3 - LLaVA - Idefics - PaliGemma - Pixtral - Molmo - DeepSeek-VL ## Iniciar un Servidor Multimodal ```bash vllm-mlx serve mlx-community/Qwen3-VL-4B-Instruct-3bit --port 8000 ``` Los modelos que contienen "VL", "Vision" o "mllm" en el nombre se detectan automáticamente como multimodales. ## Análisis de Imágenes ### Mediante el SDK de OpenAI ```python from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed") # Imagen desde URL response = client.chat.completions.create( model="default", messages=[{ "role": "user", "content": [ {"type": "text", "text": "What's in this image?"}, {"type": "image_url", "image_url": {"url": "https://example.com/image.jpg"}} ] }], max_tokens=256 ) print(response.choices[0].message.content) ``` ### Imágenes en Base64 ```python import base64 def encode_image(path): with open(path, "rb") as f: return base64.b64encode(f.read()).decode("utf-8") base64_image = encode_image("photo.jpg") response = client.chat.completions.create( model="default", messages=[{ "role": "user", "content": [ {"type": "text", "text": "Describe this image"}, {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{base64_image}"}} ] }] ) ``` ### Mediante curl ```bash curl http://localhost:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "default", "messages": [{ "role": "user", "content": [ {"type": "text", "text": "What is in this image?"}, {"type": "image_url", "image_url": {"url": "https://example.com/image.jpg"}} ] }], "max_tokens": 256 }' ``` ## Análisis de Video ### Mediante el SDK de OpenAI ```python response = client.chat.completions.create( model="default", messages=[{ "role": "user", "content": [ {"type": "text", "text": "What happens in this video?"}, {"type": "video_url", "video_url": {"url": "https://example.com/video.mp4"}} ] }], max_tokens=512 ) ``` ### Parámetros de Video Controla la extracción de fotogramas mediante parámetros adicionales en el cuerpo de la solicitud: ```python response = client.chat.completions.create( model="default", messages=[{ "role": "user", "content": [ {"type": "text", "text": "Describe this video"}, {"type": "video_url", "video_url": {"url": "video.mp4"}} ] }], extra_body={ "video_fps": 2.0, "video_max_frames": 32 } ) ``` ### Mediante curl ```bash curl http://localhost:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "default", "messages": [{ "role": "user", "content": [ {"type": "text", "text": "Describe this video"}, {"type": "video_url", "video_url": {"url": "https://example.com/video.mp4"}} ] }], "video_fps": 2.0, "video_max_frames": 16 }' ``` ## Formatos Soportados ### Imágenes | Formato | Ejemplo | |--------|---------| | URL | `{"type": "image_url", "image_url": {"url": "https://..."}}` | | Archivo local | `{"type": "image_url", "image_url": {"url": "/path/to/image.jpg"}}` | | Base64 | `{"type": "image_url", "image_url": {"url": "data:image/jpeg;base64,..."}}` | ### Videos | Formato | Ejemplo | |--------|---------| | URL | `{"type": "video_url", "video_url": {"url": "https://..."}}` | | Archivo local | `{"type": "video", "video": "/path/to/video.mp4"}` | | Base64 | `{"type": "video_url", "video_url": {"url": "data:video/mp4;base64,..."}}` | ## API de Python ```python from vllm_mlx.models import MLXMultimodalLM mllm = MLXMultimodalLM("mlx-community/Qwen3-VL-4B-Instruct-3bit") mllm.load() # Imagen description = mllm.describe_image("photo.jpg") # Video description = mllm.describe_video("video.mp4", fps=2.0) # Prompt personalizado output = mllm.generate( prompt="Compare these images", images=["img1.jpg", "img2.jpg"] ) ``` ## Consejos de Rendimiento ### Imágenes - Las resoluciones menores se procesan más rápido (224x224 vs 1920x1080) - Usa la resolución adecuada para tu tarea ### Videos - Menor FPS = procesamiento más rápido - Menos fotogramas = menor uso de memoria - 64 fotogramas es el máximo práctico (96 o más causa timeout en la GPU) ## Benchmarks Probado en Apple M4 Max con 128 GB de memoria unificada. ### Qwen3-VL-4B-Instruct-3bit | Resolución | Tiempo | Tokens | Velocidad | Memoria | |------------|------|--------|-------|--------| | 224x224 | 0.87s | 124 | 143 tok/s | 2.6 GB | | 448x448 | 1.01s | 107 | 106 tok/s | 3.1 GB | | 768x768 | 1.42s | 127 | 89 tok/s | 3.4 GB | | 1024x1024 | 1.85s | 116 | 63 tok/s | 3.6 GB | ### Qwen3-VL-8B-Instruct-4bit | Resolución | Tiempo | Tokens | Velocidad | Memoria | |------------|------|--------|-------|--------| | 224x224 | 1.08s | 78 | 73 tok/s | 5.6 GB | | 448x448 | 1.41s | 70 | 50 tok/s | 6.1 GB | | 768x768 | 2.06s | 91 | 44 tok/s | 6.5 GB | | 1024x1024 | 3.02s | 76 | 25 tok/s | 7.6 GB | ### Gemma 3 4B 4bit | Resolución | Tiempo | Tokens | Velocidad | Memoria | |------------|------|--------|-------|--------| | 224x224 | 0.95s | 30 | 32 tok/s | 5.2 GB | | 448x448 | 0.99s | 34 | 34 tok/s | 5.2 GB | | 768x768 | 0.99s | 32 | 32 tok/s | 5.2 GB | | 1024x1024 | 0.95s | 28 | 29 tok/s | 5.2 GB | ### Ejecutar Benchmarks ```bash # Benchmark rápido vllm-mlx-bench --model mlx-community/Qwen3-VL-4B-Instruct-3bit --quick # Benchmark completo con más resoluciones vllm-mlx-bench --model mlx-community/Qwen3-VL-4B-Instruct-3bit # Benchmark de video vllm-mlx-bench --model mlx-community/Qwen3-VL-4B-Instruct-3bit --video ``` ## MLLM Cache vllm-mlx incluye un sistema de prefix cache para modelos multimodales que puede acelerar significativamente las solicitudes repetidas con las mismas imágenes. ### Cómo Funciona Cuando envías una imagen al modelo, el encoder de visión la procesa y genera embeddings. Este procesamiento toma entre 1 y 2 segundos. El MLLM cache almacena esos embeddings junto con el estado del KV cache, de modo que las solicitudes posteriores con la misma imagen omiten el encoder de visión por completo. El cache utiliza hashing basado en contenido (similar a LMCache) para identificar imágenes idénticas sin importar cómo se proporcionen (URL, base64 o ruta de archivo). ### Habilitar el Cache ```bash # Habilitar con configuración predeterminada (512 MB máximo) vllm-mlx serve mlx-community/Qwen3-VL-4B-Instruct-3bit --enable-mllm-cache # Con límite de memoria personalizado vllm-mlx serve mlx-community/Qwen3-VL-4B-Instruct-3bit \ --enable-mllm-cache \ --mllm-cache-max-mb 1024 ``` ### API de Python ```python from vllm_mlx.mllm_cache import MLLMPrefixCacheManager # Crear el gestor de cache cache = MLLMPrefixCacheManager(max_memory_mb=512) # Almacenar embeddings y KV cache tras el procesamiento cache.store( images=["photo.jpg"], prompt="Describe this image", vision_embeddings=embeddings, kv_cache=kv_state, num_tokens=128 ) # Recuperar del cache en solicitudes posteriores entry, match_len = cache.fetch(images=["photo.jpg"], prompt="Describe this image") if entry: # Usar embeddings en cache, omitir el encoder de visión embeddings = entry.vision_embeddings kv_state = entry.kv_cache ``` ### Estadísticas del Cache ```python stats = cache.get_stats() print(f"Hit rate: {stats.hit_rate:.1%}") print(f"Memory used: {stats.memory_used_mb:.1f} MB") print(f"Tokens saved: {stats.tokens_saved}") ``` ### Gestión de Memoria El cache utiliza evicción LRU (Least Recently Used) cuando se alcanza el límite de memoria. Cada entrada registra: - Tamaño de los embeddings de visión - Tamaño del KV cache por capa - Frecuencia de acceso para el ordenamiento LRU Cuando hay presión de memoria, las entradas con acceso menos reciente se evictan primero. ## Gradio Chat UI Para chat multimodal interactivo: ```bash vllm-mlx-chat --served-model-name mlx-community/Qwen3-VL-4B-Instruct-3bit ``` Soporta arrastrar y soltar imágenes y videos. # Documentation page: `es/guides/python-api.md` # Python API API de Python directa para acceso programático a vllm-mlx. ## Modelos de lenguaje ### Uso básico ```python from vllm_mlx.models import MLXLanguageModel # Load model model = MLXLanguageModel("mlx-community/Llama-3.2-3B-Instruct-4bit") model.load() # Generate text output = model.generate("What is the capital of France?", max_tokens=100) print(output.text) ``` ### Generación con streaming ```python for chunk in model.stream_generate("Tell me a story about a robot"): print(chunk.text, end="", flush=True) ``` ### Interfaz de chat ```python messages = [ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Hello, who are you?"} ] response = model.chat(messages) print(response.text) ``` ### Parámetros de generación ```python output = model.generate( prompt="Write a poem", max_tokens=256, temperature=0.7, top_p=0.9, stop=["END", "\n\n"] ) ``` | Parámetro | Descripción | Valor por defecto | |-----------|-------------|---------| | `max_tokens` | Cantidad máxima de tokens a generar | 256 | | `temperature` | Temperatura de muestreo (0-2) | 0.7 | | `top_p` | Nucleus sampling | 0.9 | | `stop` | Secuencias de parada | None | ## Modelos de visión y lenguaje (VLM) ### Uso básico ```python from vllm_mlx.models import MLXMultimodalLM # Load model mllm = MLXMultimodalLM("mlx-community/Qwen3-VL-4B-Instruct-3bit") mllm.load() # Describe an image description = mllm.describe_image("photo.jpg") print(description) ``` ### Preguntas sobre imágenes ```python answer = mllm.answer_about_image("photo.jpg", "What color is the car?") print(answer) ``` ### Varias imágenes ```python output = mllm.generate( prompt="Compare these two images", images=["image1.jpg", "image2.jpg"] ) print(output.text) ``` ### Comprensión de video ```python # From local file output = mllm.generate( prompt="What is happening in this video?", videos=["video.mp4"], video_fps=2.0, video_max_frames=16 ) print(output.text) # From URL output = mllm.generate( prompt="Describe this video", videos=["https://example.com/video.mp4"], video_fps=2.0 ) # Convenience method description = mllm.describe_video("video.mp4", fps=2.0) ``` ### Parámetros de video | Parámetro | Descripción | Valor por defecto | |-----------|-------------|---------| | `video_fps` | Fotogramas por segundo a extraer | 2.0 | | `video_max_frames` | Cantidad máxima de fotogramas a procesar | 32 | ## Engine API Para casos de uso avanzados, se puede usar el engine directamente: ### Engine simple ```python from vllm_mlx.engine import SimpleEngine engine = SimpleEngine("mlx-community/Llama-3.2-3B-Instruct-4bit") await engine.start() output = await engine.generate( prompt="Hello world", max_tokens=100 ) print(output.text) await engine.stop() ``` ### Engine con batching ```python from vllm_mlx.engine import BatchedEngine engine = BatchedEngine("mlx-community/Llama-3.2-3B-Instruct-4bit") await engine.start() # Multiple concurrent requests output = await engine.generate( prompt="Hello world", max_tokens=100 ) await engine.stop() ``` ## Formato de salida Todos los métodos de generación retornan un objeto `GenerationOutput`: ```python output = model.generate("Hello") print(output.text) # Generated text print(output.prompt_tokens) # Input token count print(output.completion_tokens) # Output token count print(output.finish_reason) # "stop" or "length" ``` ## Manejo de errores ```python from vllm_mlx.models import MLXLanguageModel try: model = MLXLanguageModel("invalid-model") model.load() except Exception as e: print(f"Failed to load model: {e}") ``` # Documentation page: `es/guides/reasoning.md` # Modelos de reasoning vllm-mlx admite modelos de reasoning que muestran su proceso de thinking antes de dar una respuesta. Modelos como Qwen3 y DeepSeek-R1 envuelven su reasoning en etiquetas `...`, y vllm-mlx puede analizar estas etiquetas para separar el reasoning de la respuesta final. ## Por que usar el reasoning parser? Cuando un modelo de reasoning genera salida, normalmente luce asi: ``` Let me analyze this step by step. First, I need to consider the constraints. The answer should be a prime number less than 10. Checking: 2, 3, 5, 7 are all prime and less than 10. The prime numbers less than 10 are: 2, 3, 5, 7. ``` Sin el reasoning parser, obtienes la salida cruda con las etiquetas incluidas. Con el reasoning parser habilitado, el proceso de thinking y la respuesta final se separan en campos distintos dentro de la respuesta de la API. ## Primeros pasos ### Iniciar el servidor con el reasoning parser ```bash # For Qwen3 models vllm-mlx serve mlx-community/Qwen3-8B-4bit --reasoning-parser qwen3 # For DeepSeek-R1 models vllm-mlx serve mlx-community/DeepSeek-R1-Distill-Qwen-7B-4bit --reasoning-parser deepseek_r1 ``` ### Formato de respuesta de la API Cuando el reasoning parser esta habilitado, la respuesta de la API incluye un campo `reasoning`: **Respuesta sin streaming:** ```json { "choices": [{ "message": { "role": "assistant", "content": "The prime numbers less than 10 are: 2, 3, 5, 7.", "reasoning": "Let me analyze this step by step.\nFirst, I need to consider the constraints.\nThe answer should be a prime number less than 10.\nChecking: 2, 3, 5, 7 are all prime and less than 10." } }] } ``` **Respuesta con streaming:** Los fragmentos se envian por separado para el reasoning y el contenido. Durante la fase de reasoning, los fragmentos tienen `reasoning` con valor. Cuando el modelo pasa a la respuesta final, los fragmentos tienen `content` con valor: ```json {"delta": {"reasoning": "Let me analyze"}} {"delta": {"reasoning": " this step by step."}} {"delta": {"reasoning": "\nFirst, I need to"}} ... {"delta": {"content": "The prime"}} {"delta": {"content": " numbers less than 10"}} {"delta": {"content": " are: 2, 3, 5, 7."}} ``` ## Uso con el SDK de OpenAI ```python from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed") # Non-streaming response = client.chat.completions.create( model="default", messages=[{"role": "user", "content": "What are the prime numbers less than 10?"}] ) message = response.choices[0].message print("Reasoning:", message.reasoning) # The thinking process print("Answer:", message.content) # The final answer ``` ### Streaming con reasoning ```python reasoning_text = "" content_text = "" stream = client.chat.completions.create( model="default", messages=[{"role": "user", "content": "Solve: 2 + 2 = ?"}], stream=True ) for chunk in stream: delta = chunk.choices[0].delta if hasattr(delta, 'reasoning') and delta.reasoning: reasoning_text += delta.reasoning print(f"[Thinking] {delta.reasoning}", end="") if delta.content: content_text += delta.content print(delta.content, end="") print(f"\n\nFinal reasoning: {reasoning_text}") print(f"Final answer: {content_text}") ``` ## Parsers disponibles ### Parser de Qwen3 (`qwen3`) Para modelos Qwen3 que usan etiquetas explicitas `` y ``. - Requiere **ambas** etiquetas, la de apertura y la de cierre - Si faltan las etiquetas, la salida se trata como contenido regular - Recomendado para: Qwen3-0.6B, Qwen3-4B, Qwen3-8B y modelos similares ```bash vllm-mlx serve mlx-community/Qwen3-8B-4bit --reasoning-parser qwen3 ``` ### Parser de DeepSeek-R1 (`deepseek_r1`) Para modelos DeepSeek-R1 que pueden omitir la etiqueta de apertura ``. - Mas permisivo que el parser de Qwen3 - Maneja casos donde `` es implicita - El contenido antes de `` se trata como reasoning incluso sin `` ```bash vllm-mlx serve mlx-community/DeepSeek-R1-Distill-Qwen-7B-4bit --reasoning-parser deepseek_r1 ``` ## Como funciona El reasoning parser usa deteccion basada en texto para identificar etiquetas de thinking en la salida del modelo. Durante el streaming, rastrea la posicion actual en la salida para enrutar correctamente cada token a `reasoning` o a `content`. ``` Model Output: Step 1: analyze...The answer is 42. ├─────────────────────┤├─────────────────────┤ Parsed: │ reasoning ││ content │ └─────────────────────┘└─────────────────────┘ ``` El parsing no tiene estado y usa el texto acumulado para determinar el contexto, lo que lo hace robusto para escenarios de streaming donde los tokens pueden llegar en fragmentos arbitrarios. ## Consejos para mejores resultados ### Prompting Los modelos de reasoning funcionan mejor cuando se les anima a pensar paso a paso: ```python messages = [ {"role": "system", "content": "Think through problems step by step before answering."}, {"role": "user", "content": "What is 17 × 23?"} ] ``` ### Manejo del reasoning ausente Algunos prompts pueden no activar el reasoning. En esos casos, `reasoning` sera `None` y toda la salida va a `content`: ```python message = response.choices[0].message if message.reasoning: print(f"Model's thought process: {message.reasoning}") print(f"Answer: {message.content}") ``` ### Temperatura y reasoning Las temperaturas más bajas tienden a producir patrones de reasoning más consistentes: ```python response = client.chat.completions.create( model="default", messages=[{"role": "user", "content": "Explain quantum entanglement"}], temperature=0.3 # More focused reasoning ) ``` ## Compatibilidad con versiones anteriores Cuando no se especifica `--reasoning-parser`, el servidor se comporta como antes: - Las etiquetas de thinking se incluyen en el campo `content` - No se agrega el campo `reasoning` a las respuestas Esto garantiza que las aplicaciones existentes sigan funcionando sin cambios. ## Ejemplo: solucionador de problemas matematicos ```python from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed") def solve_math(problem: str) -> dict: """Solve a math problem and return reasoning + answer.""" response = client.chat.completions.create( model="default", messages=[ {"role": "system", "content": "You are a math tutor. Show your work."}, {"role": "user", "content": problem} ], temperature=0.2 ) message = response.choices[0].message return { "problem": problem, "work": message.reasoning, "answer": message.content } result = solve_math("If a train travels 120 km in 2 hours, what is its average speed?") print(f"Problem: {result['problem']}") print(f"\nWork shown:\n{result['work']}") print(f"\nFinal answer: {result['answer']}") ``` ## Ejemplos con curl ### Sin streaming ```bash curl http://localhost:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "default", "messages": [{"role": "user", "content": "What is 15% of 80?"}] }' ``` ### Con streaming ```bash curl http://localhost:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "default", "messages": [{"role": "user", "content": "What is 15% of 80?"}], "stream": true }' ``` ## Solucion de problemas ### No aparece el campo reasoning en la respuesta - Asegurate de haber iniciado el servidor con `--reasoning-parser` - Verifica que el modelo realmente use etiquetas de thinking (no todos los prompts activan el reasoning) ### El reasoning aparece en content - Es posible que el modelo no este usando el formato de etiquetas esperado - Prueba un parser diferente (`qwen3` vs `deepseek_r1`) ### Reasoning truncado - Aumenta `--max-tokens` si el modelo esta alcanzando el limite de tokens a mitad del thinking ## Relacionado - [Modelos admitidos](../reference/models.md) - Modelos que admiten reasoning - [Configuracion del servidor](server.md) - Todas las opciones del servidor - [Referencia de CLI](../reference/cli.md) - Opciones de línea de comandos # Documentation page: `es/guides/server.md` # Servidor compatible con OpenAI vllm-mlx provee un servidor FastAPI con compatibilidad completa con la API de OpenAI. Por defecto, el servidor escucha solo en `127.0.0.1`. Usa `--host 0.0.0.0` solo cuando quieras exponerlo fuera de la máquina local de forma intencional. ## Iniciar el servidor ### Modo simple (por defecto) Máximo rendimiento para un solo usuario: ```bash vllm-mlx serve mlx-community/Llama-3.2-3B-Instruct-4bit --port 8000 ``` ### Modo continuous batching Para múltiples usuarios concurrentes: ```bash vllm-mlx serve mlx-community/Llama-3.2-3B-Instruct-4bit --port 8000 --continuous-batching ``` ### Con paged cache Caché eficiente en memoria para producción: ```bash vllm-mlx serve mlx-community/Llama-3.2-3B-Instruct-4bit --port 8000 --continuous-batching --use-paged-cache ``` ## Opciones del servidor | Opción | Descripción | Valor por defecto | |--------|-------------|---------| | `--port` | Puerto del servidor | 8000 | | `--host` | Host del servidor | 127.0.0.1 | | `--api-key` | Clave de API para autenticación | None | | `--rate-limit` | Solicitudes por minuto por cliente (0 = desactivado) | 0 | | `--timeout` | Tiempo límite de solicitud en segundos | 300 | | `--enable-metrics` | Expone métricas de Prometheus en `/metrics` | False | | `--continuous-batching` | Activa batching para múltiples usuarios | False | | `--use-paged-cache` | Activa paged KV cache | False | | `--cache-memory-mb` | Límite de memoria de caché en MB | Auto | | `--cache-memory-percent` | Fracción de RAM para caché | 0.20 | | `--max-tokens` | Máximo de tokens por defecto | 32768 | | `--max-request-tokens` | Máximo de `max_tokens` aceptado de clientes de la API | 32768 | | `--default-temperature` | Temperatura por defecto cuando no se especifica | None | | `--default-top-p` | top_p por defecto cuando no se especifica | None | | `--stream-interval` | Tokens por fragmento de streaming | 1 | | `--mcp-config` | Ruta al archivo de configuración de MCP | None | | `--reasoning-parser` | Parser para modelos de reasoning (`qwen3`, `deepseek_r1`) | None | | `--embedding-model` | Pre-carga un modelo de embeddings al iniciar | None | | `--enable-auto-tool-choice` | Activa tool calling automático | False | | `--tool-call-parser` | Parser de tool calls (ver [Tool Calling](tool-calling.md)) | None | ## Endpoints de la API ### Chat completions ```bash POST /v1/chat/completions ``` ```python from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed") # Sin streaming response = client.chat.completions.create( model="default", messages=[{"role": "user", "content": "Hello!"}], max_tokens=100 ) # Con streaming stream = client.chat.completions.create( model="default", messages=[{"role": "user", "content": "Tell me a story"}], stream=True ) for chunk in stream: if chunk.choices[0].delta.content: print(chunk.choices[0].delta.content, end="") ``` ### Completions ```bash POST /v1/completions ``` ```python response = client.completions.create( model="default", prompt="The capital of France is", max_tokens=50 ) ``` ### Models ```bash GET /v1/models ``` Retorna los modelos disponibles. ### Embeddings ```bash POST /v1/embeddings ``` ```python response = client.embeddings.create( model="mlx-community/multilingual-e5-small-mlx", input="Hello world" ) print(response.data[0].embedding[:5]) # First 5 dimensions ``` Consulta la [Guía de Embeddings](embeddings.md) para más detalles. ### Health check ```bash GET /health ``` Retorna el estado del servidor. ### Métricas ```bash GET /metrics ``` Endpoint de scrape de Prometheus con métricas del servidor, caché, scheduler y solicitudes. El endpoint está desactivado por defecto y se habilita con `--enable-metrics`. ```bash vllm-mlx serve mlx-community/Llama-3.2-3B-Instruct-4bit \ --enable-metrics ``` `/metrics` no requiere autenticación de forma intencional. Exponlo solo en una red de confianza o detrás de un proxy inverso o firewall que limite quién puede consultarlo. ### API de mensajes de Anthropic ```bash POST /v1/messages ``` Endpoint compatible con Anthropic que permite que herramientas como Claude Code y OpenCode se conecten directamente a vllm-mlx. Internamente traduce las solicitudes de Anthropic al formato de OpenAI, ejecuta la inferencia a través del motor y convierte la respuesta de vuelta al formato de Anthropic. Capacidades: - Respuestas sin streaming y con streaming (SSE) - Mensajes de sistema (cadena de texto simple o lista de bloques de contenido) - Conversaciones multi-turno con mensajes de usuario y asistente - Tool calling con bloques de contenido `tool_use` / `tool_result` - Conteo de tokens para seguimiento de presupuesto - Contenido multimodal (imágenes mediante bloques `source`) - Detección de desconexión del cliente (retorna HTTP 499) - Filtrado automático de tokens especiales en la salida en streaming #### Sin streaming ```python from anthropic import Anthropic client = Anthropic(base_url="http://localhost:8000", api_key="not-needed") response = client.messages.create( model="default", max_tokens=256, messages=[{"role": "user", "content": "Hello!"}] ) print(response.content[0].text) # Response includes: response.id, response.model, response.stop_reason, # response.usage.input_tokens, response.usage.output_tokens ``` #### Streaming El streaming sigue el protocolo de eventos SSE de Anthropic. Los eventos se emiten en este orden: `message_start` -> `content_block_start` -> `content_block_delta` (repetido) -> `content_block_stop` -> `message_delta` -> `message_stop` ```python with client.messages.stream( model="default", max_tokens=256, messages=[{"role": "user", "content": "Tell me a story"}] ) as stream: for text in stream.text_stream: print(text, end="") ``` #### Mensajes de sistema Los mensajes de sistema pueden ser una cadena de texto simple o una lista de bloques de contenido: ```python # Plain string response = client.messages.create( model="default", max_tokens=256, system="You are a helpful coding assistant.", messages=[{"role": "user", "content": "Write a hello world in Python"}] ) # List of content blocks response = client.messages.create( model="default", max_tokens=256, system=[ {"type": "text", "text": "You are a helpful assistant."}, {"type": "text", "text": "Be concise in your answers."}, ], messages=[{"role": "user", "content": "What is 2+2?"}] ) ``` #### Tool calling Define las herramientas con `name`, `description` e `input_schema`. El modelo retorna bloques de contenido `tool_use` cuando desea llamar a una herramienta. Envía los resultados de vuelta como bloques `tool_result`. ```python # Step 1: Send request with tools response = client.messages.create( model="default", max_tokens=1024, messages=[{"role": "user", "content": "What's the weather in Paris?"}], tools=[{ "name": "get_weather", "description": "Get weather for a city", "input_schema": { "type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"] } }] ) # Step 2: Check if model wants to use tools for block in response.content: if block.type == "tool_use": print(f"Tool: {block.name}, Input: {block.input}, ID: {block.id}") # response.stop_reason will be "tool_use" # Step 3: Send tool result back response = client.messages.create( model="default", max_tokens=1024, messages=[ {"role": "user", "content": "What's the weather in Paris?"}, {"role": "assistant", "content": response.content}, {"role": "user", "content": [ { "type": "tool_result", "tool_use_id": block.id, "content": "Sunny, 22C" } ]} ], tools=[{ "name": "get_weather", "description": "Get weather for a city", "input_schema": { "type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"] } }] ) print(response.content[0].text) # "The weather in Paris is sunny, 22C." ``` Modos de selección de herramientas: | `tool_choice` | Comportamiento | |---------------|----------| | `{"type": "auto"}` | El modelo decide si llamar herramientas (por defecto) | | `{"type": "any"}` | El modelo debe llamar al menos una herramienta | | `{"type": "tool", "name": "get_weather"}` | El modelo debe llamar la herramienta especificada | | `{"type": "none"}` | El modelo no llamará ninguna herramienta | #### Conversaciones multi-turno ```python messages = [ {"role": "user", "content": "My name is Alice."}, {"role": "assistant", "content": "Nice to meet you, Alice!"}, {"role": "user", "content": "What's my name?"}, ] response = client.messages.create( model="default", max_tokens=100, messages=messages ) ``` #### Conteo de tokens ```bash POST /v1/messages/count_tokens ``` Cuenta los tokens de entrada para una solicitud de Anthropic usando el tokenizador del modelo. Útil para el seguimiento de presupuesto antes de enviar una solicitud. Cuenta tokens de mensajes de sistema, mensajes de conversación, entradas de tool_use, contenido de tool_result y definiciones de herramientas (name, description, input_schema). ```python import requests resp = requests.post("http://localhost:8000/v1/messages/count_tokens", json={ "model": "default", "messages": [{"role": "user", "content": "Hello, how are you?"}], "system": "You are helpful.", "tools": [{ "name": "search", "description": "Search the web", "input_schema": {"type": "object", "properties": {"q": {"type": "string"}}} }] }) print(resp.json()) # {"input_tokens": 42} ``` #### Ejemplos con curl Sin streaming: ```bash curl http://localhost:8000/v1/messages \ -H "Content-Type: application/json" \ -d '{ "model": "default", "max_tokens": 256, "messages": [{"role": "user", "content": "Hello!"}] }' ``` Con streaming: ```bash curl http://localhost:8000/v1/messages \ -H "Content-Type: application/json" \ -d '{ "model": "default", "max_tokens": 256, "stream": true, "messages": [{"role": "user", "content": "Tell me a joke"}] }' ``` Conteo de tokens: ```bash curl http://localhost:8000/v1/messages/count_tokens \ -H "Content-Type: application/json" \ -d '{ "model": "default", "messages": [{"role": "user", "content": "Hello!"}] }' # {"input_tokens": 12} ``` #### Campos de la solicitud | Campo | Tipo | Requerido | Valor por defecto | Descripción | |-------|------|----------|---------|-------------| | `model` | string | sí | - | Nombre del modelo (usa `"default"` para el modelo cargado) | | `messages` | list | sí | - | Mensajes de conversación con `role` y `content` | | `max_tokens` | int | sí | - | Número máximo de tokens a generar | | `system` | string o list | no | null | Prompt de sistema (cadena o lista de bloques `{"type": "text", "text": "..."}`) | | `stream` | bool | no | false | Activa el streaming SSE | | `temperature` | float | no | 0.7 | Temperatura de muestreo (0.0 = determinista, 1.0 = creativo) | | `top_p` | float | no | 0.9 | Umbral de nucleus sampling | | `top_k` | int | no | null | Top-k sampling | | `stop_sequences` | list | no | null | Secuencias que detienen la generación | | `tools` | list | no | null | Definiciones de herramientas con `name`, `description`, `input_schema` | | `tool_choice` | dict | no | null | Modo de selección de herramientas (`auto`, `any`, `tool`, `none`) | | `metadata` | dict | no | null | Metadatos arbitrarios (se pasan sin ser usados por el servidor) | #### Formato de respuesta Respuesta sin streaming: ```json { "id": "msg_abc123...", "type": "message", "role": "assistant", "model": "default", "content": [ {"type": "text", "text": "Hello! How can I help?"} ], "stop_reason": "end_turn", "stop_sequence": null, "usage": { "input_tokens": 12, "output_tokens": 8 } } ``` Cuando se llaman herramientas, `content` incluye bloques `tool_use` y `stop_reason` es `"tool_use"`: ```json { "content": [ {"type": "text", "text": "Let me check the weather."}, { "type": "tool_use", "id": "call_abc123", "name": "get_weather", "input": {"city": "Paris"} } ], "stop_reason": "tool_use" } ``` Razones de parada: | `stop_reason` | Significado | |---------------|---------| | `end_turn` | El modelo terminó de forma natural | | `tool_use` | El modelo quiere llamar una herramienta | | `max_tokens` | Se alcanzó el límite de `max_tokens` | #### Uso con Claude Code Apunta Claude Code directamente a tu servidor vllm-mlx: ```bash # Start the server vllm-mlx serve mlx-community/Qwen3-Coder-Next-235B-A22B-4bit \ --continuous-batching \ --enable-auto-tool-choice \ --tool-call-parser hermes # In another terminal, configure Claude Code export ANTHROPIC_BASE_URL=http://localhost:8000 export ANTHROPIC_API_KEY=not-needed claude ``` ### Estado del servidor ```bash GET /v1/status ``` Endpoint de monitoreo en tiempo real que retorna estadísticas generales del servidor y detalles por solicitud. Útil para depurar el rendimiento, rastrear la eficiencia de la caché y monitorear la memoria GPU Metal. ```bash curl -s http://localhost:8000/v1/status | python -m json.tool ``` Respuesta de ejemplo: ```json { "status": "running", "model": "mlx-community/Qwen3-8B-4bit", "uptime_s": 342.5, "steps_executed": 1247, "num_running": 1, "num_waiting": 0, "total_requests_processed": 15, "total_prompt_tokens": 28450, "total_completion_tokens": 3200, "metal": { "active_memory_gb": 5.2, "peak_memory_gb": 8.1, "cache_memory_gb": 2.3 }, "cache": { "type": "memory_aware_cache", "entries": 5, "hit_rate": 0.87, "memory_mb": 2350 }, "requests": [ { "request_id": "req_abc123", "phase": "generation", "tokens_per_second": 45.2, "ttft_s": 0.8, "progress": 0.35, "cache_hit_type": "prefix", "cached_tokens": 1200, "generated_tokens": 85, "max_tokens": 256 } ] } ``` Campos de la respuesta: | Campo | Descripción | |-------|-------------| | `status` | Estado del servidor: `running`, `stopped` o `not_loaded` | | `model` | Nombre del modelo cargado | | `uptime_s` | Segundos desde que el servidor inició | | `steps_executed` | Total de pasos de inferencia ejecutados | | `num_running` | Número de solicitudes generando tokens actualmente | | `num_waiting` | Número de solicitudes en cola para prefill | | `total_requests_processed` | Total de solicitudes completadas desde el inicio | | `total_prompt_tokens` | Total de tokens de prompt procesados desde el inicio | | `total_completion_tokens` | Total de tokens de completion generados desde el inicio | | `metal.active_memory_gb` | Memoria GPU Metal en uso actualmente (GB) | | `metal.peak_memory_gb` | Uso pico de memoria GPU Metal (GB) | | `metal.cache_memory_gb` | Uso de memoria de caché Metal (GB) | | `cache` | Estadísticas de caché (tipo, entradas, tasa de aciertos, uso de memoria) | | `requests` | Lista de solicitudes activas con detalles por solicitud | Campos por solicitud en `requests`: | Campo | Descripción | |-------|-------------| | `request_id` | Identificador único de la solicitud | | `phase` | Fase actual: `queued`, `prefill` o `generation` | | `tokens_per_second` | Rendimiento de generación para esta solicitud | | `ttft_s` | Tiempo hasta el primer token (segundos) | | `progress` | Porcentaje de completado (0.0 a 1.0) | | `cache_hit_type` | Tipo de coincidencia en caché: `exact`, `prefix`, `supersequence`, `lcp` o `miss` | | `cached_tokens` | Número de tokens servidos desde caché | | `generated_tokens` | Tokens generados hasta ahora | | `max_tokens` | Máximo de tokens solicitados | ## Tool Calling Activa tool calling compatible con OpenAI con `--enable-auto-tool-choice`: ```bash vllm-mlx serve mlx-community/Devstral-Small-2507-4bit \ --enable-auto-tool-choice \ --tool-call-parser mistral ``` Usa la opción `--tool-call-parser` para seleccionar el parser adecuado para tu modelo: | Parser | Modelos | |--------|--------| | `auto` | Detección automática (prueba todos los parsers) | | `mistral` | Mistral, Devstral | | `qwen` | Qwen, Qwen3 | | `llama` | Llama 3.x, 4.x | | `hermes` | Hermes, NousResearch | | `deepseek` | DeepSeek V3, R1 | | `kimi` | Kimi K2, Moonshot | | `granite` | IBM Granite 3.x, 4.x | | `nemotron` | NVIDIA Nemotron | | `xlam` | Salesforce xLAM | | `functionary` | MeetKai Functionary | | `glm47` | GLM-4.7, GLM-4.7-Flash | ```python response = client.chat.completions.create( model="default", messages=[{"role": "user", "content": "What's the weather in Paris?"}], tools=[{ "type": "function", "function": { "name": "get_weather", "description": "Get weather for a city", "parameters": { "type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"] } } }] ) if response.choices[0].message.tool_calls: for tc in response.choices[0].message.tool_calls: print(f"{tc.function.name}: {tc.function.arguments}") ``` Consulta la [Guía de Tool Calling](tool-calling.md) para la documentación completa. ## Modelos de reasoning Para modelos que muestran su proceso de pensamiento (Qwen3, DeepSeek-R1), usa `--reasoning-parser` para separar el reasoning de la respuesta final: ```bash # Qwen3 models vllm-mlx serve mlx-community/Qwen3-8B-4bit --reasoning-parser qwen3 # DeepSeek-R1 models vllm-mlx serve mlx-community/DeepSeek-R1-Distill-Qwen-7B-4bit --reasoning-parser deepseek_r1 ``` La respuesta de la API incluye un campo `reasoning` con el proceso de pensamiento del modelo: ```python response = client.chat.completions.create( model="default", messages=[{"role": "user", "content": "What is 17 × 23?"}] ) print(response.choices[0].message.reasoning) # Step-by-step thinking print(response.choices[0].message.content) # Final answer ``` En streaming, los fragmentos de reasoning llegan primero, seguidos de los fragmentos de contenido: ```python for chunk in stream: delta = chunk.choices[0].delta if delta.reasoning: print(f"[Thinking] {delta.reasoning}") if delta.content: print(delta.content, end="") ``` Consulta la [Guía de Modelos de Reasoning](reasoning.md) para todos los detalles. ## Salida estructurada (modo JSON) Obliga al modelo a retornar JSON válido usando `response_format`: ### Modo JSON Object Retorna cualquier JSON válido: ```python response = client.chat.completions.create( model="default", messages=[{"role": "user", "content": "List 3 colors"}], response_format={"type": "json_object"} ) # Output: {"colors": ["red", "blue", "green"]} ``` ### Modo JSON Schema Retorna JSON que coincide con un esquema específico: ```python response = client.chat.completions.create( model="default", messages=[{"role": "user", "content": "List 3 colors"}], response_format={ "type": "json_schema", "json_schema": { "name": "colors", "schema": { "type": "object", "properties": { "colors": { "type": "array", "items": {"type": "string"} } }, "required": ["colors"] } } } ) # Output validated against schema data = json.loads(response.choices[0].message.content) assert "colors" in data ``` ### Ejemplo con curl ```bash curl http://localhost:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "default", "messages": [{"role": "user", "content": "List 3 colors"}], "response_format": {"type": "json_object"} }' ``` ## Ejemplos con curl ### Chat ```bash curl http://localhost:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "default", "messages": [{"role": "user", "content": "Hello!"}], "max_tokens": 100 }' ``` ### Streaming ```bash curl http://localhost:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "default", "messages": [{"role": "user", "content": "Hello!"}], "stream": true }' ``` ## Configuración de streaming Controla el comportamiento del streaming con `--stream-interval`: | Valor | Comportamiento | |-------|----------| | `1` (por defecto) | Envía cada token inmediatamente | | `2-5` | Agrupa tokens antes de enviar | | `10+` | Máximo rendimiento, salida en fragmentos más grandes | ```bash # Smooth streaming vllm-mlx serve model --continuous-batching --stream-interval 1 # Batched streaming (better for high-latency networks) vllm-mlx serve model --continuous-batching --stream-interval 5 ``` ## Integración con Open WebUI ```bash # 1. Start vllm-mlx server vllm-mlx serve mlx-community/Llama-3.2-3B-Instruct-4bit --port 8000 # 2. Start Open WebUI docker run -d -p 3000:8080 \ -e OPENAI_API_BASE_URL=http://host.docker.internal:8000/v1 \ -e OPENAI_API_KEY=not-needed \ --name open-webui \ ghcr.io/open-webui/open-webui:main # 3. Open http://localhost:3000 ``` ## Despliegue en producción ### Con systemd Crea `/etc/systemd/system/vllm-mlx.service`: ```ini [Unit] Description=vLLM-MLX Server After=network.target [Service] Type=simple ExecStart=/usr/local/bin/vllm-mlx serve mlx-community/Qwen3-0.6B-8bit \ --continuous-batching --use-paged-cache --port 8000 Restart=always [Install] WantedBy=multi-user.target ``` ```bash sudo systemctl enable vllm-mlx sudo systemctl start vllm-mlx ``` ### Configuración recomendada Para producción con 50 o más usuarios concurrentes: ```bash vllm-mlx serve mlx-community/Qwen3-0.6B-8bit \ --continuous-batching \ --use-paged-cache \ --api-key your-secret-key \ --rate-limit 60 \ --timeout 120 \ --port 8000 ``` # Documentation page: `es/guides/tool-calling.md` # Tool Calling vllm-mlx soporta tool calling compatible con OpenAI (function calling) con análisis automático para muchas familias de modelos populares. ## Inicio rápido Activa el tool calling agregando la bandera `--enable-auto-tool-choice` al iniciar el servidor: ```bash vllm-mlx serve mlx-community/Devstral-Small-2507-4bit \ --enable-auto-tool-choice \ --tool-call-parser mistral ``` Luego usa herramientas con la API estándar de OpenAI: ```python from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed") response = client.chat.completions.create( model="default", messages=[{"role": "user", "content": "What's the weather in Paris?"}], tools=[{ "type": "function", "function": { "name": "get_weather", "description": "Get weather for a city", "parameters": { "type": "object", "properties": { "city": {"type": "string", "description": "City name"} }, "required": ["city"] } } }] ) # Check for tool calls if response.choices[0].message.tool_calls: for tc in response.choices[0].message.tool_calls: print(f"Function: {tc.function.name}") print(f"Arguments: {tc.function.arguments}") ``` ## Parsers disponibles Usa `--tool-call-parser` para seleccionar un tool parser según tu familia de modelos: | Parser | Alias | Modelos | Formato | |--------|-------|---------|---------| | `auto` | | Cualquier modelo | Detecta el formato automáticamente (prueba todos los parsers) | | `mistral` | | Mistral, Devstral | Arreglo JSON con `[TOOL_CALLS]` | | `qwen` | `qwen3` | Qwen, Qwen3 | XML `` o `[Calling tool:]` | | `llama` | `llama3`, `llama4` | Llama 3.x, 4.x | Etiquetas `` | | `hermes` | `nous` | Hermes, NousResearch | JSON `` dentro de XML | | `deepseek` | `deepseek_v3`, `deepseek_r1` | DeepSeek V3, R1 | Delimitadores Unicode | | `kimi` | `kimi_k2`, `moonshot` | Kimi K2, Moonshot | Tokens `<\|tool_call_begin\|>` | | `granite` | `granite3` | IBM Granite 3.x, 4.x | `<\|tool_call\|>` o `` | | `nemotron` | `nemotron3` | NVIDIA Nemotron | `` | | `xlam` | | Salesforce xLAM | JSON con arreglo `tool_calls` | | `functionary` | `meetkai` | MeetKai Functionary | Múltiples bloques de función | | `glm47` | `glm4` | GLM-4.7, GLM-4.7-Flash | `` con XML ``/`` | ## Ejemplos por modelo ### Mistral / Devstral ```bash # Devstral Small (optimizado para código y tool use) vllm-mlx serve mlx-community/Devstral-Small-2507-4bit \ --enable-auto-tool-choice --tool-call-parser mistral # Mistral Instruct vllm-mlx serve mlx-community/Mistral-7B-Instruct-v0.3-4bit \ --enable-auto-tool-choice --tool-call-parser mistral ``` ### Qwen ```bash # Qwen3 vllm-mlx serve mlx-community/Qwen3-4B-4bit \ --enable-auto-tool-choice --tool-call-parser qwen ``` ### Llama ```bash # Llama 3.2 vllm-mlx serve mlx-community/Llama-3.2-3B-Instruct-4bit \ --enable-auto-tool-choice --tool-call-parser llama ``` ### DeepSeek ```bash # DeepSeek V3 vllm-mlx serve mlx-community/DeepSeek-V3-0324-4bit \ --enable-auto-tool-choice --tool-call-parser deepseek ``` ### IBM Granite ```bash # Granite 4.0 vllm-mlx serve mlx-community/granite-4.0-tiny-preview-4bit \ --enable-auto-tool-choice --tool-call-parser granite ``` ### NVIDIA Nemotron ```bash # Nemotron 3 Nano vllm-mlx serve mlx-community/NVIDIA-Nemotron-3-Nano-30B-A3B-MLX-6Bit \ --enable-auto-tool-choice --tool-call-parser nemotron ``` ### GLM-4.7 ```bash # GLM-4.7 Flash vllm-mlx serve lmstudio-community/GLM-4.7-Flash-MLX-8bit \ --enable-auto-tool-choice --tool-call-parser glm47 ``` ### Kimi K2 ```bash # Kimi K2 vllm-mlx serve mlx-community/Kimi-K2-Instruct-4bit \ --enable-auto-tool-choice --tool-call-parser kimi ``` ### Salesforce xLAM ```bash # xLAM vllm-mlx serve mlx-community/xLAM-2-fc-r-4bit \ --enable-auto-tool-choice --tool-call-parser xlam ``` ## Parser automático Si no sabes qué parser usar, el parser `auto` intenta detectar el formato de forma automática: ```bash vllm-mlx serve mlx-community/Qwen3-4B-4bit \ --enable-auto-tool-choice --tool-call-parser auto ``` El parser automático prueba los formatos en este orden: 1. Mistral (`[TOOL_CALLS]`) 2. Qwen con corchetes (`[Calling tool:]`) 3. Nemotron (``) 4. XML de Qwen/Hermes (`{...}`) 5. Llama (`{...}`) 6. JSON sin formato ## Streaming de tool calls Los tool calls funcionan con streaming. La información del tool call se envía cuando el modelo termina de generarla: ```python stream = client.chat.completions.create( model="default", messages=[{"role": "user", "content": "What's 25 * 17?"}], tools=[{ "type": "function", "function": { "name": "calculator", "description": "Calculate math expressions", "parameters": { "type": "object", "properties": { "expression": {"type": "string"} }, "required": ["expression"] } } }], stream=True ) for chunk in stream: if chunk.choices[0].delta.tool_calls: for tc in chunk.choices[0].delta.tool_calls: print(f"Tool call: {tc.function.name}({tc.function.arguments})") ``` ## Manejo de resultados de herramientas Después de recibir un tool call, ejecuta la función y devuelve el resultado: ```python import json # Primera solicitud: el modelo decide llamar a una herramienta response = client.chat.completions.create( model="default", messages=[{"role": "user", "content": "What's the weather in Tokyo?"}], tools=[weather_tool] ) # Obtener el tool call tool_call = response.choices[0].message.tool_calls[0] tool_call_id = tool_call.id function_name = tool_call.function.name arguments = json.loads(tool_call.function.arguments) # Ejecutar la función (implementación propia) result = get_weather(**arguments) # {"temperature": 22, "condition": "sunny"} # Enviar el resultado de vuelta al modelo response = client.chat.completions.create( model="default", messages=[ {"role": "user", "content": "What's the weather in Tokyo?"}, {"role": "assistant", "tool_calls": [tool_call]}, {"role": "tool", "tool_call_id": tool_call_id, "content": json.dumps(result)} ], tools=[weather_tool] ) print(response.choices[0].message.content) # "The weather in Tokyo is sunny with a temperature of 22C." ``` ## Manejo de etiquetas de razonamiento Los modelos que producen etiquetas de razonamiento `...` (como DeepSeek-R1, Qwen3, GLM-4.7) se manejan de forma automática. El parser elimina el contenido de reasoning antes de extraer los tool calls, por lo que las etiquetas de razonamiento nunca interfieren con el análisis de tool calls. Esto funciona incluso cuando `` fue inyectado en el prompt (etiquetas implícitas con solo un cierre ``). ## Referencia de CLI | Opción | Descripción | |--------|-------------| | `--enable-auto-tool-choice` | Activa el tool calling automático | | `--tool-call-parser` | Selecciona el parser (ver tabla anterior) | Consulta la [Referencia de CLI](../reference/cli.md) para todas las opciones. # Documentation page: `es/guides/warm-prompts.md` # Warm Prompts Pre-pobla el prefix cache al iniciar el servidor para que la **primera** solicitud que envie un agent encuentre un cache caliente en lugar de pagar el prefill completo de su system prompt de varios kilobytes. ## Cuándo usar esto Las cargas de trabajo de agents, proxies hacia asistentes de código o razonamiento, servidores MCP, orquestadores multi-agent, siempre envian el mismo system prompt. Hoy, la primera solicitud desde un servidor frio paga el prefill completo de ese sistema. En un modelo de miles de millones de parámetros eso equivale a varios segundos de TTFT, justo cuando un usuario espera que su nuevo agent responda por primera vez. Si ya conoces los system prompts de tus agents al momento del despliegue, escríbelos en un archivo JSON y apunta `--warm-prompts` hacia él. El servidor ejecuta un chat completion de `max_tokens=1` para cada uno al inicio, el estado del KV cache queda en el prefix cache, y la primera solicitud real coincide via strict-prefix. Requiere `--continuous-batching` (el prefix cache vive ahí). ## Ejemplo rápido ```bash # Write the agents you care about once cat > ~/.config/vllm-mlx/agents.json <<'JSON' [ [{"role": "system", "content": "You are a code assistant..."}] ] JSON # Point the server at it vllm-mlx serve mlx-community/Qwen3-4B-4bit \ --continuous-batching \ --warm-prompts ~/.config/vllm-mlx/agents.json ``` Al iniciar verás: ``` [lifespan] Warm-up done (strict-prefix): 1 completed, 0 skipped, 1431 prompt tokens in 0.2s ``` La primera solicitud real que comparte el system prompt calentado accede al cache con `tokens_saved` cercano a la longitud del prompt de calentamiento. ## Formato del archivo Una lista JSON de nivel superior. Cada entrada es a su vez una lista de mensajes de chat, con la misma forma que `messages` en `/v1/chat/completions`. ```json [ [ {"role": "system", "content": "You are a code assistant..."} ], [ {"role": "system", "content": "You are a senior code reviewer..."} ], [ {"role": "system", "content": "You are a planner..."}, {"role": "user", "content": "hi"}, {"role": "assistant", "content": "Hello, what are we planning?"} ] ] ``` Los system prompts de un solo mensaje son el caso más común. Los historiales multi-turno son compatibles para escenarios en los que quieres calentar un inicio de conversación específico (ejemplos few-shot, una persona de asistente fija). ## Dimensionamiento Los warm prompts se procesan **de forma concurrente** via `asyncio.gather`, por lo que N entradas lanzan N prefills concurrentes al inicio. Cada prefill asigna KV cache según la longitud de su prompt. **Recomendado: 1 a 3 entradas.** Eso cubre los caminos calientes de despliegues típicos de agents (una persona por entrada). Un archivo warm-prompts muy grande en un modelo con poca memoria puede agotar el espacio disponible en el arranque. Si necesitas calentar decenas de personas, abre un issue con tu carga de trabajo y podemos agregar un limite `--warm-prompts-concurrency=N`. ## Benchmarks **Configuración.** M4 Max, 128 GB de memoria unificada. Dos servidores separados por medición (frio vs caliente), arranque frio aislado. Conjunto de prompts `long` (aprox. 2.5k tokens de usuario) antepuesto con un system prompt de aprox. 1.7k tokens para coincidir con el warm prompt. `max_tokens=128`. bench-serve con `--skip-preflight-token-count` para que el preflight de count_prompt_tokens no contamine el cache. | Model | conc | cold TTFT | warm TTFT | Speedup | |-------|-----:|----------:|----------:|--------:| | Qwen3-0.6B-8bit | 1 | 563 ms | 419 ms | 1.34x | | Qwen3-0.6B-8bit | 4 | 1 723 ms | 1 282 ms | 1.34x | | Qwen3-0.6B-8bit | 8 | 3 708 ms | 2 661 ms | 1.39x | | Llama-3.2-3B-Instruct-4bit | 1 | 1 754 ms | 1 060 ms | 1.65x | | Llama-3.2-3B-Instruct-4bit | 4 | 5 926 ms | 3 945 ms | 1.50x | | Llama-3.2-3B-Instruct-4bit | 8 | 15 161 ms | 9 820 ms | 1.54x | | Qwen3-4B-4bit | 1 | 4 937 ms | 2 191 ms | 2.25x | | Qwen3-4B-4bit | 4 | 12 535 ms | 9 623 ms | 1.30x | | Qwen3-4B-4bit | 8 | 38 148 ms | 23 878 ms | 1.60x | | Qwen3.6-35B-A3B-4bit (MoE/hybrid) | 1 | 2 400 ms | 1 603 ms | 1.50x | | Qwen3.6-35B-A3B-4bit | 4 | 8 735 ms | 6 054 ms | 1.44x | | Qwen3.6-35B-A3B-4bit | 8 | 22 419 ms | 14 409 ms | 1.56x | Las 12 configuraciones mejoran. Los ahorros de TTFT son mayores cuando la relación prompt/total es más alta (conc=1, system prompt largo) y siguen siendo significativos bajo carga concurrente. **Generation tok/s** es neutral (dentro de +-5%) para los modelos densos. Qwen3.6-35B-A3B (MoE) muestra una caida en decode del 20 al 35% con conc >= 4, que parece ser una interacción del enrutamiento MoE con el scheduling en batch. Los ahorros de TTFT siguen dominando la latencia extremo a extremo en cargas de trabajo de agents, pero toma nota de esto si tu flujo es fuertemente decode-bound a alta concurrencia. ## Cómo funciona El calentamiento naive, renderizar la plantilla de chat con un mensaje de usuario de relleno y cachear los tokens, no funciona para modelos híbridos SSM+attention (Qwen3.5-MoE, Qwen3.6-MoE). Sus capas de cache incluyen estado SSM que no puede recortarse, por lo que `memory_cache.py` deshabilita la coincidencia LCP. El contenido de usuario de relleno diverge del contenido real del usuario y una entrada cacheada a nivel de tokens ya no es un strict-prefix de ninguna solicitud real. El calentador aquí renderiza la plantilla de chat **dos veces** con dos contenidos de usuario distintos (`"__PROBE_A__"` y `"__PROBE_B__"`), encuentra la posición de carácter donde las dos cadenas divergen y trunca el primer renderizado en ese limite. Esa cadena truncada, todo lo que precede al punto donde se inserta el contenido del usuario, es lo que se envía al motor. Dado que el flujo de solicitudes reales del motor también renderiza la plantilla con `tokenize=False` y luego deja que el tokenizador codifique el resultado, los tokens del calentamiento tienen garantia de ser un strict-prefix de cualquier solicitud real con un sistema coincidente e historial de chat vacío. Las coincidencias strict-prefix funcionan en todo tipo de capas de cache, incluidos los flujos híbridos donde el LCP está deshabilitado. ## Administración ### Limpiar el prefix cache en memoria ```bash curl -X DELETE http://localhost:8000/v1/cache/prefix ``` Si el servidor se inició con `--warm-prompts`, el calentamiento se vuelve a ejecutar en segundo plano después de la limpieza. La respuesta se devuelve de inmediato sin esperar a que termine el re-calentamiento. Respuesta: ```json {"status": "cleared", "rewarm_scheduled": true} ``` ### Inspeccionar el estado del cache ```bash curl http://localhost:8000/v1/status | jq '.cache' ``` Tras el arranque con warm-prompts verás `entry_count > 0` antes de la primera solicitud del usuario. ## Benchmark de tu propia configuración Para medir el impacto en tu modelo y tus prompts, usa `bench-serve`: ```bash # Cold: no warm-prompts vllm-mlx serve MODEL --continuous-batching & vllm-mlx bench-serve --prompts long --concurrency 1,4 \ --system-prompt-file my-system.txt --tag cold \ --output cold.csv --format csv # Warm: same server config + --warm-prompts vllm-mlx serve MODEL --continuous-batching \ --warm-prompts ~/.config/vllm-mlx/agents.json & vllm-mlx bench-serve --prompts long --concurrency 1,4 \ --system-prompt-file my-system.txt --tag warm \ --output warm.csv --format csv ``` `--skip-preflight-token-count` se habilita automáticamente cuando se usa `--system-prompt-file`, por lo que el preflight de `count_prompt_tokens` no contamina el cache. Compara `cold.csv` y `warm.csv` para tu carga de trabajo. # Documentation page: `es/index.md` # Documentación de vLLM-MLX **Backend MLX para Apple Silicon en vLLM** - Aceleración GPU para texto, imagen, video y audio en Mac ## ¿Qué es vLLM-MLX? vllm-mlx incorpora aceleración GPU nativa de Apple Silicon a vLLM mediante la integración de: - **[MLX](https://github.com/ml-explore/mlx)**: El framework de ML de Apple con memoria unificada y kernels Metal - **[mlx-lm](https://github.com/ml-explore/mlx-lm)**: Inferencia LLM optimizada con KV cache y cuantización - **[mlx-vlm](https://github.com/Blaizzy/mlx-vlm)**: Modelos visión-lenguaje para inferencia multimodal - **[mlx-audio](https://github.com/Blaizzy/mlx-audio)**: TTS y STT con voces nativas - **[mlx-embeddings](https://github.com/Blaizzy/mlx-embeddings)**: Embeddings de texto para búsqueda semántica y RAG ## Características principales - **Multimodal** - Texto, imagen, video y audio en una sola plataforma - **Aceleración GPU nativa** en Apple Silicon (M1, M2, M3, M4, M5) - **Voces TTS nativas** - Español, francés, chino, japonés y 5 idiomas más - **Compatible con la API de OpenAI** - reemplazo directo del cliente de OpenAI - **Embeddings** - Endpoint `/v1/embeddings` compatible con OpenAI - **MCP Tool Calling** - integración de herramientas externas mediante el Model Context Protocol - **Paged KV Cache** - almacenamiento en caché eficiente en memoria con prefix sharing - **Continuous Batching** - alto rendimiento para múltiples usuarios concurrentes ## Enlaces rápidos ### Primeros pasos - [Instalación](getting-started/installation.md) - [Inicio rápido](getting-started/quickstart.md) ### Guías de usuario - [Servidor compatible con OpenAI](guides/server.md) - [API de Python](guides/python-api.md) - [Multimodal (imágenes y video)](guides/multimodal.md) - [Audio (STT/TTS)](guides/audio.md) - [Embeddings](guides/embeddings.md) - [Modelos de reasoning](guides/reasoning.md) - [Tool Calling](guides/tool-calling.md) - [MCP y Tool Calling](guides/mcp-tools.md) - [Continuous Batching](guides/continuous-batching.md) ### Referencia - [Comandos CLI](reference/cli.md) - [Modelos compatibles](reference/models.md) - [Configuración](reference/configuration.md) ### Benchmarks - [Benchmarks LLM](benchmarks/llm.md) - [Benchmarks de imagen](benchmarks/image.md) - [Benchmarks de video](benchmarks/video.md) - [Benchmarks de audio](benchmarks/audio.md) ### Desarrollo - [Arquitectura (en inglés)](/development/architecture/) - [Contribuir (en inglés)](/development/contributing/) ## Requisitos - macOS en Apple Silicon (M1/M2/M3/M4/M5) - Python 3.10+ - Se recomiendan 8 GB de RAM o más ## Licencia Apache 2.0. Consulta la [licencia del repositorio](https://github.com/waybarrios/vllm-mlx/blob/main/LICENSE). # Documentation page: `es/reference/cli.md` # Referencia de CLI ## Resumen de comandos | Comando | Descripcion | |---------|-------------| | `vllm-mlx serve` | Inicia el servidor compatible con OpenAI | | `vllm-mlx-bench` | Ejecuta benchmarks de rendimiento | | `vllm-mlx-chat` | Inicia la interfaz de chat con Gradio | ## `vllm-mlx serve` Inicia el servidor de API compatible con OpenAI. ### Uso ```bash vllm-mlx serve [options] ``` ### Opciones | Opcion | Descripcion | Por defecto | |--------|-------------|-------------| | `--served-model-name` | Nombre personalizado del modelo expuesto a traves de la API de OpenAI. Si no se especifica, se usa la ruta del modelo como nombre. | None | | `--port` | Puerto del servidor | 8000 | | `--host` | Host del servidor | 127.0.0.1 | | `--api-key` | Clave de API para autenticacion | None | | `--rate-limit` | Solicitudes por minuto por cliente (0 = desactivado) | 0 | | `--timeout` | Tiempo limite de solicitud en segundos | 300 | | `--enable-metrics` | Expone métricas de Prometheus en `/metrics` | False | | `--continuous-batching` | Activa continuous batching para multiples usuarios | False | | `--cache-memory-mb` | Limite de memoria para cache en MB | Auto | | `--cache-memory-percent` | Fraccion de RAM para cache | 0.20 | | `--no-memory-aware-cache` | Usa cache legacy basado en conteo de entradas | False | | `--use-paged-cache` | Activa el KV cache paginado | False | | `--max-tokens` | Maximo de tokens por defecto | 32768 | | `--max-request-tokens` | Maximo de `max_tokens` aceptado desde clientes de la API | 32768 | | `--stream-interval` | Tokens por fragmento de streaming | 1 | | `--mcp-config` | Ruta al archivo de configuración de MCP | None | | `--paged-cache-block-size` | Tokens por bloque de cache | 64 | | `--max-cache-blocks` | Maximos bloques de cache | 1000 | | `--max-num-seqs` | Maximo de secuencias concurrentes | 256 | | `--default-temperature` | Temperatura por defecto cuando no se especifica en la solicitud | None | | `--default-top-p` | top_p por defecto cuando no se especifica en la solicitud | None | | `--max-audio-upload-mb` | Tamano máximo de audio subido para `/v1/audio/transcriptions` | 25 | | `--max-tts-input-chars` | Longitud máxima de texto aceptada por `/v1/audio/speech` | 4096 | | `--reasoning-parser` | Parser para modelos de reasoning (`qwen3`, `deepseek_r1`) | None | | `--embedding-model` | Pre-carga un modelo de embeddings al iniciar | None | | `--enable-auto-tool-choice` | Activa tool calling automático | False | | `--tool-call-parser` | Parser de tool calling (`auto`, `mistral`, `qwen`, `llama`, `hermes`, `deepseek`, `kimi`, `granite`, `nemotron`, `xlam`, `functionary`, `glm47`) | None | ### Ejemplos ```bash # Modo simple (usuario único, máximo rendimiento) # La ruta del modelo se usa como nombre en la API de OpenAI (ej. model="mlx-community/Llama-3.2-3B-Instruct-4bit") vllm-mlx serve mlx-community/Llama-3.2-3B-Instruct-4bit Model will show up as 'mlx-community/Llama-3.2-3B-Instruct-4bit' in the `/v1/models` API endpoint. View with `curl http://localhost:8000/v1/models` or similar. # Con un nombre de modelo personalizado en la API (el modelo se accede como "my-model" via la API de OpenAI) # --served-model-name establece el nombre que los clientes deben usar al llamar a la API (ej. model="my-model") vllm-mlx serve --served-model-name my-model mlx-community/Llama-3.2-3B-Instruct-4bit # Note: Model will show up as 'my-model' in the `/v1/models` API endpoint. # Continuous batching (multiples usuarios) vllm-mlx serve mlx-community/Llama-3.2-3B-Instruct-4bit --continuous-batching # Con limite de memoria para modelos grandes vllm-mlx serve mlx-community/GLM-4.7-Flash-4bit \ --continuous-batching \ --cache-memory-mb 2048 # Produccion con paged cache vllm-mlx serve mlx-community/Qwen3-0.6B-8bit \ --continuous-batching \ --use-paged-cache \ --port 8000 # Con herramientas MCP vllm-mlx serve mlx-community/Qwen3-4B-4bit --mcp-config mcp.json # Modelo multimodal vllm-mlx serve mlx-community/Qwen3-VL-4B-Instruct-3bit # Modelo de reasoning (separa el pensamiento de la respuesta) vllm-mlx serve mlx-community/Qwen3-8B-4bit --reasoning-parser qwen3 # Modelo de reasoning DeepSeek vllm-mlx serve mlx-community/DeepSeek-R1-Distill-Qwen-7B-4bit --reasoning-parser deepseek_r1 # Tool calling con Mistral/Devstral vllm-mlx serve mlx-community/Devstral-Small-2507-4bit \ --enable-auto-tool-choice --tool-call-parser mistral # Tool calling con Granite vllm-mlx serve mlx-community/granite-4.0-tiny-preview-4bit \ --enable-auto-tool-choice --tool-call-parser granite # Con autenticacion por clave de API vllm-mlx serve mlx-community/Llama-3.2-3B-Instruct-4bit --api-key your-secret-key # Exponer métricas de Prometheus vllm-mlx serve mlx-community/Llama-3.2-3B-Instruct-4bit --enable-metrics # Configuracion de produccion con opciones de seguridad vllm-mlx serve mlx-community/Qwen3-4B-4bit \ --api-key your-secret-key \ --rate-limit 60 \ --timeout 120 \ --continuous-batching ``` ### Seguridad Cuando se establece `--api-key`, todas las solicitudes a la API requieren el encabezado `Authorization: Bearer `: ```python from openai import OpenAI client = OpenAI( base_url="http://localhost:8000/v1", api_key="your-secret-key" # Must match --api-key ) ``` O con curl: ```bash curl http://localhost:8000/v1/models \ -H "Authorization: Bearer your-secret-key" ``` ## `vllm-mlx-bench` Ejecuta benchmarks de rendimiento. ### Uso ```bash vllm-mlx-bench --model [options] ``` ### Opciones | Opcion | Descripcion | Por defecto | |--------|-------------|-------------| | `--model` | Nombre del modelo | Requerido | | `--prompts` | Numero de prompts | 5 | | `--max-tokens` | Maximo de tokens por prompt | 256 | | `--quick` | Modo de benchmark rápido | False | | `--video` | Ejecutar benchmark de video | False | | `--video-url` | URL de video personalizada | None | | `--video-path` | Ruta de video personalizada | None | ### Ejemplos ```bash # Benchmark de LLM vllm-mlx-bench --model mlx-community/Llama-3.2-1B-Instruct-4bit # Benchmark rápido vllm-mlx-bench --model mlx-community/Llama-3.2-1B-Instruct-4bit --quick # Benchmark de imagenes (deteccion automática para modelos VLM) vllm-mlx-bench --model mlx-community/Qwen3-VL-8B-Instruct-4bit # Benchmark de video vllm-mlx-bench --model mlx-community/Qwen3-VL-8B-Instruct-4bit --video # Video personalizado vllm-mlx-bench --model mlx-community/Qwen3-VL-8B-Instruct-4bit \ --video --video-url https://example.com/video.mp4 ``` ## `vllm-mlx-chat` Inicia la interfaz de chat con Gradio. ### Uso ```bash vllm-mlx-chat --served-model-name [options] ``` ### Opciones | Opcion | Descripcion | Por defecto | |--------|-------------|-------------| | `--model` | Nombre del modelo | Requerido | | `--port` | Puerto de Gradio | 7860 | | `--text-only` | Desactiva el modo multimodal | False | ### Ejemplos ```bash # Chat multimodal (texto + imagenes + video) vllm-mlx-chat --served-model-name mlx-community/Qwen3-VL-4B-Instruct-3bit # Chat solo de texto vllm-mlx-chat --served-model-name mlx-community/Llama-3.2-3B-Instruct-4bit --text-only ``` ## Variables de entorno | Variable | Descripcion | |----------|-------------| | `VLLM_MLX_TEST_MODEL` | Modelo para pruebas | | `HF_TOKEN` | Token de HuggingFace | # Documentation page: `es/reference/configuration.md` # Referencia de configuración ## Configuracion del servidor ### Opciones basicas | Opcion | Descripcion | Valor por defecto | |--------|-------------|---------| | `--host` | Direccion del host del servidor | `127.0.0.1` | | `--port` | Puerto del servidor | `8000` | | `--max-tokens` | Maximo de tokens por defecto | `32768` | | `--max-request-tokens` | Maximo de `max_tokens` aceptado de clientes de la API | `32768` | | `--default-temperature` | Temperatura por defecto cuando no se especifica en la solicitud | None | | `--default-top-p` | top_p por defecto cuando no se especifica en la solicitud | None | ### Opciones de seguridad | Opcion | Descripcion | Valor por defecto | |--------|-------------|---------| | `--api-key` | Clave de API para autenticacion | None | | `--rate-limit` | Solicitudes por minuto por cliente (0 = deshabilitado) | `0` | | `--timeout` | Tiempo de espera de la solicitud en segundos | `300` | | `--enable-metrics` | Expone métricas de Prometheus en `/metrics` | `false` | | `--max-audio-upload-mb` | Tamano máximo de audio subido para `/v1/audio/transcriptions` | `25` | | `--max-tts-input-chars` | Longitud máxima de texto aceptada por `/v1/audio/speech` | `4096` | ### Opciones de batching | Opcion | Descripcion | Valor por defecto | |--------|-------------|---------| | `--continuous-batching` | Habilita el continuous batching | `false` | | `--stream-interval` | Tokens por fragmento de streaming | `1` | | `--max-num-seqs` | Maximo de secuencias concurrentes | `256` | ### Opciones de cache | Opcion | Descripcion | Valor por defecto | |--------|-------------|---------| | `--cache-memory-mb` | Limite de memoria de cache en MB | Auto | | `--cache-memory-percent` | Fraccion de RAM para cache | `0.20` | | `--no-memory-aware-cache` | Usa cache de conteo de entradas heredado | `false` | | `--use-paged-cache` | Habilita el KV cache paginado | `false` | | `--paged-cache-block-size` | Tokens por bloque | `64` | | `--max-cache-blocks` | Maximo de bloques | `1000` | ### Opciones de llamado a herramientas | Opcion | Descripcion | Valor por defecto | |--------|-------------|---------| | `--enable-auto-tool-choice` | Habilita el llamado automático a herramientas | `false` | | `--tool-call-parser` | Parser de llamados a herramientas (ver [Tool Calling](../guides/tool-calling.md)) | None | ### Opciones de razonamiento | Opcion | Descripcion | Valor por defecto | |--------|-------------|---------| | `--reasoning-parser` | Parser para modelos de razonamiento (`qwen3`, `deepseek_r1`) | None | ### Opciones de embeddings | Opcion | Descripcion | Valor por defecto | |--------|-------------|---------| | `--embedding-model` | Precarga un modelo de embeddings al iniciar | None | ### Opciones de MCP | Opcion | Descripcion | Valor por defecto | |--------|-------------|---------| | `--mcp-config` | Ruta al archivo de configuración MCP | None | ## Configuracion de MCP Crear `mcp.json`: ```json { "mcpServers": { "server-name": { "command": "npx", "args": ["-y", "@modelcontextprotocol/server-name", "arg1"], "env": { "ENV_VAR": "value" } } } } ``` ### Opciones del servidor MCP | Campo | Descripcion | Requerido | |-------|-------------|----------| | `command` | Comando ejecutable | Si | | `args` | Argumentos del comando | Si | | `env` | Variables de entorno | No | ## Opciones de solicitudes a la API ### Chat Completions | Parametro | Descripcion | Valor por defecto | |-----------|-------------|---------| | `model` | Nombre del modelo | Requerido | | `messages` | Mensajes del chat | Requerido | | `max_tokens` | Maximo de tokens a generar | 256 | | `temperature` | Temperatura de muestreo | Valor por defecto del modelo | | `top_p` | Nucleus sampling | Valor por defecto del modelo | | `stream` | Habilita el streaming | `true` | | `stop` | Secuencias de detencion | None | | `tools` | Definiciones de herramientas | None | | `response_format` | Formato de salida (`json_object`, `json_schema`) | None | ### Opciones multimodales | Parametro | Descripcion | Valor por defecto | |-----------|-------------|---------| | `video_fps` | Fotogramas por segundo | 2.0 | | `video_max_frames` | Maximo de fotogramas | 32 | ## Variables de entorno | Variable | Descripcion | |----------|-------------| | `VLLM_MLX_TEST_MODEL` | Modelo por defecto para pruebas | | `HF_TOKEN` | Token de autenticacion de HuggingFace | | `OPENAI_API_KEY` | Establecer a cualquier valor para compatibilidad con el SDK | ## Configuraciones de ejemplo ### Desarrollo (usuario único) ```bash vllm-mlx serve mlx-community/Llama-3.2-3B-Instruct-4bit ``` ### Produccion (multiples usuarios) ```bash vllm-mlx serve mlx-community/Qwen3-0.6B-8bit \ --continuous-batching \ --use-paged-cache \ --api-key your-secret-key \ --rate-limit 60 \ --port 8000 ``` ### Con llamado a herramientas ```bash vllm-mlx serve mlx-community/Devstral-Small-2507-4bit \ --enable-auto-tool-choice \ --tool-call-parser mistral \ --continuous-batching ``` ### Con herramientas MCP ```bash vllm-mlx serve mlx-community/Qwen3-4B-4bit \ --mcp-config mcp.json \ --enable-auto-tool-choice \ --tool-call-parser qwen \ --continuous-batching ``` ### Modelo de razonamiento ```bash vllm-mlx serve mlx-community/Qwen3-8B-4bit \ --reasoning-parser qwen3 \ --continuous-batching ``` ### Con embeddings ```bash vllm-mlx serve mlx-community/Qwen3-4B-4bit \ --embedding-model mlx-community/multilingual-e5-small-mlx \ --continuous-batching ``` ### Alto rendimiento ```bash vllm-mlx serve mlx-community/Qwen3-0.6B-8bit \ --continuous-batching \ --stream-interval 5 \ --max-num-seqs 256 ``` # Documentation page: `es/reference/models.md` # Modelos compatibles Todos los modelos cuantizados de [mlx-community en HuggingFace](https://huggingface.co/mlx-community/models) son compatibles. Explora miles de modelos preoptimizados en: **https://huggingface.co/mlx-community/models** ## Modelos de lenguaje (vía mlx-lm) | Familia de modelos | Tamaños | Cuantización | |--------------------|---------|--------------| | Llama 3.x, 4.x | 1B, 3B, 8B, 70B | 4-bit | | Mistral / Devstral | 7B, Mixtral 8x7B | 4-bit, 8-bit | | Qwen2/Qwen3 | 0.5B a 72B | Varios | | DeepSeek V3, R1 | 7B, 33B, 67B | 4-bit | | Gemma 2, 3, 4 | 2B, 9B, 27B | 4-bit | | GLM-4.7 | Flash, Base | 4-bit, 8-bit | | Kimi K2 | Varios | 4-bit | | Phi-3 | 3.8B, 14B | 4-bit | | Granite 3.x, 4.x | Varios | 4-bit | | Nemotron | 3 Nano 30B | 6-bit | ### Modelos recomendados | Caso de uso | Modelo | Memoria | |-------------|--------|---------| | Rápido / Liviano | `mlx-community/Qwen3-0.6B-8bit` | ~0.7 GB | | Equilibrado | `mlx-community/Llama-3.2-3B-Instruct-4bit` | ~1.8 GB | | Calidad | `mlx-community/Llama-3.1-8B-Instruct-4bit` | ~4.5 GB | | Grande | `mlx-community/Qwen3-30B-A3B-4bit` | ~16 GB | ## Modelos multimodales (vía mlx-vlm) | Familia de modelos | Modelos de ejemplo | |--------------------|--------------------| | **Qwen-VL** | `Qwen3-VL-4B-Instruct-3bit`, `Qwen3-VL-8B-Instruct-4bit`, `Qwen2-VL-2B/7B-Instruct-4bit` | | **LLaVA** | `llava-1.5-7b-4bit`, `llava-v1.6-mistral-7b-4bit`, `llava-llama-3-8b-v1_1-4bit` | | **Idefics** | `Idefics3-8B-Llama3-4bit`, `idefics2-8b-4bit` | | **Gemma 4** | `gemma-4-e2b-it-mxfp4` (visión + audio) | | **PaliGemma** | `paligemma2-3b-mix-224-4bit`, `paligemma-3b-mix-224-8bit` | | **Pixtral** | `pixtral-12b-4bit`, `pixtral-12b-8bit` | | **Molmo** | `Molmo-7B-D-0924-4bit`, `Molmo-7B-D-0924-8bit` | | **Phi-3 Vision** | `Phi-3-vision-128k-instruct-4bit` | | **DeepSeek-VL** | `deepseek-vl-7b-chat-4bit`, `deepseek-vl2-small-4bit` | ### Modelos VLM recomendados | Caso de uso | Modelo | Memoria | |-------------|--------|---------| | Rápido / Liviano | `mlx-community/Qwen3-VL-4B-Instruct-3bit` | ~3 GB | | Equilibrado | `mlx-community/Qwen3-VL-8B-Instruct-4bit` | ~6 GB | | Calidad | `mlx-community/Qwen3-VL-30B-A3B-Instruct-6bit` | ~20 GB | ## Modelos de embeddings (vía mlx-embeddings) | Familia de modelos | Modelos de ejemplo | |--------------------|--------------------| | **BERT** | `mlx-community/bert-base-uncased-mlx` | | **XLM-RoBERTa** | `mlx-community/multilingual-e5-small-mlx`, `mlx-community/multilingual-e5-large-mlx` | | **ModernBERT** | `mlx-community/ModernBERT-base-mlx` | ## Modelos de audio (vía mlx-audio) | Tipo | Familia de modelos | Modelos de ejemplo | |------|--------------------|--------------------| | **STT** | Whisper | `mlx-community/whisper-large-v3-turbo` | | **STT** | Parakeet | `mlx-community/parakeet-tdt-0.6b-v2` | | **TTS** | Kokoro | `prince-canuma/Kokoro-82M` | | **TTS** | Chatterbox | `chatterbox/chatterbox-tts-0.1` | ## Detección de modelos vllm-mlx detecta automáticamente los modelos multimodales por patrones en el nombre: - Contiene "VL", "Vision", "vision" - Contiene "llava", "idefics", "paligemma" - Contiene "pixtral", "molmo", "deepseek-vl" - Contiene "MedGemma", "Gemma-3", "Gemma-4" (variantes multimodales) ## Usar modelos ### Desde HuggingFace ```bash vllm-mlx serve mlx-community/Llama-3.2-3B-Instruct-4bit ``` ### Ruta local ```bash vllm-mlx serve /path/to/local/model ``` ## Buscar modelos Filtra los modelos de mlx-community por: - **LLM**: `Llama`, `Qwen`, `Mistral`, `Phi`, `Gemma`, `DeepSeek`, `GLM`, `Kimi`, `Granite`, `Nemotron` - **VLM**: `-VL-`, `llava`, `paligemma`, `pixtral`, `molmo`, `idefics`, `deepseek-vl`, `MedGemma` - **Embedding**: `e5`, `bert`, `ModernBERT` - **Tamaño**: `1B`, `3B`, `7B`, `8B`, `70B` - **Cuantización**: `4bit`, `8bit`, `bf16` # Documentation page: `fr/benchmarks/README.md` # Benchmarks Benchmarks de performance pour vllm-mlx sur Apple Silicon. ## Types de benchmarks - [Benchmarks LLM](llm.md) - Performance de génération de texte - [Benchmarks image](image.md) - Performance de compréhension d'images - [Benchmarks vidéo](video.md) - Performance de compréhension de vidéos ## Commandes rapides ```bash # LLM benchmark vllm-mlx-bench --model mlx-community/Qwen3-0.6B-8bit # Image benchmark vllm-mlx-bench --model mlx-community/Qwen3-VL-8B-Instruct-4bit # Video benchmark vllm-mlx-bench --model mlx-community/Qwen3-VL-8B-Instruct-4bit --video ``` ## Valeurs par défaut des scripts de test autonomes Les scripts de benchmark autonomes disposent de modèles par défaut intégrés, ce qui permet de les lancer directement : ```bash python tests/test_continuous_batching.py python tests/test_prefix_cache.py ``` Valeurs par défaut : - `tests/test_continuous_batching.py` → `mlx-community/Qwen3-8B-6bit` - `tests/test_prefix_cache.py` → `mlx-community/Qwen3-0.6B-8bit` Pour tester d'autres modèles, utilisez l'option `--model` : ```bash python tests/test_continuous_batching.py --model mlx-community/Qwen3-0.6B-8bit python tests/test_prefix_cache.py --model mlx-community/Qwen3-8B-6bit ``` ## Matériel Les benchmarks ont été collectés sur les configurations Apple Silicon suivantes : | Puce | Mémoire | Python | |------|---------|--------| | Apple M4 Max | 128 Go unifiée | 3.13 | | Apple M1 Max | 64 Go unifiée | 3.12 | Les résultats varieront selon la puce Apple Silicon utilisée. ## Contribuer des benchmarks Si vous disposez d'une puce Apple Silicon différente, partagez vos résultats : ```bash vllm-mlx-bench --model mlx-community/Qwen3-0.6B-8bit --output results.json ``` Ouvrez un ticket avec vos résultats sur [GitHub Issues](https://github.com/waybarrios/vllm-mlx/issues). # Documentation page: `fr/benchmarks/audio.md` # Benchmarks Audio ## Benchmarks STT (Speech-to-Text) ### Lancer les benchmarks STT ```bash # Run with default test audio python examples/benchmark_audio.py --stt # Run with your own audio file python examples/benchmark_audio.py --stt --audio path/to/audio.wav ``` ### Résultats (M4 Max, 128 Go) **Audio de test :** 46,7 secondes de synthèse vocale | Model | Parameters | Load Time | Transcribe Time | RTF* | |-------|------------|-----------|-----------------|------| | whisper-tiny | 39M | 0.34s | 0.24s | **197x** | | whisper-small | 244M | 0.18s | 0.47s | **98x** | | whisper-medium | 769M | 0.35s | 1.15s | **41x** | | whisper-large-v3 | 1.5B | 0.50s | 1.96s | **24x** | | whisper-large-v3-turbo | 809M | 0.12s | 0.86s | **55x** | *RTF = Real-Time Factor (plus la valeur est élevée, plus c'est rapide). Un RTF de 100x signifie qu'une minute d'audio est transcrite en environ 0,6 secondes.* ### Résultats (M1 Max, 64 Go) STT avec Parakeet (environnement par défaut, Whisper indisponible en raison d'une incompatibilité de dépendance numpy) : | Model | Load Time | Transcribe Time | RTF | |-------|-----------|-----------------|-----| | parakeet-tdt-0.6b-v2 | 0.28s | 1.01s | **9.9x** | | parakeet-tdt-0.6b-v3 | 0.30s | 0.19s | **52.7x** | STT avec Whisper (`numpy==2.3.5` explicite + `uv run --no-sync`) : | Model | Load Time | Transcribe Time | RTF | |-------|-----------|-----------------|-----| | whisper-tiny | 4.02s | 1.05s | **9.5x** | | whisper-small | 10.15s | 1.03s | **9.7x** | | whisper-medium | 22.96s | 2.20s | **4.6x** | | whisper-large-v3 | 38.34s | 0.96s | **10.5x** | | whisper-large-v3-turbo | 21.79s | 0.70s | **14.3x** | | parakeet-tdt-0.6b-v2 | 0.47s | 0.18s | **54.4x** | | parakeet-tdt-0.6b-v3 | 1.13s | 0.18s | **54.6x** | ### Recommandations de modèles | Use Case | Recommended Model | Why | |----------|-------------------|-----| | **Transcription en temps réel** | whisper-tiny | Le plus rapide (RTF 197x), faible latence | | **Usage général** | whisper-large-v3-turbo | Meilleur compromis vitesse (55x) et qualité | | **Précision maximale** | whisper-large-v3 | Le plus précis, prend en charge plus de 99 langues | | **Mémoire limitée** | whisper-small | Bonne qualité à 244M paramètres | ### Qualité de transcription Tous les modèles ont correctement transcrit l'audio de test. Exemple de sortie : ``` Input text: "Welcome to this comprehensive speech to text demonstration. This audio sample is designed to test the accuracy and speed of various speech recognition models. The quick brown fox jumps over the lazy dog..." Whisper-large-v3 output: "Welcome to this comprehensive speech to text demonstration. This audio sample is designed to test the accuracy and speed of various speech recognition models. The quick brown fox jumps over the lazy dog..." (identical) ``` ### Langues prises en charge Les modèles Whisper prennent en charge plus de 99 langues, notamment : - Anglais, espagnol, français, allemand, italien, portugais - Chinois (mandarin, cantonais), japonais, coréen - Arabe, hindi, russe, turc, ukrainien - Et bien d'autres ## Benchmarks TTS (Text-to-Speech) ### Lancer les benchmarks TTS ```bash python examples/benchmark_audio.py --tts ``` ### Résultats (M4 Max, 128 Go) **Test :** Génération audio pour 3 échantillons de texte (court, moyen, long) | Model | Load Time | Chars/sec | RTF* | |-------|-----------|-----------|------| | Kokoro-82M-bf16 | 0.8s | 350+ | **22x** | | Kokoro-82M-4bit | 0.4s | 320+ | **20x** | *RTF = Real-Time Factor. Un RTF de 22x signifie qu'une seconde d'audio est générée en environ 0,045 secondes.* ### Résultats TTS (M1 Max, 64 Go) | Model | Load Time | Avg Chars/s | Avg RTF | |-------|-----------|-------------|---------| | Kokoro-82M-bf16 | 2.81s | 176.0 | **11.9x** | | Kokoro-82M-4bit | 0.22s | 225.6 | **15.5x** | ### Qualité TTS Kokoro produit une synthèse vocale au son naturel avec : - 11 voix intégrées (masculines et féminines) - Prise en charge de 8 langues (anglais, espagnol, français, japonais, chinois, italien, portugais, hindi) - 82M paramètres, rapide et léger ## Benchmarks de traitement audio ### SAM-Audio (séparation de sources) **Test :** Séparation de la batterie dans un morceau de rock de 30 secondes | Metric | Value | |--------|-------| | Model | sam-audio-large-fp16 | | Processing time | ~20s | | Peak memory | ~27 GB | | Output sample rate | 48000 Hz | ## Lancer tous les benchmarks audio ```bash # Run all benchmarks python examples/benchmark_audio.py --all # Or run individually python examples/benchmark_audio.py --stt python examples/benchmark_audio.py --tts ``` ## Modèles disponibles sur mlx-community ### Modèles STT - `mlx-community/whisper-tiny-mlx` - `mlx-community/whisper-small-mlx` - `mlx-community/whisper-medium-mlx` - `mlx-community/whisper-large-v3-mlx` - `mlx-community/whisper-large-v3-turbo` - `mlx-community/parakeet-tdt-0.6b-v2` - `mlx-community/parakeet-tdt-0.6b-v3` ### Modèles TTS - `mlx-community/Kokoro-82M-bf16` (recommandé) - `mlx-community/Kokoro-82M-4bit` - `mlx-community/chatterbox-turbo-fp16` - `mlx-community/VibeVoice-Realtime-0.5B-4bit` ### Traitement audio - `mlx-community/sam-audio-large-fp16` # Documentation page: `fr/benchmarks/image.md` # Benchmarks d'images ## Lancer les benchmarks d'images ```bash # Benchmark complet (10 résolutions) vllm-mlx-bench --model mlx-community/Qwen3-VL-8B-Instruct-4bit # Benchmark rapide (4 résolutions) vllm-mlx-bench --model mlx-community/Qwen3-VL-8B-Instruct-4bit --quick ``` ## Résultats - Qwen3-VL-8B-Instruct-4bit (M4 Max, 128GB) | Resolution | Pixels | Time | Tokens | Speed | |------------|--------|------|--------|-------| | 224x224 | 50K | 1.04s | 78 | 74.8 tok/s | | 336x336 | 113K | 0.94s | 64 | 68.3 tok/s | | 448x448 | 201K | 1.45s | 70 | 48.1 tok/s | | 512x512 | 262K | 1.58s | 99 | 62.8 tok/s | | 672x672 | 452K | 1.83s | 83 | 45.3 tok/s | | 768x768 | 590K | 2.05s | 91 | 44.3 tok/s | | 896x896 | 803K | 2.61s | 90 | 34.5 tok/s | | 1024x1024 | 1.0M | 2.79s | 76 | 27.2 tok/s | | 1280x720 | 922K | 2.97s | 96 | 32.4 tok/s | | 1920x1080 | 2.1M | 6.30s | 89 | 14.1 tok/s | **Résumé :** Moyenne de 45.2 tok/s sur toutes les résolutions. Le plus rapide à 224x224 (74.8 tok/s), le plus lent à 1920x1080 (14.1 tok/s) ## Résultats - Qwen3-VL-8B-Instruct-4bit (M1 Max, 64GB) Benchmark MLLM local : | Resolution | Pixels | Time | Tokens | Speed | |------------|--------|------|--------|-------| | 224x224 | 50K | 1.84s | 78 | 42.5 tok/s | | 448x448 | 201K | 2.28s | 70 | 30.7 tok/s | | 768x768 | 590K | 4.39s | 91 | 20.7 tok/s | | 1024x1024 | 1.0M | 6.41s | 76 | 11.9 tok/s | | Configs | Total Time (s) | Total Tokens | Aggregate Tok/s | |---------|-----------------|--------------|-----------------| | 4 | 14.92 | 315 | 21.1 | ## Résultats - Qwen3-VL-4B-Instruct-3bit Serveur (M1 Max, 64GB) | Resolution | Pixels | Time | Tokens | Speed | |------------|--------|------|--------|-------| | 224x224 | 50K | 1.65s | 113 | 68.4 tok/s | | 448x448 | 201K | 2.09s | 120 | 57.5 tok/s | | 768x768 | 590K | 2.93s | 106 | 36.2 tok/s | | 1024x1024 | 1.0M | 4.12s | 100 | 24.3 tok/s | | Configs | Total Time (s) | Total Tokens | Aggregate Tok/s | |---------|-----------------|--------------|-----------------| | 4 | 10.79 | 439 | 40.7 | ## Résultats du cache de préfixe MLLM ``` ====================================================================== MLLM PREFIX CACHE TEST ====================================================================== Model: mlx-community/Qwen3-VL-4B-Instruct-3bit Test: Verify KV cache reuse for repeated image/video + prompt combinations Expected behavior: - Same image + same prompt → cache HIT - Same image + different prompt → cache MISS - Different image + same prompt → cache MISS ---------------------------------------------------------------------- SETUP: Loading Model ---------------------------------------------------------------------- Model loaded in 0.11s ---------------------------------------------------------------------- SETUP: Creating Test Images ---------------------------------------------------------------------- Resized: 224x224, 336x336, 512x512, 768x768 ---------------------------------------------------------------------- TEST 1: Image Cache - Basic Hit/Miss ---------------------------------------------------------------------- Results: Step | Description | Expected | Actual | Time | Status -------+---------------------------+----------+--------+--------+------- 1a | First image+prompt | MISS | MISS | 0.10ms | ✓ 1b | Same image+prompt | HIT | HIT | 0.18ms | ✓ 1c | Different prompt | MISS | MISS | 0.01ms | ✓ 1d | Return to original | HIT | HIT | 0.18ms | ✓ ---------------------------------------------------------------------- TEST 2: Different Images ---------------------------------------------------------------------- Results: Step | Description | Expected | Actual | Time | Status -------+---------------------------+----------+--------+--------+------- 2a | Image A first request | MISS | MISS | 0.01ms | ✓ 2b | Image B first request | MISS | MISS | 0.01ms | ✓ 2c | Image A cached | HIT | HIT | 0.13ms | ✓ ---------------------------------------------------------------------- TEST 3: Image Resolutions ---------------------------------------------------------------------- Results: Step | Description | Expected | Actual | Time | Status -------+-----------------------+----------+--------+--------+------- 3.1a | 224x224 first | MISS | MISS | 0.01ms | ✓ 3.1b | 224x224 cached | HIT | HIT | 0.20ms | ✓ 3.2a | 336x336 first | MISS | MISS | 0.01ms | ✓ 3.2b | 336x336 cached | HIT | HIT | 0.21ms | ✓ 3.3a | 512x512 first | MISS | MISS | 0.12ms | ✓ 3.3b | 512x512 cached | HIT | HIT | 0.20ms | ✓ 3.4a | 768x768 first | MISS | MISS | 0.12ms | ✓ 3.4b | 768x768 cached | HIT | HIT | 0.24ms | ✓ ====================================================================== ``` ## Stratégie de clé de cache - **Images :** `hash(image_content) + hash(prompt)` Une même image avec le même prompt touchera toujours le cache. Une image différente ou un prompt différent provoquera un cache miss. ## Conseils de performance - Les résolutions plus petites sont traitées plus rapidement (224x224 contre 1920x1080) - Utilisez la résolution adaptée à votre tâche - Regroupez les images de taille similaire pour un débit constant ## Référence des métriques | Metric | Description | |--------|-------------| | Resolution | Dimensions de l'image (largeur x hauteur) | | Pixels | Nombre total de pixels | | Time | Durée de génération | | Tokens | Tokens de sortie générés | | Speed | Tokens par seconde (tok/s) | # Documentation page: `fr/benchmarks/llm.md` # Benchmarks LLM ## Lancer les benchmarks LLM ```bash vllm-mlx-bench --model mlx-community/Llama-3.2-1B-Instruct-4bit --prompts 5 --max-tokens 256 ``` ## Résultats (M4 Max, 128 Go) | Modèle | Vitesse de génération | TTFT* | Mémoire | |--------|-----------------------|-------|---------| | Qwen3-0.6B-8bit | 402,3 tok/s | 58,6 ms | 0,68 Go | | Llama-3.2-1B-Instruct-4bit | 463,6 tok/s | 49,2 ms | 0,69 Go | | Qwen2.5-1.5B-Instruct-4bit | 308,5 tok/s | 86,2 ms | 0,84 Go | | Llama-3.2-3B-Instruct-4bit | 200,1 tok/s | 81,4 ms | 1,79 Go | | Qwen3-30B-A3B-4bit | 123,9 tok/s | 126,9 ms | 16,05 Go | | NVIDIA-Nemotron-3-Nano-30B-A3B-MLX-6Bit | 122,9 tok/s | 72,3 ms | 23,98 Go | *TTFT = Time to First Token (latence jusqu'au premier token généré) ## Résultats (M1 Max, 64 Go) | Modèle | Requêtes | Tok. prompt | Tok. générés | Temps total (s) | TTFT moyen (ms) | TPOT moyen (ms) | Vitesse génération (tok/s) | Débit total (tok/s) | |--------|----------|-------------|--------------|-----------------|-----------------|-----------------|---------------------------|---------------------| | Qwen3-0.6B-8bit | 5 | 56 | 1280 | 5,66 | 119,0 | 3,97 | 251,9 | 236,1 | ## Résultats du continuous batching | Modèle | Requête unique | Batch (5 req) | Accélération | |--------|----------------|---------------|--------------| | Llama-3.2-1B-Instruct-4bit | 299,1 tok/s | 613,0 tok/s | **2,05x** | | Llama-3.2-3B-Instruct-4bit | 137,6 tok/s | 208,1 tok/s | **1,51x** | | Qwen3-0.6B-8bit | 328,1 tok/s | 1111,8 tok/s | **3,39x** | | Qwen3-30B-A3B-4bit | 98,1 tok/s | 233,3 tok/s | **2,38x** | | Qwen2.5-1.5B-Instruct-4bit | 196,9 tok/s | 322,2 tok/s | **1,64x** | *Le batching de 5 requêtes simultanées apporte une amélioration du throughput de 1,5 à 3x.* ### Continuous batching (M1 Max, 64 Go) | Requêtes | Tokens totaux | Temps total (s) | Throughput (tok/s) | Requêtes/sec | |----------|---------------|-----------------|--------------------|--------------| | 5 | 315 | 0,64 | 492,5 | 7,82 | ## Performances en streaming | Modèle | TTFT | Vitesse de génération | |--------|------|-----------------------| | Llama-3.2-1B-Instruct-4bit | ~4,6 ms | 218,9 tok/s | | Llama-3.2-3B-Instruct-4bit | ~10,7 ms | 93,6 tok/s | | Qwen3-0.6B-8bit | ~3,0 ms | 328,5 tok/s | | Qwen3-30B-A3B-4bit | ~10,2 ms | 98,4 tok/s | | Qwen2.5-1.5B-Instruct-4bit | ~7,1 ms | 140,3 tok/s | ### Détokeniseur en streaming (M1 Max, 64 Go) `vllm-mlx bench-detok` : | Tokens | Itérations | Temps naïf | Temps streaming | Accélération | |--------|------------|------------|-----------------|--------------| | 742 | 5 | 1,69 ms | 0,71 ms | 2,39x | `examples/benchmark_detokenizer.py` : | Séquence | Tokens | decode() | Streaming | Accélération | |----------|--------|----------|-----------|--------------| | Courte | 8 | 0,029 ms | 0,028 ms | 1,04x | | Moyenne | 103 | 0,206 ms | 0,129 ms | 1,59x | | Longue | 511 | 1,040 ms | 0,502 ms | 2,07x | | 1K | 1191 | 2,446 ms | 1,178 ms | 2,08x | | 2K | 2381 | 4,949 ms | 2,356 ms | 2,10x | | 4K | 4761 | 9,887 ms | 5,398 ms | 1,83x | Accélération moyenne : 1,79x ## Résultats du prefix cache ### Prefix cache (M4 Max, 128 Go) ``` ====================================================================== LLM PREFIX CACHE TEST ====================================================================== Model: mlx-community/Qwen3-0.6B-8bit Expected behavior: - Same prompt → cache HIT - Different prompt → cache MISS ---------------------------------------------------------------------- Results: Step | Description | Expected | Actual | Status -------+---------------------+----------+--------+------- 1a | First request | MISS | MISS | ✓ 1b | Same prompt | HIT | HIT | ✓ 1c | Different prompt | MISS | MISS | ✓ 1d | Return to prompt 1 | HIT | HIT | ✓ ====================================================================== ``` ### Prefix cache (M1 Max, 64 Go) | Test | Attendu | Réel | Temps | Statut | |------|---------|------|-------|--------| | Première requête | MISS | MISS | 203,5 ms | PASS | | Même prompt | HIT | HIT | 131,6 ms | PASS | | Prompt différent | MISS ou PREFIX_HIT | PREFIX_HIT (5 tok) | 135,3 ms | PASS | Statistiques finales du cache : | Hits cache | Misses cache | Taux de hit | Tokens économisés | Accélération avec cache | |------------|--------------|-------------|-------------------|------------------------| | 2 | 1 | 66,7 % | 20 | 1,55x | ## Résultats du paged cache *Test : 20 requêtes d'inférence réelles en 2 rounds avec un prompt système partagé d'environ 286 tokens* ``` ====================================================================== PAGED KV CACHE - REAL INFERENCE TEST ====================================================================== -------------------------------------------------- Test 1: WITHOUT Paged Cache (2 rounds of 10) -------------------------------------------------- Time: 1.47s Throughput: 681.2 tok/s Cache hits: 0 Tokens saved: 0 -------------------------------------------------- Test 2: WITH Paged Cache (2 rounds of 10) -------------------------------------------------- Time: 1.31s Throughput: 765.8 tok/s Paged Cache Stats: Blocks allocated: 25 Shared blocks: 4 Cache hits: 10 Tokens saved: 2560 ================================================== SUMMARY ================================================== Without paged cache: 681.2 tok/s With paged cache: 765.8 tok/s Speedup: 1.12x Cache hits: 10 (all Round 2 requests) Tokens saved: 2,560 (~256 tokens × 10 requests) ================================================== ``` ### KV cache paginé (M1 Max, 64 Go) Benchmark d'inférence (20 requêtes) : | Mode | Temps (s) | Throughput (tok/s) | |------|-----------|--------------------| | Sans paged cache | 3,43 | 291,8 | | Avec paged cache | 3,42 | 292,2 | | Accélération | Blocs alloués | Blocs partagés | Hits cache | Tokens économisés | |--------------|---------------|----------------|------------|-------------------| | 1,00x | 45 | 4 | 10 | 2560 | Inférence concurrente réelle (20 requêtes) : | Mode | Temps (s) | Throughput (tok/s) | |------|-----------|--------------------| | Sans paged cache | 4,32 | 231,7 | | Avec paged cache | 4,35 | 229,7 | | Accélération | Blocs alloués | Blocs partagés | Hits cache | Tokens économisés | |--------------|---------------|----------------|------------|-------------------| | 0,99x | 49 | 8 | 10 | 5120 | Démonstration des économies mémoire : | Scénario | Économies mémoire | |----------|-------------------| | Prompts système partagés | 70,8 % | | Efficacité mémoire concurrente | 83,5 % | | Branches avec partage de préfixe | 38,5 % | ## Analyse du détokeniseur en streaming *Investigation phase 9.1 : `BPEStreamingDetokenizer` de mlx-lm vs `tokenizer.decode()` naïf* ### Contexte L'approche naïve appelle `decode([token])` pour chaque token. En théorie, les détokeniseurs en streaming offrent une complexité O(T) contre O(T²) pour le décodage naïf. ### Résultats du benchmark isolé ```bash vllm-mlx bench-detok ``` En réutilisant la même instance de détokeniseur (avec `reset()` entre les utilisations) : | Séquence | Tokens | decode() naïf | Streaming | Accélération | |----------|--------|---------------|-----------|--------------| | Courte | 8 | 0,020 ms | 0,019 ms | 1,05x | | Moyenne | 103 | 0,155 ms | 0,097 ms | 1,59x | | Longue | 511 | 0,752 ms | 0,371 ms | **2,03x** | | 1K tokens | 1191 | 1,743 ms | 0,833 ms | **2,09x** | | 2K tokens | 2381 | 3,493 ms | 1,737 ms | **2,01x** | ### Constat critique : surcoût de création d'instance La création d'une nouvelle instance de `BPEStreamingDetokenizer` est **extrêmement coûteuse** : ``` 100 tokenizer.detokenizer calls: 5.266s (52.7ms each!) ``` Cela signifie que créer un nouveau détokeniseur par requête ajoute **environ 52 ms de surcoût**, annulant tout bénéfice. ### Impact en conditions réelles Intégré dans le scheduler (un détokeniseur par requête) : | Métrique | decode() naïf | Streaming (nouvelle instance) | |----------|---------------|-------------------------------| | Throughput (20 req) | 681 tok/s | 275 tok/s | | Impact | - | **-60 % plus lent** | ### Conclusion Le détokeniseur en streaming n'est **pas viable actuellement** pour un usage par requête, en raison du coût de création d'instance. L'approche naïve `decode([token])` reste plus rapide en pratique. **Optimisation future** : pré-créer un pool d'instances de détokeniseur au démarrage et les réutiliser entre les requêtes. ## Référence des métriques | Métrique | Description | |----------|-------------| | **TTFT** | Time to First Token - latence jusqu'à ce que le modèle commence à répondre (ms) | | **TPOT** | Time Per Output Token - temps entre chaque token généré (ms/token) | | **Generation TPS** | Tokens de sortie par seconde (tok/s) | | **Processing TPS** | Tokens d'entrée/prompt traités par seconde (tok/s) | | **End-to-End Latency** | Temps total de la requête à la réponse complète | | **Total Throughput** | Tokens totaux (entrée + sortie) par seconde | ## Lancer les benchmarks ```bash # Benchmark de base vllm-mlx-bench --model mlx-community/Qwen3-0.6B-8bit # Avec davantage de prompts vllm-mlx-bench --model mlx-community/Qwen3-0.6B-8bit --prompts 10 # Sauvegarder les résultats vllm-mlx-bench --model mlx-community/Qwen3-0.6B-8bit --output results.json # Test de continuous batching python tests/test_continuous_batching.py # Test de prefix cache python tests/test_prefix_cache.py # Test de paged cache python tests/test_paged_cache_real_inference.py # Benchmark du détokeniseur en streaming vllm-mlx bench-detok vllm-mlx bench-detok mlx-community/Llama-3.2-1B-Instruct-4bit --iterations 5 ``` # Documentation page: `fr/benchmarks/video.md` # Benchmarks vidéo ## Lancer les benchmarks vidéo ```bash # Benchmark complet (10 configurations, 2-64 frames) vllm-mlx-bench --model mlx-community/Qwen3-VL-8B-Instruct-4bit --video # Benchmark rapide (3 nombres de frames) vllm-mlx-bench --model mlx-community/Qwen3-VL-8B-Instruct-4bit --video --quick # Vidéo personnalisée vllm-mlx-bench --model mlx-community/Qwen3-VL-8B-Instruct-4bit --video --video-url https://example.com/video.mp4 ``` ## Résultats - Qwen3-VL-8B-Instruct-4bit (M4 Max, 128GB) | Configuration | Frames | Time | Tokens | Speed | Memory | |---------------|--------|------|--------|-------|--------| | 2 frames @ 0.5fps | 2 | 4.48s | 256 | 57.1 tok/s | 6.4 GB | | 4 frames @ 1fps | 4 | 4.65s | 256 | 55.0 tok/s | 6.4 GB | | 6 frames @ 1fps | 6 | 5.15s | 197 | 38.2 tok/s | 6.6 GB | | 8 frames @ 2fps | 8 | 6.45s | 240 | 37.2 tok/s | 6.8 GB | | 12 frames @ 2fps | 12 | 8.73s | 256 | 29.3 tok/s | 7.1 GB | | 16 frames @ 2fps | 16 | 10.96s | 256 | 23.4 tok/s | 7.6 GB | | 24 frames @ 4fps | 24 | 14.95s | 226 | 15.1 tok/s | 8.4 GB | | 32 frames @ 4fps | 32 | 20.00s | 256 | 12.8 tok/s | 9.2 GB | | 48 frames @ 8fps | 48 | 31.11s | 246 | 7.9 tok/s | 11.1 GB | | 64 frames @ 8fps | 64 | 59.81s | 256 | 4.3 tok/s | 12.9 GB | **Résumé :** Le plus rapide à 2 frames (57.1 tok/s), le plus lent à 64 frames (4.3 tok/s). La mémoire varie de 6.4 GB à 12.9 GB. > **Note :** 96 frames et plus provoque un timeout GPU sur la plupart des machines en raison des limites de mémoire et de calcul. ## Résultats - Qwen3-VL-8B-Instruct-4bit (M1 Max, 64GB) | Configuration | Frames | FPS | Time | Tokens | Speed | |---------------|--------|-----|------|--------|-------| | 4 frames @ 1fps | 4 | 1.0 | 8.84s | 256 | 29.0 tok/s | | 8 frames @ 2fps | 8 | 2.0 | 13.05s | 256 | 19.6 tok/s | | 16 frames @ 2fps | 16 | 2.0 | 21.60s | 256 | 11.9 tok/s | | Configs | Total Time (s) | Total Tokens | Aggregate Tok/s | |---------|-----------------|--------------|-----------------| | 3 | 43.48 | 768 | 17.7 | ## Résultats - Qwen3-VL-4B-Instruct-3bit (M1 Max, 64GB) | Configuration | Frames | FPS | Time | Tokens | Speed | |---------------|--------|-----|------|--------|-------| | 4 frames @ 1fps | 4 | 1.0 | 5.09s | 150 | 29.5 tok/s | | 8 frames @ 2fps | 8 | 2.0 | 8.36s | 150 | 17.9 tok/s | | 16 frames @ 2fps | 16 | 2.0 | 15.21s | 150 | 9.9 tok/s | | Configs | Total Time (s) | Total Tokens | Aggregate Tok/s | |---------|-----------------|--------------|-----------------| | 3 | 28.66 | 450 | 15.7 | ## Résultats du cache vidéo ``` ---------------------------------------------------------------------- TEST 4: Video Cache - fps/max_frames in Cache Key ---------------------------------------------------------------------- Config: fps=2.0, max_frames=16 Results: Step | Description | Expected | Actual | Time | Status -------+---------------------------+----------+--------+--------+------- 4a | Video first request | MISS | MISS | 0.03ms | ✓ 4b | Same video+params | HIT | HIT | 0.14ms | ✓ 4c | Different fps (4.0) | MISS | MISS | 0.01ms | ✓ 4d | Different max_frames (32) | MISS | MISS | 0.01ms | ✓ 4.0.5a | fps=0.5 first | MISS | MISS | 0.01ms | ✓ 4.0.5b | fps=0.5 cached | HIT | HIT | 0.14ms | ✓ 4.1.0a | fps=1.0 first | MISS | MISS | 0.01ms | ✓ 4.1.0b | fps=1.0 cached | HIT | HIT | 0.14ms | ✓ 4.2.0a | fps=2.0 first | MISS | MISS | 0.01ms | ✓ 4.2.0b | fps=2.0 cached | HIT | HIT | 0.14ms | ✓ 4.4.0a | fps=4.0 first | MISS | MISS | 0.01ms | ✓ 4.4.0b | fps=4.0 cached | HIT | HIT | 0.14ms | ✓ ---------------------------------------------------------------------- TEST 5: Additional Videos ---------------------------------------------------------------------- Results: Step | Description | Expected | Actual | Time | Status -------+---------------------------+----------+--------+--------+------- 5a | Video 1 first | MISS | MISS | 0.01ms | ✓ 5b | Video 2 first | MISS | MISS | 0.01ms | ✓ 5c | Video 1 cached | HIT | HIT | 0.13ms | ✓ 5d | Video 2 cached | HIT | HIT | 0.13ms | ✓ ``` ## Stratégie de clé de cache - **Vidéos :** `hash(video_path) + hash(fps) + hash(max_frames) + hash(prompt)` La même vidéo avec les mêmes paramètres fps, max_frames et prompt donnera un HIT dans le cache. La modification de l'un quelconque de ces paramètres provoque un MISS. ## Conseils de performance - Un FPS plus faible accélère le traitement - Moins de frames réduit l'utilisation mémoire - 64 frames est le maximum pratique - 96 frames et plus provoque un timeout GPU ## Extraction de frames | FPS | Vidéo 10s | Vidéo 30s | Vidéo 60s | |-----|-----------|-----------|-----------| | 0.5 | 5 frames | 15 frames | 30 frames | | 1.0 | 10 frames | 30 frames | 60 frames | | 2.0 | 20 frames | 60 frames | 120 frames* | | 4.0 | 40 frames | 120 frames* | 240 frames* | *Peut atteindre la limite `max_frames` ## Référence des métriques | Metric | Description | |--------|-------------| | Configuration | Paramètres FPS et nombre maximum de frames | | Frames | Nombre réel de frames extraites | | Time | Temps total de génération | | Tokens | Tokens de sortie générés | | Speed | Tokens par seconde (tok/s) | | Memory | Utilisation de la mémoire GPU | # Documentation page: `fr/getting-started/installation.md` # Installation ## Prérequis - macOS sur Apple Silicon (M1/M2/M3/M4/M5) - Python 3.10+ ## Installation avec uv (recommandée) ```bash git clone https://github.com/waybarrios/vllm-mlx.git cd vllm-mlx uv pip install -e . ``` ## Installation avec pip ```bash git clone https://github.com/waybarrios/vllm-mlx.git cd vllm-mlx pip install -e . ``` ### Optionnel : support vidéo Pour le traitement vidéo avec transformers : ```bash pip install -e ".[vision]" ``` ### Optionnel : support audio (STT/TTS) ```bash pip install mlx-audio ``` ### Optionnel : embeddings ```bash pip install mlx-embeddings ``` ## Ce qui est installé - `mlx`, `mlx-lm`, `mlx-vlm` - framework MLX et bibliothèques de modèles - `transformers`, `tokenizers` - bibliothèques HuggingFace - `opencv-python` - traitement vidéo - `gradio` - interface de chat - `psutil` - surveillance des ressources - `mlx-audio` (optionnel) - Speech-to-Text et Text-to-Speech - `mlx-embeddings` (optionnel) - embeddings de texte ## Vérifier l'installation ```bash # Check CLI commands vllm-mlx --help vllm-mlx-bench --help vllm-mlx-chat --help # Test with a small model vllm-mlx-bench --model mlx-community/Llama-3.2-1B-Instruct-4bit --prompts 1 ``` ## Dépannage ### MLX introuvable Vérifiez que vous êtes sur Apple Silicon : ```bash uname -m # Should output "arm64" ``` ### Échec du téléchargement du modèle Vérifiez votre connexion internet et vos accès HuggingFace. Certains modèles nécessitent une authentification : ```bash huggingface-cli login ``` ### Mémoire insuffisante Utilisez un modèle quantifié plus petit : ```bash vllm-mlx serve mlx-community/Llama-3.2-1B-Instruct-4bit ``` ### Interruptions du serveur pendant les exécutions longues (mise en veille de macOS) Votre machine macOS peut passer en veille pendant de longues exécutions en tant que serveur. Essayez d'utiliser `caffeinate` pour empêcher la mise en veille : ```bash caffeinate -dimsu ``` # Documentation page: `fr/getting-started/quickstart.md` # Démarrage rapide ## Option 1 : serveur compatible OpenAI Démarrez le serveur : ```bash # Simple mode - maximum throughput for single user vllm-mlx serve mlx-community/Llama-3.2-3B-Instruct-4bit --port 8000 # Continuous batching - for multiple concurrent users vllm-mlx serve mlx-community/Llama-3.2-3B-Instruct-4bit --port 8000 --continuous-batching ``` Utilisation avec le SDK Python OpenAI : ```python from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed") response = client.chat.completions.create( model="mlx-community/Llama-3.2-3B-Instruct-4bit", messages=[{"role": "user", "content": "Hello!"}], ) print(response.choices[0].message.content) ``` Ou avec curl : ```bash curl http://localhost:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{"model": "default", "messages": [{"role": "user", "content": "Hello!"}]}' ``` ## Option 2 : API Python directe ```python from vllm_mlx.models import MLXLanguageModel model = MLXLanguageModel("mlx-community/Llama-3.2-3B-Instruct-4bit") model.load() # Generate text output = model.generate("What is the capital of France?", max_tokens=100) print(output.text) # Streaming for chunk in model.stream_generate("Tell me a story"): print(chunk.text, end="", flush=True) ``` ## Option 3 : interface de chat Gradio ```bash vllm-mlx-chat --served-model-name mlx-community/Llama-3.2-3B-Instruct-4bit ``` Ouvre une interface web à l'adresse http://localhost:7860 ## Modèles multimodaux Pour la compréhension d'images et de vidéos, utilisez un modèle VLM : ```bash vllm-mlx serve mlx-community/Qwen3-VL-4B-Instruct-3bit --port 8000 ``` ```python response = client.chat.completions.create( model="default", messages=[{ "role": "user", "content": [ {"type": "text", "text": "What's in this image?"}, {"type": "image_url", "image_url": {"url": "https://example.com/image.jpg"}} ] }], max_tokens=256 ) ``` ## Modèles de raisonnement Séparez le processus de réflexion du modèle de la réponse finale : ```bash vllm-mlx serve mlx-community/Qwen3-8B-4bit --reasoning-parser qwen3 ``` ```python response = client.chat.completions.create( model="default", messages=[{"role": "user", "content": "What is 17 × 23?"}] ) print(response.choices[0].message.content) # Final answer ``` ## Embeddings Générez des embeddings textuels pour la recherche sémantique et le RAG : ```bash vllm-mlx serve mlx-community/Qwen3-4B-4bit --embedding-model mlx-community/multilingual-e5-small-mlx ``` ```python response = client.embeddings.create( model="mlx-community/multilingual-e5-small-mlx", input="Hello world" ) ``` ## Tool Calling Activez l'appel de fonctions avec tout modèle compatible : ```bash vllm-mlx serve mlx-community/Devstral-Small-2507-4bit \ --enable-auto-tool-choice --tool-call-parser mistral ``` ## Étapes suivantes - [Server Guide](../guides/server.md) - Configuration complète du serveur - [Python API](../guides/python-api.md) - Utilisation directe de l'API - [Multimodal Guide](../guides/multimodal.md) - Images et vidéos - [Audio Guide](../guides/audio.md) - Speech-to-Text et Text-to-Speech - [Embeddings Guide](../guides/embeddings.md) - Embeddings textuels - [Reasoning Models](../guides/reasoning.md) - Modèles de réflexion - [Tool Calling](../guides/tool-calling.md) - Appel de fonctions - [Supported Models](../reference/models.md) - Modèles disponibles # Documentation page: `fr/guides/audio.md` # Support Audio vllm-mlx prend en charge le traitement audio via [mlx-audio](https://github.com/Blaizzy/mlx-audio), offrant : - **STT (Speech-to-Text)** : Whisper, Parakeet - **TTS (Text-to-Speech)** : Kokoro, Chatterbox, VibeVoice, VoxCPM - **Traitement audio** : SAM-Audio (séparation vocale) ## Installation ```bash # Support audio de base pip install mlx-audio>=0.2.9 # Required dependencies for TTS pip install sounddevice soundfile scipy numba tiktoken misaki spacy num2words loguru phonemizer # Download spacy English model python -m spacy download en_core_web_sm # For non-English TTS (Spanish, French, etc.), install espeak-ng: # macOS brew install espeak-ng # Ubuntu/Debian # sudo apt-get install espeak-ng ``` Ou installez toutes les dépendances audio en une seule commande : ```bash pip install vllm-mlx[audio] python -m spacy download en_core_web_sm brew install espeak-ng # macOS, for non-English languages ``` ## Démarrage rapide ### Speech-to-Text (Transcription) ```python from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed") # Transcribe audio file with open("audio.mp3", "rb") as f: transcript = client.audio.transcriptions.create( model="whisper-large-v3", file=f, language="en" # optional ) print(transcript.text) ``` ### Text-to-Speech (Génération) ```python # Generate speech audio = client.audio.speech.create( model="kokoro", input="Hello, how are you?", voice="af_heart", speed=1.0 ) # Save to file with open("output.wav", "wb") as f: f.write(audio.content) ``` ### Séparation vocale (SAM-Audio) Isolez une voix du bruit de fond, de la musique ou d'autres sons : ```python from vllm_mlx.audio import AudioProcessor # Load SAM-Audio model processor = AudioProcessor("mlx-community/sam-audio-large-fp16") processor.load() # Separate speech from audio result = processor.separate("meeting_with_music.mp3", description="speech") # Save isolated voice and background processor.save(result.target, "voice_only.wav") processor.save(result.residual, "background_only.wav") ``` **Exemple en ligne de commande :** ```bash python examples/audio_separation_example.py meeting.mp3 --play python examples/audio_separation_example.py song.mp3 --description music -o music.wav ``` ### Démo de séparation de batterie Isolez la batterie d'une chanson rock avec SAM-Audio : | Audio | Description | Écouter | |-------|-------------|---------| | Original | "Get Ready" de David Fesliyan (30s, libre de droits) | [rock_get_ready.mp3](https://github.com/waybarrios/vllm-mlx/blob/main/examples/rock_get_ready.mp3?raw=1) | | Batterie isolée | Batterie extraite par SAM-Audio | [drums_isolated.wav](https://github.com/waybarrios/vllm-mlx/blob/main/examples/drums_isolated.wav?raw=1) | | Sans batterie | Piste sans batterie | [rock_no_drums.wav](https://github.com/waybarrios/vllm-mlx/blob/main/examples/rock_no_drums.wav?raw=1) | ```bash # Isolate drums from rock song python examples/audio_separation_example.py examples/rock_get_ready.mp3 \ --description "drums" \ --output drums_isolated.wav \ --background rock_no_drums.wav ``` **Performance :** 30 secondes d'audio traitées en environ 20 secondes sur M4 Max. ## Modèles pris en charge ### Modèles STT (Speech-to-Text) | Modèle | Alias | Langues | Vitesse | Qualité | |--------|-------|---------|---------|---------| | `mlx-community/whisper-large-v3-mlx` | `whisper-large-v3` | 99+ | Moyenne | Meilleure | | `mlx-community/whisper-large-v3-turbo` | `whisper-large-v3-turbo` | 99+ | Rapide | Excellente | | `mlx-community/whisper-medium-mlx` | `whisper-medium` | 99+ | Rapide | Bonne | | `mlx-community/whisper-small-mlx` | `whisper-small` | 99+ | Très rapide | Correcte | | `mlx-community/parakeet-tdt-0.6b-v2` | `parakeet` | Anglais | La plus rapide | Excellente | | `mlx-community/parakeet-tdt-0.6b-v3` | `parakeet-v3` | Anglais | La plus rapide | Meilleure | **Recommandations :** - Multilingue : `whisper-large-v3` - Anglais uniquement : `parakeet` (3 fois plus rapide) ### Modèles TTS (Text-to-Speech) #### Kokoro (Rapide, Léger) - Recommandé | Modèle | Alias | Taille | Langues | |--------|-------|--------|---------| | `mlx-community/Kokoro-82M-bf16` | `kokoro` | 82M | EN, ES, FR, JA, ZH, HI, IT, PT | | `mlx-community/Kokoro-82M-4bit` | `kokoro-4bit` | 82M | EN, ES, FR, JA, ZH, HI, IT, PT | **Voix (11) :** - Femme américaine : `af_heart`, `af_bella`, `af_nicole`, `af_sarah`, `af_sky` - Homme américain : `am_adam`, `am_michael` - Femme britannique : `bf_emma`, `bf_isabella` - Homme britannique : `bm_george`, `bm_lewis` **Codes de langue :** | Code | Langue | Code | Langue | |------|--------|------|--------| | `a` / `en` | English (US) | `e` / `es` | Español | | `b` / `en-gb` | English (UK) | `f` / `fr` | Français | | `j` / `ja` | 日本語 | `z` / `zh` | 中文 | | `i` / `it` | Italiano | `p` / `pt` | Português | | `h` / `hi` | हिन्दी | | | #### Chatterbox (Multilingue, Expressif) | Modèle | Alias | Taille | Langues | |--------|-------|--------|---------| | `mlx-community/chatterbox-turbo-fp16` | `chatterbox` | 134M | 15+ langues | | `mlx-community/chatterbox-turbo-4bit` | `chatterbox-4bit` | 134M | 15+ langues | **Langues prises en charge :** EN, ES, FR, DE, IT, PT, RU, JA, ZH, KO, AR, HI, NL, PL, TR #### VibeVoice (Temps réel) | Modèle | Alias | Taille | Cas d'usage | |--------|-------|--------|-------------| | `mlx-community/VibeVoice-Realtime-0.5B-4bit` | `vibevoice` | 200M | Faible latence, anglais | #### VoxCPM (Chinois/Anglais) | Modèle | Alias | Taille | Langues | |--------|-------|--------|---------| | `mlx-community/VoxCPM1.5` | `voxcpm` | 0.9B | ZH, EN | | `mlx-community/VoxCPM1.5-4bit` | `voxcpm-4bit` | 200M | ZH, EN | ### Modèles de traitement audio #### SAM-Audio (Séparation vocale) | Modèle | Taille | Cas d'usage | |--------|--------|-------------| | `mlx-community/sam-audio-large-fp16` | 3B | Meilleure qualité | | `mlx-community/sam-audio-large` | 3B | Standard | | `mlx-community/sam-audio-small-fp16` | 0.6B | Rapide | | `mlx-community/sam-audio-small` | 0.6B | Léger | ## Référence API ### POST /v1/audio/transcriptions Transcrit un fichier audio en texte (compatible API OpenAI Whisper). **Paramètres :** - `file` : Fichier audio (mp3, wav, m4a, webm) - `model` : Nom ou alias du modèle - `language` : Code de langue (optionnel, détection automatique) - `response_format` : `json` ou `text` **Limites :** - Taille maximale par défaut : 25 MiB - Modifiable avec `--max-audio-upload-mb` **Exemple :** ```bash curl http://localhost:8000/v1/audio/transcriptions \ -F file=@audio.mp3 \ -F model=whisper-large-v3 ``` ### POST /v1/audio/speech Génère de la parole à partir de texte (compatible API OpenAI TTS). **Paramètres :** - `model` : Nom ou alias du modèle - `input` : Texte à synthétiser - `voice` : Identifiant de la voix - `speed` : Vitesse de parole (0,5 à 2,0) - `response_format` : `wav`, `mp3` **Limites :** - Nombre de caractères maximal par défaut : 4096 - Modifiable avec `--max-tts-input-chars` **Exemple :** ```bash curl http://localhost:8000/v1/audio/speech \ -d '{"model": "kokoro", "input": "Hello world", "voice": "af_heart"}' \ -H "Content-Type: application/json" \ --output speech.wav ``` ### GET /v1/audio/voices Liste les voix disponibles pour un modèle. **Exemple :** ```bash curl http://localhost:8000/v1/audio/voices?model=kokoro ``` ## Exemples en ligne de commande ### Transcription en direct / Sous-titres en temps réel Transcription STT en temps réel depuis votre microphone : ```bash # Closed captions with whisper-large-v3 (best quality) python examples/closed_captions.py --language es --chunk 5 # Faster model for lower latency python examples/closed_captions.py --language en --model whisper-turbo --chunk 3 # Basic mic transcription (record then transcribe) python examples/mic_transcribe.py --language es # Real-time chunked transcription python examples/mic_realtime.py --language es --chunk 3 # Live transcription with voice activity detection python examples/mic_live.py --language es ``` **Prérequis :** ```bash pip install sounddevice soundfile numpy ``` ### TTS de base ```bash # Simple TTS example python examples/tts_example.py "Hello, how are you?" --play # With different voice python examples/tts_example.py "Hello!" --voice am_michael --play # Save to file python examples/tts_example.py "Welcome to the demo" -o greeting.wav # List available voices python examples/tts_example.py --list-voices ``` ### TTS multilingue ```bash # English (auto-selects best model) python examples/tts_multilingual.py "Hello world" --play # Spanish python examples/tts_multilingual.py "Hola mundo" --lang es --play # French python examples/tts_multilingual.py "Bonjour le monde" --lang fr --play # Japanese python examples/tts_multilingual.py "こんにちは" --lang ja --play # Chinese python examples/tts_multilingual.py "你好世界" --lang zh --play # Use specific model python examples/tts_multilingual.py "Hello" --model chatterbox --play # List all models python examples/tts_multilingual.py --list-models # List all languages python examples/tts_multilingual.py --list-languages ``` ### Exemples d'assistant vocal professionnel Exemples vocaux prégénérés avec des **voix natives** pour des cas d'usage professionnels courants : | Langue | Voix | Message | Écouter | |--------|------|---------|---------| | Anglais | af_heart | "Welcome to First National Bank. How may I assist you today?" | [assistant_bank_en.wav](https://github.com/waybarrios/vllm-mlx/blob/main/examples/assistant_bank_en.wav?raw=1) | | Espagnol | ef_dora | "Gracias por llamar a servicio al cliente. Un agente le atenderá pronto." | [assistant_service_es.wav](https://github.com/waybarrios/vllm-mlx/blob/main/examples/assistant_service_es.wav?raw=1) | | Français | ff_siwis | "Bienvenue. Votre appel est important pour nous." | [assistant_callcenter_fr.wav](https://github.com/waybarrios/vllm-mlx/blob/main/examples/assistant_callcenter_fr.wav?raw=1) | | Chinois | zf_xiaobei | "欢迎致电技术支持中心。我们将竭诚为您服务。" | [assistant_support_zh.wav](https://github.com/waybarrios/vllm-mlx/blob/main/examples/assistant_support_zh.wav?raw=1) | **Générez vos propres exemples avec des voix natives :** ```bash # English - Bank assistant (native voice: af_heart) python -m mlx_audio.tts.generate --model mlx-community/Kokoro-82M-bf16 \ --text "Welcome to First National Bank. How may I assist you today?" \ --voice af_heart --lang_code a --file_prefix assistant_bank_en # Spanish - Customer service (native voice: ef_dora) python -m mlx_audio.tts.generate --model mlx-community/Kokoro-82M-bf16 \ --text "Gracias por llamar a servicio al cliente. Un agente le atendera pronto." \ --voice ef_dora --lang_code e --file_prefix assistant_service_es # French - Call center (native voice: ff_siwis) python -m mlx_audio.tts.generate --model mlx-community/Kokoro-82M-bf16 \ --text "Bienvenue. Votre appel est important pour nous." \ --voice ff_siwis --lang_code f --file_prefix assistant_callcenter_fr # Chinese - Tech support (native voice: zf_xiaobei) python -m mlx_audio.tts.generate --model mlx-community/Kokoro-82M-bf16 \ --text "欢迎致电技术支持中心。我们将竭诚为您服务。" \ --voice zf_xiaobei --lang_code z --file_prefix assistant_support_zh ``` ### Référence des voix natives | Langue | Code | Voix | |--------|------|------| | English (US) | `a` | af_heart, af_bella, af_nicole, am_adam, am_michael | | English (UK) | `b` | bf_emma, bf_isabella, bm_george, bm_lewis | | Espagnol | `e` | ef_dora, em_alex, em_santa | | Français | `f` | ff_siwis | | Chinois | `z` | zf_xiaobei, zf_xiaoni, zf_xiaoxiao, zm_yunjian, zm_yunxi | | Japonais | `j` | jf_alpha, jf_gongitsune, jm_kumo | | Italien | `i` | if_sara, im_nicola | | Portugais | `p` | pf_dora, pm_alex | | Hindi | `h` | hf_alpha, hf_beta, hm_omega | ## API Python ### Utilisation directe (sans serveur) ```python from vllm_mlx.audio import STTEngine, TTSEngine, AudioProcessor # Speech-to-Text stt = STTEngine("mlx-community/whisper-large-v3-mlx") stt.load() result = stt.transcribe("audio.mp3") print(result.text) # Text-to-Speech tts = TTSEngine("mlx-community/Kokoro-82M-bf16") tts.load() audio = tts.generate("Hello world", voice="af_heart") tts.save(audio, "output.wav") # Voice Separation processor = AudioProcessor("mlx-community/sam-audio-large-fp16") processor.load() result = processor.separate("mixed_audio.mp3", description="speech") processor.save(result.target, "voice_only.wav") processor.save(result.residual, "background.wav") ``` ### Fonctions utilitaires ```python from vllm_mlx.audio import transcribe_audio, generate_speech, separate_voice # Quick transcription result = transcribe_audio("audio.mp3") print(result.text) # Quick TTS audio = generate_speech("Hello world", voice="af_heart") # Quick voice separation voice, background = separate_voice("mixed.mp3") ``` ## Audio dans le chat Incluez de l'audio dans les messages du chat (transcrit automatiquement) : ```python response = client.chat.completions.create( model="default", messages=[{ "role": "user", "content": [ {"type": "text", "text": "Summarize this audio"}, {"type": "audio_url", "audio_url": {"url": "file://meeting.mp3"}} ] }] ) ``` ## Benchmarks Testé sur Apple M2 Max (32 Go). ### Benchmarks TTS (Kokoro-82M-bf16) | Longueur du texte | Durée audio | Temps de génération | RTF | Caractères/s | |-------------------|-------------|---------------------|-----|--------------| | 25 caractères | 1,95 s | 0,43 s | 4,6x | 58,5 | | 88 caractères | 6,00 s | 0,32 s | 18,6x | 272,4 | | 117 caractères | 7,92 s | 0,27 s | 29,0x | 427,4 | **Résumé :** - Temps de chargement du modèle : environ 1,0 s - RTF moyen : **17,4x** (17 fois plus rapide que le temps réel) - Caractères/s moyens : **252,8** ### Benchmarks STT | Modèle | Temps de chargement | Transcription (6 s audio) | RTF | |--------|---------------------|---------------------------|-----| | whisper-small | 0,25 s | 0,20 s | 30,2x | | whisper-medium | 18,1 s | 0,38 s | 15,5x | | whisper-large-v3 | environ 30 s | environ 0,6 s | environ 10x | | parakeet | environ 0,5 s | environ 0,15 s | environ 40x | **Notes :** - Le RTF (Real-Time Factor) indique combien de fois plus rapide que le temps réel - Le premier chargement inclut le téléchargement du modèle depuis HuggingFace - Les chargements suivants utilisent les modèles mis en cache ### Recommandations par cas d'usage | Cas d'usage | Modèle recommandé | Pourquoi | |-------------|------------------|----------| | STT anglais rapide | `parakeet` | RTF 40x, faible consommation mémoire | | STT multilingue | `whisper-large-v3` | 99+ langues | | STT faible latence | `whisper-small` | RTF 30x, chargement rapide | | TTS général | `kokoro` | RTF 17x, bonne qualité | | TTS faible mémoire | `kokoro-4bit` | Quantification 4 bits | ## Conseils de performance 1. **Utilisez Parakeet pour l'anglais** : 40 fois plus rapide que le temps réel 2. **Utilisez les modèles 4 bits** pour réduire la consommation mémoire 3. **Utilisez SAM-Audio small** pour une séparation vocale plus rapide 4. **Mettez les modèles en cache** : les moteurs sont chargés à la demande et mis en cache 5. **Pré-téléchargez les modèles** pour éviter la latence au premier démarrage ## Dépannage ### mlx-audio non installé ``` pip install mlx-audio>=0.2.9 ``` ### Téléchargement du modèle lent Les modèles sont téléchargés depuis HuggingFace lors de la première utilisation. Utilisez `huggingface-cli download` pour les pré-télécharger : ```bash huggingface-cli download mlx-community/whisper-large-v3-mlx huggingface-cli download mlx-community/Kokoro-82M-bf16 ``` ### Mémoire insuffisante Utilisez des modèles plus petits ou des versions quantifiées en 4 bits : - `whisper-small-mlx` plutôt que `whisper-large-v3-mlx` - `Kokoro-82M-4bit` plutôt que `Kokoro-82M-bf16` - `sam-audio-small` plutôt que `sam-audio-large` ### Bug multilingue Kokoro (mlx-audio 0.2.9) Si vous obtenez `ValueError: too many values to unpack` en utilisant des langues autres que l'anglais (espagnol, chinois, japonais, etc.) avec Kokoro, appliquez ce correctif : ```python # Fix for mlx_audio/tts/models/kokoro/pipeline.py line 443 # Change: # ps, _ = self.g2p(chunk) # To: g2p_result = self.g2p(chunk) ps = g2p_result[0] if isinstance(g2p_result, tuple) else g2p_result ``` **Correctif en une ligne :** ```bash python -c " import os path = os.path.join(os.path.dirname(__import__('mlx_audio').__file__), 'tts/models/kokoro/pipeline.py') with open(path, 'r') as f: content = f.read() old = ' ps, _ = self.g2p(chunk)' new = ''' # Fix: handle both tuple (en) and string (zh/ja/es) returns from g2p g2p_result = self.g2p(chunk) ps = g2p_result[0] if isinstance(g2p_result, tuple) else g2p_result''' if old in content: with open(path, 'w') as f: f.write(content.replace(old, new)) print('Fix applied!') " ``` Ce bug survient car le g2p anglais retourne un tuple `(phonemes, tokens)` tandis que les autres langues retournent uniquement une chaîne de caractères. # Documentation page: `fr/guides/continuous-batching.md` # Continuous Batching Le continuous batching permet d'augmenter le throughput lors du traitement de plusieurs utilisateurs simultanés. ## Activer le Continuous Batching ```bash vllm-mlx serve mlx-community/Qwen3-0.6B-8bit --continuous-batching ``` ## Avec le Paged Cache Pour un partage mémoire efficace des préfixes : ```bash vllm-mlx serve mlx-community/Qwen3-0.6B-8bit --continuous-batching --use-paged-cache ``` ## Fonctionnement ### Mode simple (par défaut) - Une seule requête à la fois - Throughput maximal pour un utilisateur unique - Aucune surcharge liée au batching ### Mode Continuous Batching - Plusieurs requêtes traitées simultanément - Meilleur throughput pour les utilisateurs concurrents - Légère surcharge par requête ### Paged Cache - Le KV cache est stocké en blocs de taille fixe - Les prompts système identiques partagent les mêmes blocs - Économies mémoire : 80 % et plus pour 10 utilisateurs simultanés ou davantage ## Résultats de performance **Résultats du Continuous Batching (M4 Max, 128 Go) :** | Modèle | Requête unique | Batch (5 req) | Accélération | |--------|----------------|---------------|--------------| | Llama-3.2-1B-Instruct-4bit | 299.1 tok/s | 613.0 tok/s | **2.05x** | | Llama-3.2-3B-Instruct-4bit | 137.6 tok/s | 208.1 tok/s | **1.51x** | | Qwen3-0.6B-8bit | 328.1 tok/s | 1111.8 tok/s | **3.39x** | | Qwen3-30B-A3B-4bit | 98.1 tok/s | 233.3 tok/s | **2.38x** | | Qwen2.5-1.5B-Instruct-4bit | 196.9 tok/s | 322.2 tok/s | **1.64x** | *Le batching de 5 requêtes simultanées améliore le throughput d'un facteur 1,5 à 3.* ## Performance en Streaming **Performance en streaming (M4 Max, 128 Go) :** | Modèle | TTFT | Vitesse de génération | |--------|------|-----------------------| | Llama-3.2-1B-Instruct-4bit | ~4.6 ms | 218.9 tok/s | | Llama-3.2-3B-Instruct-4bit | ~10.7 ms | 93.6 tok/s | | Qwen3-0.6B-8bit | ~3.0 ms | 328.5 tok/s | | Qwen3-30B-A3B-4bit | ~10.2 ms | 98.4 tok/s | | Qwen2.5-1.5B-Instruct-4bit | ~7.1 ms | 140.3 tok/s | *TTFT = Time to First Token* ## Configuration du Streaming Contrôlez la cadence d'envoi des tokens avec `--stream-interval` : ```bash # Chaque token (le plus fluide) vllm-mlx serve model --continuous-batching --stream-interval 1 # Tokens groupés (préférable pour les connexions à latence élevée) vllm-mlx serve model --continuous-batching --stream-interval 5 ``` | Valeur | Comportement | |--------|-------------| | `1` | Envoie chaque token immédiatement | | `2-5` | Regroupe les tokens avant l'envoi | | `10+` | Throughput maximal, sortie plus fragmentée | ## Gestion de la mémoire Pour les grands modèles, le prefix cache peut consommer une quantité significative de mémoire. Le cache adaptatif la gère automatiquement : ```bash # Détection automatique (utilise 20 % de la RAM disponible) vllm-mlx serve model --continuous-batching # Limite explicite vllm-mlx serve model --continuous-batching --cache-memory-mb 2048 # Pourcentage personnalisé vllm-mlx serve model --continuous-batching --cache-memory-percent 0.10 ``` | Option | Description | |--------|-------------| | `--cache-memory-mb` | Définit une limite explicite en Mo | | `--cache-memory-percent` | Fraction de la RAM disponible (par défaut : 0,20) | | `--no-memory-aware-cache` | Utilise le cache historique basé sur le nombre d'entrées | ## Prefix Cache Le prefix caching réutilise le KV cache pour les prompts répétés. ### Fonctionnement ``` User 1: System prompt (500 tokens) → Creates 8 blocks User 2: Same system prompt → Shares 8 blocks (ref_count++) User N: Same system prompt → Shares 8 blocks (ref_count++) Memory savings: 80%+ for 10+ concurrent users ``` ### Stratégie de clé de cache - **LLM** : `hash(prompt)` - **Images** : `hash(image_content) + hash(prompt)` - **Vidéos** : `hash(video_path) + hash(fps) + hash(max_frames) + hash(prompt)` ### Tester le Prefix Cache ```bash python tests/test_prefix_cache.py ``` ``` ====================================================================== LLM PREFIX CACHE TEST ====================================================================== Model: mlx-community/Qwen3-0.6B-8bit Expected behavior: - Same prompt → cache HIT - Different prompt → cache MISS or PREFIX_HIT (shared template tokens) ---------------------------------------------------------------------- Results: Step | Description | Expected | Actual | Status -------+---------------------+----------+--------+------- 1a | First request | MISS | MISS | PASS 1b | Same prompt | HIT | HIT | PASS 1c | Different prompt | MISS | MISS | PASS 1d | Return to prompt 1 | HIT | HIT | PASS ====================================================================== ``` ## Exécuter les benchmarks ```bash # Benchmark du continuous batching python tests/test_continuous_batching.py # Test du prefix cache python tests/test_prefix_cache.py ``` ## Quand l'utiliser | Scénario | Mode | |----------|------| | Utilisateur unique, vitesse maximale | Simple (par défaut) | | Plusieurs utilisateurs simultanés | `--continuous-batching` | | Grands modèles (7B et plus) | `--continuous-batching --cache-memory-mb 2048` | | Production avec prompts partagés | `--continuous-batching --use-paged-cache` | ## Configuration en production ```bash vllm-mlx serve mlx-community/Qwen3-0.6B-8bit \ --continuous-batching \ --use-paged-cache \ --port 8000 ``` # Documentation page: `fr/guides/embeddings.md` # Embeddings vllm-mlx prend en charge les embeddings de texte via [mlx-embeddings](https://github.com/Blaizzy/mlx-embeddings), en exposant un point d'accès `/v1/embeddings` compatible OpenAI. ## Installation ```bash pip install mlx-embeddings>=0.0.5 ``` ## Démarrage rapide ### Lancer le serveur avec un modèle d'embeddings ```bash # Précharger un modèle d'embeddings spécifique au démarrage vllm-mlx serve my-llm-model --embedding-model mlx-community/all-MiniLM-L6-v2-4bit ``` Si vous n'utilisez pas `--embedding-model`, le modèle d'embeddings est chargé à la demande lors de la première requête, mais uniquement parmi les modèles autorisés par défaut. ### Générer des embeddings avec le SDK OpenAI ```python from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed") # Texte unique response = client.embeddings.create( model="mlx-community/all-MiniLM-L6-v2-4bit", input="Hello world" ) print(response.data[0].embedding[:5]) # First 5 dimensions # Lot de textes response = client.embeddings.create( model="mlx-community/all-MiniLM-L6-v2-4bit", input=[ "I love machine learning", "Deep learning is fascinating", "Natural language processing rocks" ] ) for item in response.data: print(f"Text {item.index}: {len(item.embedding)} dimensions") ``` ### Utilisation avec curl ```bash curl http://localhost:8000/v1/embeddings \ -H "Content-Type: application/json" \ -d '{ "model": "mlx-community/all-MiniLM-L6-v2-4bit", "input": ["Hello world", "How are you?"] }' ``` ## Modèles pris en charge Modèles disponibles à la demande : | Modèle | Cas d'usage | Taille | |--------|-------------|--------| | `mlx-community/all-MiniLM-L6-v2-4bit` | Rapide et compact | Small | | `mlx-community/embeddinggemma-300m-6bit` | Haute qualité | 300M | | `mlx-community/bge-large-en-v1.5-4bit` | Optimal pour l'anglais | Large | | `mlx-community/multilingual-e5-small-mlx` | Récupération multilingue | Small | | `mlx-community/multilingual-e5-large-mlx` | Récupération multilingue | Large | | `mlx-community/bert-base-uncased-mlx` | Référence BERT générale | Base | | `mlx-community/ModernBERT-base-mlx` | Référence ModernBERT | Base | Les autres modèles d'embeddings nécessitent l'option `--embedding-model` au démarrage du serveur. ## Gestion des modèles ### Chargement à la demande Par défaut, le modèle d'embeddings est chargé lors de la première requête sur `/v1/embeddings`. Vous pouvez alterner entre les modèles autorisés listés ci-dessus ; le modèle précédent est déchargé automatiquement. ### Préchargement au démarrage Utilisez `--embedding-model` pour charger un modèle au démarrage. Lorsque cette option est définie, seul ce modèle peut être utilisé pour les embeddings : ```bash vllm-mlx serve my-llm-model --embedding-model mlx-community/all-MiniLM-L6-v2-4bit ``` Toute requête utilisant un modèle différent renverra une erreur 400. ## Référence de l'API ### POST /v1/embeddings Génère des embeddings pour le ou les textes fournis. **Corps de la requête :** | Champ | Type | Requis | Description | |-------|------|--------|-------------| | `model` | string | Oui | Identifiant d'un modèle d'embeddings pris en charge, ou le modèle fixé au démarrage si `--embedding-model` est utilisé | | `input` | string ou list[string] | Oui | Texte(s) à encoder | **Réponse :** ```json { "object": "list", "data": [ {"object": "embedding", "index": 0, "embedding": [0.023, -0.982, ...]}, {"object": "embedding", "index": 1, "embedding": [0.112, -0.543, ...]} ], "model": "mlx-community/all-MiniLM-L6-v2-4bit", "usage": {"prompt_tokens": 12, "total_tokens": 12} } ``` ## API Python ### Utilisation directe sans serveur ```python from vllm_mlx.embedding import EmbeddingEngine engine = EmbeddingEngine("mlx-community/all-MiniLM-L6-v2-4bit") engine.load() vectors = engine.embed(["Hello world", "How are you?"]) print(f"Dimensions: {len(vectors[0])}") tokens = engine.count_tokens(["Hello world"]) print(f"Token count: {tokens}") ``` ## Résolution des problèmes ### mlx-embeddings non installé ``` pip install mlx-embeddings>=0.0.5 ``` ### Modèle introuvable Vérifiez que le nom du modèle correspond à l'un des identifiants autorisés listés ci-dessus, ou lancez le serveur avec `--embedding-model` pour fixer un modèle personnalisé. Vous pouvez télécharger les modèles pris en charge à l'avance : ```bash huggingface-cli download mlx-community/all-MiniLM-L6-v2-4bit ``` # Documentation page: `fr/guides/mcp-tools.md` # MCP & Tool Calling vllm-mlx prend en charge le Model Context Protocol (MCP) pour intégrer des outils externes avec des LLM. ## Fonctionnement du tool calling ``` ┌─────────────────────────────────────────────────────────────────────┐ │ Tool Calling Flow │ ├─────────────────────────────────────────────────────────────────────┤ │ │ │ 1. User Request │ │ ─────────────────► "List files in /tmp" │ │ │ │ 2. LLM Generates Tool Call │ │ ─────────────────► tool_calls: [{ │ │ name: "list_directory", │ │ arguments: {path: "/tmp"} │ │ }] │ │ │ │ 3. App Executes Tool via MCP │ │ ─────────────────► MCP Server executes list_directory │ │ Returns: ["file1.txt", "file2.txt"] │ │ │ │ 4. Tool Result Sent Back to LLM │ │ ─────────────────► role: "tool", content: [...] │ │ │ │ 5. LLM Generates Final Response │ │ ─────────────────► "The /tmp directory contains 2 files..." │ │ │ └─────────────────────────────────────────────────────────────────────┘ ``` ## Démarrage rapide ### 1. Créer la configuration MCP Créez `mcp.json` : ```json { "mcpServers": { "filesystem": { "command": "npx", "args": ["-y", "@modelcontextprotocol/server-filesystem", "/tmp"] } } } ``` ### 2. Démarrer le serveur avec MCP ```bash # Mode simple vllm-mlx serve mlx-community/Qwen3-4B-4bit --mcp-config mcp.json # Continuous batching vllm-mlx serve mlx-community/Qwen3-4B-4bit --mcp-config mcp.json --continuous-batching ``` ### 3. Vérifier l'état du MCP ```bash # Vérifier l'état du MCP curl http://localhost:8000/v1/mcp/status # Lister les outils disponibles curl http://localhost:8000/v1/mcp/tools ``` ## Exemple de tool calling ```python import json import httpx BASE_URL = "http://localhost:8000" # 1. Get available tools tools_response = httpx.get(f"{BASE_URL}/v1/mcp/tools") tools = tools_response.json()["tools"] # 2. Send request with tools response = httpx.post( f"{BASE_URL}/v1/chat/completions", json={ "model": "default", "messages": [{"role": "user", "content": "List files in /tmp"}], "tools": tools, "max_tokens": 1024 } ) result = response.json() message = result["choices"][0]["message"] # 3. Check for tool calls if message.get("tool_calls"): tool_call = message["tool_calls"][0] # 4. Execute tool via MCP exec_response = httpx.post( f"{BASE_URL}/v1/mcp/execute", json={ "server": "filesystem", "tool": tool_call["function"]["name"], "arguments": json.loads(tool_call["function"]["arguments"]) } ) tool_result = exec_response.json() # 5. Send result back to LLM messages = [ {"role": "user", "content": "List files in /tmp"}, message, { "role": "tool", "tool_call_id": tool_call["id"], "content": json.dumps(tool_result["result"]) } ] final_response = httpx.post( f"{BASE_URL}/v1/chat/completions", json={"model": "default", "messages": messages} ) print(final_response.json()["choices"][0]["message"]["content"]) ``` ## Points de terminaison MCP | Point de terminaison | Méthode | Description | |----------------------|---------|-------------| | `/v1/mcp/status` | GET | Vérifier l'état du MCP | | `/v1/mcp/tools` | GET | Lister les outils disponibles | | `/v1/mcp/execute` | POST | Exécuter un outil | ## Exemples de serveurs MCP ### Système de fichiers ```json { "mcpServers": { "filesystem": { "command": "npx", "args": ["-y", "@modelcontextprotocol/server-filesystem", "/tmp"] } } } ``` ### GitHub ```json { "mcpServers": { "github": { "command": "npx", "args": ["-y", "@modelcontextprotocol/server-github"], "env": { "GITHUB_TOKEN": "your-token" } } } } ``` ### PostgreSQL ```json { "mcpServers": { "postgres": { "command": "npx", "args": ["-y", "@modelcontextprotocol/server-postgres"], "env": { "DATABASE_URL": "postgresql://user:pass@localhost/db" } } } } ``` ### Brave Search ```json { "mcpServers": { "brave-search": { "command": "npx", "args": ["-y", "@modelcontextprotocol/server-brave-search"], "env": { "BRAVE_API_KEY": "your-key" } } } } ``` ## Plusieurs serveurs MCP ```json { "mcpServers": { "filesystem": { "command": "npx", "args": ["-y", "@modelcontextprotocol/server-filesystem", "/tmp"] }, "github": { "command": "npx", "args": ["-y", "@modelcontextprotocol/server-github"], "env": { "GITHUB_TOKEN": "your-token" } } } } ``` ## Chat MCP interactif Pour tester MCP de manière interactive : ```bash python examples/mcp_chat.py ``` ## Formats d'outils pris en charge vllm-mlx prend en charge 12 tool call parsers couvrant toutes les grandes familles de modèles. Voir [Tool Calling](tool-calling.md) pour la liste complète des parsers, alias et exemples. ## Sécurité vllm-mlx inclut des mesures de sécurité pour prévenir les attaques par injection de commandes via les serveurs MCP. ### Liste blanche de commandes Seules les commandes de confiance sont autorisées par défaut : | Catégorie | Commandes autorisées | |-----------|----------------------| | Node.js | `npx`, `npm`, `node` | | Python | `uvx`, `uv`, `python`, `python3`, `pip`, `pipx` | | Docker | `docker` | | Serveurs MCP | `mcp-server-*` (serveurs officiels) | ### Motifs bloqués Les motifs suivants sont bloqués pour prévenir les attaques par injection : - Enchaînement de commandes : `;`, `&&`, `||`, `|` - Substitution de commandes : `` ` ``, `$()` - Traversée de répertoires : `../` - Variables d'environnement dangereuses : `LD_PRELOAD`, `PATH`, `PYTHONPATH` ### Exemple : attaque bloquée ```json { "mcpServers": { "malicious": { "command": "bash", "args": ["-c", "rm -rf /"] } } } ``` Cette configuration sera rejetée : ``` ValueError: MCP server 'malicious': Command 'bash' is not in the allowed commands whitelist. ``` ### Mode développement (non sécurisé) Pour le développement uniquement, il est possible de contourner la validation de sécurité : ```json { "mcpServers": { "custom": { "command": "my-custom-server", "skip_security_validation": true } } } ``` **AVERTISSEMENT** : N'utilisez jamais `skip_security_validation` en production ! ### Liste blanche personnalisée Pour ajouter des commandes personnalisées à la liste blanche par programmation : ```python from vllm_mlx.mcp import MCPCommandValidator, set_validator # Add custom commands validator = MCPCommandValidator( custom_whitelist={"my-trusted-server", "another-server"} ) set_validator(validator) ``` ## Sandboxing de l'exécution des outils Au-delà de la validation des commandes, vllm-mlx fournit un sandboxing à l'exécution pour les exécutions d'outils. ### Fonctionnalités du sandbox | Fonctionnalité | Description | |----------------|-------------| | Liste blanche d'outils | Autoriser uniquement des outils spécifiques à s'exécuter | | Liste noire d'outils | Bloquer des outils dangereux spécifiques | | Validation des arguments | Bloquer les motifs dangereux dans les arguments des outils | | Limitation du débit | Limiter les appels d'outils par minute | | Journal d'audit | Suivre toutes les exécutions d'outils | ### Motifs d'arguments bloqués Les arguments des outils sont validés pour détecter les motifs dangereux : - Traversée de répertoires : `../` - Répertoires système : `/etc/`, `/proc/`, `/sys/` - Accès root : `/root/`, `~root` ### Détection des outils à haut risque Les outils correspondant à ces motifs déclenchent des avertissements de sécurité : - `execute`, `run_command`, `shell`, `eval`, `exec`, `system`, `subprocess` ### Configuration personnalisée du sandbox ```python from vllm_mlx.mcp import ToolSandbox, set_sandbox # Create sandbox with custom settings sandbox = ToolSandbox( # Only allow specific tools (whitelist mode) allowed_tools={"read_file", "list_directory"}, # Block specific tools (blacklist mode) blocked_tools={"execute_command", "run_shell"}, # Rate limit: max 30 calls per minute max_calls_per_minute=30, # Optional audit callback audit_callback=lambda audit: print(f"Tool: {audit.tool_name}, Success: {audit.success}"), ) set_sandbox(sandbox) ``` ### Accès aux journaux d'audit ```python from vllm_mlx.mcp import get_sandbox sandbox = get_sandbox() # Get recent audit entries entries = sandbox.get_audit_log(limit=50) # Filter by tool name file_ops = sandbox.get_audit_log(tool_filter="file") # Get only errors errors = sandbox.get_audit_log(errors_only=True) # Clear audit log sandbox.clear_audit_log() ``` ### Expurgation des données sensibles Les journaux d'audit expurgent automatiquement les champs sensibles (password, token, secret, key, credential, auth) et tronquent les valeurs volumineuses. ## Dépannage ### Le serveur MCP ne se connecte pas Vérifiez que la commande du serveur MCP est correcte : ```bash npx -y @modelcontextprotocol/server-filesystem /tmp ``` ### L'outil ne s'exécute pas Vérifiez que l'outil est disponible : ```bash curl http://localhost:8000/v1/mcp/tools | jq '.tools[].name' ``` ### L'appel d'outil n'est pas analysé Assurez-vous d'utiliser un modèle qui prend en charge le function calling (Qwen3, Llama-3.2-Instruct). ### La commande n'est pas dans la liste blanche Si vous voyez « Command X is not in the allowed commands whitelist », vous pouvez : 1. Utiliser une commande autorisée (voir la liste blanche ci-dessus) 2. Ajouter la commande à une liste blanche personnalisée 3. Utiliser `skip_security_validation: true` (développement uniquement) # Documentation page: `fr/guides/moe-top-k.md` # MoE top_k override (`--moe-top-k`) Réduit le nombre d'experts activés par token dans les modèles Mixture of Experts comme Qwen3-30B-A3B, en échangeant une légère perte de qualité contre un gain sensible de débit au décodage. > **Statut :** option à activer explicitement. Le comportement par défaut est inchangé. Les chiffres de qualité > ci-dessous concernent Qwen3-30B-A3B-4bit sur M4 Max 128 Go ; vérifiez sur votre modèle > avant de déployer en production. ## Ce que ça fait Qwen3-30B-A3B est entraîné avec `top_k=8` : chaque token sélectionne 8 experts parmi 128. Sur Apple Silicon en décodage batch=1, le produit matriciel des experts (`SwitchGLU`) représente la plus grande part du calcul de chaque couche, et ce coût évolue approximativement de façon linéaire avec `top_k`. Abaisser `top_k` à l'inférence a été démontré (LExI 2025, Lynx 2024) comme préservant l'essentiel de la qualité entraînée tout en réduisant significativement le temps de décodage. `--moe-top-k N` parcourt toutes les couches du modèle chargé et, pour chaque couche qui possède `.mlp.switch_mlp` (c'est-à-dire un bloc sparse-MoE), définit `top_k = N`. Les couches denses et les modèles denses ne sont pas modifiés ; le flag n'a aucun effet pour eux. ## Utilisation ```bash # Server vllm-mlx serve mlx-community/Qwen3-30B-A3B-4bit \ --continuous-batching \ --moe-top-k 4 # Bench vllm-mlx bench mlx-community/Qwen3-30B-A3B-4bit --moe-top-k 4 ``` Le flag est rejeté si `N` est supérieur au `top_k` d'entraînement du modèle (il ne peut que diminuer, jamais augmenter). ## Impact mesuré ### Débit de décodage (M4 Max 128 Go, batch=1, greedy) | top_k | tok/s | vs baseline | |---:|---:|---:| | 8 (baseline) | 126.5 | - | | 6 | 136.1 | +7.6% | | 5 | 140.3 | +10.9% | | 4 | 147.3 | +16.5% | ### Qualité (Qwen3-30B-A3B-4bit, lm-evaluation-harness, MLX backend) | top_k | MMLU (acc) | GSM8K (exact match) | Delta vs baseline | |---:|---:|---:|---:| | 8 | TBD | TBD | - | | 6 | TBD | TBD | TBD | | 5 | TBD | TBD | TBD | | 4 | TBD | TBD | TBD | MMLU : 200 échantillons sélectionnés aléatoirement, 0-shot. GSM8K : 100 échantillons sélectionnés aléatoirement, 0-shot, exact-match strict. Ces chiffres sont **indicatifs** ; les suites complètes sont plus grandes et feraient légèrement varier la précision absolue, mais pas le delta relatif entre configurations. ### Parité des sorties greedy Avec `top_k=4` sur le checkpoint 4-bit, nous avons observé des **16 premiers tokens générés identiques** par rapport à la baseline sur toutes les requêtes de test. Cela suggère que top_k=4 ne modifie pas l'argmax dans les premières étapes de décodage : le modèle est intrinsèquement robuste à la suppression de la moitié de ses experts activés. À `top_k=3` ou moins, la qualité commencerait à se dégrader visiblement (non mesuré ici ; déduit du papier LExI). Le flag ne peut donc pas descendre en dessous de 1 au niveau de la validation de configuration, mais le seuil recommandé pour la production est `top_k=4`. ## Quand l'utiliser, quand ne pas l'utiliser Utilisez-le quand : - Vous faites tourner un MoE Qwen3 (ou compatible : Qwen3.5 MoE, Gemma-MoE) et le débit de décodage en usage single-user est votre goulot d'étranglement. - Votre cas d'usage tolère une légère dégradation de qualité en échange d'une amélioration visible de la latence. - Vous déployez sur du matériel limité par la bande passante mémoire (Apple Silicon série M) où le gather des experts domine le temps de décodage par étape. Ne l'utilisez pas quand : - Vous servez des modèles denses : le flag n'a aucun effet. - La précision maximale sur les suites d'évaluation est une exigence. - Vous exécutez des générations longues en chaîne de pensée (mode "thinking") où la chute de qualité peut être plus prononcée que ce que suggèrent les scores MMLU 0-shot. ## Combinaison avec d'autres optimisations Ce flag se compose avec la quantification. Sur Qwen3-30B-A3B-4bit, nos mesures de combinaison sont : - 4-bit + top_k=8 : 126.5 tok/s (baseline) - 4-bit + top_k=4 : 147.3 tok/s (+16.5%) - 3-bit + top_k=8 : 138.6 tok/s (+9.6%) - 3-bit + top_k=6 : 147.1 tok/s (+16.3%) . divergence de qualité mesurable - 3-bit + top_k=4 : 157.3 tok/s (+24%) . **la qualité des sorties s'effondre** (le modèle a répondu à une question différente lors de notre test de fumée) 3-bit + top_k=4 cumule l'erreur numérique au point où l'argmax n'est plus stable. Limitez-vous à un seul réglage agressif à la fois : soit 4-bit + top_k=4, soit 3-bit + top_k=6. Les deux donnent approximativement le même tok/s (environ 147) avec des profils de qualité très différents. ## Fonctionnement interne - Fonction de patch : `vllm_mlx.scheduler.apply_moe_top_k_override(model, k)` - Appliquée dans `Scheduler.__init__` après le chargement du modèle. - Tests : `tests/test_moe_top_k.py`. couvre les modèles denses, les architectures mixtes et les chemins de validation. ## Références - LExI : Layer-Adaptive Active Experts, [arXiv 2509.02753](https://arxiv.org/html/2509.02753) - Not All Experts are Equal (NAEE), [ACL 2024](https://aclanthology.org/2024.acl-long.334.pdf) - SwiftLM (`SWIFTLM_TOP_K` env knob prior art), [github.com/SharpAI/SwiftLM](https://github.com/SharpAI/SwiftLM) # Documentation page: `fr/guides/multimodal.md` # Modèles multimodaux (images et vidéos) vllm-mlx prend en charge les VLM pour la compréhension des images et des vidéos. ## Modèles pris en charge - Qwen3-VL (recommandé) - Qwen2-VL - Gemma 3 - LLaVA - Idefics - PaliGemma - Pixtral - Molmo - DeepSeek-VL ## Démarrer un serveur multimodal ```bash vllm-mlx serve mlx-community/Qwen3-VL-4B-Instruct-3bit --port 8000 ``` Les modèles dont le nom contient « VL », « Vision » ou « mllm » sont automatiquement détectés comme multimodaux. ## Analyse d'images ### Via le SDK OpenAI ```python from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed") # Image depuis une URL response = client.chat.completions.create( model="default", messages=[{ "role": "user", "content": [ {"type": "text", "text": "What's in this image?"}, {"type": "image_url", "image_url": {"url": "https://example.com/image.jpg"}} ] }], max_tokens=256 ) print(response.choices[0].message.content) ``` ### Images en Base64 ```python import base64 def encode_image(path): with open(path, "rb") as f: return base64.b64encode(f.read()).decode("utf-8") base64_image = encode_image("photo.jpg") response = client.chat.completions.create( model="default", messages=[{ "role": "user", "content": [ {"type": "text", "text": "Describe this image"}, {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{base64_image}"}} ] }] ) ``` ### Via curl ```bash curl http://localhost:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "default", "messages": [{ "role": "user", "content": [ {"type": "text", "text": "What is in this image?"}, {"type": "image_url", "image_url": {"url": "https://example.com/image.jpg"}} ] }], "max_tokens": 256 }' ``` ## Analyse de vidéos ### Via le SDK OpenAI ```python response = client.chat.completions.create( model="default", messages=[{ "role": "user", "content": [ {"type": "text", "text": "What happens in this video?"}, {"type": "video_url", "video_url": {"url": "https://example.com/video.mp4"}} ] }], max_tokens=512 ) ``` ### Paramètres vidéo Contrôlez l'extraction des images via les paramètres du corps étendu : ```python response = client.chat.completions.create( model="default", messages=[{ "role": "user", "content": [ {"type": "text", "text": "Describe this video"}, {"type": "video_url", "video_url": {"url": "video.mp4"}} ] }], extra_body={ "video_fps": 2.0, "video_max_frames": 32 } ) ``` ### Via curl ```bash curl http://localhost:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "default", "messages": [{ "role": "user", "content": [ {"type": "text", "text": "Describe this video"}, {"type": "video_url", "video_url": {"url": "https://example.com/video.mp4"}} ] }], "video_fps": 2.0, "video_max_frames": 16 }' ``` ## Formats pris en charge ### Images | Format | Exemple | |--------|---------| | URL | `{"type": "image_url", "image_url": {"url": "https://..."}}` | | Fichier local | `{"type": "image_url", "image_url": {"url": "/path/to/image.jpg"}}` | | Base64 | `{"type": "image_url", "image_url": {"url": "data:image/jpeg;base64,..."}}` | ### Vidéos | Format | Exemple | |--------|---------| | URL | `{"type": "video_url", "video_url": {"url": "https://..."}}` | | Fichier local | `{"type": "video", "video": "/path/to/video.mp4"}` | | Base64 | `{"type": "video_url", "video_url": {"url": "data:video/mp4;base64,..."}}` | ## API Python ```python from vllm_mlx.models import MLXMultimodalLM mllm = MLXMultimodalLM("mlx-community/Qwen3-VL-4B-Instruct-3bit") mllm.load() # Image description = mllm.describe_image("photo.jpg") # Vidéo description = mllm.describe_video("video.mp4", fps=2.0) # Prompt personnalisé output = mllm.generate( prompt="Compare these images", images=["img1.jpg", "img2.jpg"] ) ``` ## Conseils de performance ### Images - Les résolutions plus petites sont traitées plus rapidement (224x224 vs 1920x1080) - Utilisez la résolution adaptée à votre tâche ### Vidéos - Un FPS plus bas accélère le traitement - Moins d'images signifie moins de mémoire utilisée - 64 images est le maximum pratique (96 et plus provoque un timeout GPU) ## Benchmarks Testés sur Apple M4 Max avec 128 Go de mémoire unifiée. ### Qwen3-VL-4B-Instruct-3bit | Résolution | Temps | Tokens | Vitesse | Mémoire | |------------|-------|--------|---------|---------| | 224x224 | 0.87s | 124 | 143 tok/s | 2.6 Go | | 448x448 | 1.01s | 107 | 106 tok/s | 3.1 Go | | 768x768 | 1.42s | 127 | 89 tok/s | 3.4 Go | | 1024x1024 | 1.85s | 116 | 63 tok/s | 3.6 Go | ### Qwen3-VL-8B-Instruct-4bit | Résolution | Temps | Tokens | Vitesse | Mémoire | |------------|-------|--------|---------|---------| | 224x224 | 1.08s | 78 | 73 tok/s | 5.6 Go | | 448x448 | 1.41s | 70 | 50 tok/s | 6.1 Go | | 768x768 | 2.06s | 91 | 44 tok/s | 6.5 Go | | 1024x1024 | 3.02s | 76 | 25 tok/s | 7.6 Go | ### Gemma 3 4B 4bit | Résolution | Temps | Tokens | Vitesse | Mémoire | |------------|-------|--------|---------|---------| | 224x224 | 0.95s | 30 | 32 tok/s | 5.2 Go | | 448x448 | 0.99s | 34 | 34 tok/s | 5.2 Go | | 768x768 | 0.99s | 32 | 32 tok/s | 5.2 Go | | 1024x1024 | 0.95s | 28 | 29 tok/s | 5.2 Go | ### Lancer les benchmarks ```bash # Benchmark rapide vllm-mlx-bench --model mlx-community/Qwen3-VL-4B-Instruct-3bit --quick # Benchmark complet avec plus de résolutions vllm-mlx-bench --model mlx-community/Qwen3-VL-4B-Instruct-3bit # Benchmark vidéo vllm-mlx-bench --model mlx-community/Qwen3-VL-4B-Instruct-3bit --video ``` ## Cache MLLM vllm-mlx inclut un système de prefix cache pour les modèles multimodaux, capable d'accélérer significativement les requêtes répétées utilisant les mêmes images. ### Fonctionnement Lorsque vous envoyez une image au modèle, l'encodeur de vision la traite en embeddings. Ce traitement prend 1 à 2 secondes. Le cache MLLM stocke ces embeddings ainsi que l'état du KV cache, de sorte que les requêtes ultérieures avec la même image contournent entièrement l'encodeur de vision. Le cache utilise un hachage basé sur le contenu (similaire à LMCache) pour identifier les images identiques, quelle que soit leur forme de transmission (URL, base64 ou chemin de fichier). ### Activer le cache ```bash # Activer avec les paramètres par défaut (512 Mo maximum) vllm-mlx serve mlx-community/Qwen3-VL-4B-Instruct-3bit --enable-mllm-cache # Avec une limite mémoire personnalisée vllm-mlx serve mlx-community/Qwen3-VL-4B-Instruct-3bit \ --enable-mllm-cache \ --mllm-cache-max-mb 1024 ``` ### API Python ```python from vllm_mlx.mllm_cache import MLLMPrefixCacheManager # Créer le gestionnaire de cache cache = MLLMPrefixCacheManager(max_memory_mb=512) # Stocker les embeddings et le KV cache après traitement cache.store( images=["photo.jpg"], prompt="Describe this image", vision_embeddings=embeddings, kv_cache=kv_state, num_tokens=128 ) # Récupérer depuis le cache lors des requêtes suivantes entry, match_len = cache.fetch(images=["photo.jpg"], prompt="Describe this image") if entry: # Utiliser les embeddings mis en cache, contourner l'encodeur de vision embeddings = entry.vision_embeddings kv_state = entry.kv_cache ``` ### Statistiques du cache ```python stats = cache.get_stats() print(f"Hit rate: {stats.hit_rate:.1%}") print(f"Memory used: {stats.memory_used_mb:.1f} MB") print(f"Tokens saved: {stats.tokens_saved}") ``` ### Gestion de la mémoire Le cache utilise une éviction LRU (Least Recently Used) lorsque la limite mémoire est atteinte. Chaque entrée suit : - La taille des embeddings de vision - La taille du KV cache par couche - La fréquence d'accès pour l'ordonnancement LRU En cas de pression mémoire, les entrées les moins récemment consultées sont évincées en premier. ## Interface de chat Gradio Pour un chat multimodal interactif : ```bash vllm-mlx-chat --served-model-name mlx-community/Qwen3-VL-4B-Instruct-3bit ``` Prend en charge le glisser-déposer d'images et de vidéos. # Documentation page: `fr/guides/python-api.md` # Python API API Python directe pour un accès programmatique à vllm-mlx. ## Modèles de langage ### Utilisation de base ```python from vllm_mlx.models import MLXLanguageModel # Load model model = MLXLanguageModel("mlx-community/Llama-3.2-3B-Instruct-4bit") model.load() # Generate text output = model.generate("What is the capital of France?", max_tokens=100) print(output.text) ``` ### Génération en streaming ```python for chunk in model.stream_generate("Tell me a story about a robot"): print(chunk.text, end="", flush=True) ``` ### Interface de chat ```python messages = [ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Hello, who are you?"} ] response = model.chat(messages) print(response.text) ``` ### Paramètres de génération ```python output = model.generate( prompt="Write a poem", max_tokens=256, temperature=0.7, top_p=0.9, stop=["END", "\n\n"] ) ``` | Paramètre | Description | Défaut | |-----------|-------------|--------| | `max_tokens` | Nombre maximum de tokens à générer | 256 | | `temperature` | Température d'échantillonnage (0-2) | 0.7 | | `top_p` | Nucleus sampling | 0.9 | | `stop` | Séquences d'arrêt | None | ## Modèles vision-langage ### Utilisation de base ```python from vllm_mlx.models import MLXMultimodalLM # Load model mllm = MLXMultimodalLM("mlx-community/Qwen3-VL-4B-Instruct-3bit") mllm.load() # Describe an image description = mllm.describe_image("photo.jpg") print(description) ``` ### Questions-réponses sur une image ```python answer = mllm.answer_about_image("photo.jpg", "What color is the car?") print(answer) ``` ### Plusieurs images ```python output = mllm.generate( prompt="Compare these two images", images=["image1.jpg", "image2.jpg"] ) print(output.text) ``` ### Compréhension vidéo ```python # From local file output = mllm.generate( prompt="What is happening in this video?", videos=["video.mp4"], video_fps=2.0, video_max_frames=16 ) print(output.text) # From URL output = mllm.generate( prompt="Describe this video", videos=["https://example.com/video.mp4"], video_fps=2.0 ) # Convenience method description = mllm.describe_video("video.mp4", fps=2.0) ``` ### Paramètres vidéo | Paramètre | Description | Défaut | |-----------|-------------|--------| | `video_fps` | Images par seconde à extraire | 2.0 | | `video_max_frames` | Nombre maximum d'images à traiter | 32 | ## API du moteur Pour les cas d'utilisation avancés, utilisez le moteur directement : ### Moteur simple ```python from vllm_mlx.engine import SimpleEngine engine = SimpleEngine("mlx-community/Llama-3.2-3B-Instruct-4bit") await engine.start() output = await engine.generate( prompt="Hello world", max_tokens=100 ) print(output.text) await engine.stop() ``` ### Moteur avec batching ```python from vllm_mlx.engine import BatchedEngine engine = BatchedEngine("mlx-community/Llama-3.2-3B-Instruct-4bit") await engine.start() # Multiple concurrent requests output = await engine.generate( prompt="Hello world", max_tokens=100 ) await engine.stop() ``` ## Format de sortie Toutes les méthodes de génération retournent un objet `GenerationOutput` : ```python output = model.generate("Hello") print(output.text) # Generated text print(output.prompt_tokens) # Input token count print(output.completion_tokens) # Output token count print(output.finish_reason) # "stop" or "length" ``` ## Gestion des erreurs ```python from vllm_mlx.models import MLXLanguageModel try: model = MLXLanguageModel("invalid-model") model.load() except Exception as e: print(f"Failed to load model: {e}") ``` # Documentation page: `fr/guides/reasoning.md` # Reasoning Models vllm-mlx prend en charge les reasoning models qui affichent leur processus de thinking avant de fournir une réponse. Des modèles comme Qwen3 et DeepSeek-R1 encapsulent leur reasoning dans des balises `...`, et vllm-mlx peut analyser ces balises pour séparer le reasoning de la réponse finale. ## Pourquoi utiliser le reasoning parsing ? Lorsqu'un reasoning model génère une sortie, elle ressemble généralement à ceci : ``` Let me analyze this step by step. First, I need to consider the constraints. The answer should be a prime number less than 10. Checking: 2, 3, 5, 7 are all prime and less than 10. The prime numbers less than 10 are: 2, 3, 5, 7. ``` Sans reasoning parsing, vous obtenez la sortie brute avec les balises incluses. Avec le reasoning parsing activé, le processus de thinking et la réponse finale sont séparés dans des champs distincts de la réponse de l'API. ## Démarrage rapide ### Démarrer le serveur avec un reasoning parser ```bash # For Qwen3 models vllm-mlx serve mlx-community/Qwen3-8B-4bit --reasoning-parser qwen3 # For DeepSeek-R1 models vllm-mlx serve mlx-community/DeepSeek-R1-Distill-Qwen-7B-4bit --reasoning-parser deepseek_r1 ``` ### Format de réponse de l'API Lorsque le reasoning parsing est activé, la réponse de l'API inclut un champ `reasoning` : **Réponse sans streaming :** ```json { "choices": [{ "message": { "role": "assistant", "content": "The prime numbers less than 10 are: 2, 3, 5, 7.", "reasoning": "Let me analyze this step by step.\nFirst, I need to consider the constraints.\nThe answer should be a prime number less than 10.\nChecking: 2, 3, 5, 7 are all prime and less than 10." } }] } ``` **Réponse en streaming :** Les fragments sont envoyés séparément pour le reasoning et le contenu. Pendant la phase de reasoning, les fragments ont le champ `reasoning` renseigné. Lorsque le modèle passe à la réponse finale, les fragments ont le champ `content` renseigné : ```json {"delta": {"reasoning": "Let me analyze"}} {"delta": {"reasoning": " this step by step."}} {"delta": {"reasoning": "\nFirst, I need to"}} ... {"delta": {"content": "The prime"}} {"delta": {"content": " numbers less than 10"}} {"delta": {"content": " are: 2, 3, 5, 7."}} ``` ## Utilisation avec le SDK OpenAI ```python from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed") # Non-streaming response = client.chat.completions.create( model="default", messages=[{"role": "user", "content": "What are the prime numbers less than 10?"}] ) message = response.choices[0].message print("Reasoning:", message.reasoning) # The thinking process print("Answer:", message.content) # The final answer ``` ### Streaming avec Reasoning ```python reasoning_text = "" content_text = "" stream = client.chat.completions.create( model="default", messages=[{"role": "user", "content": "Solve: 2 + 2 = ?"}], stream=True ) for chunk in stream: delta = chunk.choices[0].delta if hasattr(delta, 'reasoning') and delta.reasoning: reasoning_text += delta.reasoning print(f"[Thinking] {delta.reasoning}", end="") if delta.content: content_text += delta.content print(delta.content, end="") print(f"\n\nFinal reasoning: {reasoning_text}") print(f"Final answer: {content_text}") ``` ## Parsers disponibles ### Parser Qwen3 (`qwen3`) Pour les modèles Qwen3 qui utilisent explicitement les balises `` et ``. - Nécessite **les deux** balises ouvrante et fermante - Si les balises sont absentes, la sortie est traitée comme du contenu ordinaire - Recommandé pour : Qwen3-0.6B, Qwen3-4B, Qwen3-8B et les modèles similaires ```bash vllm-mlx serve mlx-community/Qwen3-8B-4bit --reasoning-parser qwen3 ``` ### Parser DeepSeek-R1 (`deepseek_r1`) Pour les modèles DeepSeek-R1 qui peuvent omettre la balise ouvrante ``. - Plus permissif que le parser Qwen3 - Gère les cas où `` est implicite - Le contenu avant `` est traité comme du reasoning même en l'absence de `` ```bash vllm-mlx serve mlx-community/DeepSeek-R1-Distill-Qwen-7B-4bit --reasoning-parser deepseek_r1 ``` ## Fonctionnement Le reasoning parser utilise une détection textuelle pour identifier les balises de thinking dans la sortie du modèle. Pendant le streaming, il suit la position courante dans la sortie afin d'acheminer chaque token vers `reasoning` ou `content`. ``` Model Output: Step 1: analyze...The answer is 42. ├─────────────────────┤├─────────────────────┤ Parsed: │ reasoning ││ content │ └─────────────────────┘└─────────────────────┘ ``` L'analyse est sans état et s'appuie sur le texte accumulé pour déterminer le contexte, ce qui la rend robuste dans les scénarios de streaming où les tokens peuvent arriver en fragments arbitraires. ## Conseils pour de meilleurs résultats ### Rédaction des prompts Les reasoning models fonctionnent mieux lorsque vous encouragez une réflexion étape par étape : ```python messages = [ {"role": "system", "content": "Think through problems step by step before answering."}, {"role": "user", "content": "What is 17 × 23?"} ] ``` ### Gestion de l'absence de reasoning Certains prompts peuvent ne pas déclencher de reasoning. Dans ce cas, `reasoning` vaut `None` et toute la sortie va dans `content` : ```python message = response.choices[0].message if message.reasoning: print(f"Model's thought process: {message.reasoning}") print(f"Answer: {message.content}") ``` ### Température et reasoning Les températures basses tendent à produire des schémas de reasoning plus cohérents : ```python response = client.chat.completions.create( model="default", messages=[{"role": "user", "content": "Explain quantum entanglement"}], temperature=0.3 # More focused reasoning ) ``` ## Compatibilité ascendante Lorsque `--reasoning-parser` n'est pas spécifié, le serveur se comporte comme avant : - Les balises de thinking sont incluses dans le champ `content` - Aucun champ `reasoning` n'est ajouté aux réponses Cela garantit que les applications existantes continuent de fonctionner sans modification. ## Exemple : résolveur de problèmes mathématiques ```python from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed") def solve_math(problem: str) -> dict: """Solve a math problem and return reasoning + answer.""" response = client.chat.completions.create( model="default", messages=[ {"role": "system", "content": "You are a math tutor. Show your work."}, {"role": "user", "content": problem} ], temperature=0.2 ) message = response.choices[0].message return { "problem": problem, "work": message.reasoning, "answer": message.content } result = solve_math("If a train travels 120 km in 2 hours, what is its average speed?") print(f"Problem: {result['problem']}") print(f"\nWork shown:\n{result['work']}") print(f"\nFinal answer: {result['answer']}") ``` ## Exemples avec curl ### Sans streaming ```bash curl http://localhost:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "default", "messages": [{"role": "user", "content": "What is 15% of 80?"}] }' ``` ### Avec streaming ```bash curl http://localhost:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "default", "messages": [{"role": "user", "content": "What is 15% of 80?"}], "stream": true }' ``` ## Résolution de problèmes ### Champ reasoning absent de la réponse - Vérifiez que le serveur a bien été démarré avec `--reasoning-parser` - Vérifiez que le modèle utilise effectivement des balises de thinking (tous les prompts ne déclenchent pas le reasoning) ### Le reasoning apparaît dans le contenu - Le modèle n'utilise peut-être pas le format de balises attendu - Essayez un autre parser (`qwen3` ou `deepseek_r1`) ### Reasoning tronqué - Augmentez `--max-tokens` si le modèle atteint la limite de tokens en plein milieu de sa réflexion ## Voir aussi - [Modèles pris en charge](../reference/models.md) - Modèles qui prennent en charge le reasoning - [Configuration du serveur](server.md) - Toutes les options du serveur - [Référence CLI](../reference/cli.md) - Options de la ligne de commande # Documentation page: `fr/guides/server.md` # Serveur compatible OpenAI vllm-mlx fournit un serveur FastAPI avec une compatibilité complète avec l'API OpenAI. Par défaut, le serveur n'écoute que sur `127.0.0.1`. Utilisez `--host 0.0.0.0` uniquement si vous souhaitez délibérément l'exposer au-delà de la machine locale. ## Démarrage du serveur ### Mode simple (par défaut) Débit maximal pour un utilisateur unique : ```bash vllm-mlx serve mlx-community/Llama-3.2-3B-Instruct-4bit --port 8000 ``` ### Mode continuous batching Pour plusieurs utilisateurs simultanés : ```bash vllm-mlx serve mlx-community/Llama-3.2-3B-Instruct-4bit --port 8000 --continuous-batching ``` ### Avec paged cache Mise en cache efficace en mémoire pour la production : ```bash vllm-mlx serve mlx-community/Llama-3.2-3B-Instruct-4bit --port 8000 --continuous-batching --use-paged-cache ``` ## Options du serveur | Option | Description | Défaut | |--------|-------------|--------| | `--port` | Port du serveur | 8000 | | `--host` | Hôte du serveur | 127.0.0.1 | | `--api-key` | Clé API pour l'authentification | None | | `--rate-limit` | Requêtes par minute par client (0 = désactivé) | 0 | | `--timeout` | Délai d'expiration des requêtes en secondes | 300 | | `--enable-metrics` | Expose les métriques Prometheus sur `/metrics` | False | | `--continuous-batching` | Active le batching pour plusieurs utilisateurs | False | | `--use-paged-cache` | Active le paged KV cache | False | | `--cache-memory-mb` | Limite mémoire du cache en Mo | Auto | | `--cache-memory-percent` | Fraction de la RAM réservée au cache | 0.20 | | `--max-tokens` | Nombre maximal de tokens par défaut | 32768 | | `--max-request-tokens` | Valeur maximale de `max_tokens` acceptée des clients API | 32768 | | `--default-temperature` | Température par défaut si non spécifiée | None | | `--default-top-p` | Valeur top_p par défaut si non spécifiée | None | | `--stream-interval` | Tokens par fragment de streaming | 1 | | `--mcp-config` | Chemin vers le fichier de configuration MCP | None | | `--reasoning-parser` | Parser pour les modèles reasoning (`qwen3`, `deepseek_r1`) | None | | `--embedding-model` | Précharge un modèle d'embeddings au démarrage | None | | `--enable-auto-tool-choice` | Active le tool calling automatique | False | | `--tool-call-parser` | Parser de tool calling (voir [Tool Calling](tool-calling.md)) | None | ## Points de terminaison de l'API ### Chat Completions ```bash POST /v1/chat/completions ``` ```python from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed") # Non-streaming response = client.chat.completions.create( model="default", messages=[{"role": "user", "content": "Hello!"}], max_tokens=100 ) # Streaming stream = client.chat.completions.create( model="default", messages=[{"role": "user", "content": "Tell me a story"}], stream=True ) for chunk in stream: if chunk.choices[0].delta.content: print(chunk.choices[0].delta.content, end="") ``` ### Completions ```bash POST /v1/completions ``` ```python response = client.completions.create( model="default", prompt="The capital of France is", max_tokens=50 ) ``` ### Modèles ```bash GET /v1/models ``` Retourne les modèles disponibles. ### Embeddings ```bash POST /v1/embeddings ``` ```python response = client.embeddings.create( model="mlx-community/multilingual-e5-small-mlx", input="Hello world" ) print(response.data[0].embedding[:5]) # First 5 dimensions ``` Voir le [Guide des embeddings](embeddings.md) pour plus de détails. ### Vérification de l'état ```bash GET /health ``` Retourne l'état du serveur. ### Métriques ```bash GET /metrics ``` Point de terminaison de collecte Prometheus pour les métriques du serveur, du cache, du scheduler et des requêtes. Le point de terminaison est désactivé par défaut et s'active avec `--enable-metrics`. ```bash vllm-mlx serve mlx-community/Llama-3.2-3B-Instruct-4bit \ --enable-metrics ``` `/metrics` est intentionnellement non authentifié. Exposez-le uniquement sur un réseau de confiance ou derrière un reverse proxy ou un pare-feu qui limite les accès. ### API Anthropic Messages ```bash POST /v1/messages ``` Point de terminaison compatible Anthropic qui permet à des outils comme Claude Code et OpenCode de se connecter directement à vllm-mlx. En interne, il traduit les requêtes Anthropic au format OpenAI, exécute l'inférence via le moteur, puis convertit la réponse au format Anthropic. Fonctionnalités : - Réponses non-streaming et streaming (SSE) - Messages système (chaîne simple ou liste de blocs de contenu) - Conversations multi-tours avec messages utilisateur et assistant - Tool calling avec blocs de contenu `tool_use` et `tool_result` - Comptage de tokens pour le suivi du budget - Contenu multimodal (images via blocs `source`) - Détection de déconnexion client (retourne HTTP 499) - Filtrage automatique des tokens spéciaux dans la sortie en streaming #### Non-streaming ```python from anthropic import Anthropic client = Anthropic(base_url="http://localhost:8000", api_key="not-needed") response = client.messages.create( model="default", max_tokens=256, messages=[{"role": "user", "content": "Hello!"}] ) print(response.content[0].text) # Response includes: response.id, response.model, response.stop_reason, # response.usage.input_tokens, response.usage.output_tokens ``` #### Streaming Le streaming suit le protocole d'événements SSE d'Anthropic. Les événements sont émis dans cet ordre : `message_start` -> `content_block_start` -> `content_block_delta` (répété) -> `content_block_stop` -> `message_delta` -> `message_stop` ```python with client.messages.stream( model="default", max_tokens=256, messages=[{"role": "user", "content": "Tell me a story"}] ) as stream: for text in stream.text_stream: print(text, end="") ``` #### Messages système Les messages système peuvent être une chaîne simple ou une liste de blocs de contenu : ```python # Plain string response = client.messages.create( model="default", max_tokens=256, system="You are a helpful coding assistant.", messages=[{"role": "user", "content": "Write a hello world in Python"}] ) # List of content blocks response = client.messages.create( model="default", max_tokens=256, system=[ {"type": "text", "text": "You are a helpful assistant."}, {"type": "text", "text": "Be concise in your answers."}, ], messages=[{"role": "user", "content": "What is 2+2?"}] ) ``` #### Tool calling Définissez les outils avec `name`, `description` et `input_schema`. Le modèle retourne des blocs de contenu `tool_use` lorsqu'il souhaite appeler un outil. Renvoyez les résultats sous forme de blocs `tool_result`. ```python # Step 1: Send request with tools response = client.messages.create( model="default", max_tokens=1024, messages=[{"role": "user", "content": "What's the weather in Paris?"}], tools=[{ "name": "get_weather", "description": "Get weather for a city", "input_schema": { "type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"] } }] ) # Step 2: Check if model wants to use tools for block in response.content: if block.type == "tool_use": print(f"Tool: {block.name}, Input: {block.input}, ID: {block.id}") # response.stop_reason will be "tool_use" # Step 3: Send tool result back response = client.messages.create( model="default", max_tokens=1024, messages=[ {"role": "user", "content": "What's the weather in Paris?"}, {"role": "assistant", "content": response.content}, {"role": "user", "content": [ { "type": "tool_result", "tool_use_id": block.id, "content": "Sunny, 22C" } ]} ], tools=[{ "name": "get_weather", "description": "Get weather for a city", "input_schema": { "type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"] } }] ) print(response.content[0].text) # "The weather in Paris is sunny, 22C." ``` Modes de sélection d'outil : | `tool_choice` | Comportement | |---------------|--------------| | `{"type": "auto"}` | Le modèle décide d'appeler ou non des outils (par défaut) | | `{"type": "any"}` | Le modèle doit appeler au moins un outil | | `{"type": "tool", "name": "get_weather"}` | Le modèle doit appeler l'outil spécifié | | `{"type": "none"}` | Le modèle n'appellera aucun outil | #### Conversations multi-tours ```python messages = [ {"role": "user", "content": "My name is Alice."}, {"role": "assistant", "content": "Nice to meet you, Alice!"}, {"role": "user", "content": "What's my name?"}, ] response = client.messages.create( model="default", max_tokens=100, messages=messages ) ``` #### Comptage de tokens ```bash POST /v1/messages/count_tokens ``` Compte les tokens d'entrée d'une requête Anthropic en utilisant le tokenizer du modèle. Utile pour le suivi du budget avant d'envoyer une requête. Comptabilise les tokens des messages système, des messages de conversation, des entrées `tool_use`, du contenu `tool_result` et des définitions d'outils (name, description, input_schema). ```python import requests resp = requests.post("http://localhost:8000/v1/messages/count_tokens", json={ "model": "default", "messages": [{"role": "user", "content": "Hello, how are you?"}], "system": "You are helpful.", "tools": [{ "name": "search", "description": "Search the web", "input_schema": {"type": "object", "properties": {"q": {"type": "string"}}} }] }) print(resp.json()) # {"input_tokens": 42} ``` #### Exemples curl Non-streaming : ```bash curl http://localhost:8000/v1/messages \ -H "Content-Type: application/json" \ -d '{ "model": "default", "max_tokens": 256, "messages": [{"role": "user", "content": "Hello!"}] }' ``` Streaming : ```bash curl http://localhost:8000/v1/messages \ -H "Content-Type: application/json" \ -d '{ "model": "default", "max_tokens": 256, "stream": true, "messages": [{"role": "user", "content": "Tell me a joke"}] }' ``` Comptage de tokens : ```bash curl http://localhost:8000/v1/messages/count_tokens \ -H "Content-Type: application/json" \ -d '{ "model": "default", "messages": [{"role": "user", "content": "Hello!"}] }' # {"input_tokens": 12} ``` #### Champs de la requête | Champ | Type | Requis | Défaut | Description | |-------|------|--------|--------|-------------| | `model` | string | oui | - | Nom du modèle (utilisez `"default"` pour le modèle chargé) | | `messages` | list | oui | - | Messages de conversation avec `role` et `content` | | `max_tokens` | int | oui | - | Nombre maximal de tokens à générer | | `system` | string or list | non | null | Invite système (chaîne ou liste de blocs `{"type": "text", "text": "..."}`) | | `stream` | bool | non | false | Active le streaming SSE | | `temperature` | float | non | 0.7 | Température d'échantillonnage (0.0 = déterministe, 1.0 = créatif) | | `top_p` | float | non | 0.9 | Seuil d'échantillonnage nucleus | | `top_k` | int | non | null | Échantillonnage top-k | | `stop_sequences` | list | non | null | Séquences qui arrêtent la génération | | `tools` | list | non | null | Définitions d'outils avec `name`, `description`, `input_schema` | | `tool_choice` | dict | non | null | Mode de sélection d'outil (`auto`, `any`, `tool`, `none`) | | `metadata` | dict | non | null | Métadonnées arbitraires (transmises telles quelles, non utilisées par le serveur) | #### Format de réponse Réponse non-streaming : ```json { "id": "msg_abc123...", "type": "message", "role": "assistant", "model": "default", "content": [ {"type": "text", "text": "Hello! How can I help?"} ], "stop_reason": "end_turn", "stop_sequence": null, "usage": { "input_tokens": 12, "output_tokens": 8 } } ``` Lorsque des outils sont appelés, `content` inclut des blocs `tool_use` et `stop_reason` vaut `"tool_use"` : ```json { "content": [ {"type": "text", "text": "Let me check the weather."}, { "type": "tool_use", "id": "call_abc123", "name": "get_weather", "input": {"city": "Paris"} } ], "stop_reason": "tool_use" } ``` Raisons d'arrêt : | `stop_reason` | Signification | |---------------|---------------| | `end_turn` | Le modèle a terminé naturellement | | `tool_use` | Le modèle souhaite appeler un outil | | `max_tokens` | La limite `max_tokens` a été atteinte | #### Utilisation avec Claude Code Pointez Claude Code directement vers votre serveur vllm-mlx : ```bash # Start the server vllm-mlx serve mlx-community/Qwen3-Coder-Next-235B-A22B-4bit \ --continuous-batching \ --enable-auto-tool-choice \ --tool-call-parser hermes # In another terminal, configure Claude Code export ANTHROPIC_BASE_URL=http://localhost:8000 export ANTHROPIC_API_KEY=not-needed claude ``` ### État du serveur ```bash GET /v1/status ``` Point de terminaison de surveillance en temps réel qui retourne des statistiques globales du serveur et des détails par requête. Utile pour déboguer les performances, suivre l'efficacité du cache et surveiller la mémoire Metal GPU. ```bash curl -s http://localhost:8000/v1/status | python -m json.tool ``` Exemple de réponse : ```json { "status": "running", "model": "mlx-community/Qwen3-8B-4bit", "uptime_s": 342.5, "steps_executed": 1247, "num_running": 1, "num_waiting": 0, "total_requests_processed": 15, "total_prompt_tokens": 28450, "total_completion_tokens": 3200, "metal": { "active_memory_gb": 5.2, "peak_memory_gb": 8.1, "cache_memory_gb": 2.3 }, "cache": { "type": "memory_aware_cache", "entries": 5, "hit_rate": 0.87, "memory_mb": 2350 }, "requests": [ { "request_id": "req_abc123", "phase": "generation", "tokens_per_second": 45.2, "ttft_s": 0.8, "progress": 0.35, "cache_hit_type": "prefix", "cached_tokens": 1200, "generated_tokens": 85, "max_tokens": 256 } ] } ``` Champs de la réponse : | Champ | Description | |-------|-------------| | `status` | État du serveur : `running`, `stopped` ou `not_loaded` | | `model` | Nom du modèle chargé | | `uptime_s` | Secondes écoulées depuis le démarrage du serveur | | `steps_executed` | Nombre total d'étapes d'inférence exécutées | | `num_running` | Nombre de requêtes en cours de génération de tokens | | `num_waiting` | Nombre de requêtes en attente de prefill | | `total_requests_processed` | Total des requêtes traitées depuis le démarrage | | `total_prompt_tokens` | Total des tokens de prompt traités depuis le démarrage | | `total_completion_tokens` | Total des tokens de complétion générés depuis le démarrage | | `metal.active_memory_gb` | Mémoire Metal GPU actuellement utilisée (Go) | | `metal.peak_memory_gb` | Pic d'utilisation de la mémoire Metal GPU (Go) | | `metal.cache_memory_gb` | Utilisation de la mémoire cache Metal (Go) | | `cache` | Statistiques du cache (type, entrées, taux de hit, utilisation mémoire) | | `requests` | Liste des requêtes actives avec détails par requête | Champs par requête dans `requests` : | Champ | Description | |-------|-------------| | `request_id` | Identifiant unique de la requête | | `phase` | Phase actuelle : `queued`, `prefill` ou `generation` | | `tokens_per_second` | Débit de génération pour cette requête | | `ttft_s` | TTFT (secondes) | | `progress` | Pourcentage de complétion (0.0 à 1.0) | | `cache_hit_type` | Type de correspondance dans le cache : `exact`, `prefix`, `supersequence`, `lcp` ou `miss` | | `cached_tokens` | Nombre de tokens servis depuis le cache | | `generated_tokens` | Tokens générés jusqu'à présent | | `max_tokens` | Nombre maximal de tokens demandés | ## Tool Calling Activez le tool calling compatible OpenAI avec `--enable-auto-tool-choice` : ```bash vllm-mlx serve mlx-community/Devstral-Small-2507-4bit \ --enable-auto-tool-choice \ --tool-call-parser mistral ``` Utilisez l'option `--tool-call-parser` pour sélectionner le parser adapté à votre modèle : | Parser | Modèles | |--------|---------| | `auto` | Détection automatique (essaie tous les parsers) | | `mistral` | Mistral, Devstral | | `qwen` | Qwen, Qwen3 | | `llama` | Llama 3.x, 4.x | | `hermes` | Hermes, NousResearch | | `deepseek` | DeepSeek V3, R1 | | `kimi` | Kimi K2, Moonshot | | `granite` | IBM Granite 3.x, 4.x | | `nemotron` | NVIDIA Nemotron | | `xlam` | Salesforce xLAM | | `functionary` | MeetKai Functionary | | `glm47` | GLM-4.7, GLM-4.7-Flash | ```python response = client.chat.completions.create( model="default", messages=[{"role": "user", "content": "What's the weather in Paris?"}], tools=[{ "type": "function", "function": { "name": "get_weather", "description": "Get weather for a city", "parameters": { "type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"] } } }] ) if response.choices[0].message.tool_calls: for tc in response.choices[0].message.tool_calls: print(f"{tc.function.name}: {tc.function.arguments}") ``` Voir le [Guide du tool calling](tool-calling.md) pour la documentation complète. ## Modèles reasoning Pour les modèles qui exposent leur processus de réflexion (Qwen3, DeepSeek-R1), utilisez `--reasoning-parser` pour séparer le reasoning de la réponse finale : ```bash # Qwen3 models vllm-mlx serve mlx-community/Qwen3-8B-4bit --reasoning-parser qwen3 # DeepSeek-R1 models vllm-mlx serve mlx-community/DeepSeek-R1-Distill-Qwen-7B-4bit --reasoning-parser deepseek_r1 ``` La réponse de l'API inclut un champ `reasoning` avec le processus de réflexion du modèle : ```python response = client.chat.completions.create( model="default", messages=[{"role": "user", "content": "What is 17 × 23?"}] ) print(response.choices[0].message.reasoning) # Step-by-step thinking print(response.choices[0].message.content) # Final answer ``` En streaming, les fragments de reasoning arrivent en premier, suivis des fragments de contenu : ```python for chunk in stream: delta = chunk.choices[0].delta if delta.reasoning: print(f"[Thinking] {delta.reasoning}") if delta.content: print(delta.content, end="") ``` Voir le [Guide des modèles reasoning](reasoning.md) pour tous les détails. ## Sortie structurée (mode JSON) Forcez le modèle à retourner du JSON valide en utilisant `response_format` : ### Mode JSON Object Retourne n'importe quel JSON valide : ```python response = client.chat.completions.create( model="default", messages=[{"role": "user", "content": "List 3 colors"}], response_format={"type": "json_object"} ) # Output: {"colors": ["red", "blue", "green"]} ``` ### Mode JSON Schema Retourne du JSON conforme à un schéma spécifique : ```python response = client.chat.completions.create( model="default", messages=[{"role": "user", "content": "List 3 colors"}], response_format={ "type": "json_schema", "json_schema": { "name": "colors", "schema": { "type": "object", "properties": { "colors": { "type": "array", "items": {"type": "string"} } }, "required": ["colors"] } } } ) # Output validated against schema data = json.loads(response.choices[0].message.content) assert "colors" in data ``` ### Exemple curl ```bash curl http://localhost:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "default", "messages": [{"role": "user", "content": "List 3 colors"}], "response_format": {"type": "json_object"} }' ``` ## Exemples curl ### Chat ```bash curl http://localhost:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "default", "messages": [{"role": "user", "content": "Hello!"}], "max_tokens": 100 }' ``` ### Streaming ```bash curl http://localhost:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "default", "messages": [{"role": "user", "content": "Hello!"}], "stream": true }' ``` ## Configuration du streaming Contrôlez le comportement du streaming avec `--stream-interval` : | Valeur | Comportement | |--------|--------------| | `1` (par défaut) | Envoie chaque token immédiatement | | `2-5` | Regroupe les tokens avant envoi | | `10+` | Débit maximal, sortie plus fragmentée | ```bash # Smooth streaming vllm-mlx serve model --continuous-batching --stream-interval 1 # Batched streaming (better for high-latency networks) vllm-mlx serve model --continuous-batching --stream-interval 5 ``` ## Intégration Open WebUI ```bash # 1. Start vllm-mlx server vllm-mlx serve mlx-community/Llama-3.2-3B-Instruct-4bit --port 8000 # 2. Start Open WebUI docker run -d -p 3000:8080 \ -e OPENAI_API_BASE_URL=http://host.docker.internal:8000/v1 \ -e OPENAI_API_KEY=not-needed \ --name open-webui \ ghcr.io/open-webui/open-webui:main # 3. Open http://localhost:3000 ``` ## Déploiement en production ### Avec systemd Créez `/etc/systemd/system/vllm-mlx.service` : ```ini [Unit] Description=vLLM-MLX Server After=network.target [Service] Type=simple ExecStart=/usr/local/bin/vllm-mlx serve mlx-community/Qwen3-0.6B-8bit \ --continuous-batching --use-paged-cache --port 8000 Restart=always [Install] WantedBy=multi-user.target ``` ```bash sudo systemctl enable vllm-mlx sudo systemctl start vllm-mlx ``` ### Paramètres recommandés Pour une production avec 50 utilisateurs simultanés ou plus : ```bash vllm-mlx serve mlx-community/Qwen3-0.6B-8bit \ --continuous-batching \ --use-paged-cache \ --api-key your-secret-key \ --rate-limit 60 \ --timeout 120 \ --port 8000 ``` # Documentation page: `fr/guides/tool-calling.md` # Tool Calling vllm-mlx prend en charge le tool calling compatible OpenAI (function calling) avec un parsing automatique pour de nombreuses familles de modèles populaires. ## Démarrage rapide Activez le tool calling en ajoutant le flag `--enable-auto-tool-choice` au démarrage du serveur : ```bash vllm-mlx serve mlx-community/Devstral-Small-2507-4bit \ --enable-auto-tool-choice \ --tool-call-parser mistral ``` Utilisez ensuite les outils avec l'API OpenAI standard : ```python from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed") response = client.chat.completions.create( model="default", messages=[{"role": "user", "content": "What's the weather in Paris?"}], tools=[{ "type": "function", "function": { "name": "get_weather", "description": "Get weather for a city", "parameters": { "type": "object", "properties": { "city": {"type": "string", "description": "City name"} }, "required": ["city"] } } }] ) # Check for tool calls if response.choices[0].message.tool_calls: for tc in response.choices[0].message.tool_calls: print(f"Function: {tc.function.name}") print(f"Arguments: {tc.function.arguments}") ``` ## Parsers disponibles Utilisez `--tool-call-parser` pour sélectionner un tool parser adapté à votre famille de modèles : | Parser | Alias | Modèles | Format | |--------|-------|---------|--------| | `auto` | | N'importe quel modèle | Détection automatique du format (essaie tous les parsers) | | `mistral` | | Mistral, Devstral | Tableau JSON `[TOOL_CALLS]` | | `qwen` | `qwen3` | Qwen, Qwen3 | XML `` ou `[Calling tool:]` | | `llama` | `llama3`, `llama4` | Llama 3.x, 4.x | Balises `` | | `hermes` | `nous` | Hermes, NousResearch | JSON `` dans XML | | `deepseek` | `deepseek_v3`, `deepseek_r1` | DeepSeek V3, R1 | Délimiteurs Unicode | | `kimi` | `kimi_k2`, `moonshot` | Kimi K2, Moonshot | Tokens `<\|tool_call_begin\|>` | | `granite` | `granite3` | IBM Granite 3.x, 4.x | `<\|tool_call\|>` ou `` | | `nemotron` | `nemotron3` | NVIDIA Nemotron | `` | | `xlam` | | Salesforce xLAM | JSON avec tableau `tool_calls` | | `functionary` | `meetkai` | MeetKai Functionary | Plusieurs blocs de fonctions | | `glm47` | `glm4` | GLM-4.7, GLM-4.7-Flash | `` avec XML ``/`` | ## Exemples par modèle ### Mistral / Devstral ```bash # Devstral Small (optimized for coding and tool use) vllm-mlx serve mlx-community/Devstral-Small-2507-4bit \ --enable-auto-tool-choice --tool-call-parser mistral # Mistral Instruct vllm-mlx serve mlx-community/Mistral-7B-Instruct-v0.3-4bit \ --enable-auto-tool-choice --tool-call-parser mistral ``` ### Qwen ```bash # Qwen3 vllm-mlx serve mlx-community/Qwen3-4B-4bit \ --enable-auto-tool-choice --tool-call-parser qwen ``` ### Llama ```bash # Llama 3.2 vllm-mlx serve mlx-community/Llama-3.2-3B-Instruct-4bit \ --enable-auto-tool-choice --tool-call-parser llama ``` ### DeepSeek ```bash # DeepSeek V3 vllm-mlx serve mlx-community/DeepSeek-V3-0324-4bit \ --enable-auto-tool-choice --tool-call-parser deepseek ``` ### IBM Granite ```bash # Granite 4.0 vllm-mlx serve mlx-community/granite-4.0-tiny-preview-4bit \ --enable-auto-tool-choice --tool-call-parser granite ``` ### NVIDIA Nemotron ```bash # Nemotron 3 Nano vllm-mlx serve mlx-community/NVIDIA-Nemotron-3-Nano-30B-A3B-MLX-6Bit \ --enable-auto-tool-choice --tool-call-parser nemotron ``` ### GLM-4.7 ```bash # GLM-4.7 Flash vllm-mlx serve lmstudio-community/GLM-4.7-Flash-MLX-8bit \ --enable-auto-tool-choice --tool-call-parser glm47 ``` ### Kimi K2 ```bash # Kimi K2 vllm-mlx serve mlx-community/Kimi-K2-Instruct-4bit \ --enable-auto-tool-choice --tool-call-parser kimi ``` ### Salesforce xLAM ```bash # xLAM vllm-mlx serve mlx-community/xLAM-2-fc-r-4bit \ --enable-auto-tool-choice --tool-call-parser xlam ``` ## Parser automatique Si vous n'êtes pas sûr du parser à utiliser, le parser `auto` tente de détecter le format automatiquement : ```bash vllm-mlx serve mlx-community/Qwen3-4B-4bit \ --enable-auto-tool-choice --tool-call-parser auto ``` Le parser automatique essaie les formats dans cet ordre : 1. Mistral (`[TOOL_CALLS]`) 2. Qwen bracket (`[Calling tool:]`) 3. Nemotron (``) 4. Qwen/Hermes XML (`{...}`) 5. Llama (`{...}`) 6. JSON brut ## Streaming tool calls Le tool calling fonctionne avec le streaming. Les informations du tool call sont envoyées lorsque le modèle a terminé de générer : ```python stream = client.chat.completions.create( model="default", messages=[{"role": "user", "content": "What's 25 * 17?"}], tools=[{ "type": "function", "function": { "name": "calculator", "description": "Calculate math expressions", "parameters": { "type": "object", "properties": { "expression": {"type": "string"} }, "required": ["expression"] } } }], stream=True ) for chunk in stream: if chunk.choices[0].delta.tool_calls: for tc in chunk.choices[0].delta.tool_calls: print(f"Tool call: {tc.function.name}({tc.function.arguments})") ``` ## Gestion des résultats de tool calls Après avoir reçu un tool call, exécutez la fonction et renvoyez le résultat : ```python import json # First request - model decides to call a tool response = client.chat.completions.create( model="default", messages=[{"role": "user", "content": "What's the weather in Tokyo?"}], tools=[weather_tool] ) # Get the tool call tool_call = response.choices[0].message.tool_calls[0] tool_call_id = tool_call.id function_name = tool_call.function.name arguments = json.loads(tool_call.function.arguments) # Execute the function (your implementation) result = get_weather(**arguments) # {"temperature": 22, "condition": "sunny"} # Send result back to model response = client.chat.completions.create( model="default", messages=[ {"role": "user", "content": "What's the weather in Tokyo?"}, {"role": "assistant", "tool_calls": [tool_call]}, {"role": "tool", "tool_call_id": tool_call_id, "content": json.dumps(result)} ], tools=[weather_tool] ) print(response.choices[0].message.content) # "The weather in Tokyo is sunny with a temperature of 22C." ``` ## Gestion des balises think Les modèles qui produisent des balises de reasoning `...` (comme DeepSeek-R1, Qwen3, GLM-4.7) sont gérés automatiquement. Le parser supprime le contenu de la réflexion avant d'extraire les tool calls, de sorte que les balises de reasoning n'interfèrent jamais avec le parsing des tool calls. Cela fonctionne même lorsque `` a été injecté dans le prompt (balises think implicites avec uniquement un `` fermant). ## Référence CLI | Option | Description | |--------|-------------| | `--enable-auto-tool-choice` | Active le tool calling automatique | | `--tool-call-parser` | Sélectionne le parser (voir tableau ci-dessus) | Voir [Référence CLI](../reference/cli.md) pour toutes les options. # Documentation page: `fr/guides/warm-prompts.md` # Warm Prompts Pré-remplissez le prefix cache au démarrage du serveur afin que la **première** requête envoyée par un agent trouve un cache déjà chaud, sans payer le coût complet du prefill pour son system prompt de plusieurs kilo-octets. ## Quand utiliser cette fonctionnalité Les charges de travail agent. proxies vers des assistants de code ou de raisonnement, serveurs MCP, orchestrateurs multi-agents. envoient toujours le même system prompt. Aujourd'hui, la première requête d'un serveur froid paie le prefill complet pour ce system prompt. Sur un modèle de plusieurs milliards de paramètres, cela représente plusieurs secondes de TTFT, précisément au moment où un utilisateur attend la première réponse de son nouvel agent. Si vous connaissez les system prompts de vos agents au moment du déploiement, écrivez-les dans un fichier JSON et pointez `--warm-prompts` dessus. Le serveur exécute une complétion de chat avec `max_tokens=1` pour chacun au démarrage, l'état KV cache est chargé dans le prefix cache, et la première vraie requête correspond via strict-prefix. Nécessite `--continuous-batching` (le prefix cache y est hébergé). ## Exemple rapide ```bash # Écrivez une seule fois les agents qui vous intéressent cat > ~/.config/vllm-mlx/agents.json <<'JSON' [ [{"role": "system", "content": "You are a code assistant..."}] ] JSON # Pointez le serveur dessus vllm-mlx serve mlx-community/Qwen3-4B-4bit \ --continuous-batching \ --warm-prompts ~/.config/vllm-mlx/agents.json ``` Au démarrage vous verrez : ``` [lifespan] Warm-up done (strict-prefix): 1 completed, 0 skipped, 1431 prompt tokens in 0.2s ``` La première vraie requête partageant le system prompt réchauffé atteint le cache avec `tokens_saved` proche de la longueur du prompt de warm-up. ## Format de fichier Une liste JSON de premier niveau. Chaque entrée est elle-même une liste de messages de chat, de même forme que `messages` dans `/v1/chat/completions`. ```json [ [ {"role": "system", "content": "You are a code assistant..."} ], [ {"role": "system", "content": "You are a senior code reviewer..."} ], [ {"role": "system", "content": "You are a planner..."}, {"role": "user", "content": "hi"}, {"role": "assistant", "content": "Hello, what are we planning?"} ] ] ``` Les system prompts à message unique sont le cas le plus courant. Les historiques multi-tours sont pris en charge pour les scénarios où vous souhaitez réchauffer un début de conversation spécifique (exemples few-shot, persona d'assistant récurrente). ## Dimensionnement Les warm prompts sont traités **en parallèle** via `asyncio.gather`, donc N entrées déclenchent N prefills simultanés au démarrage. Chaque prefill alloue du KV cache pour la longueur de son prompt. **Recommandation : 1 à 3 entrées.** Cela couvre les chemins critiques des déploiements agent typiques (une persona par entrée). Un fichier warm-prompts très grand sur un modèle à mémoire limitée peut épuiser la marge disponible au démarrage. Si vous devez réchauffer des dizaines de personas, ouvrez une issue en décrivant votre charge de travail et nous pourrons ajouter un paramètre `--warm-prompts-concurrency=N`. ## Benchmarks **Configuration.** M4 Max, 128 Go de mémoire unifiée. Deux serveurs séparés par mesure (froid et chaud), démarrage à froid isolé. Jeu de prompts `long` (~2 500 tokens utilisateur) précédé d'un system prompt d'environ 1 700 tokens correspondant au prompt de warm-up. `max_tokens=128`. bench-serve avec `--skip-preflight-token-count` afin que le preflight `count_prompt_tokens` ne pollue pas le cache. | Modèle | conc | TTFT froid | TTFT chaud | Accélération | |--------|-----:|-----------:|-----------:|-------------:| | Qwen3-0.6B-8bit | 1 | 563 ms | 419 ms | 1.34x | | Qwen3-0.6B-8bit | 4 | 1 723 ms | 1 282 ms | 1.34x | | Qwen3-0.6B-8bit | 8 | 3 708 ms | 2 661 ms | 1.39x | | Llama-3.2-3B-Instruct-4bit | 1 | 1 754 ms | 1 060 ms | 1.65x | | Llama-3.2-3B-Instruct-4bit | 4 | 5 926 ms | 3 945 ms | 1.50x | | Llama-3.2-3B-Instruct-4bit | 8 | 15 161 ms | 9 820 ms | 1.54x | | Qwen3-4B-4bit | 1 | 4 937 ms | 2 191 ms | 2.25x | | Qwen3-4B-4bit | 4 | 12 535 ms | 9 623 ms | 1.30x | | Qwen3-4B-4bit | 8 | 38 148 ms | 23 878 ms | 1.60x | | Qwen3.6-35B-A3B-4bit (MoE/hybrid) | 1 | 2 400 ms | 1 603 ms | 1.50x | | Qwen3.6-35B-A3B-4bit | 4 | 8 735 ms | 6 054 ms | 1.44x | | Qwen3.6-35B-A3B-4bit | 8 | 22 419 ms | 14 409 ms | 1.56x | Les 12 configurations s'améliorent toutes. Les gains de TTFT sont les plus importants quand le ratio prompt/total est le plus élevé (conc=1, long system prompt) et restent significatifs sous charge concurrente. **La génération tok/s** est neutre (dans ±5 %) pour les modèles denses. Qwen3.6-35B-A3B (MoE) affiche une baisse de décodage de 20 à 35 % à conc >= 4, qui semble liée à l'interaction du routage MoE avec la planification en batch. Les gains de TTFT dominent néanmoins la latence de bout en bout sur les charges agent, mais tenez-en compte si votre flux de travail est fortement limité par le décodage à forte concurrence. ## Fonctionnement interne Le warm-up naïf. rendu du template de chat avec un message utilisateur fictif et mise en cache des tokens. ne fonctionne pas pour les modèles hybrides SSM+attention (Qwen3.5-MoE, Qwen3.6-MoE). Leurs couches de cache incluent un état SSM qui ne peut pas être tronqué, si bien que `memory_cache.py` désactive la correspondance LCP. Le contenu utilisateur fictif diverge du vrai contenu utilisateur, et une entrée mise en cache au niveau des tokens n'est plus un strict prefix d'aucune vraie requête. Le warmer ici rend le template de chat **deux fois** avec deux contenus utilisateur distincts (`"__PROBE_A__"` et `"__PROBE_B__"`), trouve la position de caractère où les deux chaînes divergent, puis tronque le premier rendu à cette frontière. Cette chaîne tronquée. tout ce qui précède l'insertion du contenu utilisateur. est ce qui est envoyé au moteur. Comme le chemin de vraie requête du moteur rend aussi le template avec `tokenize=False` puis laisse le tokenizer encoder le résultat, les tokens du warm-up sont garantis d'être un strict prefix de toute vraie requête avec un system prompt correspondant et un historique de chat vide. Les correspondances strict-prefix fonctionnent sur tous les types de couche de cache, y compris les chemins hybrides où LCP est désactivé. ## Administration ### Vider le prefix cache en mémoire ```bash curl -X DELETE http://localhost:8000/v1/cache/prefix ``` Si le serveur a été démarré avec `--warm-prompts`, le warm-up se relance en arrière-plan après la suppression. La réponse est retournée immédiatement sans attendre la fin du re-warm. Réponse : ```json {"status": "cleared", "rewarm_scheduled": true} ``` ### Inspecter l'état du cache ```bash curl http://localhost:8000/v1/status | jq '.cache' ``` Après le démarrage avec warm-prompts, vous verrez `entry_count > 0` avant la première requête utilisateur. ## Mesurer l'impact sur votre configuration Pour mesurer l'impact sur votre modèle et vos prompts, utilisez `bench-serve` : ```bash # Froid : sans warm-prompts vllm-mlx serve MODEL --continuous-batching & vllm-mlx bench-serve --prompts long --concurrency 1,4 \ --system-prompt-file my-system.txt --tag cold \ --output cold.csv --format csv # Chaud : même configuration + --warm-prompts vllm-mlx serve MODEL --continuous-batching \ --warm-prompts ~/.config/vllm-mlx/agents.json & vllm-mlx bench-serve --prompts long --concurrency 1,4 \ --system-prompt-file my-system.txt --tag warm \ --output warm.csv --format csv ``` `--skip-preflight-token-count` est activé automatiquement quand `--system-prompt-file` est fourni, afin que le preflight `count_prompt_tokens` ne pollue pas le cache. Comparez `cold.csv` et `warm.csv` pour votre charge de travail. # Documentation page: `fr/index.md` # Documentation vLLM-MLX **Backend MLX pour Apple Silicon sur vLLM** - Accélération GPU pour le texte, les images, la vidéo et l'audio sur Mac ## Qu'est-ce que vLLM-MLX ? vllm-mlx apporte l'accélération GPU native d'Apple Silicon à vLLM en intégrant : - **[MLX](https://github.com/ml-explore/mlx)** : le framework ML d'Apple avec mémoire unifiée et noyaux Metal - **[mlx-lm](https://github.com/ml-explore/mlx-lm)** : inférence LLM optimisée avec KV cache et quantification - **[mlx-vlm](https://github.com/Blaizzy/mlx-vlm)** : modèles vision-langage (VLM) pour l'inférence multimodale - **[mlx-audio](https://github.com/Blaizzy/mlx-audio)** : Text-to-Speech et Speech-to-Text avec des voix natives - **[mlx-embeddings](https://github.com/Blaizzy/mlx-embeddings)** : embeddings textuels pour la recherche sémantique et le RAG ## Fonctionnalités principales - **Multimodal** - Texte, image, vidéo et audio sur une seule plateforme - **Accélération GPU native** sur Apple Silicon (M1, M2, M3, M4, M5) - **Voix TTS natives** - espagnol, français, chinois, japonais et 5 autres langues - **Compatible API OpenAI** - remplacement direct du client OpenAI - **Embeddings** - point de terminaison `/v1/embeddings` compatible OpenAI - **MCP Tool Calling** - intégration d'outils externes via le Model Context Protocol - **Paged KV Cache** - mise en cache efficace en mémoire avec partage de préfixe - **Continuous Batching** - débit élevé pour plusieurs utilisateurs simultanés ## Liens rapides ### Démarrage - [Installation](getting-started/installation.md) - [Démarrage rapide](getting-started/quickstart.md) ### Guides utilisateur - [Serveur compatible OpenAI](guides/server.md) - [API Python](guides/python-api.md) - [Multimodal (images et vidéo)](guides/multimodal.md) - [Audio (STT/TTS)](guides/audio.md) - [Embeddings](guides/embeddings.md) - [Modèles de reasoning](guides/reasoning.md) - [Tool Calling](guides/tool-calling.md) - [MCP et Tool Calling](guides/mcp-tools.md) - [Continuous Batching](guides/continuous-batching.md) ### Référence - [Commandes CLI](reference/cli.md) - [Modèles pris en charge](reference/models.md) - [Configuration](reference/configuration.md) ### Benchmarks - [Benchmarks LLM](benchmarks/llm.md) - [Benchmarks image](benchmarks/image.md) - [Benchmarks vidéo](benchmarks/video.md) - [Benchmarks audio](benchmarks/audio.md) ### Développement - [Architecture (en anglais)](/development/architecture/) - [Contribuer (en anglais)](/development/contributing/) ## Prérequis - macOS sur Apple Silicon (M1/M2/M3/M4/M5) - Python 3.10+ - 8 Go de RAM recommandés ## Licence Apache 2.0. Consultez la [licence du dépôt](https://github.com/waybarrios/vllm-mlx/blob/main/LICENSE). # Documentation page: `fr/reference/cli.md` # Référence CLI ## Vue d'ensemble des commandes | Commande | Description | |---------|-------------| | `vllm-mlx serve` | Démarrer le serveur compatible OpenAI | | `vllm-mlx-bench` | Exécuter des benchmarks de performance | | `vllm-mlx-chat` | Démarrer l'interface de chat Gradio | ## `vllm-mlx serve` Démarrer le serveur API compatible OpenAI. ### Utilisation ```bash vllm-mlx serve [options] ``` ### Options | Option | Description | Défaut | |--------|-------------|---------| | `--served-model-name` | Nom de modèle personnalisé exposé via l'API OpenAI. Si non défini, le chemin du modèle est utilisé comme nom. | None | | `--port` | Port du serveur | 8000 | | `--host` | Hôte du serveur | 127.0.0.1 | | `--api-key` | Clé API pour l'authentification | None | | `--rate-limit` | Requêtes par minute par client (0 = désactivé) | 0 | | `--timeout` | Délai d'attente des requêtes en secondes | 300 | | `--enable-metrics` | Exposer les métriques Prometheus sur `/metrics` | False | | `--continuous-batching` | Activer le continuous batching pour plusieurs utilisateurs | False | | `--cache-memory-mb` | Limite mémoire du cache en Mo | Auto | | `--cache-memory-percent` | Fraction de la RAM réservée au cache | 0.20 | | `--no-memory-aware-cache` | Utiliser le cache legacy basé sur le nombre d'entrées | False | | `--use-paged-cache` | Activer le KV cache paginé | False | | `--max-tokens` | Nombre maximum de tokens par défaut | 32768 | | `--max-request-tokens` | Valeur maximale de `max_tokens` acceptée depuis les clients API | 32768 | | `--stream-interval` | Tokens par fragment de streaming | 1 | | `--mcp-config` | Chemin vers le fichier de configuration MCP | None | | `--paged-cache-block-size` | Tokens par bloc de cache | 64 | | `--max-cache-blocks` | Nombre maximum de blocs de cache | 1000 | | `--max-num-seqs` | Nombre maximum de séquences simultanées | 256 | | `--default-temperature` | Température par défaut si non spécifiée dans la requête | None | | `--default-top-p` | Valeur top_p par défaut si non spécifiée dans la requête | None | | `--max-audio-upload-mb` | Taille maximale des fichiers audio téléversés pour `/v1/audio/transcriptions` | 25 | | `--max-tts-input-chars` | Longueur maximale du texte acceptée par `/v1/audio/speech` | 4096 | | `--reasoning-parser` | Analyseur pour les modèles de reasoning (`qwen3`, `deepseek_r1`) | None | | `--embedding-model` | Pré-charger un modèle d'embeddings au démarrage | None | | `--enable-auto-tool-choice` | Activer le tool calling automatique | False | | `--tool-call-parser` | Analyseur d'appels d'outils (`auto`, `mistral`, `qwen`, `llama`, `hermes`, `deepseek`, `kimi`, `granite`, `nemotron`, `xlam`, `functionary`, `glm47`) | None | ### Exemples ```bash # Simple mode (single user, max throughput) # Model path is used as the model name in the OpenAI API (e.g. model="mlx-community/Llama-3.2-3B-Instruct-4bit") vllm-mlx serve mlx-community/Llama-3.2-3B-Instruct-4bit Model will show up as 'mlx-community/Llama-3.2-3B-Instruct-4bit' in the `/v1/models` API endpoint. View with `curl http://localhost:8000/v1/models` or similar. # With a custom API model name (model is accessed as "my-model" via the OpenAI API) # --served-model-name sets the name clients must use when calling the API (e.g. model="my-model") vllm-mlx serve --served-model-name my-model mlx-community/Llama-3.2-3B-Instruct-4bit # Note: Model will show up as 'my-model' in the `/v1/models` API endpoint. # Continuous batching (multiple users) vllm-mlx serve mlx-community/Llama-3.2-3B-Instruct-4bit --continuous-batching # With memory limit for large models vllm-mlx serve mlx-community/GLM-4.7-Flash-4bit \ --continuous-batching \ --cache-memory-mb 2048 # Production with paged cache vllm-mlx serve mlx-community/Qwen3-0.6B-8bit \ --continuous-batching \ --use-paged-cache \ --port 8000 # With MCP tools vllm-mlx serve mlx-community/Qwen3-4B-4bit --mcp-config mcp.json # Multimodal model vllm-mlx serve mlx-community/Qwen3-VL-4B-Instruct-3bit # Reasoning model (separates thinking from answer) vllm-mlx serve mlx-community/Qwen3-8B-4bit --reasoning-parser qwen3 # DeepSeek reasoning model vllm-mlx serve mlx-community/DeepSeek-R1-Distill-Qwen-7B-4bit --reasoning-parser deepseek_r1 # Tool calling with Mistral/Devstral vllm-mlx serve mlx-community/Devstral-Small-2507-4bit \ --enable-auto-tool-choice --tool-call-parser mistral # Tool calling with Granite vllm-mlx serve mlx-community/granite-4.0-tiny-preview-4bit \ --enable-auto-tool-choice --tool-call-parser granite # With API key authentication vllm-mlx serve mlx-community/Llama-3.2-3B-Instruct-4bit --api-key your-secret-key # Expose Prometheus metrics vllm-mlx serve mlx-community/Llama-3.2-3B-Instruct-4bit --enable-metrics # Production setup with security options vllm-mlx serve mlx-community/Qwen3-4B-4bit \ --api-key your-secret-key \ --rate-limit 60 \ --timeout 120 \ --continuous-batching ``` ### Sécurité Lorsque `--api-key` est défini, toutes les requêtes API requièrent l'en-tête `Authorization: Bearer ` : ```python from openai import OpenAI client = OpenAI( base_url="http://localhost:8000/v1", api_key="your-secret-key" # Must match --api-key ) ``` Ou avec curl : ```bash curl http://localhost:8000/v1/models \ -H "Authorization: Bearer your-secret-key" ``` ## `vllm-mlx-bench` Exécuter des benchmarks de performance. ### Utilisation ```bash vllm-mlx-bench --model [options] ``` ### Options | Option | Description | Défaut | |--------|-------------|---------| | `--model` | Nom du modèle | Requis | | `--prompts` | Nombre de prompts | 5 | | `--max-tokens` | Nombre maximum de tokens par prompt | 256 | | `--quick` | Mode benchmark rapide | False | | `--video` | Exécuter le benchmark vidéo | False | | `--video-url` | URL vidéo personnalisée | None | | `--video-path` | Chemin vidéo personnalisé | None | ### Exemples ```bash # LLM benchmark vllm-mlx-bench --model mlx-community/Llama-3.2-1B-Instruct-4bit # Quick benchmark vllm-mlx-bench --model mlx-community/Llama-3.2-1B-Instruct-4bit --quick # Image benchmark (auto-detected for VLM models) vllm-mlx-bench --model mlx-community/Qwen3-VL-8B-Instruct-4bit # Video benchmark vllm-mlx-bench --model mlx-community/Qwen3-VL-8B-Instruct-4bit --video # Custom video vllm-mlx-bench --model mlx-community/Qwen3-VL-8B-Instruct-4bit \ --video --video-url https://example.com/video.mp4 ``` ## `vllm-mlx-chat` Démarrer l'interface de chat Gradio. ### Utilisation ```bash vllm-mlx-chat --served-model-name [options] ``` ### Options | Option | Description | Défaut | |--------|-------------|---------| | `--model` | Nom du modèle | Requis | | `--port` | Port Gradio | 7860 | | `--text-only` | Désactiver le multimodal | False | ### Exemples ```bash # Multimodal chat (text + images + video) vllm-mlx-chat --served-model-name mlx-community/Qwen3-VL-4B-Instruct-3bit # Text-only chat vllm-mlx-chat --served-model-name mlx-community/Llama-3.2-3B-Instruct-4bit --text-only ``` ## Variables d'environnement | Variable | Description | |----------|-------------| | `VLLM_MLX_TEST_MODEL` | Modèle utilisé pour les tests | | `HF_TOKEN` | Jeton HuggingFace | # Documentation page: `fr/reference/configuration.md` # Référence de configuration ## Configuration du serveur ### Options de base | Option | Description | Défaut | |--------|-------------|---------| | `--host` | Adresse hôte du serveur | `127.0.0.1` | | `--port` | Port du serveur | `8000` | | `--max-tokens` | Nombre maximum de tokens par défaut | `32768` | | `--max-request-tokens` | Valeur maximale de `max_tokens` acceptée depuis les clients API | `32768` | | `--default-temperature` | Température par défaut si non spécifiée dans la requête | None | | `--default-top-p` | top_p par défaut si non spécifié dans la requête | None | ### Options de sécurité | Option | Description | Défaut | |--------|-------------|---------| | `--api-key` | Clé API pour l'authentification | None | | `--rate-limit` | Requêtes par minute par client (0 = désactivé) | `0` | | `--timeout` | Délai d'expiration des requêtes en secondes | `300` | | `--enable-metrics` | Expose les métriques Prometheus sur `/metrics` | `false` | | `--max-audio-upload-mb` | Taille maximale du fichier audio téléversé pour `/v1/audio/transcriptions` | `25` | | `--max-tts-input-chars` | Longueur maximale du texte acceptée par `/v1/audio/speech` | `4096` | ### Options de batching | Option | Description | Défaut | |--------|-------------|---------| | `--continuous-batching` | Active le continuous batching | `false` | | `--stream-interval` | Tokens par fragment de streaming | `1` | | `--max-num-seqs` | Nombre maximum de séquences simultanées | `256` | ### Options de cache | Option | Description | Défaut | |--------|-------------|---------| | `--cache-memory-mb` | Limite mémoire du cache en Mo | Auto | | `--cache-memory-percent` | Fraction de la RAM allouée au cache | `0.20` | | `--no-memory-aware-cache` | Utilise le cache legacy basé sur le nombre d'entrées | `false` | | `--use-paged-cache` | Active le KV cache paginé | `false` | | `--paged-cache-block-size` | Tokens par bloc | `64` | | `--max-cache-blocks` | Nombre maximum de blocs | `1000` | ### Options d'appel d'outils | Option | Description | Défaut | |--------|-------------|---------| | `--enable-auto-tool-choice` | Active l'appel automatique d'outils | `false` | | `--tool-call-parser` | Parseur d'appels d'outils (voir [Appel d'outils](../guides/tool-calling.md)) | None | ### Options de raisonnement | Option | Description | Défaut | |--------|-------------|---------| | `--reasoning-parser` | Parseur pour les modèles de raisonnement (`qwen3`, `deepseek_r1`) | None | ### Options d'embeddings | Option | Description | Défaut | |--------|-------------|---------| | `--embedding-model` | Précharge un modèle d'embedding au démarrage | None | ### Options MCP | Option | Description | Défaut | |--------|-------------|---------| | `--mcp-config` | Chemin vers le fichier de configuration MCP | None | ## Configuration MCP Créez le fichier `mcp.json` : ```json { "mcpServers": { "server-name": { "command": "npx", "args": ["-y", "@modelcontextprotocol/server-name", "arg1"], "env": { "ENV_VAR": "value" } } } } ``` ### Options du serveur MCP | Champ | Description | Obligatoire | |-------|-------------|-------------| | `command` | Commande exécutable | Oui | | `args` | Arguments de la commande | Oui | | `env` | Variables d'environnement | Non | ## Options des requêtes API ### Complétions de chat | Paramètre | Description | Défaut | |-----------|-------------|---------| | `model` | Nom du modèle | Obligatoire | | `messages` | Messages du chat | Obligatoire | | `max_tokens` | Nombre maximum de tokens à générer | 256 | | `temperature` | Température d'échantillonnage | Défaut du modèle | | `top_p` | Échantillonnage par noyau | Défaut du modèle | | `stream` | Active le streaming | `true` | | `stop` | Séquences d'arrêt | None | | `tools` | Définitions des outils | None | | `response_format` | Format de sortie (`json_object`, `json_schema`) | None | ### Options multimodales | Paramètre | Description | Défaut | |-----------|-------------|---------| | `video_fps` | Images par seconde | 2.0 | | `video_max_frames` | Nombre maximum d'images | 32 | ## Variables d'environnement | Variable | Description | |----------|-------------| | `VLLM_MLX_TEST_MODEL` | Modèle par défaut pour les tests | | `HF_TOKEN` | Token d'authentification HuggingFace | | `OPENAI_API_KEY` | À définir avec n'importe quelle valeur pour la compatibilité SDK | ## Exemples de configurations ### Développement (utilisateur unique) ```bash vllm-mlx serve mlx-community/Llama-3.2-3B-Instruct-4bit ``` ### Production (utilisateurs multiples) ```bash vllm-mlx serve mlx-community/Qwen3-0.6B-8bit \ --continuous-batching \ --use-paged-cache \ --api-key your-secret-key \ --rate-limit 60 \ --port 8000 ``` ### Avec appel d'outils ```bash vllm-mlx serve mlx-community/Devstral-Small-2507-4bit \ --enable-auto-tool-choice \ --tool-call-parser mistral \ --continuous-batching ``` ### Avec les outils MCP ```bash vllm-mlx serve mlx-community/Qwen3-4B-4bit \ --mcp-config mcp.json \ --enable-auto-tool-choice \ --tool-call-parser qwen \ --continuous-batching ``` ### Modèle de raisonnement ```bash vllm-mlx serve mlx-community/Qwen3-8B-4bit \ --reasoning-parser qwen3 \ --continuous-batching ``` ### Avec embeddings ```bash vllm-mlx serve mlx-community/Qwen3-4B-4bit \ --embedding-model mlx-community/multilingual-e5-small-mlx \ --continuous-batching ``` ### Débit élevé ```bash vllm-mlx serve mlx-community/Qwen3-0.6B-8bit \ --continuous-batching \ --stream-interval 5 \ --max-num-seqs 256 ``` # Documentation page: `fr/reference/models.md` # Modèles pris en charge Tous les modèles quantifiés de [mlx-community sur HuggingFace](https://huggingface.co/mlx-community/models) sont compatibles. Parcourez des milliers de modèles pré-optimisés à l'adresse : **https://huggingface.co/mlx-community/models** ## Modèles de langage (via mlx-lm) | Famille de modèles | Tailles | Quantification | |--------------------|---------|----------------| | Llama 3.x, 4.x | 1B, 3B, 8B, 70B | 4-bit | | Mistral / Devstral | 7B, Mixtral 8x7B | 4-bit, 8-bit | | Qwen2/Qwen3 | 0.5B à 72B | Variable | | DeepSeek V3, R1 | 7B, 33B, 67B | 4-bit | | Gemma 2, 3, 4 | 2B, 9B, 27B | 4-bit | | GLM-4.7 | Flash, Base | 4-bit, 8-bit | | Kimi K2 | Variable | 4-bit | | Phi-3 | 3.8B, 14B | 4-bit | | Granite 3.x, 4.x | Variable | 4-bit | | Nemotron | 3 Nano 30B | 6-bit | ### Modèles recommandés | Cas d'utilisation | Modèle | Mémoire | |-------------------|--------|---------| | Rapide / léger | `mlx-community/Qwen3-0.6B-8bit` | ~0,7 Go | | Équilibré | `mlx-community/Llama-3.2-3B-Instruct-4bit` | ~1,8 Go | | Qualité | `mlx-community/Llama-3.1-8B-Instruct-4bit` | ~4,5 Go | | Grand modèle | `mlx-community/Qwen3-30B-A3B-4bit` | ~16 Go | ## Modèles multimodaux (via mlx-vlm) | Famille de modèles | Exemples de modèles | |--------------------|---------------------| | **Qwen-VL** | `Qwen3-VL-4B-Instruct-3bit`, `Qwen3-VL-8B-Instruct-4bit`, `Qwen2-VL-2B/7B-Instruct-4bit` | | **LLaVA** | `llava-1.5-7b-4bit`, `llava-v1.6-mistral-7b-4bit`, `llava-llama-3-8b-v1_1-4bit` | | **Idefics** | `Idefics3-8B-Llama3-4bit`, `idefics2-8b-4bit` | | **Gemma 4** | `gemma-4-e2b-it-mxfp4` (vision + audio) | | **PaliGemma** | `paligemma2-3b-mix-224-4bit`, `paligemma-3b-mix-224-8bit` | | **Pixtral** | `pixtral-12b-4bit`, `pixtral-12b-8bit` | | **Molmo** | `Molmo-7B-D-0924-4bit`, `Molmo-7B-D-0924-8bit` | | **Phi-3 Vision** | `Phi-3-vision-128k-instruct-4bit` | | **DeepSeek-VL** | `deepseek-vl-7b-chat-4bit`, `deepseek-vl2-small-4bit` | ### Modèles VLM recommandés | Cas d'utilisation | Modèle | Mémoire | |-------------------|--------|---------| | Rapide / léger | `mlx-community/Qwen3-VL-4B-Instruct-3bit` | ~3 Go | | Équilibré | `mlx-community/Qwen3-VL-8B-Instruct-4bit` | ~6 Go | | Qualité | `mlx-community/Qwen3-VL-30B-A3B-Instruct-6bit` | ~20 Go | ## Modèles d'embeddings (via mlx-embeddings) | Famille de modèles | Exemples de modèles | |--------------------|---------------------| | **BERT** | `mlx-community/bert-base-uncased-mlx` | | **XLM-RoBERTa** | `mlx-community/multilingual-e5-small-mlx`, `mlx-community/multilingual-e5-large-mlx` | | **ModernBERT** | `mlx-community/ModernBERT-base-mlx` | ## Modèles audio (via mlx-audio) | Type | Famille de modèles | Exemples de modèles | |------|--------------------|---------------------| | **STT** | Whisper | `mlx-community/whisper-large-v3-turbo` | | **STT** | Parakeet | `mlx-community/parakeet-tdt-0.6b-v2` | | **TTS** | Kokoro | `prince-canuma/Kokoro-82M` | | **TTS** | Chatterbox | `chatterbox/chatterbox-tts-0.1` | ## Détection automatique des modèles vllm-mlx détecte automatiquement les modèles multimodaux selon des motifs dans leur nom : - Contient "VL", "Vision", "vision" - Contient "llava", "idefics", "paligemma" - Contient "pixtral", "molmo", "deepseek-vl" - Contient "MedGemma", "Gemma-3", "Gemma-4" (variantes multimodales) ## Utilisation des modèles ### Depuis HuggingFace ```bash vllm-mlx serve mlx-community/Llama-3.2-3B-Instruct-4bit ``` ### Chemin local ```bash vllm-mlx serve /path/to/local/model ``` ## Recherche de modèles Filtrez les modèles mlx-community par : - **LLM** : `Llama`, `Qwen`, `Mistral`, `Phi`, `Gemma`, `DeepSeek`, `GLM`, `Kimi`, `Granite`, `Nemotron` - **VLM** : `-VL-`, `llava`, `paligemma`, `pixtral`, `molmo`, `idefics`, `deepseek-vl`, `MedGemma` - **Embedding** : `e5`, `bert`, `ModernBERT` - **Taille** : `1B`, `3B`, `7B`, `8B`, `70B` - **Quantification** : `4bit`, `8bit`, `bf16` # Documentation page: `getting-started/installation.md` # Installation ## Requirements - macOS on Apple Silicon (M1/M2/M3/M4/M5) - Python 3.10+ ## Install with uv (Recommended) ```bash git clone https://github.com/waybarrios/vllm-mlx.git cd vllm-mlx uv pip install -e . ``` ## Install with pip ```bash git clone https://github.com/waybarrios/vllm-mlx.git cd vllm-mlx pip install -e . ``` ### Optional: Vision Support For video processing with transformers: ```bash pip install -e ".[vision]" ``` ### Optional: Audio Support (STT/TTS) ```bash pip install mlx-audio ``` ### Optional: Embeddings ```bash pip install mlx-embeddings ``` ## What Gets Installed - `mlx`, `mlx-lm`, `mlx-vlm` - MLX framework and model libraries - `transformers`, `tokenizers` - HuggingFace libraries - `opencv-python` - Video processing - `gradio` - Chat UI - `psutil` - Resource monitoring - `mlx-audio` (optional) - Speech-to-Text and Text-to-Speech - `mlx-embeddings` (optional) - Text embeddings ## Verify Installation ```bash # Check CLI commands vllm-mlx --help vllm-mlx-bench --help vllm-mlx-chat --help # Test with a small model vllm-mlx-bench --model mlx-community/Llama-3.2-1B-Instruct-4bit --prompts 1 ``` ## Troubleshooting ### MLX not found Ensure you're on Apple Silicon: ```bash uname -m # Should output "arm64" ``` ### Model download fails Check your internet connection and HuggingFace access. Some models require authentication: ```bash huggingface-cli login ``` You can inspect and stage models before serving: ```bash vllm-mlx model inspect mlx-community/Llama-3.2-3B-Instruct-4bit vllm-mlx model acquire mlx-community/Llama-3.2-3B-Instruct-4bit \ --target-dir ./models/llama-3b-4bit ``` ### Out of memory Use a smaller quantized model: ```bash vllm-mlx serve mlx-community/Llama-3.2-1B-Instruct-4bit ``` ### Server interruptions during long runs (macOS sleep) Your macOS machine may go to sleep during long-running server sessions. Try using `caffeinate` to prevent sleep: ```bash caffeinate -dimsu ``` # Documentation page: `getting-started/quickstart.md` # Quick Start ## Option 1: OpenAI-Compatible Server Start the server: ```bash # Simple mode - maximum throughput for single user vllm-mlx serve mlx-community/Llama-3.2-3B-Instruct-4bit --port 8000 # Continuous batching - for multiple concurrent users vllm-mlx serve mlx-community/Llama-3.2-3B-Instruct-4bit --port 8000 --continuous-batching ``` Use with OpenAI Python SDK: ```python from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed") response = client.chat.completions.create( model="mlx-community/Llama-3.2-3B-Instruct-4bit", messages=[{"role": "user", "content": "Hello!"}], ) print(response.choices[0].message.content) ``` Or with curl: ```bash curl http://localhost:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{"model": "default", "messages": [{"role": "user", "content": "Hello!"}]}' ``` ## Option 2: Direct Python API ```python from vllm_mlx.models import MLXLanguageModel model = MLXLanguageModel("mlx-community/Llama-3.2-3B-Instruct-4bit") model.load() # Generate text output = model.generate("What is the capital of France?", max_tokens=100) print(output.text) # Streaming for chunk in model.stream_generate("Tell me a story"): print(chunk.text, end="", flush=True) ``` ## Option 3: Gradio Chat UI ```bash vllm-mlx-chat --served-model-name mlx-community/Llama-3.2-3B-Instruct-4bit ``` Opens a web interface at http://localhost:7860 ## Multimodal Models For image/video understanding, use a VLM model: ```bash vllm-mlx serve mlx-community/Qwen3-VL-4B-Instruct-3bit --port 8000 ``` ```python response = client.chat.completions.create( model="default", messages=[{ "role": "user", "content": [ {"type": "text", "text": "What's in this image?"}, {"type": "image_url", "image_url": {"url": "https://example.com/image.jpg"}} ] }], max_tokens=256 ) ``` ## Reasoning Models Separate the model's thinking process from the final answer: ```bash vllm-mlx serve mlx-community/Qwen3-8B-4bit --reasoning-parser qwen3 ``` ```python response = client.chat.completions.create( model="default", messages=[{"role": "user", "content": "What is 17 × 23?"}] ) print(response.choices[0].message.content) # Final answer ``` ## Embeddings Generate text embeddings for semantic search and RAG: ```bash vllm-mlx serve mlx-community/Qwen3-4B-4bit --embedding-model mlx-community/multilingual-e5-small-mlx ``` ```python response = client.embeddings.create( model="mlx-community/multilingual-e5-small-mlx", input="Hello world" ) ``` ## Tool Calling Enable function calling with any supported model: ```bash vllm-mlx serve mlx-community/Devstral-Small-2507-4bit \ --enable-auto-tool-choice --tool-call-parser mistral ``` ## Next Steps - [Server Guide](../guides/server.md) - Full server configuration - [Python API](../guides/python-api.md) - Direct API usage - [Multimodal Guide](../guides/multimodal.md) - Images and video - [Audio Guide](../guides/audio.md) - Speech-to-Text and Text-to-Speech - [Embeddings Guide](../guides/embeddings.md) - Text embeddings - [Reasoning Models](../guides/reasoning.md) - Thinking models - [Tool Calling](../guides/tool-calling.md) - Function calling - [Supported Models](../reference/models.md) - Available models # Documentation page: `guides/audio.md` # Audio Support vllm-mlx supports audio processing using [mlx-audio](https://github.com/Blaizzy/mlx-audio), providing: - **STT (Speech-to-Text)**: Whisper, Parakeet - **TTS (Text-to-Speech)**: Kokoro, Chatterbox, VibeVoice, VoxCPM - **Audio Processing**: SAM-Audio (voice separation) ## Installation ```bash # Core audio support pip install mlx-audio>=0.2.9 # Required dependencies for TTS pip install sounddevice soundfile scipy numba tiktoken misaki spacy num2words loguru phonemizer # Download spacy English model python -m spacy download en_core_web_sm # For non-English TTS (Spanish, French, etc.), install espeak-ng: # macOS brew install espeak-ng # Ubuntu/Debian # sudo apt-get install espeak-ng ``` Or install all audio dependencies at once: ```bash pip install vllm-mlx[audio] python -m spacy download en_core_web_sm brew install espeak-ng # macOS, for non-English languages ``` ## Quick Start ### Speech-to-Text (Transcription) ```python from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed") # Transcribe audio file with open("audio.mp3", "rb") as f: transcript = client.audio.transcriptions.create( model="whisper-large-v3", file=f, language="en" # optional ) print(transcript.text) ``` ### Text-to-Speech (Generation) ```python # Generate speech audio = client.audio.speech.create( model="kokoro", input="Hello, how are you?", voice="af_heart", speed=1.0 ) # Save to file with open("output.wav", "wb") as f: f.write(audio.content) ``` ### Voice Separation (SAM-Audio) Isolate voice from background noise, music, or other sounds: ```python from vllm_mlx.audio import AudioProcessor # Load SAM-Audio model processor = AudioProcessor("mlx-community/sam-audio-large-fp16") processor.load() # Separate speech from audio result = processor.separate("meeting_with_music.mp3", description="speech") # Save isolated voice and background processor.save(result.target, "voice_only.wav") processor.save(result.residual, "background_only.wav") ``` **CLI Example:** ```bash python examples/audio_separation_example.py meeting.mp3 --play python examples/audio_separation_example.py song.mp3 --description music -o music.wav ``` ### Drums Separation Demo Isolate drums from a rock song using SAM-Audio: | Audio | Description | Listen | |-------|-------------|--------| | Original | "Get Ready" by David Fesliyan (30s, royalty-free) | [🎵 rock_get_ready.mp3](https://github.com/waybarrios/vllm-mlx/blob/main/examples/rock_get_ready.mp3?raw=1) | | Isolated Drums | Drums extracted by SAM-Audio | [🥁 drums_isolated.wav](https://github.com/waybarrios/vllm-mlx/blob/main/examples/drums_isolated.wav?raw=1) | | Without Drums | Track with drums removed | [🎸 rock_no_drums.wav](https://github.com/waybarrios/vllm-mlx/blob/main/examples/rock_no_drums.wav?raw=1) | ```bash # Isolate drums from rock song python examples/audio_separation_example.py examples/rock_get_ready.mp3 \ --description "drums" \ --output drums_isolated.wav \ --background rock_no_drums.wav ``` **Performance:** 30s audio processed in ~20 seconds on M4 Max. ## Supported Models ### STT Models (Speech-to-Text) | Model | Alias | Languages | Speed | Quality | |-------|-------|-----------|-------|---------| | `mlx-community/whisper-large-v3-mlx` | `whisper-large-v3` | 99+ | Medium | Best | | `mlx-community/whisper-large-v3-turbo` | `whisper-large-v3-turbo` | 99+ | Fast | Great | | `mlx-community/whisper-medium-mlx` | `whisper-medium` | 99+ | Fast | Good | | `mlx-community/whisper-small-mlx` | `whisper-small` | 99+ | Very Fast | OK | | `mlx-community/parakeet-tdt-0.6b-v2` | `parakeet` | English | Fastest | Great | | `mlx-community/parakeet-tdt-0.6b-v3` | `parakeet-v3` | English | Fastest | Best | **Recommendation:** - Multilingual: `whisper-large-v3` - English only: `parakeet` (3x faster) ### TTS Models (Text-to-Speech) #### Kokoro (Fast, Lightweight) - Recommended | Model | Alias | Size | Languages | |-------|-------|------|-----------| | `mlx-community/Kokoro-82M-bf16` | `kokoro` | 82M | EN, ES, FR, JA, ZH, HI, IT, PT | | `mlx-community/Kokoro-82M-4bit` | `kokoro-4bit` | 82M | EN, ES, FR, JA, ZH, HI, IT, PT | **Voices (11):** - Female American: `af_heart`, `af_bella`, `af_nicole`, `af_sarah`, `af_sky` - Male American: `am_adam`, `am_michael` - Female British: `bf_emma`, `bf_isabella` - Male British: `bm_george`, `bm_lewis` **Language Codes:** | Code | Language | Code | Language | |------|----------|------|----------| | `a` / `en` | English (US) | `e` / `es` | Español | | `b` / `en-gb` | English (UK) | `f` / `fr` | Français | | `j` / `ja` | 日本語 | `z` / `zh` | 中文 | | `i` / `it` | Italiano | `p` / `pt` | Português | | `h` / `hi` | हिन्दी | | | #### Chatterbox (Multilingual, Expressive) | Model | Alias | Size | Languages | |-------|-------|------|-----------| | `mlx-community/chatterbox-turbo-fp16` | `chatterbox` | 134M | 15+ languages | | `mlx-community/chatterbox-turbo-4bit` | `chatterbox-4bit` | 134M | 15+ languages | **Supported Languages:** EN, ES, FR, DE, IT, PT, RU, JA, ZH, KO, AR, HI, NL, PL, TR #### VibeVoice (Realtime) | Model | Alias | Size | Use Case | |-------|-------|------|----------| | `mlx-community/VibeVoice-Realtime-0.5B-4bit` | `vibevoice` | 200M | Low latency, English | #### VoxCPM (Chinese/English) | Model | Alias | Size | Languages | |-------|-------|------|-----------| | `mlx-community/VoxCPM1.5` | `voxcpm` | 0.9B | ZH, EN | | `mlx-community/VoxCPM1.5-4bit` | `voxcpm-4bit` | 200M | ZH, EN | ### Audio Processing Models #### SAM-Audio (Voice Separation) | Model | Size | Use Case | |-------|------|----------| | `mlx-community/sam-audio-large-fp16` | 3B | Best quality | | `mlx-community/sam-audio-large` | 3B | Standard | | `mlx-community/sam-audio-small-fp16` | 0.6B | Fast | | `mlx-community/sam-audio-small` | 0.6B | Lightweight | ## API Reference ### POST /v1/audio/transcriptions Transcribe audio to text (OpenAI Whisper API compatible). **Parameters:** - `file`: Audio file (mp3, wav, m4a, webm) - `model`: Model name or alias - `language`: Language code (optional, auto-detected) - `response_format`: `json` or `text` **Limits:** - Default upload cap: 25 MiB - Override with `--max-audio-upload-mb` **Example:** ```bash curl http://localhost:8000/v1/audio/transcriptions \ -F file=@audio.mp3 \ -F model=whisper-large-v3 ``` ### POST /v1/audio/speech Generate speech from text (OpenAI TTS API compatible). **Parameters:** - `model`: Model name or alias - `input`: Text to synthesize - `voice`: Voice ID - `speed`: Speech speed (0.5 to 2.0) - `response_format`: `wav`, `mp3` **Limits:** - Default input cap: 4096 characters - Override with `--max-tts-input-chars` **Example:** ```bash curl http://localhost:8000/v1/audio/speech \ -d '{"model": "kokoro", "input": "Hello world", "voice": "af_heart"}' \ -H "Content-Type: application/json" \ --output speech.wav ``` ### GET /v1/audio/voices List available voices for a model. **Example:** ```bash curl http://localhost:8000/v1/audio/voices?model=kokoro ``` ## CLI Examples ### Live Transcription / Closed Captions Real-time speech-to-text transcription from your microphone: ```bash # Closed captions with whisper-large-v3 (best quality) python examples/closed_captions.py --language es --chunk 5 # Faster model for lower latency python examples/closed_captions.py --language en --model whisper-turbo --chunk 3 # Basic mic transcription (record then transcribe) python examples/mic_transcribe.py --language es # Real-time chunked transcription python examples/mic_realtime.py --language es --chunk 3 # Live transcription with voice activity detection python examples/mic_live.py --language es ``` **Requirements:** ```bash pip install sounddevice soundfile numpy ``` ### Basic TTS ```bash # Simple TTS example python examples/tts_example.py "Hello, how are you?" --play # With different voice python examples/tts_example.py "Hello!" --voice am_michael --play # Save to file python examples/tts_example.py "Welcome to the demo" -o greeting.wav # List available voices python examples/tts_example.py --list-voices ``` ### Multilingual TTS ```bash # English (auto-selects best model) python examples/tts_multilingual.py "Hello world" --play # Spanish python examples/tts_multilingual.py "Hola mundo" --lang es --play # French python examples/tts_multilingual.py "Bonjour le monde" --lang fr --play # Japanese python examples/tts_multilingual.py "こんにちは" --lang ja --play # Chinese python examples/tts_multilingual.py "你好世界" --lang zh --play # Use specific model python examples/tts_multilingual.py "Hello" --model chatterbox --play # List all models python examples/tts_multilingual.py --list-models # List all languages python examples/tts_multilingual.py --list-languages ``` ### Business Assistant Voice Examples Pre-generated voice samples with **native voices** for common business use cases: | Language | Voice | Message | Listen | |----------|-------|---------|--------| | 🇺🇸 English | af_heart | "Welcome to First National Bank. How may I assist you today?" | [▶️ assistant_bank_en.wav](https://github.com/waybarrios/vllm-mlx/blob/main/examples/assistant_bank_en.wav?raw=1) | | 🇪🇸 Spanish | ef_dora | "Gracias por llamar a servicio al cliente. Un agente le atenderá pronto." | [▶️ assistant_service_es.wav](https://github.com/waybarrios/vllm-mlx/blob/main/examples/assistant_service_es.wav?raw=1) | | 🇫🇷 French | ff_siwis | "Bienvenue. Votre appel est important pour nous." | [▶️ assistant_callcenter_fr.wav](https://github.com/waybarrios/vllm-mlx/blob/main/examples/assistant_callcenter_fr.wav?raw=1) | | 🇨🇳 Chinese | zf_xiaobei | "欢迎致电技术支持中心。我们将竭诚为您服务。" | [▶️ assistant_support_zh.wav](https://github.com/waybarrios/vllm-mlx/blob/main/examples/assistant_support_zh.wav?raw=1) | **Generate your own with native voices:** ```bash # English - Bank assistant (native voice: af_heart) python -m mlx_audio.tts.generate --model mlx-community/Kokoro-82M-bf16 \ --text "Welcome to First National Bank. How may I assist you today?" \ --voice af_heart --lang_code a --file_prefix assistant_bank_en # Spanish - Customer service (native voice: ef_dora) python -m mlx_audio.tts.generate --model mlx-community/Kokoro-82M-bf16 \ --text "Gracias por llamar a servicio al cliente. Un agente le atendera pronto." \ --voice ef_dora --lang_code e --file_prefix assistant_service_es # French - Call center (native voice: ff_siwis) python -m mlx_audio.tts.generate --model mlx-community/Kokoro-82M-bf16 \ --text "Bienvenue. Votre appel est important pour nous." \ --voice ff_siwis --lang_code f --file_prefix assistant_callcenter_fr # Chinese - Tech support (native voice: zf_xiaobei) python -m mlx_audio.tts.generate --model mlx-community/Kokoro-82M-bf16 \ --text "欢迎致电技术支持中心。我们将竭诚为您服务。" \ --voice zf_xiaobei --lang_code z --file_prefix assistant_support_zh ``` ### Native Voice Reference | Language | Code | Voices | |----------|------|--------| | English (US) | `a` | af_heart, af_bella, af_nicole, am_adam, am_michael | | English (UK) | `b` | bf_emma, bf_isabella, bm_george, bm_lewis | | Spanish | `e` | ef_dora, em_alex, em_santa | | French | `f` | ff_siwis | | Chinese | `z` | zf_xiaobei, zf_xiaoni, zf_xiaoxiao, zm_yunjian, zm_yunxi | | Japanese | `j` | jf_alpha, jf_gongitsune, jm_kumo | | Italian | `i` | if_sara, im_nicola | | Portuguese | `p` | pf_dora, pm_alex | | Hindi | `h` | hf_alpha, hf_beta, hm_omega | ## Python API ### Direct Usage (without server) ```python from vllm_mlx.audio import STTEngine, TTSEngine, AudioProcessor # Speech-to-Text stt = STTEngine("mlx-community/whisper-large-v3-mlx") stt.load() result = stt.transcribe("audio.mp3") print(result.text) # Text-to-Speech tts = TTSEngine("mlx-community/Kokoro-82M-bf16") tts.load() audio = tts.generate("Hello world", voice="af_heart") tts.save(audio, "output.wav") # Voice Separation processor = AudioProcessor("mlx-community/sam-audio-large-fp16") processor.load() result = processor.separate("mixed_audio.mp3", description="speech") processor.save(result.target, "voice_only.wav") processor.save(result.residual, "background.wav") ``` ### Convenience Functions ```python from vllm_mlx.audio import transcribe_audio, generate_speech, separate_voice # Quick transcription result = transcribe_audio("audio.mp3") print(result.text) # Quick TTS audio = generate_speech("Hello world", voice="af_heart") # Quick voice separation voice, background = separate_voice("mixed.mp3") ``` ## Audio in Chat Include audio in chat messages (transcribed automatically): ```python response = client.chat.completions.create( model="default", messages=[{ "role": "user", "content": [ {"type": "text", "text": "Summarize this audio"}, {"type": "audio_url", "audio_url": {"url": "file://meeting.mp3"}} ] }] ) ``` ## Benchmarks Tested on Apple M2 Max (32GB). ### TTS Benchmarks (Kokoro-82M-bf16) | Text Length | Audio Duration | Gen Time | RTF | Chars/sec | |-------------|----------------|----------|-----|-----------| | 25 chars | 1.95s | 0.43s | 4.6x | 58.5 | | 88 chars | 6.00s | 0.32s | 18.6x | 272.4 | | 117 chars | 7.92s | 0.27s | 29.0x | 427.4 | **Summary:** - Model load time: ~1.0s - Average RTF: **17.4x** (17x faster than real-time) - Average chars/sec: **252.8** ### STT Benchmarks | Model | Load Time | Transcribe (6s audio) | RTF | |-------|-----------|----------------------|-----| | whisper-small | 0.25s | 0.20s | 30.2x | | whisper-medium | 18.1s | 0.38s | 15.5x | | whisper-large-v3 | ~30s | ~0.6s | ~10x | | parakeet | ~0.5s | ~0.15s | ~40x | **Notes:** - RTF (Real-Time Factor) indicates how many times faster than real-time - First load includes model download from HuggingFace - Subsequent loads use cached models ### Recommendations by Use Case | Use Case | Recommended Model | Why | |----------|------------------|-----| | Fast English STT | `parakeet` | 40x RTF, low memory | | Multilingual STT | `whisper-large-v3` | 99+ languages | | Low-latency STT | `whisper-small` | 30x RTF, quick load | | General TTS | `kokoro` | 17x RTF, good quality | | Low memory TTS | `kokoro-4bit` | 4-bit quantized | ## Performance Tips 1. **Use Parakeet for English** - 40x faster than real-time 2. **Use 4-bit models** for lower memory usage 3. **Use SAM-Audio small** for faster voice separation 4. **Cache models** - engines are lazy-loaded and cached 5. **Pre-download models** to avoid first-run latency ## Troubleshooting ### mlx-audio not installed ``` pip install mlx-audio>=0.2.9 ``` ### Model download slow Models are downloaded from HuggingFace on first use. Use `huggingface-cli download` to pre-download: ```bash huggingface-cli download mlx-community/whisper-large-v3-mlx huggingface-cli download mlx-community/Kokoro-82M-bf16 ``` ### Out of memory Use smaller models or 4-bit quantized versions: - `whisper-small-mlx` instead of `whisper-large-v3-mlx` - `Kokoro-82M-4bit` instead of `Kokoro-82M-bf16` - `sam-audio-small` instead of `sam-audio-large` ### Kokoro multilingual bug (mlx-audio 0.2.9) If you get `ValueError: too many values to unpack` when using non-English languages (Spanish, Chinese, Japanese, etc.) with Kokoro, apply this fix: ```python # Fix for mlx_audio/tts/models/kokoro/pipeline.py line 443 # Change: # ps, _ = self.g2p(chunk) # To: g2p_result = self.g2p(chunk) ps = g2p_result[0] if isinstance(g2p_result, tuple) else g2p_result ``` **One-liner fix:** ```bash python -c " import os path = os.path.join(os.path.dirname(__import__('mlx_audio').__file__), 'tts/models/kokoro/pipeline.py') with open(path, 'r') as f: content = f.read() old = ' ps, _ = self.g2p(chunk)' new = ''' # Fix: handle both tuple (en) and string (zh/ja/es) returns from g2p g2p_result = self.g2p(chunk) ps = g2p_result[0] if isinstance(g2p_result, tuple) else g2p_result''' if old in content: with open(path, 'w') as f: f.write(content.replace(old, new)) print('Fix applied!') " ``` This bug occurs because English g2p returns a tuple `(phonemes, tokens)` while other languages return just a string. # Documentation page: `guides/continuous-batching.md` # Continuous Batching Continuous batching enables higher throughput when serving multiple concurrent users. ## Enabling Continuous Batching ```bash vllm-mlx serve mlx-community/Qwen3-0.6B-8bit --continuous-batching ``` ## With Paged Cache For memory-efficient prefix sharing: ```bash vllm-mlx serve mlx-community/Qwen3-0.6B-8bit --continuous-batching --use-paged-cache ``` ## How It Works ### Simple Mode (Default) - One request at a time - Maximum throughput for single user - No overhead from batching ### Continuous Batching Mode - Multiple requests processed together - Better throughput for concurrent users - Small overhead per request ### MLLM MTP and Prefill Notes For multimodal models served through the batched MLLM scheduler, MTP is currently a conservative greedy-only optimization. The MLLM MTP verifier is used only when the active batch has one request, `temperature=0`, `top_p=1`, `top_k=0`, `min_p=0`, and no request-local logits processors. Requests outside that envelope fall back to the normal scheduler path instead of using MTP. This keeps sampling correctness ahead of throughput until the MLLM verifier is sampler-aware. Injected Qwen3.5 and Qwen3.6 MTP checkpoints use the established post-norm backbone state by default. A checkpoint qualified for pre-norm input can opt in by setting `mtp_hidden_state_mode` to `pre_norm` in its `text_config`. Leave the setting unset for existing checkpoints. Unknown values fall back to post-norm and emit a warning. Thinking/logits processors stay active by default for the whole request. The experimental retirement-to-MTP handoff is opt-in via `VLLM_MLX_ENABLE_THINKING_RETIREMENT_RESUME=1`; leave it unset unless you have validated that the processor advertises a safe `is_retired` transition. MLLM prefill uses the regular scheduler `prefill_step_size` unless a future MLLM-specific override is provided. This value controls the language-model prefill chunk size; image/video preprocessing remains per request. MLX generation streams are thread-local. The runtime rebinds mlx-lm/mlx-vlm generation streams at worker-entry boundaries so generation does not reuse a stream created on a different thread. This is a correctness guard for worker ownership, not a throughput feature. ### Paged Cache - KV cache stored in fixed-size blocks - Shared system prompts use same blocks - Memory savings: 80%+ for 10+ concurrent users ## Performance Results **Continuous Batching Results (M4 Max, 128GB):** | Model | Single Request | Batch (5 req) | Speedup | |-------|----------------|---------------|---------| | Llama-3.2-1B-Instruct-4bit | 299.1 tok/s | 613.0 tok/s | **2.05x** | | Llama-3.2-3B-Instruct-4bit | 137.6 tok/s | 208.1 tok/s | **1.51x** | | Qwen3-0.6B-8bit | 328.1 tok/s | 1111.8 tok/s | **3.39x** | | Qwen3-30B-A3B-4bit | 98.1 tok/s | 233.3 tok/s | **2.38x** | | Qwen2.5-1.5B-Instruct-4bit | 196.9 tok/s | 322.2 tok/s | **1.64x** | *Batching 5 concurrent requests shows 1.5-3x throughput improvement.* ## Streaming Performance **Streaming Performance (M4 Max, 128GB):** | Model | TTFT | Generation Speed | |-------|------|------------------| | Llama-3.2-1B-Instruct-4bit | ~4.6ms | 218.9 tok/s | | Llama-3.2-3B-Instruct-4bit | ~10.7ms | 93.6 tok/s | | Qwen3-0.6B-8bit | ~3.0ms | 328.5 tok/s | | Qwen3-30B-A3B-4bit | ~10.2ms | 98.4 tok/s | | Qwen2.5-1.5B-Instruct-4bit | ~7.1ms | 140.3 tok/s | *TTFT = Time to First Token* ## Streaming Configuration Control token delivery with `--stream-interval`: ```bash # Every token (smoothest) vllm-mlx serve model --continuous-batching --stream-interval 1 # Batch tokens (better for high-latency) vllm-mlx serve model --continuous-batching --stream-interval 5 ``` | Value | Behavior | |-------|----------| | `1` | Send every token immediately | | `2-5` | Batch tokens before sending | | `10+` | Maximum throughput, chunkier output | ## Memory Management For large models, the prefix cache can consume significant memory. The memory-aware cache automatically manages this: ```bash # Auto-detect (uses 20% of available RAM) vllm-mlx serve model --continuous-batching # Explicit limit vllm-mlx serve model --continuous-batching --cache-memory-mb 2048 # Custom percentage vllm-mlx serve model --continuous-batching --cache-memory-percent 0.10 ``` | Option | Description | |--------|-------------| | `--cache-memory-mb` | Set explicit limit in MB | | `--cache-memory-percent` | Fraction of available RAM (default: 0.20) | | `--no-memory-aware-cache` | Use legacy entry-count based cache | ## Prefix Cache Prefix caching reuses KV cache for repeated prompts. ### How It Works ``` User 1: System prompt (500 tokens) → Creates 8 blocks User 2: Same system prompt → Shares 8 blocks (ref_count++) User N: Same system prompt → Shares 8 blocks (ref_count++) Memory savings: 80%+ for 10+ concurrent users ``` ### Cache Key Strategy - **LLM**: `hash(prompt)` - **Images**: `hash(image_content) + hash(prompt)` - **Videos**: `hash(video_path) + hash(fps) + hash(max_frames) + hash(prompt)` ### Testing Prefix Cache ```bash python tests/test_prefix_cache.py ``` ``` ====================================================================== LLM PREFIX CACHE TEST ====================================================================== Model: mlx-community/Qwen3-0.6B-8bit Expected behavior: - Same prompt → cache HIT - Different prompt → cache MISS or PREFIX_HIT (shared template tokens) ---------------------------------------------------------------------- Results: Step | Description | Expected | Actual | Status -------+---------------------+----------+--------+------- 1a | First request | MISS | MISS | PASS 1b | Same prompt | HIT | HIT | PASS 1c | Different prompt | MISS | MISS | PASS 1d | Return to prompt 1 | HIT | HIT | PASS ====================================================================== ``` ## Running Benchmarks ```bash # Continuous batching benchmark python tests/test_continuous_batching.py # Prefix cache test python tests/test_prefix_cache.py ``` ## When to Use | Scenario | Mode | |----------|------| | Single user, maximum speed | Simple (default) | | Multiple concurrent users | `--continuous-batching` | | Large models (7B+) | `--continuous-batching --cache-memory-mb 2048` | | Production with shared prompts | `--continuous-batching --use-paged-cache` | ## Production Setup ```bash vllm-mlx serve mlx-community/Qwen3-0.6B-8bit \ --continuous-batching \ --use-paged-cache \ --port 8000 ``` # Documentation page: `guides/embeddings.md` # Embeddings vllm-mlx supports text embeddings using [mlx-embeddings](https://github.com/Blaizzy/mlx-embeddings), providing an OpenAI-compatible `/v1/embeddings` endpoint. ## Installation ```bash pip install mlx-embeddings>=0.0.5 ``` ## Quick Start ### Start the server with an embedding model ```bash # Pre-load a specific embedding model at startup vllm-mlx serve my-llm-model --embedding-model mlx-community/all-MiniLM-L6-v2-4bit ``` If you don't use `--embedding-model`, the embedding model is loaded lazily on the first request, but only from the built-in request-time allowlist. ### Generate embeddings with the OpenAI SDK ```python from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed") # Single text response = client.embeddings.create( model="mlx-community/all-MiniLM-L6-v2-4bit", input="Hello world" ) print(response.data[0].embedding[:5]) # First 5 dimensions # Batch of texts response = client.embeddings.create( model="mlx-community/all-MiniLM-L6-v2-4bit", input=[ "I love machine learning", "Deep learning is fascinating", "Natural language processing rocks" ] ) for item in response.data: print(f"Text {item.index}: {len(item.embedding)} dimensions") ``` ### Using curl ```bash curl http://localhost:8000/v1/embeddings \ -H "Content-Type: application/json" \ -d '{ "model": "mlx-community/all-MiniLM-L6-v2-4bit", "input": ["Hello world", "How are you?"] }' ``` ## Supported Models Supported request-time models: | Model | Use Case | Size | |-------|----------|------| | `mlx-community/all-MiniLM-L6-v2-4bit` | Fast, compact | Small | | `mlx-community/embeddinggemma-300m-6bit` | High quality | 300M | | `mlx-community/bge-large-en-v1.5-4bit` | Best for English | Large | | `mlx-community/multilingual-e5-small-mlx` | Multilingual retrieval | Small | | `mlx-community/multilingual-e5-large-mlx` | Multilingual retrieval | Large | | `mlx-community/bert-base-uncased-mlx` | General BERT baseline | Base | | `mlx-community/ModernBERT-base-mlx` | ModernBERT baseline | Base | Other embedding models require `--embedding-model` at server startup. ## Model Management ### Lazy loading By default, the embedding model is loaded on the first `/v1/embeddings` request. You can switch between the supported request-time models above, and the previous model will be unloaded automatically. ### Pre-loading at startup Use `--embedding-model` to load a model at startup. When this flag is set, only that specific model can be used for embeddings: ```bash vllm-mlx serve my-llm-model --embedding-model mlx-community/all-MiniLM-L6-v2-4bit ``` Requesting a different model will return a 400 error. ## API Reference ### POST /v1/embeddings Create embeddings for the given input text(s). **Request body:** | Field | Type | Required | Description | |-------|------|----------|-------------| | `model` | string | Yes | Supported embedding model ID, or the startup-pinned model when `--embedding-model` is used | | `input` | string or list[string] | Yes | Text(s) to embed | **Response:** ```json { "object": "list", "data": [ {"object": "embedding", "index": 0, "embedding": [0.023, -0.982, ...]}, {"object": "embedding", "index": 1, "embedding": [0.112, -0.543, ...]} ], "model": "mlx-community/all-MiniLM-L6-v2-4bit", "usage": {"prompt_tokens": 12, "total_tokens": 12} } ``` ## Python API ### Direct usage without server ```python from vllm_mlx.embedding import EmbeddingEngine engine = EmbeddingEngine("mlx-community/all-MiniLM-L6-v2-4bit") engine.load() vectors = engine.embed(["Hello world", "How are you?"]) print(f"Dimensions: {len(vectors[0])}") tokens = engine.count_tokens(["Hello world"]) print(f"Token count: {tokens}") ``` ## Troubleshooting ### mlx-embeddings not installed ``` pip install mlx-embeddings>=0.0.5 ``` ### Model not found Make sure the model name matches one of the supported request-time IDs above, or start the server with `--embedding-model` to pin a custom model. You can pre-download supported models: ```bash huggingface-cli download mlx-community/all-MiniLM-L6-v2-4bit ``` # Documentation page: `guides/mcp-tools.md` # MCP & Tool Calling vllm-mlx supports the Model Context Protocol (MCP) for integrating external tools with LLMs. ## How Tool Calling Works ``` ┌─────────────────────────────────────────────────────────────────────┐ │ Tool Calling Flow │ ├─────────────────────────────────────────────────────────────────────┤ │ │ │ 1. User Request │ │ ─────────────────► "List files in /tmp" │ │ │ │ 2. LLM Generates Tool Call │ │ ─────────────────► tool_calls: [{ │ │ name: "list_directory", │ │ arguments: {path: "/tmp"} │ │ }] │ │ │ │ 3. App Executes Tool via MCP │ │ ─────────────────► MCP Server executes list_directory │ │ Returns: ["file1.txt", "file2.txt"] │ │ │ │ 4. Tool Result Sent Back to LLM │ │ ─────────────────► role: "tool", content: [...] │ │ │ │ 5. LLM Generates Final Response │ │ ─────────────────► "The /tmp directory contains 2 files..." │ │ │ └─────────────────────────────────────────────────────────────────────┘ ``` ## Quick Start ### 1. Create MCP Config Create `mcp.json`: ```json { "mcpServers": { "filesystem": { "command": "npx", "args": ["-y", "@modelcontextprotocol/server-filesystem", "/tmp"] } } } ``` ### 2. Start Server with MCP ```bash # Simple mode vllm-mlx serve mlx-community/Qwen3-4B-4bit --mcp-config mcp.json # Continuous batching vllm-mlx serve mlx-community/Qwen3-4B-4bit --mcp-config mcp.json --continuous-batching ``` ### 3. Verify MCP Status ```bash # Check MCP status curl http://localhost:8000/v1/mcp/status # List available tools curl http://localhost:8000/v1/mcp/tools ``` ## Tool Calling Example ```python import json import httpx BASE_URL = "http://localhost:8000" # 1. Get available tools tools_response = httpx.get(f"{BASE_URL}/v1/mcp/tools") tools = tools_response.json()["tools"] # 2. Send request with tools response = httpx.post( f"{BASE_URL}/v1/chat/completions", json={ "model": "default", "messages": [{"role": "user", "content": "List files in /tmp"}], "tools": tools, "max_tokens": 1024 } ) result = response.json() message = result["choices"][0]["message"] # 3. Check for tool calls if message.get("tool_calls"): tool_call = message["tool_calls"][0] # 4. Execute tool via MCP exec_response = httpx.post( f"{BASE_URL}/v1/mcp/execute", json={ "server": "filesystem", "tool": tool_call["function"]["name"], "arguments": json.loads(tool_call["function"]["arguments"]) } ) tool_result = exec_response.json() # 5. Send result back to LLM messages = [ {"role": "user", "content": "List files in /tmp"}, message, { "role": "tool", "tool_call_id": tool_call["id"], "content": json.dumps(tool_result["result"]) } ] final_response = httpx.post( f"{BASE_URL}/v1/chat/completions", json={"model": "default", "messages": messages} ) print(final_response.json()["choices"][0]["message"]["content"]) ``` ## MCP Endpoints | Endpoint | Method | Description | |----------|--------|-------------| | `/v1/mcp/status` | GET | Check MCP status | | `/v1/mcp/tools` | GET | List available tools | | `/v1/mcp/execute` | POST | Execute a tool | ## Example MCP Servers ### Filesystem ```json { "mcpServers": { "filesystem": { "command": "npx", "args": ["-y", "@modelcontextprotocol/server-filesystem", "/tmp"] } } } ``` ### GitHub ```json { "mcpServers": { "github": { "command": "npx", "args": ["-y", "@modelcontextprotocol/server-github"], "env": { "GITHUB_TOKEN": "your-token" } } } } ``` ### PostgreSQL ```json { "mcpServers": { "postgres": { "command": "npx", "args": ["-y", "@modelcontextprotocol/server-postgres"], "env": { "DATABASE_URL": "postgresql://user:pass@localhost/db" } } } } ``` ### Brave Search ```json { "mcpServers": { "brave-search": { "command": "npx", "args": ["-y", "@modelcontextprotocol/server-brave-search"], "env": { "BRAVE_API_KEY": "your-key" } } } } ``` ## Multiple MCP Servers ```json { "mcpServers": { "filesystem": { "command": "npx", "args": ["-y", "@modelcontextprotocol/server-filesystem", "/tmp"] }, "github": { "command": "npx", "args": ["-y", "@modelcontextprotocol/server-github"], "env": { "GITHUB_TOKEN": "your-token" } } } } ``` ## Interactive MCP Chat For testing MCP interactively: ```bash python examples/mcp_chat.py ``` ## Supported Tool Formats vllm-mlx supports 12 tool call parsers covering all major model families. See [Tool Calling](tool-calling.md) for the full list of parsers, aliases, and examples. ## Security vllm-mlx includes security measures to prevent command injection attacks via MCP servers. ### Command Whitelist Only trusted commands are allowed by default: | Category | Allowed Commands | |----------|-----------------| | Node.js | `npx`, `npm`, `node` | | Python | `uvx`, `uv`, `python`, `python3`, `pip`, `pipx` | | Docker | `docker` | | MCP Servers | `mcp-server-*` (official servers) | ### Blocked Patterns The following patterns are blocked to prevent injection attacks: - Command chaining: `;`, `&&`, `||`, `|` - Command substitution: `` ` ``, `$()` - Path traversal: `../` - Dangerous env vars: `LD_PRELOAD`, `PATH`, `PYTHONPATH` ### Example: Blocked Attack ```json { "mcpServers": { "malicious": { "command": "bash", "args": ["-c", "rm -rf /"] } } } ``` This config will be rejected: ``` ValueError: MCP server 'malicious': Command 'bash' is not in the allowed commands whitelist. ``` ### Development Mode (Unsafe) For development only, you can bypass security validation: ```json { "mcpServers": { "custom": { "command": "my-custom-server", "skip_security_validation": true } } } ``` **WARNING**: Never use `skip_security_validation` in production! ### Custom Whitelist To add custom commands to the whitelist programmatically: ```python from vllm_mlx.mcp import MCPCommandValidator, set_validator # Add custom commands validator = MCPCommandValidator( custom_whitelist={"my-trusted-server", "another-server"} ) set_validator(validator) ``` ## Tool Execution Sandboxing Beyond command validation, vllm-mlx provides runtime sandboxing for tool executions: ### Sandbox Features | Feature | Description | |---------|-------------| | Tool Allowlisting | Only permit specific tools to execute | | Tool Blocklisting | Block specific dangerous tools | | Argument Validation | Block dangerous patterns in tool arguments | | Rate Limiting | Limit tool calls per minute | | Audit Logging | Track all tool executions | ### Blocked Argument Patterns Tool arguments are validated for dangerous patterns: - Path traversal: `../` - System directories: `/etc/`, `/proc/`, `/sys/` - Root access: `/root/`, `~root` ### High-Risk Tool Detection Tools matching these patterns trigger security warnings: - `execute`, `run_command`, `shell`, `eval`, `exec`, `system`, `subprocess` ### Custom Sandbox Configuration ```python from vllm_mlx.mcp import ToolSandbox, set_sandbox # Create sandbox with custom settings sandbox = ToolSandbox( # Only allow specific tools (whitelist mode) allowed_tools={"read_file", "list_directory"}, # Block specific tools (blacklist mode) blocked_tools={"execute_command", "run_shell"}, # Rate limit: max 30 calls per minute max_calls_per_minute=30, # Optional audit callback audit_callback=lambda audit: print(f"Tool: {audit.tool_name}, Success: {audit.success}"), ) set_sandbox(sandbox) ``` ### Accessing Audit Logs ```python from vllm_mlx.mcp import get_sandbox sandbox = get_sandbox() # Get recent audit entries entries = sandbox.get_audit_log(limit=50) # Filter by tool name file_ops = sandbox.get_audit_log(tool_filter="file") # Get only errors errors = sandbox.get_audit_log(errors_only=True) # Clear audit log sandbox.clear_audit_log() ``` ### Sensitive Data Redaction Audit logs automatically redact sensitive fields (password, token, secret, key, credential, auth) and truncate large values. ## Troubleshooting ### MCP server not connecting Check that the MCP server command is correct: ```bash npx -y @modelcontextprotocol/server-filesystem /tmp ``` ### Tool not executing Verify tool is available: ```bash curl http://localhost:8000/v1/mcp/tools | jq '.tools[].name' ``` ### Tool call not parsed Ensure you're using a model that supports function calling (Qwen3, Llama-3.2-Instruct). ### Command not in whitelist If you see "Command X is not in the allowed commands whitelist", either: 1. Use an allowed command (see whitelist above) 2. Add the command to a custom whitelist 3. Use `skip_security_validation: true` (development only) # Documentation page: `guides/model-registry.md` # Multi-Model Serving `vllm-mlx` can serve a registry of named models behind one process and one OpenAI-compatible API surface. This mode is designed for Apple Silicon machines where unified memory is the main constraint: - models load lazily on first use - idle models are evicted with an LRU policy under a memory budget - contention can be configured to wait, fail fast, or preempt active models - `/v1/models` reflects the configured registry instead of a single default model ## When to Use It Use registry-backed serving when you want one server to expose multiple models such as: - a small low-latency chat model - a larger reasoning or coding model - a multimodal model for image or video requests Keep single-model serving when you want the smallest operational surface and the highest per-model simplicity. ## Start the Server ```bash vllm-mlx serve --models-config /etc/vllm-mlx/models.yaml --host 0.0.0.0 --port 8000 ``` You can still use global serve flags such as: - `--api-key` - `--rate-limit` - `--timeout` - `--default-temperature` - `--default-top-p` - `--reasoning-parser` - `--enable-auto-tool-choice` - `--tool-call-parser` Do not combine `--models-config` with: - a positional model argument - `--served-model-name` ## Registry File The registry is a YAML file with two top-level sections: - `manager`: global budget and contention behavior - `models`: named model entries that clients select via the OpenAI `model` field Example: ```yaml manager: memory_budget_gb: 100 contention_policy: strategy: wait_then_preempt wait_timeout_s: 45 preempt_after_s: 15 models: - name: fast path: /Users/david/ai-models/mlx_models/gemma-4-E2B-it-5bit preload: true continuous_batching: false estimated_memory_gb: 4 - name: smart path: /Users/david/ai-models/mlx_models/Qwen3.5-27B-VLM-MTP-8bit continuous_batching: true enable_mtp: true estimated_memory_gb: 36 - name: vision path: /Users/david/ai-models/mlx_models/gemma-4-31B-it-6bit mllm: true continuous_batching: true estimated_memory_gb: 44 ``` ## Manager Settings ### `memory_budget_gb` Total resident-model budget for the registry manager. **This budget counts model weights only.** It is the number the manager compares against when deciding whether a new model fits or an idle one must be evicted. It does not include, and does not reserve room for: - KV cache - activations during prefill and decode - OS / filesystem cache - other colocated services On a 128 GB machine, a practical starting point is often `80-100 GB`. ### Budget vs. the Metal allocation ceiling The manager budget and the MLX allocation ceiling are two separate numbers, and the budget does not derive from the ceiling. The ceiling is installed at engine start from `--gpu-memory-utilization`: ``` allocation_ceiling = gpu_memory_utilization x device_working_set_size ``` The weights *plus* the KV cache *plus* activations all have to fit under that ceiling, while the budget only accounts for the weights. If the budget is set above what is actually allocatable, the manager's arithmetic says N models fit, it keeps them all resident, and MLX hits the ceiling — so you get a hard out-of-memory failure instead of the graceful eviction the budget exists to provide. The invariant to maintain is: ``` memory_budget_gb <= gpu_memory_utilization x device_RAM - KV/activation headroom - prefix cache actually resident ``` The server reconciles the two process-wide terms at startup and logs them together with the prefix-cache setting: ``` Registry memory budget: 68.0 GB of model weights; Metal allocation ceiling 64.0 GB (50% of 128.0 GB, from serve default); prefix-cache maximum 20.0 GB per continuous-batching engine (--cache-memory-mb, 2 of 3 entries) ``` When the weights budget alone does not fit below the ceiling, startup warns: ``` WARNING models-config manager.memory_budget_gb (68.0 GB) exceeds the Metal allocation ceiling (64.0 GB). ... ``` This is a diagnostic, not a clamp — the server still starts with the budget you configured. It is also a *necessary, not sufficient* condition: passing the check does not mean you will not run out of memory, because the KV cache, prefix cache and activations all come out of the same ceiling and are workload-dependent. Treat the ceiling as an upper bound and leave real margin below it. Notes on how the check is computed: - The Metal limit is installed only by continuous-batching entries — that is the one path calling `mx.set_memory_limit`, and simple-mode entries are not even constructed with a `gpu_memory_utilization`. The check therefore considers only the effective utilization of continuous-batching entries, taking the *lowest*, since each such load re-installs the process-wide limit. A `gpu_memory_utilization` set on a simple-mode entry has no effect on the ceiling and is ignored here. - A registry with no continuous-batching entries gets **no** attributed ceiling: nothing installs one, so the report says so rather than deriving a figure from a value that is never applied. The serve default likewise only competes when some continuous-batching entry actually inherits it. - The conflict check compares **only** the weights budget against the ceiling, because both are process-wide totals and therefore directly comparable. - `--cache-memory-mb` is **not** subtracted from the ceiling. It is a per-engine maximum: it is cloned into each resident continuous-batching engine and allocated lazily, and simple-mode entries never receive it at all. Subtracting it once would understate capacity with one resident model and overstate it with several, so it is reported next to the ceiling rather than folded into it. It is reported only when it can actually bind — that is, for continuous-batching entries using the memory-aware prefix cache (not `--use-paged-cache`). - A separate warning fires when `--cache-memory-mb` alone is at or above the ceiling, which is a configuration error in its own right. - On hosts where MLX cannot report a Metal working-set size, the check reports that the budget could not be reconciled and issues no warning. ### `contention_policy` Controls what happens when a request needs a model that does not currently fit. Supported strategies: - `fail`: return capacity failure immediately - `wait`: wait for capacity to free up - `preempt`: cancel active requests on other models and evict them - `wait_then_fail`: wait up to `wait_timeout_s`, then fail - `wait_then_preempt`: wait up to `preempt_after_s`, then start preempting, and stop waiting at `wait_timeout_s` Recommended defaults: - shared internal service: `wait_then_preempt` - user-facing low-latency API: `wait_then_fail` - strict isolation / no interruption: `wait` ## Model Entry Fields Required: - `name`: request-time model id - one of `path`, `source`, or `model` Optional: - `preload`: load this model at startup - `continuous_batching`: override the global mode for this model - `mllm`: force multimodal loading when autodetect is not enough - `enable_mtp`: enable native MTP for this model - `prefill_step_size` - `specprefill` - `specprefill_threshold` - `specprefill_keep_pct` - `specprefill_draft_model` - `stream_interval` - `gpu_memory_utilization` - `estimated_memory_gb` ## Sizing Rules For deterministic eviction behavior: - local models should have real weight files on disk - non-local model ids should set `estimated_memory_gb` If a registry entry points at a non-local source and no `estimated_memory_gb` is provided, startup will reject the config. This prevents the manager from making bad eviction decisions from guesswork. Both sizing paths are **weight estimates, not total runtime memory**: - for a local source, the estimate is the summed on-disk size of the entry's `.safetensors` / `.gguf` files - for a declared model id, the estimate is the operator-supplied `estimated_memory_gb` Neither includes KV cache or activations, so a model's real peak footprint is larger than the number the manager charges against `memory_budget_gb`. Size the budget with that gap in mind — see [Budget vs. the Metal allocation ceiling](#budget-vs-the-metal-allocation-ceiling). ## Request Routing Clients select a registry entry through the normal OpenAI `model` field: ```python from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed") resp = client.chat.completions.create( model="smart", messages=[{"role": "user", "content": "Explain speculative decoding."}], ) ``` If the requested model is not registered, the server returns `404` and lists the configured model ids. ## Operational Checks ### Inspect registry state ```bash curl http://localhost:8000/v1/models ``` Registry-backed responses include the configured model ids and current state such as: - `loaded` - `loading` - `unloaded` - `preempting` ### Verify a cold-load path ```bash curl http://localhost:8000/v1/chat/completions \ -H 'Content-Type: application/json' \ -d '{ "model": "fast", "messages": [{"role": "user", "content": "hello"}], "max_tokens": 32 }' ``` Then repeat with a second model id to verify: - lazy load works - the memory budget is enforced - the selected contention policy behaves as expected ## Recommended Rollout 1. Start with local-disk model paths, not remote model ids. 2. Set `estimated_memory_gb` for every large model, even when local, so your operational budget stays explicit. 3. Preload only the model that must be instantly available. 4. Verify `/v1/models` before exposing the endpoint to shared traffic. 5. Exercise the configured contention strategy under load before production cutover. ## Failure Modes to Expect - Bad or missing `estimated_memory_gb` on non-local sources: config load failure - Too-small `memory_budget_gb`: repeated capacity failures or unnecessary preemption - Too-large `memory_budget_gb` relative to `--gpu-memory-utilization`: MLX out-of-memory instead of eviction (the startup log warns about this) - Over-aggressive `preempt` policy: active requests get cancelled during model swaps - Too many `preload: true` entries: startup load storm and immediate budget pressure ## Choosing Per-Model Overrides Use global defaults for the common case, then override only the model-specific performance knobs that materially differ. Good candidates for per-model overrides: - `continuous_batching` - `enable_mtp` - `mllm` - `prefill_step_size` - `stream_interval` Keep these global unless you have a strong reason not to: - auth - rate limits - request timeout - reasoning parser selection - tool parser selection - manager memory budget / contention policy # Documentation page: `guides/moe-top-k.md` # MoE top_k override (`--moe-top-k`) Reduces the number of experts activated per token in Mixture-of-Experts models like Qwen3-30B-A3B, trading a small amount of quality for meaningfully higher decode throughput. > **Status:** opt-in flag. Default behaviour is unchanged. Quality numbers > below are for Qwen3-30B-A3B-4bit on M4 Max 128 GB — verify on your model > before shipping this to production workloads. ## What it does Qwen3-30B-A3B is trained with `top_k=8` — every token picks 8 out of 128 experts. On Apple Silicon at batch=1 decode the expert matmul (`SwitchGLU`) is the single biggest chunk of each layer's compute, and that cost scales roughly linearly with `top_k`. Lowering `top_k` at inference time has been shown (LExI 2025, Lynx 2024) to preserve most of the trained quality while cutting decode time materially. `--moe-top-k N` iterates every layer of the loaded model, and on each layer that has `.mlp.switch_mlp` (i.e. a sparse-MoE block) sets `top_k = N`. Dense layers and dense models are untouched — the flag is a no-op for them. ## Usage ```bash # Server vllm-mlx serve mlx-community/Qwen3-30B-A3B-4bit \ --continuous-batching \ --moe-top-k 4 # Bench vllm-mlx bench mlx-community/Qwen3-30B-A3B-4bit --moe-top-k 4 ``` The flag is rejected if `N` is greater than the model's trained `top_k` (it only makes sense to lower, never to raise). ## Measured impact ### Decode throughput (M4 Max 128 GB, batch=1, greedy) | top_k | tok/s | vs baseline | |---:|---:|---:| | 8 (baseline) | 126.5 | — | | 6 | 136.1 | +7.6% | | 5 | 140.3 | +10.9% | | 4 | 147.3 | +16.5% | ### Quality (Qwen3-30B-A3B-4bit, lm-evaluation-harness, MLX backend) | top_k | MMLU (acc) | GSM8K (exact match) | Δ vs baseline | |---:|---:|---:|---:| | 8 | TBD | TBD | — | | 6 | TBD | TBD | TBD | | 5 | TBD | TBD | TBD | | 4 | TBD | TBD | TBD | MMLU: 200 randomly-selected samples, 0-shot. GSM8K: 100 randomly-selected samples, 0-shot, exact-match strict. These numbers are **directional** — full suites are larger and would shift the absolute accuracy but not the relative delta between configs by much. ### Greedy output parity With `top_k=4` on the 4-bit checkpoint we observed **identical first 16 generated tokens** vs the baseline across every probe prompt we tried. This suggests top_k=4 does not change the argmax in the early decode steps — the model is internally robust to dropping half its activated experts. At `top_k=3` or lower quality would start to degrade visibly (not measured here; inferred from LExI paper), so the flag is intentionally not lowered below 1 at the config validation layer but the recommended floor for production is `top_k=4`. ## When to use it, when not to Use it when: - You run a Qwen3 MoE (or compatible: Qwen3.5 MoE, Gemma-MoE) and single-user decode throughput is your bottleneck. - You have a workload where a small quality drop is acceptable in exchange for a visible latency improvement. - You're deploying on memory-bandwidth-bound hardware (M-series Apple Silicon) where expert gather dominates per-step decode time. Skip it when: - You serve dense models — flag is a no-op, adds nothing. - You care about top-1% leaderboard accuracy on eval suites. - You run long chain-of-thought / "thinking mode" generations where the quality cliff may be steeper than 0-shot MMLU suggests. ## Stacking with other optimizations This flag composes with quantization. On Qwen3-30B-A3B-4bit our measured stack is: - 4-bit + top_k=8: 126.5 tok/s (baseline) - 4-bit + top_k=4: 147.3 tok/s (+16.5%) - 3-bit + top_k=8: 138.6 tok/s (+9.6%) - 3-bit + top_k=6: 147.1 tok/s (+16.3%) — quality divergence measurable - 3-bit + top_k=4: 157.3 tok/s (+24%) — **output quality breaks** (model answered a different question in our smoke test) 3-bit + top_k=4 compounded the numerical error past the point where the argmax is stable. Stick to at most one aggressive knob: either 4-bit + top_k=4 or 3-bit + top_k=6. Both give approximately the same tok/s (~147) with very different quality profiles. ## Internals - Patch helper: `vllm_mlx.scheduler.apply_moe_top_k_override(model, k)` - Applied in `Scheduler.__init__` after the model is loaded. - Tests: `tests/test_moe_top_k.py` — covers dense models, mixed architectures, and validation paths. ## References - LExI: Layer-Adaptive Active Experts, [arXiv 2509.02753](https://arxiv.org/html/2509.02753) - Not All Experts are Equal (NAEE), [ACL 2024](https://aclanthology.org/2024.acl-long.334.pdf) - SwiftLM (`SWIFTLM_TOP_K` env knob prior art), [github.com/SharpAI/SwiftLM](https://github.com/SharpAI/SwiftLM) # Documentation page: `guides/multimodal.md` # Multimodal Models (Images & Video) vllm-mlx supports vision-language models for image and video understanding. ## Supported Models - Qwen3-VL (recommended) - Qwen2-VL - Gemma 3 - LLaVA - Idefics - PaliGemma - Pixtral - Molmo - DeepSeek-VL ## Starting a Multimodal Server ```bash vllm-mlx serve mlx-community/Qwen3-VL-4B-Instruct-3bit --port 8000 ``` Models with "VL", "Vision", or "mllm" in the name are auto-detected as multimodal. ## Image Analysis ### Via OpenAI SDK ```python from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed") # Image from URL response = client.chat.completions.create( model="default", messages=[{ "role": "user", "content": [ {"type": "text", "text": "What's in this image?"}, {"type": "image_url", "image_url": {"url": "https://example.com/image.jpg"}} ] }], max_tokens=256 ) print(response.choices[0].message.content) ``` ### Base64 Images ```python import base64 def encode_image(path): with open(path, "rb") as f: return base64.b64encode(f.read()).decode("utf-8") base64_image = encode_image("photo.jpg") response = client.chat.completions.create( model="default", messages=[{ "role": "user", "content": [ {"type": "text", "text": "Describe this image"}, {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{base64_image}"}} ] }] ) ``` ### Via curl ```bash curl http://localhost:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "default", "messages": [{ "role": "user", "content": [ {"type": "text", "text": "What is in this image?"}, {"type": "image_url", "image_url": {"url": "https://example.com/image.jpg"}} ] }], "max_tokens": 256 }' ``` ## Video Analysis ### Via OpenAI SDK ```python response = client.chat.completions.create( model="default", messages=[{ "role": "user", "content": [ {"type": "text", "text": "What happens in this video?"}, {"type": "video_url", "video_url": {"url": "https://example.com/video.mp4"}} ] }], max_tokens=512 ) ``` ### Video Parameters Control frame extraction via extra body parameters: ```python response = client.chat.completions.create( model="default", messages=[{ "role": "user", "content": [ {"type": "text", "text": "Describe this video"}, {"type": "video_url", "video_url": {"url": "video.mp4"}} ] }], extra_body={ "video_fps": 2.0, "video_max_frames": 32 } ) ``` ### Via curl ```bash curl http://localhost:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "default", "messages": [{ "role": "user", "content": [ {"type": "text", "text": "Describe this video"}, {"type": "video_url", "video_url": {"url": "https://example.com/video.mp4"}} ] }], "video_fps": 2.0, "video_max_frames": 16 }' ``` ## Supported Formats ### Images | Format | Example | |--------|---------| | URL | `{"type": "image_url", "image_url": {"url": "https://..."}}` | | Local file | `{"type": "image_url", "image_url": {"url": "/path/to/image.jpg"}}` | | Base64 | `{"type": "image_url", "image_url": {"url": "data:image/jpeg;base64,..."}}` | ### Videos | Format | Example | |--------|---------| | URL | `{"type": "video_url", "video_url": {"url": "https://..."}}` | | Local file | `{"type": "video", "video": "/path/to/video.mp4"}` | | Base64 | `{"type": "video_url", "video_url": {"url": "data:video/mp4;base64,..."}}` | ### Remote URL safety Remote image, video, and audio URLs are checked before each fetch and redirect hop. URLs that resolve to localhost, link-local, private, or otherwise non-global addresses are rejected with a generic client error while detailed diagnostics stay in server logs. This validation does not pin the IP address used by the later HTTP transport connection. In environments where DNS rebinding or split-horizon DNS is in scope, run vllm-mlx behind network egress controls or fetch media through a trusted proxy that enforces the destination policy at connect time. ## Python API ```python from vllm_mlx.models import MLXMultimodalLM mllm = MLXMultimodalLM("mlx-community/Qwen3-VL-4B-Instruct-3bit") mllm.load() # Image description = mllm.describe_image("photo.jpg") # Video description = mllm.describe_video("video.mp4", fps=2.0) # Custom prompt output = mllm.generate( prompt="Compare these images", images=["img1.jpg", "img2.jpg"] ) ``` ## Performance Tips ### Images - Smaller resolutions process faster (224x224 vs 1920x1080) - Use appropriate resolution for your task ### Videos - Lower FPS = faster processing - Fewer frames = less memory usage - 64 frames is practical maximum (96+ causes GPU timeout) ## Benchmarks Tested on Apple M4 Max with 128 GB unified memory. ### Qwen3-VL-4B-Instruct-3bit | Resolution | Time | Tokens | Speed | Memory | |------------|------|--------|-------|--------| | 224x224 | 0.87s | 124 | 143 tok/s | 2.6 GB | | 448x448 | 1.01s | 107 | 106 tok/s | 3.1 GB | | 768x768 | 1.42s | 127 | 89 tok/s | 3.4 GB | | 1024x1024 | 1.85s | 116 | 63 tok/s | 3.6 GB | ### Qwen3-VL-8B-Instruct-4bit | Resolution | Time | Tokens | Speed | Memory | |------------|------|--------|-------|--------| | 224x224 | 1.08s | 78 | 73 tok/s | 5.6 GB | | 448x448 | 1.41s | 70 | 50 tok/s | 6.1 GB | | 768x768 | 2.06s | 91 | 44 tok/s | 6.5 GB | | 1024x1024 | 3.02s | 76 | 25 tok/s | 7.6 GB | ### Gemma 3 4B 4bit | Resolution | Time | Tokens | Speed | Memory | |------------|------|--------|-------|--------| | 224x224 | 0.95s | 30 | 32 tok/s | 5.2 GB | | 448x448 | 0.99s | 34 | 34 tok/s | 5.2 GB | | 768x768 | 0.99s | 32 | 32 tok/s | 5.2 GB | | 1024x1024 | 0.95s | 28 | 29 tok/s | 5.2 GB | ### Running Benchmarks ```bash # Quick benchmark vllm-mlx-bench --model mlx-community/Qwen3-VL-4B-Instruct-3bit --quick # Full benchmark with more resolutions vllm-mlx-bench --model mlx-community/Qwen3-VL-4B-Instruct-3bit # Video benchmark vllm-mlx-bench --model mlx-community/Qwen3-VL-4B-Instruct-3bit --video ``` ## MLLM Cache vllm-mlx includes a prefix cache system for multimodal models that can significantly speed up repeated requests with the same images. ### How It Works When you send an image to the model, the vision encoder processes it into embeddings. This processing takes 1-2 seconds. The MLLM cache stores these embeddings along with the KV cache state, so subsequent requests with the same image skip the vision encoder entirely. The cache uses content-based hashing (similar to LMCache) to identify identical images regardless of how they're provided (URL, base64, or file path). ### Enabling the Cache ```bash # Enable with default settings (512 MB max) vllm-mlx serve mlx-community/Qwen3-VL-4B-Instruct-3bit --enable-mllm-cache # With custom memory limit vllm-mlx serve mlx-community/Qwen3-VL-4B-Instruct-3bit \ --enable-mllm-cache \ --mllm-cache-max-mb 1024 ``` ### Python API ```python from vllm_mlx.mllm_cache import MLLMPrefixCacheManager # Create cache manager cache = MLLMPrefixCacheManager(max_memory_mb=512) # Store embeddings and KV cache after processing cache.store( images=["photo.jpg"], prompt="Describe this image", vision_embeddings=embeddings, kv_cache=kv_state, num_tokens=128 ) # Fetch from cache on subsequent requests entry, match_len = cache.fetch(images=["photo.jpg"], prompt="Describe this image") if entry: # Use cached embeddings, skip vision encoder embeddings = entry.vision_embeddings kv_state = entry.kv_cache ``` ### Cache Statistics ```python stats = cache.get_stats() print(f"Hit rate: {stats.hit_rate:.1%}") print(f"Memory used: {stats.memory_used_mb:.1f} MB") print(f"Tokens saved: {stats.tokens_saved}") ``` ### Memory Management The cache uses LRU (Least Recently Used) eviction when memory limit is reached. Each entry tracks: - Vision embeddings size - KV cache size per layer - Access frequency for LRU ordering When memory pressure occurs, least recently accessed entries are evicted first. ## Gradio Chat UI For interactive multimodal chat: ```bash vllm-mlx-chat --served-model-name mlx-community/Qwen3-VL-4B-Instruct-3bit ``` Supports drag-and-drop images and videos. # Documentation page: `guides/python-api.md` # Python API Direct Python API for programmatic access to vllm-mlx. ## Language Models ### Basic Usage ```python from vllm_mlx.models import MLXLanguageModel # Load model model = MLXLanguageModel("mlx-community/Llama-3.2-3B-Instruct-4bit") model.load() # Generate text output = model.generate("What is the capital of France?", max_tokens=100) print(output.text) ``` ### Streaming Generation ```python for chunk in model.stream_generate("Tell me a story about a robot"): print(chunk.text, end="", flush=True) ``` ### Chat Interface ```python messages = [ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Hello, who are you?"} ] response = model.chat(messages) print(response.text) ``` ### Generation Parameters ```python output = model.generate( prompt="Write a poem", max_tokens=256, temperature=0.7, top_p=0.9, stop=["END", "\n\n"] ) ``` | Parameter | Description | Default | |-----------|-------------|---------| | `max_tokens` | Maximum tokens to generate | 256 | | `temperature` | Sampling temperature (0-2) | 0.7 | | `top_p` | Nucleus sampling | 0.9 | | `stop` | Stop sequences | None | ## Vision-Language Models ### Basic Usage ```python from vllm_mlx.models import MLXMultimodalLM # Load model mllm = MLXMultimodalLM("mlx-community/Qwen3-VL-4B-Instruct-3bit") mllm.load() # Describe an image description = mllm.describe_image("photo.jpg") print(description) ``` ### Question Answering ```python answer = mllm.answer_about_image("photo.jpg", "What color is the car?") print(answer) ``` ### Multiple Images ```python output = mllm.generate( prompt="Compare these two images", images=["image1.jpg", "image2.jpg"] ) print(output.text) ``` ### Video Understanding ```python # From local file output = mllm.generate( prompt="What is happening in this video?", videos=["video.mp4"], video_fps=2.0, video_max_frames=16 ) print(output.text) # From URL output = mllm.generate( prompt="Describe this video", videos=["https://example.com/video.mp4"], video_fps=2.0 ) # Convenience method description = mllm.describe_video("video.mp4", fps=2.0) ``` ### Video Parameters | Parameter | Description | Default | |-----------|-------------|---------| | `video_fps` | Frames per second to extract | 2.0 | | `video_max_frames` | Maximum frames to process | 32 | ## Engine API For advanced use cases, use the engine directly: ### Simple Engine ```python from vllm_mlx.engine import SimpleEngine engine = SimpleEngine("mlx-community/Llama-3.2-3B-Instruct-4bit") await engine.start() output = await engine.generate( prompt="Hello world", max_tokens=100 ) print(output.text) await engine.stop() ``` ### Batched Engine ```python from vllm_mlx.engine import BatchedEngine engine = BatchedEngine("mlx-community/Llama-3.2-3B-Instruct-4bit") await engine.start() # Multiple concurrent requests output = await engine.generate( prompt="Hello world", max_tokens=100 ) await engine.stop() ``` ## Output Format All generation methods return a `GenerationOutput`: ```python output = model.generate("Hello") print(output.text) # Generated text print(output.prompt_tokens) # Input token count print(output.completion_tokens) # Output token count print(output.finish_reason) # "stop" or "length" ``` ## Error Handling ```python from vllm_mlx.models import MLXLanguageModel try: model = MLXLanguageModel("invalid-model") model.load() except Exception as e: print(f"Failed to load model: {e}") ``` # Documentation page: `guides/reasoning.md` # Reasoning Models vllm-mlx supports reasoning models that show their thinking process before giving an answer. Models like Qwen3 and DeepSeek-R1 wrap their reasoning in `...` tags, and vllm-mlx can parse these tags to separate the reasoning from the final response. ## Why Use Reasoning Parsing? When a reasoning model generates output, it typically looks like this: ``` Let me analyze this step by step. First, I need to consider the constraints. The answer should be a prime number less than 10. Checking: 2, 3, 5, 7 are all prime and less than 10. The prime numbers less than 10 are: 2, 3, 5, 7. ``` Without reasoning parsing, you get the raw output with the tags included. With reasoning parsing enabled, the thinking process and final answer are separated into distinct fields in the API response. ## Getting Started ### Start the Server with Reasoning Parser ```bash # For Qwen3 models vllm-mlx serve mlx-community/Qwen3-8B-4bit --reasoning-parser qwen3 # For DeepSeek-R1 models vllm-mlx serve mlx-community/DeepSeek-R1-Distill-Qwen-7B-4bit --reasoning-parser deepseek_r1 ``` ### API Response Format When reasoning parsing is enabled, the API response includes a `reasoning` field: **Non-streaming response:** ```json { "choices": [{ "message": { "role": "assistant", "content": "The prime numbers less than 10 are: 2, 3, 5, 7.", "reasoning": "Let me analyze this step by step.\nFirst, I need to consider the constraints.\nThe answer should be a prime number less than 10.\nChecking: 2, 3, 5, 7 are all prime and less than 10." } }] } ``` **Streaming response:** Chunks are sent separately for reasoning and content. During the reasoning phase, chunks have `reasoning` populated. When the model transitions to the final answer, chunks have `content` populated: ```json {"delta": {"reasoning": "Let me analyze"}} {"delta": {"reasoning": " this step by step."}} {"delta": {"reasoning": "\nFirst, I need to"}} ... {"delta": {"content": "The prime"}} {"delta": {"content": " numbers less than 10"}} {"delta": {"content": " are: 2, 3, 5, 7."}} ``` ## Using with OpenAI SDK ```python from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed") # Non-streaming response = client.chat.completions.create( model="default", messages=[{"role": "user", "content": "What are the prime numbers less than 10?"}] ) message = response.choices[0].message print("Reasoning:", message.reasoning) # The thinking process print("Answer:", message.content) # The final answer ``` ### Streaming with Reasoning ```python reasoning_text = "" content_text = "" stream = client.chat.completions.create( model="default", messages=[{"role": "user", "content": "Solve: 2 + 2 = ?"}], stream=True ) for chunk in stream: delta = chunk.choices[0].delta if hasattr(delta, 'reasoning') and delta.reasoning: reasoning_text += delta.reasoning print(f"[Thinking] {delta.reasoning}", end="") if delta.content: content_text += delta.content print(delta.content, end="") print(f"\n\nFinal reasoning: {reasoning_text}") print(f"Final answer: {content_text}") ``` ## Supported Parsers ### Qwen3 Parser (`qwen3`) For Qwen3 models that use explicit `` and `` tags. - Requires **both** opening and closing tags - If tags are missing, output is treated as regular content - Best for: Qwen3-0.6B, Qwen3-4B, Qwen3-8B and similar models ```bash vllm-mlx serve mlx-community/Qwen3-8B-4bit --reasoning-parser qwen3 ``` ### DeepSeek-R1 Parser (`deepseek_r1`) For DeepSeek-R1 models that may omit the opening `` tag. - More lenient than Qwen3 parser - Handles cases where `` is implicit - Content before `` is treated as reasoning even without `` ```bash vllm-mlx serve mlx-community/DeepSeek-R1-Distill-Qwen-7B-4bit --reasoning-parser deepseek_r1 ``` ## How It Works The reasoning parser uses text-based detection to identify thinking tags in the model output. During streaming, it tracks the current position in the output to correctly route each token to either `reasoning` or `content`. ``` Model Output: Step 1: analyze...The answer is 42. ├─────────────────────┤├─────────────────────┤ Parsed: │ reasoning ││ content │ └─────────────────────┘└─────────────────────┘ ``` The parsing is stateless and uses the accumulated text to determine context, making it robust for streaming scenarios where tokens may arrive in arbitrary chunks. ## Tips for Best Results ### Prompting Reasoning models work best when you encourage step-by-step thinking: ```python messages = [ {"role": "system", "content": "Think through problems step by step before answering."}, {"role": "user", "content": "What is 17 × 23?"} ] ``` ### Handling Missing Reasoning Some prompts may not trigger reasoning. In these cases, `reasoning` will be `None` and all output goes to `content`: ```python message = response.choices[0].message if message.reasoning: print(f"Model's thought process: {message.reasoning}") print(f"Answer: {message.content}") ``` ### Temperature and Reasoning Lower temperatures tend to produce more consistent reasoning patterns: ```python response = client.chat.completions.create( model="default", messages=[{"role": "user", "content": "Explain quantum entanglement"}], temperature=0.3 # More focused reasoning ) ``` ## Backward Compatibility When `--reasoning-parser` is not specified, the server behaves as before: - Thinking tags are included in the `content` field - No `reasoning` field is added to responses This ensures existing applications continue to work without changes. ## Example: Math Problem Solver ```python from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed") def solve_math(problem: str) -> dict: """Solve a math problem and return reasoning + answer.""" response = client.chat.completions.create( model="default", messages=[ {"role": "system", "content": "You are a math tutor. Show your work."}, {"role": "user", "content": problem} ], temperature=0.2 ) message = response.choices[0].message return { "problem": problem, "work": message.reasoning, "answer": message.content } result = solve_math("If a train travels 120 km in 2 hours, what is its average speed?") print(f"Problem: {result['problem']}") print(f"\nWork shown:\n{result['work']}") print(f"\nFinal answer: {result['answer']}") ``` ## Curl Examples ### Non-streaming ```bash curl http://localhost:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "default", "messages": [{"role": "user", "content": "What is 15% of 80?"}] }' ``` ### Streaming ```bash curl http://localhost:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "default", "messages": [{"role": "user", "content": "What is 15% of 80?"}], "stream": true }' ``` ## Troubleshooting ### No reasoning field in response - Make sure you started the server with `--reasoning-parser` - Check that the model actually uses thinking tags (not all prompts trigger reasoning) ### Reasoning appears in content - The model may not be using the expected tag format - Try a different parser (`qwen3` vs `deepseek_r1`) ### Truncated reasoning - Increase `--max-tokens` if the model is hitting the token limit mid-thought ## Related - [Supported Models](../reference/models.md) - Models that support reasoning - [Server Configuration](server.md) - All server options - [CLI Reference](../reference/cli.md) - Command line options # Documentation page: `guides/server.md` # OpenAI-Compatible Server vllm-mlx provides a FastAPI server with full OpenAI API compatibility. By default the server binds only to `127.0.0.1`. Use `--host 0.0.0.0` only when you intentionally want to expose it beyond the local machine. ## Starting the Server ### Simple Mode (Default) Maximum throughput for single user: ```bash vllm-mlx serve mlx-community/Llama-3.2-3B-Instruct-4bit --port 8000 ``` ### Continuous Batching Mode For multiple concurrent users: ```bash vllm-mlx serve mlx-community/Llama-3.2-3B-Instruct-4bit --port 8000 --continuous-batching ``` ### With Paged Cache Memory-efficient caching for production: ```bash vllm-mlx serve mlx-community/Llama-3.2-3B-Instruct-4bit --port 8000 --continuous-batching --use-paged-cache ``` ### Registry-Backed Multi-Model Serving Serve a named registry of models behind one endpoint: ```bash vllm-mlx serve --models-config /etc/vllm-mlx/models.yaml --port 8000 ``` Clients route requests by setting the OpenAI `model` field to one of the configured registry names. See [Multi-Model Serving](model-registry.md) for the registry file format, eviction policy, and rollout guidance. ### With Server-Wide Chat Template Defaults Set server defaults for chat template kwargs. Request-level `chat_template_kwargs` values still win per key. ```bash vllm-mlx serve mlx-community/Qwen3-8B-4bit \ --reasoning-parser qwen3 \ --default-chat-template-kwargs '{"enable_thinking": false}' ``` ## Server Options | Option | Description | Default | |--------|-------------|---------| | `--port` | Server port | 8000 | | `--host` | Server host | 127.0.0.1 | | `--api-key` | API key for authentication | None | | `--rate-limit` | Requests per minute per client (0 = disabled) | 0 | | `--timeout` | Request timeout in seconds | 300 | | `--enable-metrics` | Expose Prometheus metrics on `/metrics` | False | | `--continuous-batching` | Enable batching for multi-user | False | | `--use-paged-cache` | Enable paged KV cache | False | | `--cache-memory-mb` | Cache memory limit in MB | Auto | | `--cache-memory-percent` | Fraction of RAM for cache | 0.20 | | `--max-tokens` | Default max tokens | 32768 | | `--max-request-tokens` | Maximum `max_tokens` accepted from API clients | 32768 | | `--default-temperature` | Default temperature when not specified | None | | `--default-top-p` | Default top_p when not specified | None | | `--default-chat-template-kwargs` | Default chat template kwargs used when request `chat_template_kwargs` is omitted (JSON object) | None | | `--stream-interval` | Tokens per stream chunk | 1 | | `--mcp-config` | Path to MCP config file | None | | `--reasoning-parser` | Parser for reasoning models (`qwen3`, `deepseek_r1`) | None | | `--embedding-model` | Pre-load an embedding model at startup | None | | `--enable-auto-tool-choice` | Enable automatic tool calling | False | | `--tool-call-parser` | Tool call parser (see [Tool Calling](tool-calling.md)) | None | | `--models-config` | YAML registry file for multi-model serving | None | ## API Endpoints ### Chat Completions ```bash POST /v1/chat/completions ``` ```python from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed") # Non-streaming response = client.chat.completions.create( model="default", messages=[{"role": "user", "content": "Hello!"}], max_tokens=100 ) # Streaming stream = client.chat.completions.create( model="default", messages=[{"role": "user", "content": "Tell me a story"}], stream=True ) for chunk in stream: if chunk.choices[0].delta.content: print(chunk.choices[0].delta.content, end="") ``` ### Completions ```bash POST /v1/completions ``` ```python response = client.completions.create( model="default", prompt="The capital of France is", max_tokens=50 ) ``` ### Models ```bash GET /v1/models ``` Returns available models. ### Embeddings ```bash POST /v1/embeddings ``` ```python response = client.embeddings.create( model="mlx-community/multilingual-e5-small-mlx", input="Hello world" ) print(response.data[0].embedding[:5]) # First 5 dimensions ``` See [Embeddings Guide](embeddings.md) for details. ### Health Check ```bash GET /health ``` Returns server status. ### Metrics ```bash GET /metrics ``` Prometheus scrape endpoint for server, cache, scheduler, and request metrics. The endpoint is disabled by default and is enabled with `--enable-metrics`. ```bash vllm-mlx serve mlx-community/Llama-3.2-3B-Instruct-4bit \ --enable-metrics ``` `/metrics` is intentionally unauthenticated. Expose it only on a trusted network or behind a reverse proxy / firewall that limits who can scrape it. ### Anthropic Messages API ```bash POST /v1/messages ``` Anthropic-compatible endpoint that allows tools like Claude Code and OpenCode to connect directly to vllm-mlx. Internally it translates Anthropic requests to OpenAI format, runs inference through the engine, and converts the response back to Anthropic format. Capabilities: - Non-streaming and streaming responses (SSE) - System messages (plain string or list of content blocks) - Multi-turn conversations with user and assistant messages - Tool calling with `tool_use` / `tool_result` content blocks - Token counting for budget tracking - Multimodal content (images via `source` blocks) - Client disconnect detection (returns HTTP 499) - Automatic special token filtering in streamed output #### Non-streaming ```python from anthropic import Anthropic client = Anthropic(base_url="http://localhost:8000", api_key="not-needed") response = client.messages.create( model="default", max_tokens=256, messages=[{"role": "user", "content": "Hello!"}] ) print(response.content[0].text) # Response includes: response.id, response.model, response.stop_reason, # response.usage.input_tokens, response.usage.output_tokens ``` #### Streaming Streaming follows the Anthropic SSE event protocol. Events are emitted in this order: `message_start` -> `content_block_start` -> `content_block_delta` (repeated) -> `content_block_stop` -> `message_delta` -> `message_stop` ```python with client.messages.stream( model="default", max_tokens=256, messages=[{"role": "user", "content": "Tell me a story"}] ) as stream: for text in stream.text_stream: print(text, end="") ``` #### System messages System messages can be a plain string or a list of content blocks: ```python # Plain string response = client.messages.create( model="default", max_tokens=256, system="You are a helpful coding assistant.", messages=[{"role": "user", "content": "Write a hello world in Python"}] ) # List of content blocks response = client.messages.create( model="default", max_tokens=256, system=[ {"type": "text", "text": "You are a helpful assistant."}, {"type": "text", "text": "Be concise in your answers."}, ], messages=[{"role": "user", "content": "What is 2+2?"}] ) ``` #### Tool calling Define tools with `name`, `description`, and `input_schema`. The model returns `tool_use` content blocks when it wants to call a tool. Send results back as `tool_result` blocks. ```python # Step 1: Send request with tools response = client.messages.create( model="default", max_tokens=1024, messages=[{"role": "user", "content": "What's the weather in Paris?"}], tools=[{ "name": "get_weather", "description": "Get weather for a city", "input_schema": { "type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"] } }] ) # Step 2: Check if model wants to use tools for block in response.content: if block.type == "tool_use": print(f"Tool: {block.name}, Input: {block.input}, ID: {block.id}") # response.stop_reason will be "tool_use" # Step 3: Send tool result back response = client.messages.create( model="default", max_tokens=1024, messages=[ {"role": "user", "content": "What's the weather in Paris?"}, {"role": "assistant", "content": response.content}, {"role": "user", "content": [ { "type": "tool_result", "tool_use_id": block.id, "content": "Sunny, 22C" } ]} ], tools=[{ "name": "get_weather", "description": "Get weather for a city", "input_schema": { "type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"] } }] ) print(response.content[0].text) # "The weather in Paris is sunny, 22C." ``` Tool choice modes: | `tool_choice` | Behavior | |---------------|----------| | `{"type": "auto"}` | Model decides whether to call tools (default) | | `{"type": "any"}` | Model must call at least one tool | | `{"type": "tool", "name": "get_weather"}` | Model must call the specified tool | | `{"type": "none"}` | Model will not call any tools | #### Multi-turn conversations ```python messages = [ {"role": "user", "content": "My name is Alice."}, {"role": "assistant", "content": "Nice to meet you, Alice!"}, {"role": "user", "content": "What's my name?"}, ] response = client.messages.create( model="default", max_tokens=100, messages=messages ) ``` #### Token counting ```bash POST /v1/messages/count_tokens ``` Counts input tokens for an Anthropic request using the model's tokenizer. Useful for budget tracking before sending a request. Counts tokens from system messages, conversation messages, tool_use inputs, tool_result content, and tool definitions (name, description, input_schema). ```python import requests resp = requests.post("http://localhost:8000/v1/messages/count_tokens", json={ "model": "default", "messages": [{"role": "user", "content": "Hello, how are you?"}], "system": "You are helpful.", "tools": [{ "name": "search", "description": "Search the web", "input_schema": {"type": "object", "properties": {"q": {"type": "string"}}} }] }) print(resp.json()) # {"input_tokens": 42} ``` #### curl examples Non-streaming: ```bash curl http://localhost:8000/v1/messages \ -H "Content-Type: application/json" \ -d '{ "model": "default", "max_tokens": 256, "messages": [{"role": "user", "content": "Hello!"}] }' ``` Streaming: ```bash curl http://localhost:8000/v1/messages \ -H "Content-Type: application/json" \ -d '{ "model": "default", "max_tokens": 256, "stream": true, "messages": [{"role": "user", "content": "Tell me a joke"}] }' ``` Token counting: ```bash curl http://localhost:8000/v1/messages/count_tokens \ -H "Content-Type: application/json" \ -d '{ "model": "default", "messages": [{"role": "user", "content": "Hello!"}] }' # {"input_tokens": 12} ``` #### Request fields | Field | Type | Required | Default | Description | |-------|------|----------|---------|-------------| | `model` | string | yes | - | Model name (use `"default"` for the loaded model) | | `messages` | list | yes | - | Conversation messages with `role` and `content` | | `max_tokens` | int | yes | - | Maximum number of tokens to generate | | `system` | string or list | no | null | System prompt (string or list of `{"type": "text", "text": "..."}` blocks) | | `stream` | bool | no | false | Enable SSE streaming | | `temperature` | float | no | 0.7 | Sampling temperature (0.0 = deterministic, 1.0 = creative) | | `top_p` | float | no | 0.9 | Nucleus sampling threshold | | `top_k` | int | no | null | Top-k sampling | | `stop_sequences` | list | no | null | Sequences that stop generation | | `tools` | list | no | null | Tool definitions with `name`, `description`, `input_schema` | | `tool_choice` | dict | no | null | Tool selection mode (`auto`, `any`, `tool`, `none`) | | `metadata` | dict | no | null | Arbitrary metadata (passed through, not used by server) | #### Response format Non-streaming response: ```json { "id": "msg_abc123...", "type": "message", "role": "assistant", "model": "default", "content": [ {"type": "text", "text": "Hello! How can I help?"} ], "stop_reason": "end_turn", "stop_sequence": null, "usage": { "input_tokens": 12, "output_tokens": 8 } } ``` When tools are called, `content` includes `tool_use` blocks and `stop_reason` is `"tool_use"`: ```json { "content": [ {"type": "text", "text": "Let me check the weather."}, { "type": "tool_use", "id": "call_abc123", "name": "get_weather", "input": {"city": "Paris"} } ], "stop_reason": "tool_use" } ``` Stop reasons: | `stop_reason` | Meaning | |---------------|---------| | `end_turn` | Model finished naturally | | `tool_use` | Model wants to call a tool | | `max_tokens` | Hit the `max_tokens` limit | #### Using with Claude Code Point Claude Code directly at your vllm-mlx server: ```bash # Start the server vllm-mlx serve mlx-community/Qwen3-Coder-Next-235B-A22B-4bit \ --continuous-batching \ --enable-auto-tool-choice \ --tool-call-parser hermes # In another terminal, configure Claude Code export ANTHROPIC_BASE_URL=http://localhost:8000 export ANTHROPIC_API_KEY=not-needed claude ``` ### Server Status ```bash GET /v1/status ``` Real-time monitoring endpoint that returns server-wide statistics and per-request details. Useful for debugging performance, tracking cache efficiency, and monitoring Metal GPU memory. ```bash curl -s http://localhost:8000/v1/status | python -m json.tool ``` Example response: ```json { "status": "running", "model": "mlx-community/Qwen3-8B-4bit", "uptime_s": 342.5, "steps_executed": 1247, "num_running": 1, "num_waiting": 0, "total_requests_processed": 15, "total_prompt_tokens": 28450, "total_completion_tokens": 3200, "metal": { "active_memory_gb": 5.2, "peak_memory_gb": 8.1, "cache_memory_gb": 2.3 }, "cache": { "type": "memory_aware_cache", "entries": 5, "hit_rate": 0.87, "memory_mb": 2350 }, "requests": [ { "request_id": "req_abc123", "phase": "generation", "tokens_per_second": 45.2, "ttft_s": 0.8, "progress": 0.35, "cache_hit_type": "prefix", "cached_tokens": 1200, "generated_tokens": 85, "max_tokens": 256 } ] } ``` Response fields: | Field | Description | |-------|-------------| | `status` | Server state: `running`, `stopped`, or `not_loaded` | | `model` | Name of the loaded model | | `uptime_s` | Seconds since the server started | | `steps_executed` | Total inference steps executed | | `num_running` | Number of requests currently generating tokens | | `num_waiting` | Number of requests queued for prefill | | `total_requests_processed` | Total requests completed since startup | | `total_prompt_tokens` | Total prompt tokens processed since startup | | `total_completion_tokens` | Total completion tokens generated since startup | | `metal.active_memory_gb` | Current Metal GPU memory in use (GB) | | `metal.peak_memory_gb` | Peak Metal GPU memory usage (GB) | | `metal.cache_memory_gb` | Metal cache memory usage (GB) | | `cache` | Cache statistics (type, entries, hit rate, memory usage) | | `requests` | List of active requests with per-request details | Per-request fields in `requests`: | Field | Description | |-------|-------------| | `request_id` | Unique request identifier | | `phase` | Current phase: `queued`, `prefill`, or `generation` | | `tokens_per_second` | Generation throughput for this request | | `ttft_s` | Time to first token (seconds) | | `progress` | Completion percentage (0.0 to 1.0) | | `cache_hit_type` | Cache match type: `exact`, `prefix`, `supersequence`, `lcp`, or `miss` | | `cached_tokens` | Number of tokens served from cache | | `generated_tokens` | Tokens generated so far | | `max_tokens` | Maximum tokens requested | ## Tool Calling Enable OpenAI-compatible tool calling with `--enable-auto-tool-choice`: ```bash vllm-mlx serve mlx-community/Devstral-Small-2507-4bit \ --enable-auto-tool-choice \ --tool-call-parser mistral ``` Use the `--tool-call-parser` option to select the parser for your model: | Parser | Models | |--------|--------| | `auto` | Auto-detect (tries all parsers) | | `mistral` | Mistral, Devstral | | `qwen` | Qwen, Qwen3 | | `llama` | Llama 3.x, 4.x | | `hermes` | Hermes, NousResearch | | `deepseek` | DeepSeek V3, R1 | | `kimi` | Kimi K2, Moonshot | | `granite` | IBM Granite 3.x, 4.x | | `nemotron` | NVIDIA Nemotron | | `xlam` | Salesforce xLAM | | `functionary` | MeetKai Functionary | | `glm47` | GLM-4.7, GLM-4.7-Flash | ```python response = client.chat.completions.create( model="default", messages=[{"role": "user", "content": "What's the weather in Paris?"}], tools=[{ "type": "function", "function": { "name": "get_weather", "description": "Get weather for a city", "parameters": { "type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"] } } }] ) if response.choices[0].message.tool_calls: for tc in response.choices[0].message.tool_calls: print(f"{tc.function.name}: {tc.function.arguments}") ``` See [Tool Calling Guide](tool-calling.md) for full documentation. ## Reasoning Models For models that show their thinking process (Qwen3, DeepSeek-R1), use `--reasoning-parser` to separate reasoning from the final answer: ```bash # Qwen3 models vllm-mlx serve mlx-community/Qwen3-8B-4bit --reasoning-parser qwen3 # DeepSeek-R1 models vllm-mlx serve mlx-community/DeepSeek-R1-Distill-Qwen-7B-4bit --reasoning-parser deepseek_r1 ``` The API response includes a `reasoning` field with the model's thought process: ```python response = client.chat.completions.create( model="default", messages=[{"role": "user", "content": "What is 17 × 23?"}] ) print(response.choices[0].message.reasoning) # Step-by-step thinking print(response.choices[0].message.content) # Final answer ``` For streaming, reasoning chunks arrive first, followed by content chunks: ```python for chunk in stream: delta = chunk.choices[0].delta if delta.reasoning: print(f"[Thinking] {delta.reasoning}") if delta.content: print(delta.content, end="") ``` See [Reasoning Models Guide](reasoning.md) for full details. ## Structured Output (JSON Mode) Force the model to return valid JSON using `response_format`: ### JSON Object Mode Returns any valid JSON: ```python response = client.chat.completions.create( model="default", messages=[{"role": "user", "content": "List 3 colors"}], response_format={"type": "json_object"} ) # Output: {"colors": ["red", "blue", "green"]} ``` ### JSON Schema Mode Returns JSON matching a specific schema: ```python response = client.chat.completions.create( model="default", messages=[{"role": "user", "content": "List 3 colors"}], response_format={ "type": "json_schema", "json_schema": { "name": "colors", "schema": { "type": "object", "properties": { "colors": { "type": "array", "items": {"type": "string"} } }, "required": ["colors"] } } } ) # Output validated against schema data = json.loads(response.choices[0].message.content) assert "colors" in data ``` ### Curl Example ```bash curl http://localhost:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "default", "messages": [{"role": "user", "content": "List 3 colors"}], "response_format": {"type": "json_object"} }' ``` ## Curl Examples ### Chat ```bash curl http://localhost:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "default", "messages": [{"role": "user", "content": "Hello!"}], "max_tokens": 100 }' ``` ### Streaming ```bash curl http://localhost:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "default", "messages": [{"role": "user", "content": "Hello!"}], "stream": true }' ``` ## Streaming Configuration Control streaming behavior with `--stream-interval`: | Value | Behavior | |-------|----------| | `1` (default) | Send every token immediately | | `2-5` | Batch tokens before sending | | `10+` | Maximum throughput, chunkier output | ```bash # Smooth streaming vllm-mlx serve model --continuous-batching --stream-interval 1 # Batched streaming (better for high-latency networks) vllm-mlx serve model --continuous-batching --stream-interval 5 ``` ## Open WebUI Integration ```bash # 1. Start vllm-mlx server vllm-mlx serve mlx-community/Llama-3.2-3B-Instruct-4bit --port 8000 # 2. Start Open WebUI docker run -d -p 3000:8080 \ -e OPENAI_API_BASE_URL=http://host.docker.internal:8000/v1 \ -e OPENAI_API_KEY=not-needed \ --name open-webui \ ghcr.io/open-webui/open-webui:main # 3. Open http://localhost:3000 ``` ## Production Deployment ### With systemd Create `/etc/systemd/system/vllm-mlx.service`: ```ini [Unit] Description=vLLM-MLX Server After=network.target [Service] Type=simple ExecStart=/usr/local/bin/vllm-mlx serve mlx-community/Qwen3-0.6B-8bit \ --continuous-batching --use-paged-cache --port 8000 Restart=always [Install] WantedBy=multi-user.target ``` ```bash sudo systemctl enable vllm-mlx sudo systemctl start vllm-mlx ``` ### Recommended Settings For production with 50+ concurrent users: ```bash vllm-mlx serve mlx-community/Qwen3-0.6B-8bit \ --continuous-batching \ --use-paged-cache \ --api-key your-secret-key \ --rate-limit 60 \ --timeout 120 \ --port 8000 ``` # Documentation page: `guides/tool-calling.md` # Tool Calling vllm-mlx supports OpenAI-compatible tool calling (function calling) with automatic parsing for many popular model families. ## Quick Start Enable tool calling by adding the `--enable-auto-tool-choice` flag when starting the server: ```bash vllm-mlx serve mlx-community/Devstral-Small-2507-4bit \ --enable-auto-tool-choice \ --tool-call-parser mistral ``` Then use tools with the standard OpenAI API: ```python from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed") response = client.chat.completions.create( model="default", messages=[{"role": "user", "content": "What's the weather in Paris?"}], tools=[{ "type": "function", "function": { "name": "get_weather", "description": "Get weather for a city", "parameters": { "type": "object", "properties": { "city": {"type": "string", "description": "City name"} }, "required": ["city"] } } }] ) # Check for tool calls if response.choices[0].message.tool_calls: for tc in response.choices[0].message.tool_calls: print(f"Function: {tc.function.name}") print(f"Arguments: {tc.function.arguments}") ``` ## Supported Parsers Use `--tool-call-parser` to select a parser for your model family: | Parser | Aliases | Models | Format | |--------|---------|--------|--------| | `auto` | | Any model | Auto-detects format (tries all parsers) | | `mistral` | | Mistral, Devstral | `[TOOL_CALLS]` JSON array | | `qwen` | `qwen3` | Qwen, Qwen3 | `` XML or `[Calling tool:]` | | `llama` | `llama3`, `llama4` | Llama 3.x, 4.x | `` tags | | `hermes` | `nous` | Hermes, NousResearch | `` JSON in XML | | `deepseek` | `deepseek_v3`, `deepseek_r1` | DeepSeek V3, R1 | Unicode delimiters | | `kimi` | `kimi_k2`, `moonshot` | Kimi K2, Moonshot | `<\|tool_call_begin\|>` tokens | | `granite` | `granite3` | IBM Granite 3.x, 4.x | `<\|tool_call\|>` or `` | | `nemotron` | `nemotron3` | NVIDIA Nemotron | `` | | `xlam` | | Salesforce xLAM | JSON with `tool_calls` array | | `functionary` | `meetkai` | MeetKai Functionary | Multiple function blocks | | `glm47` | `glm4` | GLM-4.7, GLM-4.7-Flash | `` with ``/`` XML | ## Model Examples ### Mistral / Devstral ```bash # Devstral Small (optimized for coding and tool use) vllm-mlx serve mlx-community/Devstral-Small-2507-4bit \ --enable-auto-tool-choice --tool-call-parser mistral # Mistral Instruct vllm-mlx serve mlx-community/Mistral-7B-Instruct-v0.3-4bit \ --enable-auto-tool-choice --tool-call-parser mistral ``` ### Qwen ```bash # Qwen3 vllm-mlx serve mlx-community/Qwen3-4B-4bit \ --enable-auto-tool-choice --tool-call-parser qwen ``` ### Llama ```bash # Llama 3.2 vllm-mlx serve mlx-community/Llama-3.2-3B-Instruct-4bit \ --enable-auto-tool-choice --tool-call-parser llama ``` ### DeepSeek ```bash # DeepSeek V3 vllm-mlx serve mlx-community/DeepSeek-V3-0324-4bit \ --enable-auto-tool-choice --tool-call-parser deepseek ``` ### IBM Granite ```bash # Granite 4.0 vllm-mlx serve mlx-community/granite-4.0-tiny-preview-4bit \ --enable-auto-tool-choice --tool-call-parser granite ``` ### NVIDIA Nemotron ```bash # Nemotron 3 Nano vllm-mlx serve mlx-community/NVIDIA-Nemotron-3-Nano-30B-A3B-MLX-6Bit \ --enable-auto-tool-choice --tool-call-parser nemotron ``` ### GLM-4.7 ```bash # GLM-4.7 Flash vllm-mlx serve lmstudio-community/GLM-4.7-Flash-MLX-8bit \ --enable-auto-tool-choice --tool-call-parser glm47 ``` ### Kimi K2 ```bash # Kimi K2 vllm-mlx serve mlx-community/Kimi-K2-Instruct-4bit \ --enable-auto-tool-choice --tool-call-parser kimi ``` ### Salesforce xLAM ```bash # xLAM vllm-mlx serve mlx-community/xLAM-2-fc-r-4bit \ --enable-auto-tool-choice --tool-call-parser xlam ``` ## Auto Parser If you're not sure which parser to use, the `auto` parser tries to detect the format automatically: ```bash vllm-mlx serve mlx-community/Qwen3-4B-4bit \ --enable-auto-tool-choice --tool-call-parser auto ``` The auto parser tries formats in this order: 1. Mistral (`[TOOL_CALLS]`) 2. Qwen bracket (`[Calling tool:]`) 3. Nemotron (``) 4. Qwen/Hermes XML (`{...}`) 5. Llama (`{...}`) 6. Raw JSON ## Streaming Tool Calls Tool calls work with streaming. The tool call information is sent when the model finishes generating: ```python stream = client.chat.completions.create( model="default", messages=[{"role": "user", "content": "What's 25 * 17?"}], tools=[{ "type": "function", "function": { "name": "calculator", "description": "Calculate math expressions", "parameters": { "type": "object", "properties": { "expression": {"type": "string"} }, "required": ["expression"] } } }], stream=True ) for chunk in stream: if chunk.choices[0].delta.tool_calls: for tc in chunk.choices[0].delta.tool_calls: print(f"Tool call: {tc.function.name}({tc.function.arguments})") ``` ## Handling Tool Results After receiving a tool call, execute the function and send the result back: ```python import json # First request - model decides to call a tool response = client.chat.completions.create( model="default", messages=[{"role": "user", "content": "What's the weather in Tokyo?"}], tools=[weather_tool] ) # Get the tool call tool_call = response.choices[0].message.tool_calls[0] tool_call_id = tool_call.id function_name = tool_call.function.name arguments = json.loads(tool_call.function.arguments) # Execute the function (your implementation) result = get_weather(**arguments) # {"temperature": 22, "condition": "sunny"} # Send result back to model response = client.chat.completions.create( model="default", messages=[ {"role": "user", "content": "What's the weather in Tokyo?"}, {"role": "assistant", "tool_calls": [tool_call]}, {"role": "tool", "tool_call_id": tool_call_id, "content": json.dumps(result)} ], tools=[weather_tool] ) print(response.choices[0].message.content) # "The weather in Tokyo is sunny with a temperature of 22C." ``` ## Think Tag Handling Models that produce `...` reasoning tags (like DeepSeek-R1, Qwen3, GLM-4.7) are handled automatically. The parser strips thinking content before extracting tool calls, so reasoning tags never interfere with tool call parsing. This works even when `` was injected in the prompt (implicit think tags with only a closing ``). ## CLI Reference | Option | Description | |--------|-------------| | `--enable-auto-tool-choice` | Enable automatic tool calling | | `--tool-call-parser` | Select parser (see table above) | See [CLI Reference](../reference/cli.md) for all options. # Documentation page: `guides/warm-prompts.md` # Warm Prompts Pre-populate the prefix cache at server startup so the **first** request an agent sends hits a warm cache instead of paying the full prefill for its multi-kilobyte system prompt. ## When to use this Agent workloads — proxies to coding/reasoning assistants, MCP servers, multi-agent orchestrators — always send the same system prompt. Today the first request from a cold server pays the full prefill for that system. On a multi-billion-parameter model that is several seconds of TTFT, landing exactly when a user is waiting for their new agent to respond for the first time. If you already know your agents' system prompts at deploy time, write them to a JSON file and point `--warm-prompts` at it. The server runs a `max_tokens=1` chat completion for each at startup, the KV state lands in the prefix cache, and the first real request matches via strict-prefix. Requires `--continuous-batching` (the prefix cache lives there). ## Quick example ```bash # Write the agents you care about once cat > ~/.config/vllm-mlx/agents.json <<'JSON' [ [{"role": "system", "content": "You are a code assistant..."}] ] JSON # Point the server at it vllm-mlx serve mlx-community/Qwen3-4B-4bit \ --continuous-batching \ --warm-prompts ~/.config/vllm-mlx/agents.json ``` On start you'll see: ``` [lifespan] Warm-up done (strict-prefix): 1 completed, 0 skipped, 1431 prompt tokens in 0.2s ``` The first real request that shares the warmed system prompt hits the cache with `tokens_saved` close to the warm-up prompt length. ## File format A top-level JSON list. Each entry is itself a list of chat messages — same shape as `messages` in `/v1/chat/completions`. ```json [ [ {"role": "system", "content": "You are a code assistant..."} ], [ {"role": "system", "content": "You are a senior code reviewer..."} ], [ {"role": "system", "content": "You are a planner..."}, {"role": "user", "content": "hi"}, {"role": "assistant", "content": "Hello, what are we planning?"} ] ] ``` Single-message system prompts are the common case. Multi-turn histories are supported for scenarios where you want to warm a specific conversation start (few-shot examples, a running assistant persona). ## Sizing Warm-up prompts are processed **concurrently** via `asyncio.gather`, so N entries fire N concurrent prefills at startup. Each prefill allocates KV cache for its prompt length. **Recommended: 1–3 entries.** That covers the hot paths for typical agent deployments (one persona per entry). A very large warm-prompts file on a memory-tight model can exhaust headroom at boot. If you need to warm dozens of personas, open an issue with your workload and we can add a `--warm-prompts-concurrency=N` cap. ## Benchmarks **Setup.** M4 Max, 128 GB unified memory. Two separate servers per measurement (cold vs warm), isolated cold start. `long` prompt set (~2.5k user tokens) prepended with a ~1.7k-token system prompt to match the warm-up prompt. `max_tokens=128`. bench-serve with `--skip-preflight-token-count` so the count_prompt_tokens preflight does not pollute the cache. | Model | conc | cold TTFT | warm TTFT | Speedup | |-------|-----:|----------:|----------:|--------:| | Qwen3-0.6B-8bit | 1 | 563 ms | 419 ms | 1.34x | | Qwen3-0.6B-8bit | 4 | 1 723 ms | 1 282 ms | 1.34x | | Qwen3-0.6B-8bit | 8 | 3 708 ms | 2 661 ms | 1.39x | | Llama-3.2-3B-Instruct-4bit | 1 | 1 754 ms | 1 060 ms | 1.65x | | Llama-3.2-3B-Instruct-4bit | 4 | 5 926 ms | 3 945 ms | 1.50x | | Llama-3.2-3B-Instruct-4bit | 8 | 15 161 ms | 9 820 ms | 1.54x | | Qwen3-4B-4bit | 1 | 4 937 ms | 2 191 ms | 2.25x | | Qwen3-4B-4bit | 4 | 12 535 ms | 9 623 ms | 1.30x | | Qwen3-4B-4bit | 8 | 38 148 ms | 23 878 ms | 1.60x | | Qwen3.6-35B-A3B-4bit (MoE/hybrid) | 1 | 2 400 ms | 1 603 ms | 1.50x | | Qwen3.6-35B-A3B-4bit | 4 | 8 735 ms | 6 054 ms | 1.44x | | Qwen3.6-35B-A3B-4bit | 8 | 22 419 ms | 14 409 ms | 1.56x | All 12 configurations improve. TTFT savings are largest when the prompt-to-total ratio is highest (conc=1, long system prompt) and still meaningful under concurrent load. **Generation tok/s** is neutral (within ±5%) for the dense models. Qwen3.6-35B-A3B (MoE) shows a 20–35% decode drop at conc ≥ 4 that appears to be MoE routing interaction with batched scheduling. TTFT savings still dominate end-to-end latency on agent workloads, but note this if your workflow is heavily decode-bound at high concurrency. ## How it works The naive warm-up — render the chat template with a placeholder user message and cache the tokens — does not work for hybrid SSM+attention models (Qwen3.5-MoE, Qwen3.6-MoE). Their cache layers include SSM state that cannot be trimmed, so `memory_cache.py` disables LCP matching. The placeholder user content diverges from real user content, and a tokens-level cached entry is no longer a strict prefix of any real request. The warmer here renders the chat template **twice** with two distinct user contents (`"__PROBE_A__"` and `"__PROBE_B__"`), finds the character position where the two strings diverge, and truncates the first rendering at that boundary. That truncated string — everything up to the point where user content gets inserted — is what goes to the engine. Because the engine's real-request path also renders the template with `tokenize=False` and then lets the tokenizer encode the result, the warm-up's tokens are guaranteed to be a strict prefix of any real request with a matching system and empty chat history. Strict prefix matches work on every cache layer type, including the hybrid paths where LCP is disabled. ## Admin ### Clear the in-memory prefix cache ```bash curl -X DELETE http://localhost:8000/v1/cache/prefix ``` If the server was started with `--warm-prompts`, the warm-up re-runs in the background after clear. The response returns immediately without waiting for re-warm. Response: ```json {"status": "cleared", "rewarm_scheduled": true} ``` ### Inspect cache state ```bash curl http://localhost:8000/v1/status | jq '.cache' ``` After startup with warm-prompts you will see `entry_count > 0` before the first user request. ## Benchmarking your own setup To measure the impact on your model and prompts, use `bench-serve`: ```bash # Cold: no warm-prompts vllm-mlx serve MODEL --continuous-batching & vllm-mlx bench-serve --prompts long --concurrency 1,4 \ --system-prompt-file my-system.txt --tag cold \ --output cold.csv --format csv # Warm: same server config + --warm-prompts vllm-mlx serve MODEL --continuous-batching \ --warm-prompts ~/.config/vllm-mlx/agents.json & vllm-mlx bench-serve --prompts long --concurrency 1,4 \ --system-prompt-file my-system.txt --tag warm \ --output warm.csv --format csv ``` `--skip-preflight-token-count` is auto-enabled when `--system-prompt-file` is set, so the `count_prompt_tokens` preflight does not pollute the cache. Compare `cold.csv` and `warm.csv` for your workload. # Documentation page: `index.md` # vllm-mlx documentation
## OpenAI and Anthropic compatible inference on Apple Silicon Serve text, image, video, audio, embeddings, and reranking from one local process. vllm-mlx combines continuous batching, efficient KV caches, tool calling, structured output, and model residency with native MLX acceleration. ```bash pip install vllm-mlx vllm-mlx serve \ mlx-community/Llama-3.2-3B-Instruct-4bit \ --port 8000 ``` [Get started](getting-started/quickstart.md){ .md-button .md-button--primary } [Browse the Python API](reference/api/index.md){ .md-button }
## Start here
- **Install and serve** Set up the supported environment and start your first local model. [Installation](getting-started/installation.md) · [Quickstart](getting-started/quickstart.md) - **Connect a client** Use OpenAI, Anthropic, Responses, audio, embedding, reranking, or MCP routes. [Server guide](guides/server.md) · [HTTP API](reference/http-api.md) - **Use the Python API** Integrate generation directly and inspect exact runtime interfaces. [Python guide](guides/python-api.md) · [API reference](reference/api/index.md) - **Understand the runtime** Follow requests through scheduling, batching, caches, parsers, and model execution. [Core concepts](concepts/index.md) · [Architecture](concepts/runtime-architecture.md) - **Find exact source behavior** Search every module, class, function, method, CLI option, and source range. [Source inventory](reference/source/index.md) · [CLI options](reference/cli-options.md) - **Work with LLMs and agents** Load compact, full-corpus, or JSON documentation designed for machine context. [`llms.txt`](llms.txt) · [Agent guide](development/agent-guide.md)
## What you can run - **Language models:** text generation, reasoning, structured output, and tool calling - **Multimodal models:** image and video understanding - **Audio models:** speech-to-text and text-to-speech - **Embedding models:** OpenAI-compatible vector generation - **Rerankers:** query-document relevance scoring - **Model registries:** multiple resident or dynamically loaded models ## Core runtime capabilities - Continuous batching and paged KV cache management - Prefix reuse, prompt warmup, and optional SSD cache support - OpenAI, Anthropic, Responses, audio, reranking, and MCP protocols - Reasoning parsers, tool parsers, constrained decoding, and JSON schemas - Metrics, health checks, cancellation, lifecycle control, and model residency ## Requirements - macOS on Apple Silicon - Python 3.10 or newer - At least 8 GB of unified memory See the [installation guide](getting-started/installation.md) for supported dependency and environment details. # Documentation page: `reference/api/index.md` # Python API reference The API reference is generated directly from every tracked `vllm_mlx/**/*.py` file during the MkDocs build. It is exhaustive by construction and does not import MLX on the Linux documentation runner. Use the [Python symbol index](../python-symbols.md) to filter every class, function, method, and nested helper by name, kind, or exact callable signature. Each result shows its inputs and links directly to the detailed API record. ## What each module page contains 1. The module's purpose and complete source link. 2. Full signatures, type annotations, parameters, defaults, and return annotations. 3. Parsed docstrings, including parameter descriptions, returns, yields, warnings, examples, and exceptions when supplied by the source. 4. A source map for every class, function, method, nested class, and nested helper. 5. Exact GitHub links in `#Lx-Ly` form, pinned to the immutable commit used for the deployed build. 6. Static implementation facts for every definition, including calls, state reads and writes, return expressions, direct raises, decorators, async work, and yields. 7. Inline source rendering for addressable classes, functions, methods, and module attributes. Private implementation objects are included because this reference also serves maintainers and coding agents. Public classes and functions must have human-authored docstrings. Private and nested helpers without parameter prose receive their exact AST signature, input kinds, annotations, defaults, direct exceptions, return paths, and conservative implementation facts. The generator does not invent semantic claims that are absent from source. The coverage check fails when a new public object lacks an explanation or a source definition is absent from the reference. ## Machine-readable forms - [`/llms.txt`](../../llms.txt) is the compact documentation index. - [`/llms-full.txt`](https://vllm-mlx.is-a.dev/llms-full.txt) contains all hand-written pages and every symbol record in one Markdown file. - [`/api-inventory.json`](https://vllm-mlx.is-a.dev/api-inventory.json) contains module paths, symbols, kinds, signatures, docstrings, visibility, addressability, and exact source URLs. - [`/source-inventory.json`](https://vllm-mlx.is-a.dev/source-inventory.json) adds maintenance scripts and runnable examples to the runtime inventory. Maintenance scripts and examples also have [generated source-reference pages](../source/index.md). ## Navigation Open **Reference → Python modules → vllm_mlx** in the site navigation. The complete package tree is indexed there, including **models → llm** and every other runtime module. Package overview pages correspond to `__init__.py`; all other module names map directly to their Python paths. Use [Python symbol index](../python-symbols.md) when you know a class, function, method, parameter, or partial signature but not its module. For example, filtering for `stream_outputs` links directly to that callable's expandable contract. Generated Markdown mirrors are also available at predictable paths. For example: ```text https://vllm-mlx.is-a.dev/reference/api/vllm_mlx/server.md https://vllm-mlx.is-a.dev/reference/api/vllm_mlx/scheduler.md https://vllm-mlx.is-a.dev/reference/api/vllm_mlx/tool_parsers/qwen_tool_parser.md ``` # Documentation page: `reference/cli.md` # CLI Reference This guide explains the supported vllm-mlx command workflows. The [complete generated option inventory](cli-options.md) lists every argparse declaration across runtime commands, maintenance scripts, and runnable examples with defaults, choices, help text, and exact `#Lx-Ly` source links. Agents can also consume [`cli-inventory.json`](https://vllm-mlx.is-a.dev/cli-inventory.json). ## Commands Overview | Command | Description | |---------|-------------| | `vllm-mlx serve` | Start OpenAI-compatible server | | `vllm-mlx model` | Inspect, acquire, or convert model artifacts | | `vllm-mlx bench-serve` | Benchmark a running server with prompt sweeps or workload contracts | | `vllm-mlx-bench` | Run performance benchmarks | | `vllm-mlx-chat` | Start Gradio chat interface | ## `vllm-mlx serve` Start the OpenAI-compatible API server. ### Usage ```bash vllm-mlx serve [options] vllm-mlx serve --models-config [options] ``` ### Options | Option | Description | Default | |--------|-------------|---------| | `--served-model-name` | Custom model name exposed through the OpenAI API. If not set, the model path is used as the name. | None | | `--port` | Server port | 8000 | | `--host` | Server host | 127.0.0.1 | | `--api-key` | API key for authentication | None | | `--rate-limit` | Requests per minute per client (0 = disabled) | 0 | | `--timeout` | Request timeout in seconds | 300 | | `--enable-metrics` | Expose Prometheus metrics on `/metrics` | False | | `--continuous-batching` | Enable batching for multi-user | False | | `--cache-memory-mb` | Cache memory limit in MB | Auto | | `--cache-memory-percent` | Fraction of RAM for cache | 0.20 | | `--no-memory-aware-cache` | Use legacy entry-count cache | False | | `--use-paged-cache` | Enable paged KV cache | False | | `--max-tokens` | Default max tokens | 32768 | | `--max-request-tokens` | Maximum `max_tokens` accepted from API clients | 32768 | | `--stream-interval` | Tokens per stream chunk | 1 | | `--mcp-config` | Path to MCP config file | None | | `--paged-cache-block-size` | Tokens per cache block | 64 | | `--max-cache-blocks` | Maximum cache blocks | 1000 | | `--max-num-seqs` | Max concurrent sequences | 256 | | `--default-temperature` | Default temperature when not specified in request | None | | `--default-top-p` | Default top_p when not specified in request | None | | `--default-chat-template-kwargs` | Default chat template kwargs applied when request `chat_template_kwargs` is omitted (JSON object) | None | | `--max-audio-upload-mb` | Maximum uploaded audio size for `/v1/audio/transcriptions` | 25 | | `--max-tts-input-chars` | Maximum text length accepted by `/v1/audio/speech` | 4096 | | `--reasoning-parser` | Parser for reasoning models (`qwen3`, `deepseek_r1`) | None | | `--embedding-model` | Pre-load an embedding model at startup | None | | `--enable-auto-tool-choice` | Enable automatic tool calling | False | | `--tool-call-parser` | Tool call parser (`auto`, `mistral`, `qwen`, `llama`, `hermes`, `deepseek`, `kimi`, `granite`, `nemotron`, `xlam`, `functionary`, `glm47`) | None | | `--models-config` | YAML registry file for multi-model serving | None | ### Examples ```bash # Simple mode (single user, max throughput) # Model path is used as the model name in the OpenAI API (e.g. model="mlx-community/Llama-3.2-3B-Instruct-4bit") vllm-mlx serve mlx-community/Llama-3.2-3B-Instruct-4bit Model will show up as 'mlx-community/Llama-3.2-3B-Instruct-4bit' in the `/v1/models` API endpoint. View with `curl http://localhost:8000/v1/models` or similar. # With a custom API model name (model is accessed as "my-model" via the OpenAI API) # --served-model-name sets the name clients must use when calling the API (e.g. model="my-model") vllm-mlx serve --served-model-name my-model mlx-community/Llama-3.2-3B-Instruct-4bit # Note: Model will show up as 'my-model' in the `/v1/models` API endpoint. # Continuous batching (multiple users) vllm-mlx serve mlx-community/Llama-3.2-3B-Instruct-4bit --continuous-batching # With memory limit for large models vllm-mlx serve mlx-community/GLM-4.7-Flash-4bit \ --continuous-batching \ --cache-memory-mb 2048 # Production with paged cache vllm-mlx serve mlx-community/Qwen3-0.6B-8bit \ --continuous-batching \ --use-paged-cache \ --port 8000 # With MCP tools vllm-mlx serve mlx-community/Qwen3-4B-4bit --mcp-config mcp.json # Multimodal model vllm-mlx serve mlx-community/Qwen3-VL-4B-Instruct-3bit # Reasoning model (separates thinking from answer) vllm-mlx serve mlx-community/Qwen3-8B-4bit --reasoning-parser qwen3 # Disable server-wide thinking by default (request-level chat_template_kwargs still override) vllm-mlx serve mlx-community/Qwen3-8B-4bit \ --reasoning-parser qwen3 \ --default-chat-template-kwargs '{"enable_thinking": false}' # DeepSeek reasoning model vllm-mlx serve mlx-community/DeepSeek-R1-Distill-Qwen-7B-4bit --reasoning-parser deepseek_r1 # Tool calling with Mistral/Devstral vllm-mlx serve mlx-community/Devstral-Small-2507-4bit \ --enable-auto-tool-choice --tool-call-parser mistral # Tool calling with Granite vllm-mlx serve mlx-community/granite-4.0-tiny-preview-4bit \ --enable-auto-tool-choice --tool-call-parser granite # With API key authentication vllm-mlx serve mlx-community/Llama-3.2-3B-Instruct-4bit --api-key your-secret-key # Registry-backed multi-model serving vllm-mlx serve --models-config /etc/vllm-mlx/models.yaml --continuous-batching # Expose Prometheus metrics vllm-mlx serve mlx-community/Llama-3.2-3B-Instruct-4bit --enable-metrics # Production setup with security options vllm-mlx serve mlx-community/Qwen3-4B-4bit \ --api-key your-secret-key \ --rate-limit 60 \ --timeout 120 \ --continuous-batching ``` For registry-backed serving, see [Multi-Model Serving](../guides/model-registry.md). ### Security When `--api-key` is set, all API requests require the `Authorization: Bearer ` header: ```python from openai import OpenAI client = OpenAI( base_url="http://localhost:8000/v1", api_key="your-secret-key" # Must match --api-key ) ``` Or with curl: ```bash curl http://localhost:8000/v1/models \ -H "Authorization: Bearer your-secret-key" ``` ## `vllm-mlx model` Inspect, acquire, and convert model artifacts without serving them. These commands are intended to make model setup auditable: inspect before download, download into a finalized artifact manifest, then convert through `mlx-lm` with the exact recipe recorded. ### Usage ```bash vllm-mlx model inspect vllm-mlx model acquire [--target-dir ] vllm-mlx model convert --output [--quantize] ``` ### Options | Command | Option | Description | |---------|--------|-------------| | `inspect` | `--revision` | Hugging Face revision to inspect | | `inspect` | `--local-files-only` | Inspect only local Hugging Face cache files | | `acquire` | `--target-dir` | Final local directory for a staged download | | `acquire` | `--staging-dir` | Temporary directory used before finalizing `--target-dir` | | `acquire` | `--mllm` | Download multimodal file patterns | | `acquire` | `--no-fast-transfer` | Do not set `HF_HUB_ENABLE_HF_TRANSFER=1` | | `convert` | `--output` | Output directory for the converted MLX model | | `convert` | `--quantize` | Enable `mlx-lm` quantization | | `convert` | `--q-bits`, `--q-group-size`, `--q-mode` | Quantization recipe | | `convert` | `--quant-predicate` | `mlx-lm` mixed-bit quantization recipe | | `convert` | `--dtype` | Dtype for non-quantized parameters | | `convert` | `--dry-run` | Print command and manifest without executing conversion | ### Examples ```bash vllm-mlx model inspect mlx-community/Llama-3.2-3B-Instruct-4bit vllm-mlx model acquire mlx-community/Llama-3.2-3B-Instruct-4bit \ --target-dir ./models/llama-3b-4bit vllm-mlx model convert meta-llama/Llama-3.2-3B-Instruct \ --output ./models/llama-3b-mlx-q4 \ --quantize --q-bits 4 --q-group-size 64 --q-mode affine ``` ## `vllm-mlx bench-serve` Benchmark a running vllm-mlx server over HTTP. Prompt-sweep mode measures TTFT, TPOT, throughput, cache deltas, and Metal memory. Workload mode adds per-case quality checks, repeated samples for variance, and comparison-only product policy timeouts. Workload cases can embed `messages` directly or point `request_path` at an existing OpenAI-compatible request JSON. ### Usage ```bash vllm-mlx bench-serve --url http://localhost:8000 [options] ``` ### Options | Option | Description | Default | |--------|-------------|---------| | `--url` | Running server base URL | `http://127.0.0.1:8080` | | `--model` | API model id | Auto-detect | | `--prompts` | Comma-separated prompt sets or files for sweep mode | `short,medium,long` | | `--workload` | Declarative workload JSON for contract mode | None | | `--concurrency` | Comma-separated concurrency levels for sweep mode | `1,4` | | `--max-tokens` | Max tokens for sweep mode | `256` | | `--repetitions` | Repetitions per sweep configuration or workload case | `3` | | `--enable-thinking` | `true`, `false`, or `true,false` sweep | None | | `--scrape-metrics` | Scrape `/metrics` before/after runs | `true` | | `--include-content` | Include full generated content in workload JSON | False | | `--request-timeout-s` | Workload HTTP transport timeout, `0` disables | `300` | | `--cache-policy` | Workload cache handling: `preserve`, `before-run`, `before-case` | Workload default or `preserve` | | `--output` | Output file | stdout | | `--format` | Output format: `auto`, `table`, `json`, `csv`, `sql`, `sqlite` | `auto` = `table` for prompt sweeps, `json` for workloads | In workload mode, `--request-timeout-s` is the HTTP transport ceiling for each request. Product policy timeouts should live in the workload as `policy_timeout_ms`. Workload `required_regex` and `forbidden_regex` values are Python regex patterns, so literal strings are valid. Workload JSON may spell cache policy values with underscores, such as `before_case`; they normalize to the hyphenated CLI values. ### Examples ```bash # Prompt sweep vllm-mlx bench-serve --url http://localhost:8000 \ --prompts short,long --concurrency 1,4 --format json --output bench.json # Contract workload with quality checks and policy-timeout evidence vllm-mlx bench-serve --url http://localhost:8000 \ --workload workload.json --repetitions 5 --output workload-results.json # Append contract rows directly into SQLite for longitudinal comparisons vllm-mlx bench-serve --url http://localhost:8000 \ --workload workload.json --repetitions 5 --format sqlite --output bench.db ``` ## `vllm-mlx-bench` Run performance benchmarks. # Documentation page: `reference/configuration.md` # Configuration Reference ## Server Configuration ### Basic Options | Option | Description | Default | |--------|-------------|---------| | `--host` | Server host address | `127.0.0.1` | | `--port` | Server port | `8000` | | `--max-tokens` | Default max tokens | `32768` | | `--max-request-tokens` | Maximum `max_tokens` accepted from API clients | `32768` | | `--default-temperature` | Default temperature when not specified in request | None | | `--default-top-p` | Default top_p when not specified in request | None | | `--default-chat-template-kwargs` | Default chat template kwargs used when request `chat_template_kwargs` is omitted (JSON object) | None | ### Security Options | Option | Description | Default | |--------|-------------|---------| | `--api-key` | API key for authentication | None | | `--rate-limit` | Requests per minute per client (0 = disabled) | `0` | | `--timeout` | Request timeout in seconds | `300` | | `--enable-metrics` | Expose Prometheus metrics on `/metrics` | `false` | | `--max-audio-upload-mb` | Maximum uploaded audio size for `/v1/audio/transcriptions` | `25` | | `--max-tts-input-chars` | Maximum text length accepted by `/v1/audio/speech` | `4096` | ### Batching Options | Option | Description | Default | |--------|-------------|---------| | `--continuous-batching` | Enable batching | `false` | | `--stream-interval` | Tokens per stream chunk | `1` | | `--max-num-seqs` | Max concurrent sequences | `256` | ### Cache Options | Option | Description | Default | |--------|-------------|---------| | `--cache-memory-mb` | Cache memory limit in MB | Auto | | `--cache-memory-percent` | Fraction of RAM for cache | `0.20` | | `--no-memory-aware-cache` | Use legacy entry-count cache | `false` | | `--use-paged-cache` | Enable paged KV cache | `false` | | `--paged-cache-block-size` | Tokens per block | `64` | | `--max-cache-blocks` | Maximum blocks | `1000` | ### Tool Calling Options | Option | Description | Default | |--------|-------------|---------| | `--enable-auto-tool-choice` | Enable automatic tool calling | `false` | | `--tool-call-parser` | Tool call parser (see [Tool Calling](../guides/tool-calling.md)) | None | ### Reasoning Options | Option | Description | Default | |--------|-------------|---------| | `--reasoning-parser` | Parser for reasoning models (`qwen3`, `deepseek_r1`) | None | ### Embedding Options | Option | Description | Default | |--------|-------------|---------| | `--embedding-model` | Pre-load an embedding model at startup | None | ### MCP Options | Option | Description | Default | |--------|-------------|---------| | `--mcp-config` | Path to MCP config file | None | ## MCP Configuration Create `mcp.json`: ```json { "mcpServers": { "server-name": { "command": "npx", "args": ["-y", "@modelcontextprotocol/server-name", "arg1"], "env": { "ENV_VAR": "value" } } } } ``` ### MCP Server Options | Field | Description | Required | |-------|-------------|----------| | `command` | Executable command | Yes | | `args` | Command arguments | Yes | | `env` | Environment variables | No | ## API Request Options ### Chat Completions | Parameter | Description | Default | |-----------|-------------|---------| | `model` | Model name | Required | | `messages` | Chat messages | Required | | `max_tokens` | Max tokens to generate | 256 | | `temperature` | Sampling temperature | Model default | | `top_p` | Nucleus sampling | Model default | | `stream` | Enable streaming | `true` | | `stop` | Stop sequences | None | | `tools` | Tool definitions | None | | `response_format` | Output format (`json_object`, `json_schema`) | None | ### Multimodal Options | Parameter | Description | Default | |-----------|-------------|---------| | `video_fps` | Frames per second | 2.0 | | `video_max_frames` | Max frames | 32 | ## Environment Variables | Variable | Description | |----------|-------------| | `VLLM_MLX_TEST_MODEL` | Default model for tests | | `HF_TOKEN` | HuggingFace authentication token | | `OPENAI_API_KEY` | Set to any value for SDK compatibility | ## Example Configurations ### Development (Single User) ```bash vllm-mlx serve mlx-community/Llama-3.2-3B-Instruct-4bit ``` ### Production (Multiple Users) ```bash vllm-mlx serve mlx-community/Qwen3-0.6B-8bit \ --continuous-batching \ --use-paged-cache \ --api-key your-secret-key \ --rate-limit 60 \ --port 8000 ``` ### With Tool Calling ```bash vllm-mlx serve mlx-community/Devstral-Small-2507-4bit \ --enable-auto-tool-choice \ --tool-call-parser mistral \ --continuous-batching ``` ### With MCP Tools ```bash vllm-mlx serve mlx-community/Qwen3-4B-4bit \ --mcp-config mcp.json \ --enable-auto-tool-choice \ --tool-call-parser qwen \ --continuous-batching ``` ### Reasoning Model ```bash vllm-mlx serve mlx-community/Qwen3-8B-4bit \ --reasoning-parser qwen3 \ --continuous-batching ``` ### With Embeddings ```bash vllm-mlx serve mlx-community/Qwen3-4B-4bit \ --embedding-model mlx-community/multilingual-e5-small-mlx \ --continuous-batching ``` ### High Throughput ```bash vllm-mlx serve mlx-community/Qwen3-0.6B-8bit \ --continuous-batching \ --stream-interval 5 \ --max-num-seqs 256 ``` # Documentation page: `reference/http-api.md` # HTTP API reference The server implements OpenAI-compatible, Anthropic-compatible, operational, cache, audio, reranking, and MCP routes in [`vllm_mlx.server`](api/vllm_mlx/server.md). Request and response schemas are documented in [`vllm_mlx.api`](api/vllm_mlx/api/index.md). Default base URL: ```text http://127.0.0.1:8000 ``` When `--api-key` is configured, protected routes require `Authorization: Bearer `. ## Generation protocols | Method | Path | Purpose | Implementation | | --- | --- | --- | --- | | `POST` | `/v1/completions` | OpenAI-compatible text completions | [`create_completion`](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L4763-L4909) | | `POST` | `/v1/chat/completions` | OpenAI-compatible chat, tools, reasoning, and multimodal input | [`create_chat_completion`](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L4916-L5114) | | `POST` | `/v1/responses` | OpenAI-compatible Responses API | [`create_response`](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L5194-L5214) | | `POST` | `/v1/messages` | Anthropic-compatible Messages API | [`create_anthropic_message`](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L5357-L5578) | | `POST` | `/v1/messages/count_tokens` | Count Anthropic message tokens | [`count_anthropic_tokens`](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L5585-L5666) | OpenAI streaming uses Server-Sent Events and terminates with `data: [DONE]`. Anthropic and Responses API streams emit their protocol-specific typed terminal events. ## Vector and ranking protocols | Method | Path | Purpose | Implementation | | --- | --- | --- | --- | | `POST` | `/v1/embeddings` | OpenAI-compatible embeddings | [`create_embeddings`](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L3787-L3908) | | `POST` | `/v1/rerank` | Score query-document relevance | [`rerank_documents`](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L3920-L4038) | ## Audio protocols | Method | Path | Purpose | Implementation | | --- | --- | --- | --- | | `POST` | `/v1/audio/transcriptions` | Speech-to-text transcription | [`create_transcription`](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L4130-L4196) | | `POST` | `/v1/audio/speech` | Text-to-speech synthesis | [`create_speech`](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L4200-L4254) | | `GET` | `/v1/audio/voices` | List available voices for a TTS model | [`list_voices`](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L4258-L4267) | Upload and input limits are applied before model execution. Audio dependencies are installed separately with the `audio` extra. ## MCP protocols | Method | Path | Purpose | Implementation | | --- | --- | --- | --- | | `GET` | `/v1/mcp/tools` | List tools discovered from configured MCP servers | [`list_mcp_tools`](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L4047-L4063) | | `GET` | `/v1/mcp/servers` | List MCP server connection state | [`list_mcp_servers`](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L4067-L4084) | | `POST` | `/v1/mcp/execute` | Execute a named MCP tool | [`execute_mcp_tool`](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L4088-L4117) | MCP execution is a trust boundary. Use explicit server configuration and review the [MCP guide](../guides/mcp-tools.md) before exposing it to untrusted clients. ## Models and operations | Method | Path | Purpose | Implementation | | --- | --- | --- | --- | | `GET` | `/health` | Basic health and residency readiness | [`health`](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L3489-L3544) | | `GET` | `/metrics` | Prometheus metrics when enabled | [`metrics`](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L3476-L3485) | | `GET` | `/v1/status` | Server, model, engine, and lifecycle status | [`status`](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L3548-L3597) | | `GET` | `/v1/models` | List API-visible model IDs | [`list_models`](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L3760-L3775) | | `POST` | `/v1/requests/{request_id}/cancel` | Cancel active generation | [`cancel_request`](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L3720-L3747) | | `DELETE` | `/v1/requests/{request_id}` | Delete or cancel active generation | [`delete_request`](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L3754-L3756) | ## Cache operations | Method | Path | Purpose | Implementation | | --- | --- | --- | --- | | `GET` | `/v1/cache/stats` | Return active engine cache statistics | [`cache_stats`](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L3601-L3627) | | `DELETE` | `/v1/cache` | Clear all supported engine caches | [`clear_cache`](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L3631-L3659) | | `DELETE` | `/v1/cache/prefix` | Clear the prefix cache | [`clear_prefix_cache`](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L3663-L3713) | ## Error behavior Validation failures use HTTP 4xx responses. Authentication failures return 401. Rate limits return 429. Busy, timeout, model-loading, and internal generation failures are mapped by the endpoint to a protocol-compatible error body where possible. For complete fields and validation rules, inspect the Pydantic model reference rather than inferring support from an upstream OpenAI or Anthropic schema. # Documentation page: `reference/models.md` # Supported Models All quantized models from [mlx-community on HuggingFace](https://huggingface.co/mlx-community/models) are compatible. Browse thousands of pre-optimized models at: **https://huggingface.co/mlx-community/models** ## Language Models (via mlx-lm) | Model Family | Sizes | Quantization | |--------------|-------|--------------| | Llama 3.x, 4.x | 1B, 3B, 8B, 70B | 4-bit | | Mistral / Devstral | 7B, Mixtral 8x7B | 4-bit, 8-bit | | Qwen2/Qwen3 | 0.5B to 72B | Various | | DeepSeek V3, R1 | 7B, 33B, 67B | 4-bit | | Gemma 2, 3, 4 | 2B, 9B, 27B | 4-bit | | GLM-4.7 | Flash, Base | 4-bit, 8-bit | | Kimi K2 | Various | 4-bit | | Phi-3 | 3.8B, 14B | 4-bit | | Granite 3.x, 4.x | Various | 4-bit | | Nemotron | 3 Nano 30B | 6-bit | ### Recommended Models | Use Case | Model | Memory | |----------|-------|--------| | Fast/Light | `mlx-community/Qwen3-0.6B-8bit` | ~0.7 GB | | Balanced | `mlx-community/Llama-3.2-3B-Instruct-4bit` | ~1.8 GB | | Quality | `mlx-community/Llama-3.1-8B-Instruct-4bit` | ~4.5 GB | | Large | `mlx-community/Qwen3-30B-A3B-4bit` | ~16 GB | ## Multimodal Models (via mlx-vlm) | Model Family | Example Models | |--------------|----------------| | **Qwen-VL** | `Qwen3-VL-4B-Instruct-3bit`, `Qwen3-VL-8B-Instruct-4bit`, `Qwen2-VL-2B/7B-Instruct-4bit` | | **LLaVA** | `llava-1.5-7b-4bit`, `llava-v1.6-mistral-7b-4bit`, `llava-llama-3-8b-v1_1-4bit` | | **Idefics** | `Idefics3-8B-Llama3-4bit`, `idefics2-8b-4bit` | | **Gemma 4** | `gemma-4-e2b-it-mxfp4` (vision + audio) | | **PaliGemma** | `paligemma2-3b-mix-224-4bit`, `paligemma-3b-mix-224-8bit` | | **Pixtral** | `pixtral-12b-4bit`, `pixtral-12b-8bit` | | **Molmo** | `Molmo-7B-D-0924-4bit`, `Molmo-7B-D-0924-8bit` | | **Phi-3 Vision** | `Phi-3-vision-128k-instruct-4bit` | | **DeepSeek-VL** | `deepseek-vl-7b-chat-4bit`, `deepseek-vl2-small-4bit` | ### Recommended VLM Models | Use Case | Model | Memory | |----------|-------|--------| | Fast/Light | `mlx-community/Qwen3-VL-4B-Instruct-3bit` | ~3 GB | | Balanced | `mlx-community/Qwen3-VL-8B-Instruct-4bit` | ~6 GB | | Quality | `mlx-community/Qwen3-VL-30B-A3B-Instruct-6bit` | ~20 GB | ## Embedding Models (via mlx-embeddings) | Model Family | Example Models | |--------------|----------------| | **BERT** | `mlx-community/bert-base-uncased-mlx` | | **XLM-RoBERTa** | `mlx-community/multilingual-e5-small-mlx`, `mlx-community/multilingual-e5-large-mlx` | | **ModernBERT** | `mlx-community/ModernBERT-base-mlx` | ## Audio Models (via mlx-audio) | Type | Model Family | Example Models | |------|--------------|----------------| | **STT** | Whisper | `mlx-community/whisper-large-v3-turbo` | | **STT** | Parakeet | `mlx-community/parakeet-tdt-0.6b-v2` | | **TTS** | Kokoro | `prince-canuma/Kokoro-82M` | | **TTS** | Chatterbox | `chatterbox/chatterbox-tts-0.1` | ## Model Detection vllm-mlx auto-detects multimodal models by name patterns: - Contains "VL", "Vision", "vision" - Contains "llava", "idefics", "paligemma" - Contains "pixtral", "molmo", "deepseek-vl" - Contains "MedGemma", "Gemma-3", "Gemma-4" (multimodal variants) ## Using Models ### From HuggingFace ```bash vllm-mlx serve mlx-community/Llama-3.2-3B-Instruct-4bit ``` ### Local Path ```bash vllm-mlx serve /path/to/local/model ``` ## Finding Models Filter mlx-community models by: - **LLM**: `Llama`, `Qwen`, `Mistral`, `Phi`, `Gemma`, `DeepSeek`, `GLM`, `Kimi`, `Granite`, `Nemotron` - **VLM**: `-VL-`, `llava`, `paligemma`, `pixtral`, `molmo`, `idefics`, `deepseek-vl`, `MedGemma` - **Embedding**: `e5`, `bert`, `ModernBERT` - **Size**: `1B`, `3B`, `7B`, `8B`, `70B` - **Quantization**: `4bit`, `8bit`, `bf16` # Documentation page: `reference/source/index.md` # Repository source reference This section documents executable Python outside the installable `vllm_mlx` package. It includes maintenance tools under `scripts/` and runnable programs under `examples/`. Every page contains: - A complete module source link. - Exact `#Lx-Ly` links for every class, function, method, and nested helper. - Human-authored docstrings when present. - Conservative implementation facts generated from calls, state access, return expressions, direct raises, decorators, awaits, and yields. - Inline static source rendering. The machine-readable [`source-inventory.json`](https://vllm-mlx.is-a.dev/source-inventory.json) combines this section with the complete runtime package. Tests are intentionally excluded from public source reference pages because they document verification scenarios rather than supported runtime or user-facing programs. Use the navigation beneath this page to browse `scripts` and `examples`. # Documentation page: `zh/benchmarks/README.md` # 基准测试 vllm-mlx 在 Apple Silicon 上的性能基准测试。 ## 基准测试类型 - [LLM 基准测试](llm.md) - 文本生成性能 - [图像基准测试](image.md) - 图像理解性能 - [视频基准测试](video.md) - 视频理解性能 ## 常用命令 ```bash # LLM benchmark vllm-mlx-bench --model mlx-community/Qwen3-0.6B-8bit # Image benchmark vllm-mlx-bench --model mlx-community/Qwen3-VL-8B-Instruct-4bit # Video benchmark vllm-mlx-bench --model mlx-community/Qwen3-VL-8B-Instruct-4bit --video ``` ## 独立测试默认值 独立基准测试脚本内置了默认模型，可以直接运行： ```bash python tests/test_continuous_batching.py python tests/test_prefix_cache.py ``` 默认模型： - `tests/test_continuous_batching.py` 对应 `mlx-community/Qwen3-8B-6bit` - `tests/test_prefix_cache.py` 对应 `mlx-community/Qwen3-0.6B-8bit` 如需测试其他模型，使用可选的 `--model` 参数： ```bash python tests/test_continuous_batching.py --model mlx-community/Qwen3-0.6B-8bit python tests/test_prefix_cache.py --model mlx-community/Qwen3-8B-6bit ``` ## 硬件配置 以下 Apple Silicon 配置已收录基准测试结果： | 芯片 | 内存 | Python | |------|--------|--------| | Apple M4 Max | 128 GB unified | 3.13 | | Apple M1 Max | 64 GB unified | 3.12 | 不同 Apple Silicon 芯片的测试结果会有所差异。 ## 贡献基准测试 如果您使用的是其他 Apple Silicon 芯片，欢迎分享您的测试结果： ```bash vllm-mlx-bench --model mlx-community/Qwen3-0.6B-8bit --output results.json ``` 请在 [GitHub Issues](https://github.com/waybarrios/vllm-mlx/issues) 中提交您的结果。 # Documentation page: `zh/benchmarks/audio.md` # 音频基准测试 ## 语音转文本 (STT) 基准测试 ### 运行 STT 基准测试 ```bash # Run with default test audio python examples/benchmark_audio.py --stt # Run with your own audio file python examples/benchmark_audio.py --stt --audio path/to/audio.wav ``` ### 测试结果（M4 Max，128GB） **测试音频：** 46.7 秒的合成语音 | Model | Parameters | Load Time | Transcribe Time | RTF* | |-------|------------|-----------|-----------------|------| | whisper-tiny | 39M | 0.34s | 0.24s | **197x** | | whisper-small | 244M | 0.18s | 0.47s | **98x** | | whisper-medium | 769M | 0.35s | 1.15s | **41x** | | whisper-large-v3 | 1.5B | 0.50s | 1.96s | **24x** | | whisper-large-v3-turbo | 809M | 0.12s | 0.86s | **55x** | *RTF = 实时倍率（值越高速度越快）。RTF 为 100x 表示 1 分钟的音频约在 0.6 秒内转录完成。* ### 测试结果（M1 Max，64GB） 使用 Parakeet 进行 STT（默认环境，Whisper 因 numpy 依赖版本不匹配而不可用）： | Model | Load Time | Transcribe Time | RTF | |-------|-----------|-----------------|-----| | parakeet-tdt-0.6b-v2 | 0.28s | 1.01s | **9.9x** | | parakeet-tdt-0.6b-v3 | 0.30s | 0.19s | **52.7x** | 使用 Whisper 进行 STT（显式指定 `numpy==2.3.5` 并搭配 `uv run --no-sync`）： | Model | Load Time | Transcribe Time | RTF | |-------|-----------|-----------------|-----| | whisper-tiny | 4.02s | 1.05s | **9.5x** | | whisper-small | 10.15s | 1.03s | **9.7x** | | whisper-medium | 22.96s | 2.20s | **4.6x** | | whisper-large-v3 | 38.34s | 0.96s | **10.5x** | | whisper-large-v3-turbo | 21.79s | 0.70s | **14.3x** | | parakeet-tdt-0.6b-v2 | 0.47s | 0.18s | **54.4x** | | parakeet-tdt-0.6b-v3 | 1.13s | 0.18s | **54.6x** | ### 模型推荐 | 使用场景 | 推荐模型 | 原因 | |----------|----------|------| | **实时转录** | whisper-tiny | 速度最快（197x RTF），延迟低 | | **通用场景** | whisper-large-v3-turbo | 速度（55x）与质量兼顾，综合表现最佳 | | **最高精度** | whisper-large-v3 | 准确率最高，支持 99 种以上语言 | | **低内存** | whisper-small | 244M 参数，质量良好 | ### 转录质量 所有模型均能正确转录测试音频。示例输出： ``` Input text: "Welcome to this comprehensive speech to text demonstration. This audio sample is designed to test the accuracy and speed of various speech recognition models. The quick brown fox jumps over the lazy dog..." Whisper-large-v3 output: "Welcome to this comprehensive speech to text demonstration. This audio sample is designed to test the accuracy and speed of various speech recognition models. The quick brown fox jumps over the lazy dog..." (identical) ``` ### 支持的语言 Whisper 模型支持 99 种以上语言，包括： - 英语、西班牙语、法语、德语、意大利语、葡萄牙语 - 中文（普通话、粤语）、日语、韩语 - 阿拉伯语、印地语、俄语、土耳其语、乌克兰语 - 以及更多语言 ## 文本转语音 (TTS) 基准测试 ### 运行 TTS 基准测试 ```bash python examples/benchmark_audio.py --tts ``` ### 测试结果（M4 Max，128GB） **测试内容：** 为 3 段文本样本（短、中、长）生成音频 | Model | Load Time | Chars/sec | RTF* | |-------|-----------|-----------|------| | Kokoro-82M-bf16 | 0.8s | 350+ | **22x** | | Kokoro-82M-4bit | 0.4s | 320+ | **20x** | *RTF = 实时倍率。RTF 为 22x 表示 1 秒的音频约在 0.045 秒内生成完毕。* ### TTS 测试结果（M1 Max，64GB） | Model | Load Time | Avg Chars/s | Avg RTF | |-------|-----------|-------------|---------| | Kokoro-82M-bf16 | 2.81s | 176.0 | **11.9x** | | Kokoro-82M-4bit | 0.22s | 225.6 | **15.5x** | ### TTS 质量 Kokoro 可生成自然流畅的语音，具备以下特性： - 11 种内置音色（男声与女声） - 支持 8 种语言（英语、西班牙语、法语、日语、中文、意大利语、葡萄牙语、印地语） - 82M 参数，轻量且高效 ## 音频处理基准测试 ### SAM-Audio（音源分离） **测试内容：** 从 30 秒摇滚歌曲中分离鼓声 | Metric | Value | |--------|-------| | Model | sam-audio-large-fp16 | | Processing time | ~20s | | Peak memory | ~27 GB | | Output sample rate | 48000 Hz | ## 运行全部音频基准测试 ```bash # Run all benchmarks python examples/benchmark_audio.py --all # Or run individually python examples/benchmark_audio.py --stt python examples/benchmark_audio.py --tts ``` ## mlx-community 上的可用模型 ### STT 模型 - `mlx-community/whisper-tiny-mlx` - `mlx-community/whisper-small-mlx` - `mlx-community/whisper-medium-mlx` - `mlx-community/whisper-large-v3-mlx` - `mlx-community/whisper-large-v3-turbo` - `mlx-community/parakeet-tdt-0.6b-v2` - `mlx-community/parakeet-tdt-0.6b-v3` ### TTS 模型 - `mlx-community/Kokoro-82M-bf16`（推荐） - `mlx-community/Kokoro-82M-4bit` - `mlx-community/chatterbox-turbo-fp16` - `mlx-community/VibeVoice-Realtime-0.5B-4bit` ### 音频处理 - `mlx-community/sam-audio-large-fp16` # Documentation page: `zh/benchmarks/image.md` # 图像基准测试 ## 运行图像基准测试 ```bash # 完整基准测试（10 种分辨率） vllm-mlx-bench --model mlx-community/Qwen3-VL-8B-Instruct-4bit # 快速基准测试（4 种分辨率） vllm-mlx-bench --model mlx-community/Qwen3-VL-8B-Instruct-4bit --quick ``` ## 测试结果 - Qwen3-VL-8B-Instruct-4bit（M4 Max，128GB） | Resolution | Pixels | Time | Tokens | Speed | |------------|--------|------|--------|-------| | 224x224 | 50K | 1.04s | 78 | 74.8 tok/s | | 336x336 | 113K | 0.94s | 64 | 68.3 tok/s | | 448x448 | 201K | 1.45s | 70 | 48.1 tok/s | | 512x512 | 262K | 1.58s | 99 | 62.8 tok/s | | 672x672 | 452K | 1.83s | 83 | 45.3 tok/s | | 768x768 | 590K | 2.05s | 91 | 44.3 tok/s | | 896x896 | 803K | 2.61s | 90 | 34.5 tok/s | | 1024x1024 | 1.0M | 2.79s | 76 | 27.2 tok/s | | 1280x720 | 922K | 2.97s | 96 | 32.4 tok/s | | 1920x1080 | 2.1M | 6.30s | 89 | 14.1 tok/s | **摘要：** 所有分辨率的平均速度为 45.2 tok/s。最快为 224x224（74.8 tok/s），最慢为 1920x1080（14.1 tok/s）。 ## 测试结果 - Qwen3-VL-8B-Instruct-4bit（M1 Max，64GB） 本地 MLLM 基准测试： | Resolution | Pixels | Time | Tokens | Speed | |------------|--------|------|--------|-------| | 224x224 | 50K | 1.84s | 78 | 42.5 tok/s | | 448x448 | 201K | 2.28s | 70 | 30.7 tok/s | | 768x768 | 590K | 4.39s | 91 | 20.7 tok/s | | 1024x1024 | 1.0M | 6.41s | 76 | 11.9 tok/s | | Configs | Total Time (s) | Total Tokens | Aggregate Tok/s | |---------|-----------------|--------------|-----------------| | 4 | 14.92 | 315 | 21.1 | ## 测试结果 - Qwen3-VL-4B-Instruct-3bit 服务端（M1 Max，64GB） | Resolution | Pixels | Time | Tokens | Speed | |------------|--------|------|--------|-------| | 224x224 | 50K | 1.65s | 113 | 68.4 tok/s | | 448x448 | 201K | 2.09s | 120 | 57.5 tok/s | | 768x768 | 590K | 2.93s | 106 | 36.2 tok/s | | 1024x1024 | 1.0M | 4.12s | 100 | 24.3 tok/s | | Configs | Total Time (s) | Total Tokens | Aggregate Tok/s | |---------|-----------------|--------------|-----------------| | 4 | 10.79 | 439 | 40.7 | ## MLLM 前缀缓存测试结果 ``` ====================================================================== MLLM PREFIX CACHE TEST ====================================================================== Model: mlx-community/Qwen3-VL-4B-Instruct-3bit Test: Verify KV cache reuse for repeated image/video + prompt combinations Expected behavior: - Same image + same prompt → cache HIT - Same image + different prompt → cache MISS - Different image + same prompt → cache MISS ---------------------------------------------------------------------- SETUP: Loading Model ---------------------------------------------------------------------- Model loaded in 0.11s ---------------------------------------------------------------------- SETUP: Creating Test Images ---------------------------------------------------------------------- Resized: 224x224, 336x336, 512x512, 768x768 ---------------------------------------------------------------------- TEST 1: Image Cache - Basic Hit/Miss ---------------------------------------------------------------------- Results: Step | Description | Expected | Actual | Time | Status -------+---------------------------+----------+--------+--------+------- 1a | First image+prompt | MISS | MISS | 0.10ms | ✓ 1b | Same image+prompt | HIT | HIT | 0.18ms | ✓ 1c | Different prompt | MISS | MISS | 0.01ms | ✓ 1d | Return to original | HIT | HIT | 0.18ms | ✓ ---------------------------------------------------------------------- TEST 2: Different Images ---------------------------------------------------------------------- Results: Step | Description | Expected | Actual | Time | Status -------+---------------------------+----------+--------+--------+------- 2a | Image A first request | MISS | MISS | 0.01ms | ✓ 2b | Image B first request | MISS | MISS | 0.01ms | ✓ 2c | Image A cached | HIT | HIT | 0.13ms | ✓ ---------------------------------------------------------------------- TEST 3: Image Resolutions ---------------------------------------------------------------------- Results: Step | Description | Expected | Actual | Time | Status -------+-----------------------+----------+--------+--------+------- 3.1a | 224x224 first | MISS | MISS | 0.01ms | ✓ 3.1b | 224x224 cached | HIT | HIT | 0.20ms | ✓ 3.2a | 336x336 first | MISS | MISS | 0.01ms | ✓ 3.2b | 336x336 cached | HIT | HIT | 0.21ms | ✓ 3.3a | 512x512 first | MISS | MISS | 0.12ms | ✓ 3.3b | 512x512 cached | HIT | HIT | 0.20ms | ✓ 3.4a | 768x768 first | MISS | MISS | 0.12ms | ✓ 3.4b | 768x768 cached | HIT | HIT | 0.24ms | ✓ ====================================================================== ``` ## 缓存键策略 - **图像：** `hash(image_content) + hash(prompt)` 相同图像与相同提示词始终命中缓存。不同图像或不同提示词将不命中缓存。 ## 性能提示 - 分辨率越小，处理速度越快（如 224x224 对比 1920x1080） - 请根据任务需求选择合适的分辨率 - 批量处理尺寸相近的图像，以获得稳定的吞吐量 ## 指标说明 | Metric | Description | |--------|-------------| | Resolution | 图像尺寸（宽 x 高） | | Pixels | 总像素数 | | Time | 生成时间 | | Tokens | 生成的输出 token 数量 | | Speed | 每秒 token 数（tok/s） | # Documentation page: `zh/benchmarks/llm.md` # LLM 基准测试 ## 运行 LLM 基准测试 ```bash vllm-mlx-bench --model mlx-community/Llama-3.2-1B-Instruct-4bit --prompts 5 --max-tokens 256 ``` ## 测试结果 (M4 Max, 128GB) | Model | Gen Speed | TTFT* | Memory | |-------|-----------|-------|--------| | Qwen3-0.6B-8bit | 402.3 tok/s | 58.6 ms | 0.68 GB | | Llama-3.2-1B-Instruct-4bit | 463.6 tok/s | 49.2 ms | 0.69 GB | | Qwen2.5-1.5B-Instruct-4bit | 308.5 tok/s | 86.2 ms | 0.84 GB | | Llama-3.2-3B-Instruct-4bit | 200.1 tok/s | 81.4 ms | 1.79 GB | | Qwen3-30B-A3B-4bit | 123.9 tok/s | 126.9 ms | 16.05 GB | | NVIDIA-Nemotron-3-Nano-30B-A3B-MLX-6Bit | 122.9 tok/s | 72.3 ms | 23.98 GB | *TTFT = 首个 token 的生成时间（模型开始输出前的延迟） ## 测试结果 (M1 Max, 64GB) | Model | Runs | Prompt Tok | Gen Tok | Total Time (s) | TTFT Mean (ms) | TPOT Mean (ms) | Gen Speed (tok/s) | Total Throughput (tok/s) | |-------|------|------------|---------|-----------------|-----------------|-----------------|-------------------|--------------------------| | Qwen3-0.6B-8bit | 5 | 56 | 1280 | 5.66 | 119.0 | 3.97 | 251.9 | 236.1 | ## Continuous Batching 测试结果 | Model | Single Request | Batch (5 req) | Speedup | |-------|----------------|---------------|---------| | Llama-3.2-1B-Instruct-4bit | 299.1 tok/s | 613.0 tok/s | **2.05x** | | Llama-3.2-3B-Instruct-4bit | 137.6 tok/s | 208.1 tok/s | **1.51x** | | Qwen3-0.6B-8bit | 328.1 tok/s | 1111.8 tok/s | **3.39x** | | Qwen3-30B-A3B-4bit | 98.1 tok/s | 233.3 tok/s | **2.38x** | | Qwen2.5-1.5B-Instruct-4bit | 196.9 tok/s | 322.2 tok/s | **1.64x** | *批量处理 5 个并发请求可将 throughput 提升 1.5 到 3 倍。* ### Continuous Batching (M1 Max, 64GB) | Requests | Total Tokens | Total Time (s) | Throughput (tok/s) | Requests/sec | |----------|--------------|-----------------|--------------------|--------------| | 5 | 315 | 0.64 | 492.5 | 7.82 | ## Streaming 性能 | Model | TTFT | Generation Speed | |-------|------|------------------| | Llama-3.2-1B-Instruct-4bit | ~4.6ms | 218.9 tok/s | | Llama-3.2-3B-Instruct-4bit | ~10.7ms | 93.6 tok/s | | Qwen3-0.6B-8bit | ~3.0ms | 328.5 tok/s | | Qwen3-30B-A3B-4bit | ~10.2ms | 98.4 tok/s | | Qwen2.5-1.5B-Instruct-4bit | ~7.1ms | 140.3 tok/s | ### Streaming 解码器 (M1 Max, 64GB) `vllm-mlx bench-detok`: | Tokens | Iterations | Naive Time | Streaming Time | Speedup | |--------|------------|------------|----------------|---------| | 742 | 5 | 1.69ms | 0.71ms | 2.39x | `examples/benchmark_detokenizer.py`: | Sequence | Tokens | decode() | Streaming | Speedup | |----------|--------|----------|-----------|---------| | Short | 8 | 0.029ms | 0.028ms | 1.04x | | Medium | 103 | 0.206ms | 0.129ms | 1.59x | | Long | 511 | 1.040ms | 0.502ms | 2.07x | | 1K | 1191 | 2.446ms | 1.178ms | 2.08x | | 2K | 2381 | 4.949ms | 2.356ms | 2.10x | | 4K | 4761 | 9.887ms | 5.398ms | 1.83x | 平均加速比：1.79x ## Prefix Cache 测试结果 ### Prefix Cache (M4 Max, 128GB) ``` ====================================================================== LLM PREFIX CACHE TEST ====================================================================== Model: mlx-community/Qwen3-0.6B-8bit Expected behavior: - Same prompt → cache HIT - Different prompt → cache MISS ---------------------------------------------------------------------- Results: Step | Description | Expected | Actual | Status -------+---------------------+----------+--------+------- 1a | First request | MISS | MISS | ✓ 1b | Same prompt | HIT | HIT | ✓ 1c | Different prompt | MISS | MISS | ✓ 1d | Return to prompt 1 | HIT | HIT | ✓ ====================================================================== ``` ### Prefix Cache (M1 Max, 64GB) | Test | Expected | Actual | Time | Status | |------|----------|--------|------|--------| | First request | MISS | MISS | 203.5ms | PASS | | Same prompt | HIT | HIT | 131.6ms | PASS | | Different prompt | MISS or PREFIX_HIT | PREFIX_HIT (5 tok) | 135.3ms | PASS | 最终缓存统计： | Cache Hits | Cache Misses | Hit Rate | Tokens Saved | Cached Speedup | |------------|--------------|----------|--------------|----------------| | 2 | 1 | 66.7% | 20 | 1.55x | ## Paged Cache 测试结果 *测试：2 轮共 20 个真实推理请求，系统提示约 286 个 token* ``` ====================================================================== PAGED KV CACHE - REAL INFERENCE TEST ====================================================================== -------------------------------------------------- Test 1: WITHOUT Paged Cache (2 rounds of 10) -------------------------------------------------- Time: 1.47s Throughput: 681.2 tok/s Cache hits: 0 Tokens saved: 0 -------------------------------------------------- Test 2: WITH Paged Cache (2 rounds of 10) -------------------------------------------------- Time: 1.31s Throughput: 765.8 tok/s Paged Cache Stats: Blocks allocated: 25 Shared blocks: 4 Cache hits: 10 Tokens saved: 2560 ================================================== SUMMARY ================================================== Without paged cache: 681.2 tok/s With paged cache: 765.8 tok/s Speedup: 1.12x Cache hits: 10 (all Round 2 requests) Tokens saved: 2,560 (~256 tokens × 10 requests) ================================================== ``` ### Paged KV Cache (M1 Max, 64GB) 推理基准测试（20 个请求）： | Mode | Time (s) | Throughput (tok/s) | |------|----------|--------------------| | Without paged cache | 3.43 | 291.8 | | With paged cache | 3.42 | 292.2 | | Speedup | Blocks Allocated | Shared Blocks | Cache Hits | Tokens Saved | |---------|------------------|---------------|------------|--------------| | 1.00x | 45 | 4 | 10 | 2560 | 真实并发推理（20 个请求）： | Mode | Time (s) | Throughput (tok/s) | |------|----------|--------------------| | Without paged cache | 4.32 | 231.7 | | With paged cache | 4.35 | 229.7 | | Speedup | Blocks Allocated | Shared Blocks | Cache Hits | Tokens Saved | |---------|------------------|---------------|------------|--------------| | 0.99x | 49 | 8 | 10 | 5120 | 内存节省示例： | Scenario | Memory Savings | |----------|----------------| | Shared system prompts | 70.8% | | Concurrent memory efficiency | 83.5% | | Prefix sharing branches | 38.5% | ## Streaming 解码器分析 *第 9.1 阶段调查：mlx-lm 的 `BPEStreamingDetokenizer` 与朴素 `tokenizer.decode()` 对比* ### 背景 朴素方法对每个 token 调用 `decode([token])`。理论上，streaming 解码器的时间复杂度为 O(T)，而朴素解码为 O(T²)。 ### 孤立基准测试结果 ```bash vllm-mlx bench-detok ``` 复用同一解码器实例时（每次使用前调用 `reset()`）： | Sequence | Tokens | Naive decode() | Streaming | Speedup | |----------|--------|----------------|-----------|---------| | Short | 8 | 0.020ms | 0.019ms | 1.05x | | Medium | 103 | 0.155ms | 0.097ms | 1.59x | | Long | 511 | 0.752ms | 0.371ms | **2.03x** | | 1K tokens | 1191 | 1.743ms | 0.833ms | **2.09x** | | 2K tokens | 2381 | 3.493ms | 1.737ms | **2.01x** | ### 关键发现：实例创建开销 创建新的 `BPEStreamingDetokenizer` 实例**极其昂贵**： ``` 100 tokenizer.detokenizer calls: 5.266s (52.7ms each!) ``` 这意味着每个请求新建一个解码器实例会增加约 **52ms 的额外开销**，从而抵消所有性能收益。 ### 实际影响 集成到调度器后（每个请求一个解码器实例）： | Metric | Naive decode() | Streaming (new instance) | |--------|----------------|--------------------------| | Throughput (20 req) | 681 tok/s | 275 tok/s | | Impact | - | **慢 60%** | ### 结论 由于实例创建成本过高，streaming 解码器**目前不适合**在每个请求中独立使用。朴素的 `decode([token])` 方法在实践中仍然更快。 **未来优化方向**：在启动时预先创建一个解码器实例池，并在请求间复用这些实例。 ## 指标参考 | Metric | Description | |--------|-------------| | **TTFT** | Time to First Token，模型开始响应前的延迟 (ms) | | **TPOT** | Time Per Output Token，每个生成 token 之间的间隔 (ms/token) | | **Generation TPS** | 每秒输出 token 数 (tok/s) | | **Processing TPS** | 每秒处理输入或提示词 token 数 (tok/s) | | **End-to-End Latency** | 从请求发出到收到完整响应的总时间 | | **Total Throughput** | 每秒处理的总 token 数（输入加输出） | ## 运行基准测试 ```bash # Basic benchmark vllm-mlx-bench --model mlx-community/Qwen3-0.6B-8bit # With more prompts vllm-mlx-bench --model mlx-community/Qwen3-0.6B-8bit --prompts 10 # Save results vllm-mlx-bench --model mlx-community/Qwen3-0.6B-8bit --output results.json # Continuous batching test python tests/test_continuous_batching.py # Prefix cache test python tests/test_prefix_cache.py # Paged cache test python tests/test_paged_cache_real_inference.py # Streaming detokenizer benchmark vllm-mlx bench-detok vllm-mlx bench-detok mlx-community/Llama-3.2-1B-Instruct-4bit --iterations 5 ``` # Documentation page: `zh/benchmarks/video.md` # 视频基准测试 ## 运行视频基准测试 ```bash # 完整基准测试（10 种配置，2-64 帧） vllm-mlx-bench --model mlx-community/Qwen3-VL-8B-Instruct-4bit --video # 快速基准测试（3 种帧数） vllm-mlx-bench --model mlx-community/Qwen3-VL-8B-Instruct-4bit --video --quick # 自定义视频 vllm-mlx-bench --model mlx-community/Qwen3-VL-8B-Instruct-4bit --video --video-url https://example.com/video.mp4 ``` ## 结果 - Qwen3-VL-8B-Instruct-4bit（M4 Max，128GB） | Configuration | Frames | Time | Tokens | Speed | Memory | |---------------|--------|------|--------|-------|--------| | 2 frames @ 0.5fps | 2 | 4.48s | 256 | 57.1 tok/s | 6.4 GB | | 4 frames @ 1fps | 4 | 4.65s | 256 | 55.0 tok/s | 6.4 GB | | 6 frames @ 1fps | 6 | 5.15s | 197 | 38.2 tok/s | 6.6 GB | | 8 frames @ 2fps | 8 | 6.45s | 240 | 37.2 tok/s | 6.8 GB | | 12 frames @ 2fps | 12 | 8.73s | 256 | 29.3 tok/s | 7.1 GB | | 16 frames @ 2fps | 16 | 10.96s | 256 | 23.4 tok/s | 7.6 GB | | 24 frames @ 4fps | 24 | 14.95s | 226 | 15.1 tok/s | 8.4 GB | | 32 frames @ 4fps | 32 | 20.00s | 256 | 12.8 tok/s | 9.2 GB | | 48 frames @ 8fps | 48 | 31.11s | 246 | 7.9 tok/s | 11.1 GB | | 64 frames @ 8fps | 64 | 59.81s | 256 | 4.3 tok/s | 12.9 GB | **总结：** 2 帧时速度最快（57.1 tok/s），64 帧时速度最慢（4.3 tok/s）。内存占用从 6.4 GB 增长至 12.9 GB。 > **注意：** 96 帧及以上会因内存或计算资源限制，在大多数硬件上导致 GPU 超时。 ## 结果 - Qwen3-VL-8B-Instruct-4bit（M1 Max，64GB） | Configuration | Frames | FPS | Time | Tokens | Speed | |---------------|--------|-----|------|--------|-------| | 4 frames @ 1fps | 4 | 1.0 | 8.84s | 256 | 29.0 tok/s | | 8 frames @ 2fps | 8 | 2.0 | 13.05s | 256 | 19.6 tok/s | | 16 frames @ 2fps | 16 | 2.0 | 21.60s | 256 | 11.9 tok/s | | Configs | Total Time (s) | Total Tokens | Aggregate Tok/s | |---------|-----------------|--------------|-----------------| | 3 | 43.48 | 768 | 17.7 | ## 结果 - Qwen3-VL-4B-Instruct-3bit（M1 Max，64GB） | Configuration | Frames | FPS | Time | Tokens | Speed | |---------------|--------|-----|------|--------|-------| | 4 frames @ 1fps | 4 | 1.0 | 5.09s | 150 | 29.5 tok/s | | 8 frames @ 2fps | 8 | 2.0 | 8.36s | 150 | 17.9 tok/s | | 16 frames @ 2fps | 16 | 2.0 | 15.21s | 150 | 9.9 tok/s | | Configs | Total Time (s) | Total Tokens | Aggregate Tok/s | |---------|-----------------|--------------|-----------------| | 3 | 28.66 | 450 | 15.7 | ## 视频缓存结果 ``` ---------------------------------------------------------------------- TEST 4: Video Cache - fps/max_frames in Cache Key ---------------------------------------------------------------------- Config: fps=2.0, max_frames=16 Results: Step | Description | Expected | Actual | Time | Status -------+---------------------------+----------+--------+--------+------- 4a | Video first request | MISS | MISS | 0.03ms | ✓ 4b | Same video+params | HIT | HIT | 0.14ms | ✓ 4c | Different fps (4.0) | MISS | MISS | 0.01ms | ✓ 4d | Different max_frames (32) | MISS | MISS | 0.01ms | ✓ 4.0.5a | fps=0.5 first | MISS | MISS | 0.01ms | ✓ 4.0.5b | fps=0.5 cached | HIT | HIT | 0.14ms | ✓ 4.1.0a | fps=1.0 first | MISS | MISS | 0.01ms | ✓ 4.1.0b | fps=1.0 cached | HIT | HIT | 0.14ms | ✓ 4.2.0a | fps=2.0 first | MISS | MISS | 0.01ms | ✓ 4.2.0b | fps=2.0 cached | HIT | HIT | 0.14ms | ✓ 4.4.0a | fps=4.0 first | MISS | MISS | 0.01ms | ✓ 4.4.0b | fps=4.0 cached | HIT | HIT | 0.14ms | ✓ ---------------------------------------------------------------------- TEST 5: Additional Videos ---------------------------------------------------------------------- Results: Step | Description | Expected | Actual | Time | Status -------+---------------------------+----------+--------+--------+------- 5a | Video 1 first | MISS | MISS | 0.01ms | ✓ 5b | Video 2 first | MISS | MISS | 0.01ms | ✓ 5c | Video 1 cached | HIT | HIT | 0.13ms | ✓ 5d | Video 2 cached | HIT | HIT | 0.13ms | ✓ ``` ## 缓存键策略 - **视频：** `hash(video_path) + hash(fps) + hash(max_frames) + hash(prompt)` 相同视频在 fps、max_frames 和 prompt 均相同时会命中缓存。任意参数发生变化则导致缓存未命中。 ## 性能建议 - FPS 越低，处理速度越快。 - 帧数越少，内存占用越小。 - 64 帧是实际可用的最大值。 - 96 帧及以上会导致 GPU 超时。 ## 帧提取参考 | FPS | 10s 视频 | 30s 视频 | 60s 视频 | |-----|-----------|-----------|-----------| | 0.5 | 5 frames | 15 frames | 30 frames | | 1.0 | 10 frames | 30 frames | 60 frames | | 2.0 | 20 frames | 60 frames | 120 frames* | | 4.0 | 40 frames | 120 frames* | 240 frames* | *可能触及 `max_frames` 上限 ## 指标说明 | Metric | 说明 | |--------|-------------| | Configuration | FPS 与最大帧数设置 | | Frames | 实际提取的帧数 | | Time | 总生成时长 | | Tokens | 生成的输出 token 数 | | Speed | 每秒 token 数（tok/s） | | Memory | GPU 内存占用 | # Documentation page: `zh/getting-started/installation.md` # 安装 ## 系统要求 - 搭载 Apple Silicon（M1/M2/M3/M4/M5）的 macOS - Python 3.10+ ## 使用 uv 安装（推荐） ```bash git clone https://github.com/waybarrios/vllm-mlx.git cd vllm-mlx uv pip install -e . ``` ## 使用 pip 安装 ```bash git clone https://github.com/waybarrios/vllm-mlx.git cd vllm-mlx pip install -e . ``` ### 可选：视觉支持 使用 transformers 进行视频处理： ```bash pip install -e ".[vision]" ``` ### 可选：音频支持（STT/TTS） ```bash pip install mlx-audio ``` ### 可选：向量嵌入 ```bash pip install mlx-embeddings ``` ## 安装内容说明 - `mlx`, `mlx-lm`, `mlx-vlm` - MLX 框架及模型库 - `transformers`, `tokenizers` - HuggingFace 库 - `opencv-python` - 视频处理 - `gradio` - 对话界面 - `psutil` - 资源监控 - `mlx-audio`（可选）- 语音转文字与文字转语音 - `mlx-embeddings`（可选）- 文本向量嵌入 ## 验证安装 ```bash # Check CLI commands vllm-mlx --help vllm-mlx-bench --help vllm-mlx-chat --help # Test with a small model vllm-mlx-bench --model mlx-community/Llama-3.2-1B-Instruct-4bit --prompts 1 ``` ## 故障排查 ### 找不到 MLX 请确认您使用的是 Apple Silicon 设备： ```bash uname -m # Should output "arm64" ``` ### 模型下载失败 请检查网络连接及 HuggingFace 访问权限。部分模型需要身份验证： ```bash huggingface-cli login ``` ### 内存不足 请使用更小的量化模型： ```bash vllm-mlx serve mlx-community/Llama-3.2-1B-Instruct-4bit ``` ### 长时间运行时服务器可能因 macOS 进入睡眠而中断 长时间作为服务器运行时，macOS 设备可能会进入睡眠。可以使用 `caffeinate` 来阻止进入睡眠： ```bash caffeinate -dimsu ``` # Documentation page: `zh/getting-started/quickstart.md` # 快速开始 ## 选项 1：兼容 OpenAI 的服务器 启动服务器： ```bash # Simple mode - maximum throughput for single user vllm-mlx serve mlx-community/Llama-3.2-3B-Instruct-4bit --port 8000 # Continuous batching - for multiple concurrent users vllm-mlx serve mlx-community/Llama-3.2-3B-Instruct-4bit --port 8000 --continuous-batching ``` 使用 OpenAI Python SDK： ```python from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed") response = client.chat.completions.create( model="mlx-community/Llama-3.2-3B-Instruct-4bit", messages=[{"role": "user", "content": "Hello!"}], ) print(response.choices[0].message.content) ``` 或使用 curl： ```bash curl http://localhost:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{"model": "default", "messages": [{"role": "user", "content": "Hello!"}]}' ``` ## 选项 2：直接使用 Python API ```python from vllm_mlx.models import MLXLanguageModel model = MLXLanguageModel("mlx-community/Llama-3.2-3B-Instruct-4bit") model.load() # Generate text output = model.generate("What is the capital of France?", max_tokens=100) print(output.text) # Streaming for chunk in model.stream_generate("Tell me a story"): print(chunk.text, end="", flush=True) ``` ## 选项 3：Gradio 聊天界面 ```bash vllm-mlx-chat --served-model-name mlx-community/Llama-3.2-3B-Instruct-4bit ``` 在 http://localhost:7860 打开网页界面。 ## 多模态模型 如需图像或视频理解，请使用 VLM 模型： ```bash vllm-mlx serve mlx-community/Qwen3-VL-4B-Instruct-3bit --port 8000 ``` ```python response = client.chat.completions.create( model="default", messages=[{ "role": "user", "content": [ {"type": "text", "text": "What's in this image?"}, {"type": "image_url", "image_url": {"url": "https://example.com/image.jpg"}} ] }], max_tokens=256 ) ``` ## 推理模型 将模型的思考过程与最终答案分离： ```bash vllm-mlx serve mlx-community/Qwen3-8B-4bit --reasoning-parser qwen3 ``` ```python response = client.chat.completions.create( model="default", messages=[{"role": "user", "content": "What is 17 × 23?"}] ) print(response.choices[0].message.content) # Final answer ``` ## Embeddings 为语义搜索和 RAG 生成文本 embeddings： ```bash vllm-mlx serve mlx-community/Qwen3-4B-4bit --embedding-model mlx-community/multilingual-e5-small-mlx ``` ```python response = client.embeddings.create( model="mlx-community/multilingual-e5-small-mlx", input="Hello world" ) ``` ## 工具调用 为任意支持的模型启用函数调用： ```bash vllm-mlx serve mlx-community/Devstral-Small-2507-4bit \ --enable-auto-tool-choice --tool-call-parser mistral ``` ## 下一步 - [服务器指南](../guides/server.md) - 完整的服务器配置 - [Python API](../guides/python-api.md) - 直接使用 API - [多模态指南](../guides/multimodal.md) - 图像与视频 - [音频指南](../guides/audio.md) - STT 与 TTS - [Embeddings 指南](../guides/embeddings.md) - 文本 embeddings - [推理模型](../guides/reasoning.md) - 思考模型 - [工具调用](../guides/tool-calling.md) - 函数调用 - [支持的模型](../reference/models.md) - 可用模型 # Documentation page: `zh/guides/audio.md` # 音频支持 vllm-mlx 通过 [mlx-audio](https://github.com/Blaizzy/mlx-audio) 支持音频处理，提供以下功能： - **STT（语音转文字）**：whisper、Parakeet - **TTS（文字转语音）**：Kokoro、Chatterbox、VibeVoice、VoxCPM - **音频处理**：SAM-Audio（人声分离） ## 安装 ```bash # 核心音频支持 pip install mlx-audio>=0.2.9 # TTS 所需依赖 pip install sounddevice soundfile scipy numba tiktoken misaki spacy num2words loguru phonemizer # 下载 spacy 英文模型 python -m spacy download en_core_web_sm # 非英语 TTS（西班牙语、法语等）需安装 espeak-ng： # macOS brew install espeak-ng # Ubuntu/Debian # sudo apt-get install espeak-ng ``` 或一次性安装所有音频依赖： ```bash pip install vllm-mlx[audio] python -m spacy download en_core_web_sm brew install espeak-ng # macOS，用于非英语语言 ``` ## 快速开始 ### STT（语音转录） ```python from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed") # 转录音频文件 with open("audio.mp3", "rb") as f: transcript = client.audio.transcriptions.create( model="whisper-large-v3", file=f, language="en" # optional ) print(transcript.text) ``` ### TTS（语音合成） ```python # 生成语音 audio = client.audio.speech.create( model="kokoro", input="Hello, how are you?", voice="af_heart", speed=1.0 ) # 保存到文件 with open("output.wav", "wb") as f: f.write(audio.content) ``` ### 人声分离（SAM-Audio） 从背景噪声、音乐或其他声音中提取人声： ```python from vllm_mlx.audio import AudioProcessor # 加载 SAM-Audio 模型 processor = AudioProcessor("mlx-community/sam-audio-large-fp16") processor.load() # 从音频中分离语音 result = processor.separate("meeting_with_music.mp3", description="speech") # 保存分离后的人声和背景音 processor.save(result.target, "voice_only.wav") processor.save(result.residual, "background_only.wav") ``` **命令行示例：** ```bash python examples/audio_separation_example.py meeting.mp3 --play python examples/audio_separation_example.py song.mp3 --description music -o music.wav ``` ### 鼓声分离演示 使用 SAM-Audio 从摇滚歌曲中分离鼓声： | 音频 | 说明 | 收听 | |-------|-------------|--------| | 原始音频 | David Fesliyan 的"Get Ready"（30 秒，免版权） | [rock_get_ready.mp3](https://github.com/waybarrios/vllm-mlx/blob/main/examples/rock_get_ready.mp3?raw=1) | | 分离鼓声 | SAM-Audio 提取的鼓声 | [drums_isolated.wav](https://github.com/waybarrios/vllm-mlx/blob/main/examples/drums_isolated.wav?raw=1) | | 去除鼓声 | 移除鼓声后的音轨 | [rock_no_drums.wav](https://github.com/waybarrios/vllm-mlx/blob/main/examples/rock_no_drums.wav?raw=1) | ```bash # 从摇滚歌曲中分离鼓声 python examples/audio_separation_example.py examples/rock_get_ready.mp3 \ --description "drums" \ --output drums_isolated.wav \ --background rock_no_drums.wav ``` **性能：** 在 M4 Max 上处理 30 秒音频约需 20 秒。 ## 支持的模型 ### STT 模型（语音转文字） | 模型 | 别名 | 语言数 | 速度 | 质量 | |-------|-------|-----------|-------|---------| | `mlx-community/whisper-large-v3-mlx` | `whisper-large-v3` | 99+ | 中等 | 最佳 | | `mlx-community/whisper-large-v3-turbo` | `whisper-large-v3-turbo` | 99+ | 快速 | 优秀 | | `mlx-community/whisper-medium-mlx` | `whisper-medium` | 99+ | 快速 | 良好 | | `mlx-community/whisper-small-mlx` | `whisper-small` | 99+ | 极快 | 一般 | | `mlx-community/parakeet-tdt-0.6b-v2` | `parakeet` | 仅英语 | 最快 | 优秀 | | `mlx-community/parakeet-tdt-0.6b-v3` | `parakeet-v3` | 仅英语 | 最快 | 最佳 | **推荐方案：** - 多语言场景：`whisper-large-v3` - 仅英语场景：`parakeet`（速度快 3 倍） ### TTS 模型（文字转语音） #### Kokoro（快速、轻量）- 推荐 | 模型 | 别名 | 参数量 | 支持语言 | |-------|-------|------|-----------| | `mlx-community/Kokoro-82M-bf16` | `kokoro` | 82M | EN、ES、FR、JA、ZH、HI、IT、PT | | `mlx-community/Kokoro-82M-4bit` | `kokoro-4bit` | 82M | EN、ES、FR、JA、ZH、HI、IT、PT | **声音（11 种）：** - 美式女声：`af_heart`、`af_bella`、`af_nicole`、`af_sarah`、`af_sky` - 美式男声：`am_adam`、`am_michael` - 英式女声：`bf_emma`、`bf_isabella` - 英式男声：`bm_george`、`bm_lewis` **语言代码：** | 代码 | 语言 | 代码 | 语言 | |------|----------|------|----------| | `a` / `en` | 英语（美国） | `e` / `es` | Español | | `b` / `en-gb` | 英语（英国） | `f` / `fr` | Français | | `j` / `ja` | 日本語 | `z` / `zh` | 中文 | | `i` / `it` | Italiano | `p` / `pt` | Português | | `h` / `hi` | हिन्दी | | | #### Chatterbox（多语言、表现力强） | 模型 | 别名 | 参数量 | 支持语言 | |-------|-------|------|-----------| | `mlx-community/chatterbox-turbo-fp16` | `chatterbox` | 134M | 15+ 种语言 | | `mlx-community/chatterbox-turbo-4bit` | `chatterbox-4bit` | 134M | 15+ 种语言 | **支持语言：** EN、ES、FR、DE、IT、PT、RU、JA、ZH、KO、AR、HI、NL、PL、TR #### VibeVoice（实时） | 模型 | 别名 | 参数量 | 适用场景 | |-------|-------|------|----------| | `mlx-community/VibeVoice-Realtime-0.5B-4bit` | `vibevoice` | 200M | 低延迟、仅英语 | #### VoxCPM（中英双语） | 模型 | 别名 | 参数量 | 支持语言 | |-------|-------|------|-----------| | `mlx-community/VoxCPM1.5` | `voxcpm` | 0.9B | ZH、EN | | `mlx-community/VoxCPM1.5-4bit` | `voxcpm-4bit` | 200M | ZH、EN | ### 音频处理模型 #### SAM-Audio（人声分离） | 模型 | 参数量 | 适用场景 | |-------|------|----------| | `mlx-community/sam-audio-large-fp16` | 3B | 最佳质量 | | `mlx-community/sam-audio-large` | 3B | 标准 | | `mlx-community/sam-audio-small-fp16` | 0.6B | 快速 | | `mlx-community/sam-audio-small` | 0.6B | 轻量 | ## API 参考 ### POST /v1/audio/transcriptions 将音频转录为文字（兼容 OpenAI Whisper API）。 **参数：** - `file`：音频文件（mp3、wav、m4a、webm） - `model`：模型名称或别名 - `language`：语言代码（可选，自动检测） - `response_format`：`json` 或 `text` **限制：** - 默认上传上限：25 MiB - 可通过 `--max-audio-upload-mb` 修改 **示例：** ```bash curl http://localhost:8000/v1/audio/transcriptions \ -F file=@audio.mp3 \ -F model=whisper-large-v3 ``` ### POST /v1/audio/speech 从文字生成语音（兼容 OpenAI TTS API）。 **参数：** - `model`：模型名称或别名 - `input`：待合成文本 - `voice`：声音 ID - `speed`：语速（0.5 到 2.0） - `response_format`：`wav`、`mp3` **限制：** - 默认输入上限：4096 个字符 - 可通过 `--max-tts-input-chars` 修改 **示例：** ```bash curl http://localhost:8000/v1/audio/speech \ -d '{"model": "kokoro", "input": "Hello world", "voice": "af_heart"}' \ -H "Content-Type: application/json" \ --output speech.wav ``` ### GET /v1/audio/voices 列出模型可用的声音。 **示例：** ```bash curl http://localhost:8000/v1/audio/voices?model=kokoro ``` ## 命令行示例 ### 实时转录与字幕 从麦克风进行实时 STT 转录： ```bash # 使用 whisper-large-v3 生成字幕（最佳质量） python examples/closed_captions.py --language es --chunk 5 # 使用更快的模型降低延迟 python examples/closed_captions.py --language en --model whisper-turbo --chunk 3 # 基础麦克风转录（先录音再转录） python examples/mic_transcribe.py --language es # 实时分块转录 python examples/mic_realtime.py --language es --chunk 3 # 带语音活动检测的实时转录 python examples/mic_live.py --language es ``` **依赖安装：** ```bash pip install sounddevice soundfile numpy ``` ### 基础 TTS ```bash # 简单 TTS 示例 python examples/tts_example.py "Hello, how are you?" --play # 使用不同声音 python examples/tts_example.py "Hello!" --voice am_michael --play # 保存到文件 python examples/tts_example.py "Welcome to the demo" -o greeting.wav # 列出可用声音 python examples/tts_example.py --list-voices ``` ### 多语言 TTS ```bash # 英语（自动选择最佳模型） python examples/tts_multilingual.py "Hello world" --play # 西班牙语 python examples/tts_multilingual.py "Hola mundo" --lang es --play # 法语 python examples/tts_multilingual.py "Bonjour le monde" --lang fr --play # 日语 python examples/tts_multilingual.py "こんにちは" --lang ja --play # 中文 python examples/tts_multilingual.py "你好世界" --lang zh --play # 指定模型 python examples/tts_multilingual.py "Hello" --model chatterbox --play # 列出所有模型 python examples/tts_multilingual.py --list-models # 列出所有语言 python examples/tts_multilingual.py --list-languages ``` ### 商务助手语音示例 使用**原生声音**预生成的常见商务场景语音样本： | 语言 | 声音 | 内容 | 收听 | |----------|-------|---------|--------| | 英语 | af_heart | "Welcome to First National Bank. How may I assist you today?" | [assistant_bank_en.wav](https://github.com/waybarrios/vllm-mlx/blob/main/examples/assistant_bank_en.wav?raw=1) | | 西班牙语 | ef_dora | "Gracias por llamar a servicio al cliente. Un agente le atenderá pronto." | [assistant_service_es.wav](https://github.com/waybarrios/vllm-mlx/blob/main/examples/assistant_service_es.wav?raw=1) | | 法语 | ff_siwis | "Bienvenue. Votre appel est important pour nous." | [assistant_callcenter_fr.wav](https://github.com/waybarrios/vllm-mlx/blob/main/examples/assistant_callcenter_fr.wav?raw=1) | | 中文 | zf_xiaobei | "欢迎致电技术支持中心。我们将竭诚为您服务。" | [assistant_support_zh.wav](https://github.com/waybarrios/vllm-mlx/blob/main/examples/assistant_support_zh.wav?raw=1) | **使用原生声音自行生成：** ```bash # 英语 - 银行助手（原生声音：af_heart） python -m mlx_audio.tts.generate --model mlx-community/Kokoro-82M-bf16 \ --text "Welcome to First National Bank. How may I assist you today?" \ --voice af_heart --lang_code a --file_prefix assistant_bank_en # 西班牙语 - 客服（原生声音：ef_dora） python -m mlx_audio.tts.generate --model mlx-community/Kokoro-82M-bf16 \ --text "Gracias por llamar a servicio al cliente. Un agente le atendera pronto." \ --voice ef_dora --lang_code e --file_prefix assistant_service_es # 法语 - 呼叫中心（原生声音：ff_siwis） python -m mlx_audio.tts.generate --model mlx-community/Kokoro-82M-bf16 \ --text "Bienvenue. Votre appel est important pour nous." \ --voice ff_siwis --lang_code f --file_prefix assistant_callcenter_fr # 中文 - 技术支持（原生声音：zf_xiaobei） python -m mlx_audio.tts.generate --model mlx-community/Kokoro-82M-bf16 \ --text "欢迎致电技术支持中心。我们将竭诚为您服务。" \ --voice zf_xiaobei --lang_code z --file_prefix assistant_support_zh ``` ### 原生声音参考 | 语言 | 代码 | 声音 | |----------|------|--------| | 英语（美国） | `a` | af_heart、af_bella、af_nicole、am_adam、am_michael | | 英语（英国） | `b` | bf_emma、bf_isabella、bm_george、bm_lewis | | 西班牙语 | `e` | ef_dora、em_alex、em_santa | | 法语 | `f` | ff_siwis | | 中文 | `z` | zf_xiaobei、zf_xiaoni、zf_xiaoxiao、zm_yunjian、zm_yunxi | | 日语 | `j` | jf_alpha、jf_gongitsune、jm_kumo | | 意大利语 | `i` | if_sara、im_nicola | | 葡萄牙语 | `p` | pf_dora、pm_alex | | 印地语 | `h` | hf_alpha、hf_beta、hm_omega | ## Python API ### 直接调用（不启动服务器） ```python from vllm_mlx.audio import STTEngine, TTSEngine, AudioProcessor # Speech-to-Text stt = STTEngine("mlx-community/whisper-large-v3-mlx") stt.load() result = stt.transcribe("audio.mp3") print(result.text) # Text-to-Speech tts = TTSEngine("mlx-community/Kokoro-82M-bf16") tts.load() audio = tts.generate("Hello world", voice="af_heart") tts.save(audio, "output.wav") # Voice Separation processor = AudioProcessor("mlx-community/sam-audio-large-fp16") processor.load() result = processor.separate("mixed_audio.mp3", description="speech") processor.save(result.target, "voice_only.wav") processor.save(result.residual, "background.wav") ``` ### 便捷函数 ```python from vllm_mlx.audio import transcribe_audio, generate_speech, separate_voice # 快速转录 result = transcribe_audio("audio.mp3") print(result.text) # 快速 TTS audio = generate_speech("Hello world", voice="af_heart") # 快速人声分离 voice, background = separate_voice("mixed.mp3") ``` ## 在对话中使用音频 在对话消息中附带音频（自动转录）： ```python response = client.chat.completions.create( model="default", messages=[{ "role": "user", "content": [ {"type": "text", "text": "Summarize this audio"}, {"type": "audio_url", "audio_url": {"url": "file://meeting.mp3"}} ] }] ) ``` ## 基准测试 测试环境：Apple M2 Max（32GB）。 ### TTS 基准测试（Kokoro-82M-bf16） | 文本长度 | 音频时长 | 生成耗时 | RTF | 字符/秒 | |-------------|----------------|----------|-----|-----------| | 25 字符 | 1.95s | 0.43s | 4.6x | 58.5 | | 88 字符 | 6.00s | 0.32s | 18.6x | 272.4 | | 117 字符 | 7.92s | 0.27s | 29.0x | 427.4 | **汇总：** - 模型加载时间：约 1.0 秒 - 平均 RTF：**17.4x**（比实时快 17 倍） - 平均字符/秒：**252.8** ### STT 基准测试 | 模型 | 加载时间 | 转录耗时（6 秒音频） | RTF | |-------|-----------|----------------------|-----| | whisper-small | 0.25s | 0.20s | 30.2x | | whisper-medium | 18.1s | 0.38s | 15.5x | | whisper-large-v3 | ~30s | ~0.6s | ~10x | | parakeet | ~0.5s | ~0.15s | ~40x | **说明：** - RTF（实时倍率）表示处理速度相对于实时的倍数 - 首次加载包含从 HuggingFace 下载模型的时间 - 后续加载使用已缓存的模型 ### 按场景推荐 | 场景 | 推荐模型 | 原因 | |----------|------------------|-----| | 英语 STT（快速） | `parakeet` | RTF 40x，内存占用低 | | 多语言 STT | `whisper-large-v3` | 支持 99+ 种语言 | | 低延迟 STT | `whisper-small` | RTF 30x，加载快 | | 通用 TTS | `kokoro` | RTF 17x，质量良好 | | 低内存 TTS | `kokoro-4bit` | 4-bit 量化 | ## 性能建议 1. **英语场景使用 Parakeet**，比实时快 40 倍 2. **使用 4-bit 模型**降低内存占用 3. **使用 SAM-Audio small**加快人声分离速度 4. **模型缓存**，引擎采用懒加载并自动缓存 5. **提前下载模型**，避免首次运行时的延迟 ## 常见问题 ### mlx-audio 未安装 ``` pip install mlx-audio>=0.2.9 ``` ### 模型下载缓慢 模型首次使用时从 HuggingFace 下载。可使用 `huggingface-cli download` 提前下载： ```bash huggingface-cli download mlx-community/whisper-large-v3-mlx huggingface-cli download mlx-community/Kokoro-82M-bf16 ``` ### 内存不足 请使用较小的模型或 4-bit 量化版本： - 用 `whisper-small-mlx` 替代 `whisper-large-v3-mlx` - 用 `Kokoro-82M-4bit` 替代 `Kokoro-82M-bf16` - 用 `sam-audio-small` 替代 `sam-audio-large` ### Kokoro 多语言问题（mlx-audio 0.2.9） 使用非英语语言（西班牙语、中文、日语等）时若出现 `ValueError: too many values to unpack`，请应用以下修复： ```python # Fix for mlx_audio/tts/models/kokoro/pipeline.py line 443 # Change: # ps, _ = self.g2p(chunk) # To: g2p_result = self.g2p(chunk) ps = g2p_result[0] if isinstance(g2p_result, tuple) else g2p_result ``` **一键修复：** ```bash python -c " import os path = os.path.join(os.path.dirname(__import__('mlx_audio').__file__), 'tts/models/kokoro/pipeline.py') with open(path, 'r') as f: content = f.read() old = ' ps, _ = self.g2p(chunk)' new = ''' # Fix: handle both tuple (en) and string (zh/ja/es) returns from g2p g2p_result = self.g2p(chunk) ps = g2p_result[0] if isinstance(g2p_result, tuple) else g2p_result''' if old in content: with open(path, 'w') as f: f.write(content.replace(old, new)) print('Fix applied!') " ``` 此问题的原因是英语 g2p 返回元组 `(phonemes, tokens)`，而其他语言仅返回字符串。 # Documentation page: `zh/guides/continuous-batching.md` # Continuous Batching Continuous batching 在同时服务多个用户时能显著提升 throughput。 ## 启用 Continuous Batching ```bash vllm-mlx serve mlx-community/Qwen3-0.6B-8bit --continuous-batching ``` ## 与 Paged Cache 配合使用 启用高效内存的前缀共享： ```bash vllm-mlx serve mlx-community/Qwen3-0.6B-8bit --continuous-batching --use-paged-cache ``` ## 工作原理 ### 简单模式（默认） - 每次处理一个请求 - 单用户场景下 throughput 最高 - 无 batching 额外开销 ### Continuous Batching 模式 - 多个请求同时处理 - 并发用户场景下 throughput 更高 - 每个请求存在少量额外开销 ### Paged Cache - KV cache 以固定大小的块存储 - 相同的系统提示词共享同一批块 - 10 个以上并发用户时节省 80% 以上内存 ## 性能测试结果 **Continuous Batching 测试结果（M4 Max，128GB）：** | 模型 | 单请求 | Batch（5 个请求） | 加速比 | |-------|----------------|---------------|---------| | Llama-3.2-1B-Instruct-4bit | 299.1 tok/s | 613.0 tok/s | **2.05x** | | Llama-3.2-3B-Instruct-4bit | 137.6 tok/s | 208.1 tok/s | **1.51x** | | Qwen3-0.6B-8bit | 328.1 tok/s | 1111.8 tok/s | **3.39x** | | Qwen3-30B-A3B-4bit | 98.1 tok/s | 233.3 tok/s | **2.38x** | | Qwen2.5-1.5B-Instruct-4bit | 196.9 tok/s | 322.2 tok/s | **1.64x** | *5 个并发请求的 batching 可将 throughput 提升 1.5 到 3 倍。* ## Streaming 性能 **Streaming 性能（M4 Max，128GB）：** | 模型 | TTFT | 生成速度 | |-------|------|------------------| | Llama-3.2-1B-Instruct-4bit | ~4.6ms | 218.9 tok/s | | Llama-3.2-3B-Instruct-4bit | ~10.7ms | 93.6 tok/s | | Qwen3-0.6B-8bit | ~3.0ms | 328.5 tok/s | | Qwen3-30B-A3B-4bit | ~10.2ms | 98.4 tok/s | | Qwen2.5-1.5B-Instruct-4bit | ~7.1ms | 140.3 tok/s | *TTFT = Time to First Token（首 token 延迟）* ## Streaming 配置 使用 `--stream-interval` 控制 token 发送频率： ```bash # 每个 token 立即发送（最流畅） vllm-mlx serve model --continuous-batching --stream-interval 1 # 批量发送 token（适合高延迟场景） vllm-mlx serve model --continuous-batching --stream-interval 5 ``` | 值 | 行为 | |-------|----------| | `1` | 每个 token 立即发送 | | `2-5` | 攒批后再发送 | | `10+` | 最大化 throughput，输出颗粒度更大 | ## 内存管理 对于大型模型，prefix cache 可能占用大量内存。内存感知缓存会自动进行管理： ```bash # 自动检测（使用可用内存的 20%） vllm-mlx serve model --continuous-batching # 显式限制 vllm-mlx serve model --continuous-batching --cache-memory-mb 2048 # 自定义百分比 vllm-mlx serve model --continuous-batching --cache-memory-percent 0.10 ``` | 选项 | 说明 | |--------|-------------| | `--cache-memory-mb` | 以 MB 为单位设置显式上限 | | `--cache-memory-percent` | 可用内存的占比（默认值：0.20） | | `--no-memory-aware-cache` | 使用基于条目数量的旧式缓存 | ## Prefix Cache Prefix caching 对重复提示词复用 KV cache。 ### 工作原理 ``` User 1: System prompt (500 tokens) → Creates 8 blocks User 2: Same system prompt → Shares 8 blocks (ref_count++) User N: Same system prompt → Shares 8 blocks (ref_count++) Memory savings: 80%+ for 10+ concurrent users ``` ### 缓存键策略 - **LLM**：`hash(prompt)` - **图片**：`hash(image_content) + hash(prompt)` - **视频**：`hash(video_path) + hash(fps) + hash(max_frames) + hash(prompt)` ### 测试 Prefix Cache ```bash python tests/test_prefix_cache.py ``` ``` ====================================================================== LLM PREFIX CACHE TEST ====================================================================== Model: mlx-community/Qwen3-0.6B-8bit Expected behavior: - Same prompt → cache HIT - Different prompt → cache MISS or PREFIX_HIT (shared template tokens) ---------------------------------------------------------------------- Results: Step | Description | Expected | Actual | Status -------+---------------------+----------+--------+------- 1a | First request | MISS | MISS | PASS 1b | Same prompt | HIT | HIT | PASS 1c | Different prompt | MISS | MISS | PASS 1d | Return to prompt 1 | HIT | HIT | PASS ====================================================================== ``` ## 运行基准测试 ```bash # Continuous batching 基准测试 python tests/test_continuous_batching.py # Prefix cache 测试 python tests/test_prefix_cache.py ``` ## 适用场景 | 场景 | 模式 | |----------|------| | 单用户，追求最高速度 | 简单模式（默认） | | 多用户并发 | `--continuous-batching` | | 大型模型（7B+） | `--continuous-batching --cache-memory-mb 2048` | | 生产环境，提示词共享 | `--continuous-batching --use-paged-cache` | ## 生产环境配置 ```bash vllm-mlx serve mlx-community/Qwen3-0.6B-8bit \ --continuous-batching \ --use-paged-cache \ --port 8000 ``` # Documentation page: `zh/guides/embeddings.md` # Embeddings vllm-mlx 通过 [mlx-embeddings](https://github.com/Blaizzy/mlx-embeddings) 支持文本 embeddings，提供与 OpenAI 兼容的 `/v1/embeddings` 接口。 ## 安装 ```bash pip install mlx-embeddings>=0.0.5 ``` ## 快速入门 ### 启动带有 embedding 模型的服务器 ```bash # 启动时预加载指定的 embedding 模型 vllm-mlx serve my-llm-model --embedding-model mlx-community/all-MiniLM-L6-v2-4bit ``` 如果不使用 `--embedding-model`，embedding 模型会在第一次请求时按需加载，但仅限于内置的请求时许可列表中的模型。 ### 使用 OpenAI SDK 生成 embeddings ```python from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed") # 单条文本 response = client.embeddings.create( model="mlx-community/all-MiniLM-L6-v2-4bit", input="Hello world" ) print(response.data[0].embedding[:5]) # First 5 dimensions # 批量文本 response = client.embeddings.create( model="mlx-community/all-MiniLM-L6-v2-4bit", input=[ "I love machine learning", "Deep learning is fascinating", "Natural language processing rocks" ] ) for item in response.data: print(f"Text {item.index}: {len(item.embedding)} dimensions") ``` ### 使用 curl ```bash curl http://localhost:8000/v1/embeddings \ -H "Content-Type: application/json" \ -d '{ "model": "mlx-community/all-MiniLM-L6-v2-4bit", "input": ["Hello world", "How are you?"] }' ``` ## 支持的模型 请求时支持的模型： | 模型 | 适用场景 | 规模 | |-------|----------|------| | `mlx-community/all-MiniLM-L6-v2-4bit` | 快速、轻量 | 小 | | `mlx-community/embeddinggemma-300m-6bit` | 高质量 | 300M | | `mlx-community/bge-large-en-v1.5-4bit` | 英文效果最佳 | 大 | | `mlx-community/multilingual-e5-small-mlx` | 多语言检索 | 小 | | `mlx-community/multilingual-e5-large-mlx` | 多语言检索 | 大 | | `mlx-community/bert-base-uncased-mlx` | 通用 BERT 基准 | 基础 | | `mlx-community/ModernBERT-base-mlx` | ModernBERT 基准 | 基础 | 其他 embedding 模型需要在启动服务器时通过 `--embedding-model` 指定。 ## 模型管理 ### 按需加载 默认情况下，embedding 模型在第一次收到 `/v1/embeddings` 请求时加载。你可以在上述请求时支持的模型之间切换，切换后旧模型会自动卸载。 ### 启动时预加载 使用 `--embedding-model` 可在启动时加载模型。设置该参数后，只有该指定模型可用于 embeddings： ```bash vllm-mlx serve my-llm-model --embedding-model mlx-community/all-MiniLM-L6-v2-4bit ``` 请求其他模型将返回 400 错误。 ## API 参考 ### POST /v1/embeddings 为给定的输入文本生成 embeddings。 **请求体：** | 字段 | 类型 | 是否必填 | 描述 | |-------|------|----------|-------------| | `model` | string | 是 | 支持的 embedding 模型 ID，或使用 `--embedding-model` 时启动时固定的模型 | | `input` | string 或 list[string] | 是 | 待嵌入的文本 | **响应：** ```json { "object": "list", "data": [ {"object": "embedding", "index": 0, "embedding": [0.023, -0.982, ...]}, {"object": "embedding", "index": 1, "embedding": [0.112, -0.543, ...]} ], "model": "mlx-community/all-MiniLM-L6-v2-4bit", "usage": {"prompt_tokens": 12, "total_tokens": 12} } ``` ## Python API ### 不启动服务器直接使用 ```python from vllm_mlx.embedding import EmbeddingEngine engine = EmbeddingEngine("mlx-community/all-MiniLM-L6-v2-4bit") engine.load() vectors = engine.embed(["Hello world", "How are you?"]) print(f"Dimensions: {len(vectors[0])}") tokens = engine.count_tokens(["Hello world"]) print(f"Token count: {tokens}") ``` ## 常见问题 ### mlx-embeddings 未安装 ``` pip install mlx-embeddings>=0.0.5 ``` ### 找不到模型 请确认模型名称与上方请求时支持的 ID 之一匹配，或在启动服务器时通过 `--embedding-model` 指定自定义模型。你也可以提前下载支持的模型： ```bash huggingface-cli download mlx-community/all-MiniLM-L6-v2-4bit ``` # Documentation page: `zh/guides/mcp-tools.md` # MCP 与 tool calling vllm-mlx 支持 Model Context Protocol (MCP)，用于将外部工具与 LLM 集成。 ## tool calling 工作原理 ``` ┌─────────────────────────────────────────────────────────────────────┐ │ Tool Calling Flow │ ├─────────────────────────────────────────────────────────────────────┤ │ │ │ 1. User Request │ │ ─────────────────► "List files in /tmp" │ │ │ │ 2. LLM Generates Tool Call │ │ ─────────────────► tool_calls: [{ │ │ name: "list_directory", │ │ arguments: {path: "/tmp"} │ │ }] │ │ │ │ 3. App Executes Tool via MCP │ │ ─────────────────► MCP Server executes list_directory │ │ Returns: ["file1.txt", "file2.txt"] │ │ │ │ 4. Tool Result Sent Back to LLM │ │ ─────────────────► role: "tool", content: [...] │ │ │ │ 5. LLM Generates Final Response │ │ ─────────────────► "The /tmp directory contains 2 files..." │ │ │ └─────────────────────────────────────────────────────────────────────┘ ``` ## 快速开始 ### 1. 创建 MCP 配置 创建 `mcp.json`： ```json { "mcpServers": { "filesystem": { "command": "npx", "args": ["-y", "@modelcontextprotocol/server-filesystem", "/tmp"] } } } ``` ### 2. 启动带 MCP 的服务器 ```bash # 简单模式 vllm-mlx serve mlx-community/Qwen3-4B-4bit --mcp-config mcp.json # 连续批处理 vllm-mlx serve mlx-community/Qwen3-4B-4bit --mcp-config mcp.json --continuous-batching ``` ### 3. 验证 MCP 状态 ```bash # 查看 MCP 状态 curl http://localhost:8000/v1/mcp/status # 列出可用工具 curl http://localhost:8000/v1/mcp/tools ``` ## tool calling 示例 ```python import json import httpx BASE_URL = "http://localhost:8000" # 1. Get available tools tools_response = httpx.get(f"{BASE_URL}/v1/mcp/tools") tools = tools_response.json()["tools"] # 2. Send request with tools response = httpx.post( f"{BASE_URL}/v1/chat/completions", json={ "model": "default", "messages": [{"role": "user", "content": "List files in /tmp"}], "tools": tools, "max_tokens": 1024 } ) result = response.json() message = result["choices"][0]["message"] # 3. Check for tool calls if message.get("tool_calls"): tool_call = message["tool_calls"][0] # 4. Execute tool via MCP exec_response = httpx.post( f"{BASE_URL}/v1/mcp/execute", json={ "server": "filesystem", "tool": tool_call["function"]["name"], "arguments": json.loads(tool_call["function"]["arguments"]) } ) tool_result = exec_response.json() # 5. Send result back to LLM messages = [ {"role": "user", "content": "List files in /tmp"}, message, { "role": "tool", "tool_call_id": tool_call["id"], "content": json.dumps(tool_result["result"]) } ] final_response = httpx.post( f"{BASE_URL}/v1/chat/completions", json={"model": "default", "messages": messages} ) print(final_response.json()["choices"][0]["message"]["content"]) ``` ## MCP 接口端点 | 端点 | 方法 | 说明 | |----------|--------|-------------| | `/v1/mcp/status` | GET | 查看 MCP 状态 | | `/v1/mcp/tools` | GET | 列出可用工具 | | `/v1/mcp/execute` | POST | 执行工具 | ## MCP 服务器示例 ### 文件系统 ```json { "mcpServers": { "filesystem": { "command": "npx", "args": ["-y", "@modelcontextprotocol/server-filesystem", "/tmp"] } } } ``` ### GitHub ```json { "mcpServers": { "github": { "command": "npx", "args": ["-y", "@modelcontextprotocol/server-github"], "env": { "GITHUB_TOKEN": "your-token" } } } } ``` ### PostgreSQL ```json { "mcpServers": { "postgres": { "command": "npx", "args": ["-y", "@modelcontextprotocol/server-postgres"], "env": { "DATABASE_URL": "postgresql://user:pass@localhost/db" } } } } ``` ### Brave Search ```json { "mcpServers": { "brave-search": { "command": "npx", "args": ["-y", "@modelcontextprotocol/server-brave-search"], "env": { "BRAVE_API_KEY": "your-key" } } } } ``` ## 使用多个 MCP 服务器 ```json { "mcpServers": { "filesystem": { "command": "npx", "args": ["-y", "@modelcontextprotocol/server-filesystem", "/tmp"] }, "github": { "command": "npx", "args": ["-y", "@modelcontextprotocol/server-github"], "env": { "GITHUB_TOKEN": "your-token" } } } } ``` ## 交互式 MCP 聊天 如需交互式测试 MCP： ```bash python examples/mcp_chat.py ``` ## 支持的工具格式 vllm-mlx 支持 12 种 tool call parser，覆盖所有主流模型系列。完整的 parser 列表、别名及示例请参见 [Tool Calling](tool-calling.md)。 ## 安全性 vllm-mlx 内置安全措施，防止通过 MCP 服务器进行命令注入攻击。 ### 命令白名单 默认情况下，仅允许可信命令： | 类别 | 允许的命令 | |----------|-----------------| | Node.js | `npx`、`npm`、`node` | | Python | `uvx`、`uv`、`python`、`python3`、`pip`、`pipx` | | Docker | `docker` | | MCP 服务器 | `mcp-server-*`（官方服务器） | ### 屏蔽模式 以下模式会被屏蔽以防止注入攻击： - 命令链接：`;`、`&&`、`||`、`|` - 命令替换：`` ` ``、`$()` - 路径穿越：`../` - 危险环境变量：`LD_PRELOAD`、`PATH`、`PYTHONPATH` ### 示例：被屏蔽的攻击 ```json { "mcpServers": { "malicious": { "command": "bash", "args": ["-c", "rm -rf /"] } } } ``` 此配置将被拒绝： ``` ValueError: MCP server 'malicious': Command 'bash' is not in the allowed commands whitelist. ``` ### 开发模式（不安全） 仅限开发环境，可绕过安全校验： ```json { "mcpServers": { "custom": { "command": "my-custom-server", "skip_security_validation": true } } } ``` **警告**：切勿在生产环境中使用 `skip_security_validation`。 ### 自定义白名单 如需通过编程方式向白名单添加自定义命令： ```python from vllm_mlx.mcp import MCPCommandValidator, set_validator # Add custom commands validator = MCPCommandValidator( custom_whitelist={"my-trusted-server", "another-server"} ) set_validator(validator) ``` ## 工具执行沙箱 除命令校验外，vllm-mlx 还为工具执行提供运行时沙箱。 ### 沙箱功能 | 功能 | 说明 | |---------|-------------| | 工具白名单 | 仅允许特定工具执行 | | 工具黑名单 | 屏蔽特定危险工具 | | 参数校验 | 屏蔽工具参数中的危险模式 | | 频率限制 | 限制每分钟的工具调用次数 | | 审计日志 | 记录所有工具执行情况 | ### 屏蔽的参数模式 工具参数会针对以下危险模式进行校验： - 路径穿越：`../` - 系统目录：`/etc/`、`/proc/`、`/sys/` - root 访问：`/root/`、`~root` ### 高风险工具检测 匹配以下模式的工具会触发安全警告： - `execute`、`run_command`、`shell`、`eval`、`exec`、`system`、`subprocess` ### 自定义沙箱配置 ```python from vllm_mlx.mcp import ToolSandbox, set_sandbox # Create sandbox with custom settings sandbox = ToolSandbox( # Only allow specific tools (whitelist mode) allowed_tools={"read_file", "list_directory"}, # Block specific tools (blacklist mode) blocked_tools={"execute_command", "run_shell"}, # Rate limit: max 30 calls per minute max_calls_per_minute=30, # Optional audit callback audit_callback=lambda audit: print(f"Tool: {audit.tool_name}, Success: {audit.success}"), ) set_sandbox(sandbox) ``` ### 访问审计日志 ```python from vllm_mlx.mcp import get_sandbox sandbox = get_sandbox() # Get recent audit entries entries = sandbox.get_audit_log(limit=50) # Filter by tool name file_ops = sandbox.get_audit_log(tool_filter="file") # Get only errors errors = sandbox.get_audit_log(errors_only=True) # Clear audit log sandbox.clear_audit_log() ``` ### 敏感数据脱敏 审计日志会自动对敏感字段（password、token、secret、key、credential、auth）进行脱敏处理，并对过大的值进行截断。 ## 故障排查 ### MCP 服务器无法连接 检查 MCP 服务器命令是否正确： ```bash npx -y @modelcontextprotocol/server-filesystem /tmp ``` ### 工具无法执行 验证工具是否可用： ```bash curl http://localhost:8000/v1/mcp/tools | jq '.tools[].name' ``` ### tool call 未被解析 请确保所用模型支持函数调用（如 Qwen3、Llama-3.2-Instruct）。 ### 命令不在白名单中 如果看到 "Command X is not in the allowed commands whitelist"，可采取以下措施之一： 1. 使用允许的命令（参见上方白名单） 2. 将该命令添加到自定义白名单 3. 使用 `skip_security_validation: true`（仅限开发环境） # Documentation page: `zh/guides/moe-top-k.md` # MoE top_k 覆盖参数（`--moe-top-k`） 减少 Mixture of Experts 模型（如 Qwen3-30B-A3B）每个 token 激活的 expert 数量，以少量质量损失换取明显更高的解码吞吐量。 > **状态：** 可选参数，默认行为不变。以下质量数据基于 Qwen3-30B-A3B-4bit 在 M4 Max 128 GB 上的测试结果，在将其用于生产环境前请在你的模型上自行验证。 ## 功能说明 Qwen3-30B-A3B 使用 `top_k=8` 训练，即每个 token 从 128 个 expert 中选取 8 个。在 Apple Silicon 上进行 batch=1 解码时，expert 矩阵乘法（`SwitchGLU`）是每层计算中占比最大的部分，其开销与 `top_k` 大致呈线性关系。在推理阶段降低 `top_k` 已被证明（LExI 2025，Lynx 2024）能在保留大部分训练质量的同时，有效缩短解码时间。 `--moe-top-k N` 会遍历已加载模型的每一层，对含有 `.mlp.switch_mlp`（即稀疏 MoE 块）的层将 `top_k` 设置为 N。密集层和密集模型不受影响，该参数对它们是空操作。 ## 用法 ```bash # Server vllm-mlx serve mlx-community/Qwen3-30B-A3B-4bit \ --continuous-batching \ --moe-top-k 4 # Bench vllm-mlx bench mlx-community/Qwen3-30B-A3B-4bit --moe-top-k 4 ``` 若 N 大于模型训练时的 `top_k`，该参数会被拒绝，因为只有降低才有意义，不支持提高。 ## 实测影响 ### 解码吞吐量（M4 Max 128 GB，batch=1，贪心解码） | top_k | tok/s | 对比基线 | |---:|---:|---:| | 8（基线） | 126.5 | - | | 6 | 136.1 | +7.6% | | 5 | 140.3 | +10.9% | | 4 | 147.3 | +16.5% | ### 质量评估（Qwen3-30B-A3B-4bit，lm-evaluation-harness，MLX backend） | top_k | MMLU (acc) | GSM8K (exact match) | Δ vs baseline | |---:|---:|---:|---:| | 8 | TBD | TBD | - | | 6 | TBD | TBD | TBD | | 5 | TBD | TBD | TBD | | 4 | TBD | TBD | TBD | MMLU：随机抽取 200 个样本，0-shot。 GSM8K：随机抽取 100 个样本，0-shot，严格 exact-match。 以上数据具有**方向性参考价值**，完整评测集规模更大，会改变绝对精度数值，但各配置间的相对差距不会有太大变化。 ### 贪心输出一致性 在 4-bit 检查点上使用 `top_k=4` 时，我们测试的所有探针提示中，生成的**前 16 个 token 与基线完全一致**。这表明 top_k=4 不会改变早期解码步骤中的 argmax，模型对减少一半激活 expert 具有内在的鲁棒性。 当 `top_k=3` 或更低时，质量会出现可见的下降（此处未测量，基于 LExI 论文推断），因此该参数在配置校验层刻意不允许低于 1，但生产环境推荐的最低值为 `top_k=4`。 ## 适用场景与不适用场景 适合使用的情况： - 运行 Qwen3 MoE（或兼容模型：Qwen3.5 MoE、Gemma-MoE），且单用户解码吞吐量是瓶颈。 - 工作负载允许少量质量损失，以换取明显的延迟改善。 - 部署在受内存带宽限制的硬件上（M 系列 Apple Silicon），expert gather 主导每步解码时间。 不适合使用的情况： - 运行密集模型，该参数是空操作，没有任何效果。 - 对评测集排行榜精度有顶尖要求。 - 运行长链式推理或"思考模式"生成，质量下降幅度可能比 0-shot MMLU 所示更陡。 ## 与其他优化叠加使用 该参数可与量化叠加使用。在 Qwen3-30B-A3B-4bit 上的实测叠加结果如下： - 4-bit + top_k=8：126.5 tok/s（基线） - 4-bit + top_k=4：147.3 tok/s（+16.5%） - 3-bit + top_k=8：138.6 tok/s（+9.6%） - 3-bit + top_k=6：147.1 tok/s（+16.3%），质量差异可测量 - 3-bit + top_k=4：157.3 tok/s（+24%），**输出质量严重下降**（在冒烟测试中模型回答了不同的问题） 3-bit + top_k=4 的数值误差累积超出了 argmax 稳定的临界点。最多只应使用一个激进参数：4-bit + top_k=4 或 3-bit + top_k=6。两者的 tok/s 大致相同（约 147），但质量表现差异显著。 ## 内部实现 - 补丁辅助函数：`vllm_mlx.scheduler.apply_moe_top_k_override(model, k)` - 在 `Scheduler.__init__` 中于模型加载完成后执行。 - 测试：`tests/test_moe_top_k.py`，覆盖密集模型、混合架构及校验路径。 ## 参考资料 - LExI: Layer-Adaptive Active Experts, [arXiv 2509.02753](https://arxiv.org/html/2509.02753) - Not All Experts are Equal (NAEE), [ACL 2024](https://aclanthology.org/2024.acl-long.334.pdf) - SwiftLM (`SWIFTLM_TOP_K` env knob prior art), [github.com/SharpAI/SwiftLM](https://github.com/SharpAI/SwiftLM) # Documentation page: `zh/guides/multimodal.md` # 多模态模型（图像与视频） vllm-mlx 支持用于图像和视频理解的视觉语言模型。 ## 支持的模型 - Qwen3-VL（推荐） - Qwen2-VL - Gemma 3 - LLaVA - Idefics - PaliGemma - Pixtral - Molmo - DeepSeek-VL ## 启动多模态服务器 ```bash vllm-mlx serve mlx-community/Qwen3-VL-4B-Instruct-3bit --port 8000 ``` 名称中包含 "VL"、"Vision" 或 "mllm" 的模型会被自动识别为多模态模型。 ## 图像分析 ### 通过 OpenAI SDK ```python from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed") # Image from URL response = client.chat.completions.create( model="default", messages=[{ "role": "user", "content": [ {"type": "text", "text": "What's in this image?"}, {"type": "image_url", "image_url": {"url": "https://example.com/image.jpg"}} ] }], max_tokens=256 ) print(response.choices[0].message.content) ``` ### Base64 图像 ```python import base64 def encode_image(path): with open(path, "rb") as f: return base64.b64encode(f.read()).decode("utf-8") base64_image = encode_image("photo.jpg") response = client.chat.completions.create( model="default", messages=[{ "role": "user", "content": [ {"type": "text", "text": "Describe this image"}, {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{base64_image}"}} ] }] ) ``` ### 通过 curl ```bash curl http://localhost:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "default", "messages": [{ "role": "user", "content": [ {"type": "text", "text": "What is in this image?"}, {"type": "image_url", "image_url": {"url": "https://example.com/image.jpg"}} ] }], "max_tokens": 256 }' ``` ## 视频分析 ### 通过 OpenAI SDK ```python response = client.chat.completions.create( model="default", messages=[{ "role": "user", "content": [ {"type": "text", "text": "What happens in this video?"}, {"type": "video_url", "video_url": {"url": "https://example.com/video.mp4"}} ] }], max_tokens=512 ) ``` ### 视频参数 通过额外的请求体参数控制帧提取： ```python response = client.chat.completions.create( model="default", messages=[{ "role": "user", "content": [ {"type": "text", "text": "Describe this video"}, {"type": "video_url", "video_url": {"url": "video.mp4"}} ] }], extra_body={ "video_fps": 2.0, "video_max_frames": 32 } ) ``` ### 通过 curl ```bash curl http://localhost:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "default", "messages": [{ "role": "user", "content": [ {"type": "text", "text": "Describe this video"}, {"type": "video_url", "video_url": {"url": "https://example.com/video.mp4"}} ] }], "video_fps": 2.0, "video_max_frames": 16 }' ``` ## 支持的格式 ### 图像 | 格式 | 示例 | |------|------| | URL | `{"type": "image_url", "image_url": {"url": "https://..."}}` | | 本地文件 | `{"type": "image_url", "image_url": {"url": "/path/to/image.jpg"}}` | | Base64 | `{"type": "image_url", "image_url": {"url": "data:image/jpeg;base64,..."}}` | ### 视频 | 格式 | 示例 | |------|------| | URL | `{"type": "video_url", "video_url": {"url": "https://..."}}` | | 本地文件 | `{"type": "video", "video": "/path/to/video.mp4"}` | | Base64 | `{"type": "video_url", "video_url": {"url": "data:video/mp4;base64,..."}}` | ## Python API ```python from vllm_mlx.models import MLXMultimodalLM mllm = MLXMultimodalLM("mlx-community/Qwen3-VL-4B-Instruct-3bit") mllm.load() # Image description = mllm.describe_image("photo.jpg") # Video description = mllm.describe_video("video.mp4", fps=2.0) # Custom prompt output = mllm.generate( prompt="Compare these images", images=["img1.jpg", "img2.jpg"] ) ``` ## 性能建议 ### 图像 - 分辨率越小，处理速度越快（224x224 对比 1920x1080） - 根据任务选择合适的分辨率 ### 视频 - 帧率越低，处理速度越快 - 帧数越少，内存占用越低 - 64 帧是实际可用的最大值（96 帧及以上会导致 GPU 超时） ## 基准测试 在配备 128 GB 统一内存的 Apple M4 Max 上测试。 ### Qwen3-VL-4B-Instruct-3bit | 分辨率 | 耗时 | token 数 | 速度 | 内存 | |--------|------|----------|------|------| | 224x224 | 0.87s | 124 | 143 tok/s | 2.6 GB | | 448x448 | 1.01s | 107 | 106 tok/s | 3.1 GB | | 768x768 | 1.42s | 127 | 89 tok/s | 3.4 GB | | 1024x1024 | 1.85s | 116 | 63 tok/s | 3.6 GB | ### Qwen3-VL-8B-Instruct-4bit | 分辨率 | 耗时 | token 数 | 速度 | 内存 | |--------|------|----------|------|------| | 224x224 | 1.08s | 78 | 73 tok/s | 5.6 GB | | 448x448 | 1.41s | 70 | 50 tok/s | 6.1 GB | | 768x768 | 2.06s | 91 | 44 tok/s | 6.5 GB | | 1024x1024 | 3.02s | 76 | 25 tok/s | 7.6 GB | ### Gemma 3 4B 4bit | 分辨率 | 耗时 | token 数 | 速度 | 内存 | |--------|------|----------|------|------| | 224x224 | 0.95s | 30 | 32 tok/s | 5.2 GB | | 448x448 | 0.99s | 34 | 34 tok/s | 5.2 GB | | 768x768 | 0.99s | 32 | 32 tok/s | 5.2 GB | | 1024x1024 | 0.95s | 28 | 29 tok/s | 5.2 GB | ### 运行基准测试 ```bash # Quick benchmark vllm-mlx-bench --model mlx-community/Qwen3-VL-4B-Instruct-3bit --quick # Full benchmark with more resolutions vllm-mlx-bench --model mlx-community/Qwen3-VL-4B-Instruct-3bit # Video benchmark vllm-mlx-bench --model mlx-community/Qwen3-VL-4B-Instruct-3bit --video ``` ## MLLM Cache vllm-mlx 为多模态模型内置了 prefix cache 系统，可以显著加速对相同图像的重复请求。 ### 工作原理 向模型发送图像时，视觉编码器会将其处理为嵌入向量，该过程通常需要 1 到 2 秒。MLLM Cache 会同时存储这些嵌入向量和 KV cache 状态，因此后续使用相同图像的请求可以完全跳过视觉编码器。 该缓存采用基于内容的哈希（类似 LMCache）来识别相同图像，无论图像以何种方式提供（URL、base64 还是文件路径）。 ### 启用缓存 ```bash # Enable with default settings (512 MB max) vllm-mlx serve mlx-community/Qwen3-VL-4B-Instruct-3bit --enable-mllm-cache # With custom memory limit vllm-mlx serve mlx-community/Qwen3-VL-4B-Instruct-3bit \ --enable-mllm-cache \ --mllm-cache-max-mb 1024 ``` ### Python API ```python from vllm_mlx.mllm_cache import MLLMPrefixCacheManager # Create cache manager cache = MLLMPrefixCacheManager(max_memory_mb=512) # Store embeddings and KV cache after processing cache.store( images=["photo.jpg"], prompt="Describe this image", vision_embeddings=embeddings, kv_cache=kv_state, num_tokens=128 ) # Fetch from cache on subsequent requests entry, match_len = cache.fetch(images=["photo.jpg"], prompt="Describe this image") if entry: # Use cached embeddings, skip vision encoder embeddings = entry.vision_embeddings kv_state = entry.kv_cache ``` ### 缓存统计 ```python stats = cache.get_stats() print(f"Hit rate: {stats.hit_rate:.1%}") print(f"Memory used: {stats.memory_used_mb:.1f} MB") print(f"Tokens saved: {stats.tokens_saved}") ``` ### 内存管理 当达到内存上限时，缓存采用 LRU（最近最少使用）策略进行淘汰。每个缓存条目记录以下信息： - 视觉嵌入向量大小 - 每层 KV cache 大小 - 用于 LRU 排序的访问频率 当内存压力出现时，最近最少访问的条目会被优先淘汰。 ## Gradio 聊天界面 如需交互式多模态对话： ```bash vllm-mlx-chat --served-model-name mlx-community/Qwen3-VL-4B-Instruct-3bit ``` 支持拖放图像和视频。 # Documentation page: `zh/guides/python-api.md` # Python API 通过 Python API 直接以编程方式访问 vllm-mlx。 ## Language Models ### 基本用法 ```python from vllm_mlx.models import MLXLanguageModel # Load model model = MLXLanguageModel("mlx-community/Llama-3.2-3B-Instruct-4bit") model.load() # Generate text output = model.generate("What is the capital of France?", max_tokens=100) print(output.text) ``` ### Streaming Generation ```python for chunk in model.stream_generate("Tell me a story about a robot"): print(chunk.text, end="", flush=True) ``` ### Chat 接口 ```python messages = [ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Hello, who are you?"} ] response = model.chat(messages) print(response.text) ``` ### 生成参数 ```python output = model.generate( prompt="Write a poem", max_tokens=256, temperature=0.7, top_p=0.9, stop=["END", "\n\n"] ) ``` | 参数 | 描述 | 默认值 | |-----------|-------------|---------| | `max_tokens` | 最大生成 token 数量 | 256 | | `temperature` | 采样温度（0-2） | 0.7 | | `top_p` | Nucleus sampling | 0.9 | | `stop` | 停止序列 | None | ## Vision-Language Models ### 基本用法 ```python from vllm_mlx.models import MLXMultimodalLM # Load model mllm = MLXMultimodalLM("mlx-community/Qwen3-VL-4B-Instruct-3bit") mllm.load() # Describe an image description = mllm.describe_image("photo.jpg") print(description) ``` ### 问答 ```python answer = mllm.answer_about_image("photo.jpg", "What color is the car?") print(answer) ``` ### 多图片输入 ```python output = mllm.generate( prompt="Compare these two images", images=["image1.jpg", "image2.jpg"] ) print(output.text) ``` ### 视频理解 ```python # From local file output = mllm.generate( prompt="What is happening in this video?", videos=["video.mp4"], video_fps=2.0, video_max_frames=16 ) print(output.text) # From URL output = mllm.generate( prompt="Describe this video", videos=["https://example.com/video.mp4"], video_fps=2.0 ) # Convenience method description = mllm.describe_video("video.mp4", fps=2.0) ``` ### 视频参数 | 参数 | 描述 | 默认值 | |-----------|-------------|---------| | `video_fps` | 每秒提取帧数 | 2.0 | | `video_max_frames` | 最大处理帧数 | 32 | ## Engine API 针对高级使用场景，可直接使用 engine： ### Simple Engine ```python from vllm_mlx.engine import SimpleEngine engine = SimpleEngine("mlx-community/Llama-3.2-3B-Instruct-4bit") await engine.start() output = await engine.generate( prompt="Hello world", max_tokens=100 ) print(output.text) await engine.stop() ``` ### Batched Engine ```python from vllm_mlx.engine import BatchedEngine engine = BatchedEngine("mlx-community/Llama-3.2-3B-Instruct-4bit") await engine.start() # Multiple concurrent requests output = await engine.generate( prompt="Hello world", max_tokens=100 ) await engine.stop() ``` ## 输出格式 所有生成方法均返回 `GenerationOutput`： ```python output = model.generate("Hello") print(output.text) # Generated text print(output.prompt_tokens) # Input token count print(output.completion_tokens) # Output token count print(output.finish_reason) # "stop" or "length" ``` ## 错误处理 ```python from vllm_mlx.models import MLXLanguageModel try: model = MLXLanguageModel("invalid-model") model.load() except Exception as e: print(f"Failed to load model: {e}") ``` # Documentation page: `zh/guides/reasoning.md` # Reasoning 模型 vllm-mlx 支持在给出答案之前展示 thinking 过程的 reasoning 模型。Qwen3 和 DeepSeek-R1 等模型会将 reasoning 内容包裹在 `...` 标签中，vllm-mlx 可以解析这些标签，将 reasoning 与最终回答分离。 ## 为什么使用 Reasoning 解析？ reasoning 模型生成的原始输出通常如下所示： ``` Let me analyze this step by step. First, I need to consider the constraints. The answer should be a prime number less than 10. Checking: 2, 3, 5, 7 are all prime and less than 10. The prime numbers less than 10 are: 2, 3, 5, 7. ``` 不启用 reasoning 解析时，响应中会包含原始标签。启用 reasoning parsing 后，thinking 过程与最终回答会被分离到 API 响应的不同字段中。 ## 快速开始 ### 启动服务器并指定 Reasoning Parser ```bash # For Qwen3 models vllm-mlx serve mlx-community/Qwen3-8B-4bit --reasoning-parser qwen3 # For DeepSeek-R1 models vllm-mlx serve mlx-community/DeepSeek-R1-Distill-Qwen-7B-4bit --reasoning-parser deepseek_r1 ``` ### API 响应格式 启用 reasoning parsing 后，API 响应中会包含 `reasoning` 字段。 **非 streaming 响应：** ```json { "choices": [{ "message": { "role": "assistant", "content": "The prime numbers less than 10 are: 2, 3, 5, 7.", "reasoning": "Let me analyze this step by step.\nFirst, I need to consider the constraints.\nThe answer should be a prime number less than 10.\nChecking: 2, 3, 5, 7 are all prime and less than 10." } }] } ``` **Streaming 响应：** reasoning 和正文内容分块独立发送。在 reasoning 阶段，数据块的 `reasoning` 字段有内容；当模型进入最终回答阶段后，数据块的 `content` 字段有内容： ```json {"delta": {"reasoning": "Let me analyze"}} {"delta": {"reasoning": " this step by step."}} {"delta": {"reasoning": "\nFirst, I need to"}} ... {"delta": {"content": "The prime"}} {"delta": {"content": " numbers less than 10"}} {"delta": {"content": " are: 2, 3, 5, 7."}} ``` ## 与 OpenAI SDK 配合使用 ```python from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed") # Non-streaming response = client.chat.completions.create( model="default", messages=[{"role": "user", "content": "What are the prime numbers less than 10?"}] ) message = response.choices[0].message print("Reasoning:", message.reasoning) # The thinking process print("Answer:", message.content) # The final answer ``` ### Streaming 与 Reasoning ```python reasoning_text = "" content_text = "" stream = client.chat.completions.create( model="default", messages=[{"role": "user", "content": "Solve: 2 + 2 = ?"}], stream=True ) for chunk in stream: delta = chunk.choices[0].delta if hasattr(delta, 'reasoning') and delta.reasoning: reasoning_text += delta.reasoning print(f"[Thinking] {delta.reasoning}", end="") if delta.content: content_text += delta.content print(delta.content, end="") print(f"\n\nFinal reasoning: {reasoning_text}") print(f"Final answer: {content_text}") ``` ## 支持的 Parser ### Qwen3 Parser（`qwen3`） 适用于使用显式 `` 和 `` 标签的 Qwen3 模型。 - 需要开标签和闭标签**同时存在** - 如果标签缺失，输出将被视为普通内容 - 适合：Qwen3-0.6B、Qwen3-4B、Qwen3-8B 及同系列模型 ```bash vllm-mlx serve mlx-community/Qwen3-8B-4bit --reasoning-parser qwen3 ``` ### DeepSeek-R1 Parser（`deepseek_r1`） 适用于可能省略开标签 `` 的 DeepSeek-R1 模型。 - 比 Qwen3 parser 更宽松 - 能处理 `` 为隐式的情况 - 即使没有 ``，`` 之前的内容也会被视为 reasoning ```bash vllm-mlx serve mlx-community/DeepSeek-R1-Distill-Qwen-7B-4bit --reasoning-parser deepseek_r1 ``` ## 工作原理 reasoning parser 通过基于文本的检测来识别模型输出中的 thinking 标签。在 streaming 过程中，它会追踪当前在输出中的位置，将每个 token 正确路由到 `reasoning` 或 `content` 字段。 ``` Model Output: Step 1: analyze...The answer is 42. ├─────────────────────┤├─────────────────────┤ Parsed: │ reasoning ││ content │ └─────────────────────┘└─────────────────────┘ ``` 解析过程是无状态的，通过累积文本来判断上下文，在 token 以任意分块到达的 streaming 场景下也能稳定工作。 ## 最佳使用建议 ### 提示词写法 引导模型逐步思考，reasoning 模型的效果更好： ```python messages = [ {"role": "system", "content": "Think through problems step by step before answering."}, {"role": "user", "content": "What is 17 × 23?"} ] ``` ### 处理缺失的 Reasoning 某些提示词可能不会触发 reasoning。此时 `reasoning` 值为 `None`，所有输出都进入 `content`： ```python message = response.choices[0].message if message.reasoning: print(f"Model's thought process: {message.reasoning}") print(f"Answer: {message.content}") ``` ### 温度参数与 Reasoning 较低的温度通常会产生更稳定的 reasoning 模式： ```python response = client.chat.completions.create( model="default", messages=[{"role": "user", "content": "Explain quantum entanglement"}], temperature=0.3 # More focused reasoning ) ``` ## 向后兼容性 未指定 `--reasoning-parser` 时，服务器行为与之前一致：thinking 标签包含在 `content` 字段中，响应中不会添加 `reasoning` 字段。这确保现有应用无需修改即可继续正常使用。 ## 示例：数学题求解器 ```python from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed") def solve_math(problem: str) -> dict: """Solve a math problem and return reasoning + answer.""" response = client.chat.completions.create( model="default", messages=[ {"role": "system", "content": "You are a math tutor. Show your work."}, {"role": "user", "content": problem} ], temperature=0.2 ) message = response.choices[0].message return { "problem": problem, "work": message.reasoning, "answer": message.content } result = solve_math("If a train travels 120 km in 2 hours, what is its average speed?") print(f"Problem: {result['problem']}") print(f"\nWork shown:\n{result['work']}") print(f"\nFinal answer: {result['answer']}") ``` ## Curl 示例 ### 非 Streaming ```bash curl http://localhost:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "default", "messages": [{"role": "user", "content": "What is 15% of 80?"}] }' ``` ### Streaming ```bash curl http://localhost:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "default", "messages": [{"role": "user", "content": "What is 15% of 80?"}], "stream": true }' ``` ## 常见问题排查 ### 响应中没有 reasoning 字段 - 确认启动服务器时指定了 `--reasoning-parser` - 检查模型是否实际使用了 thinking 标签（并非所有提示词都会触发 reasoning） ### Reasoning 出现在 content 中 - 模型可能没有使用预期的标签格式 - 尝试换用其他 parser（`qwen3` 或 `deepseek_r1`） ### Reasoning 被截断 - 如果模型在 thinking 过程中触及了 token 上限，请增大 `--max-tokens` ## 相关链接 - [支持的模型](../reference/models.md). 支持 reasoning 的模型列表 - [服务器配置](server.md). 所有服务器选项 - [CLI 参考](../reference/cli.md). 命令行选项 # Documentation page: `zh/guides/server.md` # OpenAI 兼容服务器 vllm-mlx 提供一个具备完整 OpenAI API 兼容性的 FastAPI 服务器。 默认情况下，服务器仅绑定到 `127.0.0.1`。只有在明确需要将其暴露到本机以外的网络时，才使用 `--host 0.0.0.0`。 ## 启动服务器 ### 简单模式（默认） 单用户最大吞吐量： ```bash vllm-mlx serve mlx-community/Llama-3.2-3B-Instruct-4bit --port 8000 ``` ### continuous batching 模式 适用于多个并发用户： ```bash vllm-mlx serve mlx-community/Llama-3.2-3B-Instruct-4bit --port 8000 --continuous-batching ``` ### 启用 paged cache 适用于生产环境的高效内存缓存： ```bash vllm-mlx serve mlx-community/Llama-3.2-3B-Instruct-4bit --port 8000 --continuous-batching --use-paged-cache ``` ## 服务器选项 | 选项 | 说明 | 默认值 | |------|------|--------| | `--port` | 服务器端口 | 8000 | | `--host` | 服务器主机 | 127.0.0.1 | | `--api-key` | 身份验证用 API key | None | | `--rate-limit` | 每客户端每分钟请求数（0 表示禁用） | 0 | | `--timeout` | 请求超时时间（秒） | 300 | | `--enable-metrics` | 在 `/metrics` 上暴露 Prometheus 指标 | False | | `--continuous-batching` | 为多用户启用 batching | False | | `--use-paged-cache` | 启用 paged KV cache | False | | `--cache-memory-mb` | 缓存内存上限（MB） | Auto | | `--cache-memory-percent` | 用于缓存的 RAM 比例 | 0.20 | | `--max-tokens` | 默认最大 token 数 | 32768 | | `--max-request-tokens` | API 客户端可传入的 `max_tokens` 最大值 | 32768 | | `--default-temperature` | 未指定时的默认 temperature | None | | `--default-top-p` | 未指定时的默认 top_p | None | | `--stream-interval` | 每个 streaming chunk 包含的 token 数 | 1 | | `--mcp-config` | MCP 配置文件路径 | None | | `--reasoning-parser` | reasoning 模型解析器（`qwen3`、`deepseek_r1`） | None | | `--embedding-model` | 启动时预加载 embeddings 模型 | None | | `--enable-auto-tool-choice` | 启用自动 tool calling | False | | `--tool-call-parser` | tool call 解析器（参见 [Tool Calling](tool-calling.md)） | None | ## API 端点 ### Chat Completions ```bash POST /v1/chat/completions ``` ```python from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed") # Non-streaming response = client.chat.completions.create( model="default", messages=[{"role": "user", "content": "Hello!"}], max_tokens=100 ) # Streaming stream = client.chat.completions.create( model="default", messages=[{"role": "user", "content": "Tell me a story"}], stream=True ) for chunk in stream: if chunk.choices[0].delta.content: print(chunk.choices[0].delta.content, end="") ``` ### Completions ```bash POST /v1/completions ``` ```python response = client.completions.create( model="default", prompt="The capital of France is", max_tokens=50 ) ``` ### Models ```bash GET /v1/models ``` 返回可用模型列表。 ### Embeddings ```bash POST /v1/embeddings ``` ```python response = client.embeddings.create( model="mlx-community/multilingual-e5-small-mlx", input="Hello world" ) print(response.data[0].embedding[:5]) # First 5 dimensions ``` 详情参见 [Embeddings 指南](embeddings.md)。 ### 健康检查 ```bash GET /health ``` 返回服务器状态。 ### 指标 ```bash GET /metrics ``` Prometheus 抓取端点，提供服务器、缓存、scheduler 及请求指标。该端点默认禁用，需通过 `--enable-metrics` 启用。 ```bash vllm-mlx serve mlx-community/Llama-3.2-3B-Instruct-4bit \ --enable-metrics ``` `/metrics` 端点有意不做身份验证。请仅在受信任的网络中暴露，或通过反向代理、防火墙限制访问来源。 ### Anthropic Messages API ```bash POST /v1/messages ``` 兼容 Anthropic 协议的端点，允许 Claude Code、OpenCode 等工具直接连接 vllm-mlx。内部将 Anthropic 请求转换为 OpenAI 格式，经引擎推理后再将响应转换回 Anthropic 格式。 功能： - 非 streaming 与 streaming 响应（SSE） - 系统消息（纯字符串或内容块列表） - 包含用户和助手消息的多轮对话 - 使用 `tool_use` / `tool_result` 内容块进行 tool calling - 用于预算追踪的 token 计数 - 多模态内容（通过 `source` 块传入图片） - 客户端断开检测（返回 HTTP 499） - streaming 输出中的特殊 token 自动过滤 #### 非 streaming ```python from anthropic import Anthropic client = Anthropic(base_url="http://localhost:8000", api_key="not-needed") response = client.messages.create( model="default", max_tokens=256, messages=[{"role": "user", "content": "Hello!"}] ) print(response.content[0].text) # Response includes: response.id, response.model, response.stop_reason, # response.usage.input_tokens, response.usage.output_tokens ``` #### Streaming streaming 遵循 Anthropic SSE 事件协议，事件按以下顺序发出： `message_start` -> `content_block_start` -> `content_block_delta`（重复）-> `content_block_stop` -> `message_delta` -> `message_stop` ```python with client.messages.stream( model="default", max_tokens=256, messages=[{"role": "user", "content": "Tell me a story"}] ) as stream: for text in stream.text_stream: print(text, end="") ``` #### 系统消息 系统消息可以是纯字符串，也可以是内容块列表： ```python # Plain string response = client.messages.create( model="default", max_tokens=256, system="You are a helpful coding assistant.", messages=[{"role": "user", "content": "Write a hello world in Python"}] ) # List of content blocks response = client.messages.create( model="default", max_tokens=256, system=[ {"type": "text", "text": "You are a helpful assistant."}, {"type": "text", "text": "Be concise in your answers."}, ], messages=[{"role": "user", "content": "What is 2+2?"}] ) ``` #### Tool calling 使用 `name`、`description` 和 `input_schema` 定义工具。模型在需要调用工具时会返回 `tool_use` 内容块。将结果以 `tool_result` 块的形式返回。 ```python # Step 1: Send request with tools response = client.messages.create( model="default", max_tokens=1024, messages=[{"role": "user", "content": "What's the weather in Paris?"}], tools=[{ "name": "get_weather", "description": "Get weather for a city", "input_schema": { "type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"] } }] ) # Step 2: Check if model wants to use tools for block in response.content: if block.type == "tool_use": print(f"Tool: {block.name}, Input: {block.input}, ID: {block.id}") # response.stop_reason will be "tool_use" # Step 3: Send tool result back response = client.messages.create( model="default", max_tokens=1024, messages=[ {"role": "user", "content": "What's the weather in Paris?"}, {"role": "assistant", "content": response.content}, {"role": "user", "content": [ { "type": "tool_result", "tool_use_id": block.id, "content": "Sunny, 22C" } ]} ], tools=[{ "name": "get_weather", "description": "Get weather for a city", "input_schema": { "type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"] } }] ) print(response.content[0].text) # "The weather in Paris is sunny, 22C." ``` Tool choice 模式： | `tool_choice` | 行为 | |---------------|------| | `{"type": "auto"}` | 由模型决定是否调用工具（默认） | | `{"type": "any"}` | 模型必须至少调用一个工具 | | `{"type": "tool", "name": "get_weather"}` | 模型必须调用指定工具 | | `{"type": "none"}` | 模型不调用任何工具 | #### 多轮对话 ```python messages = [ {"role": "user", "content": "My name is Alice."}, {"role": "assistant", "content": "Nice to meet you, Alice!"}, {"role": "user", "content": "What's my name?"}, ] response = client.messages.create( model="default", max_tokens=100, messages=messages ) ``` #### Token 计数 ```bash POST /v1/messages/count_tokens ``` 使用模型的 tokenizer 统计 Anthropic 请求的输入 token 数。适用于在发送请求前进行预算追踪。可统计系统消息、对话消息、tool_use 输入、tool_result 内容及工具定义（name、description、input_schema）中的 token。 ```python import requests resp = requests.post("http://localhost:8000/v1/messages/count_tokens", json={ "model": "default", "messages": [{"role": "user", "content": "Hello, how are you?"}], "system": "You are helpful.", "tools": [{ "name": "search", "description": "Search the web", "input_schema": {"type": "object", "properties": {"q": {"type": "string"}}} }] }) print(resp.json()) # {"input_tokens": 42} ``` #### curl 示例 非 streaming： ```bash curl http://localhost:8000/v1/messages \ -H "Content-Type: application/json" \ -d '{ "model": "default", "max_tokens": 256, "messages": [{"role": "user", "content": "Hello!"}] }' ``` Streaming： ```bash curl http://localhost:8000/v1/messages \ -H "Content-Type: application/json" \ -d '{ "model": "default", "max_tokens": 256, "stream": true, "messages": [{"role": "user", "content": "Tell me a joke"}] }' ``` Token 计数： ```bash curl http://localhost:8000/v1/messages/count_tokens \ -H "Content-Type: application/json" \ -d '{ "model": "default", "messages": [{"role": "user", "content": "Hello!"}] }' # {"input_tokens": 12} ``` #### 请求字段 | 字段 | 类型 | 是否必填 | 默认值 | 说明 | |------|------|----------|--------|------| | `model` | string | 是 | - | 模型名称（使用 `"default"` 指向已加载的模型） | | `messages` | list | 是 | - | 包含 `role` 和 `content` 的对话消息 | | `max_tokens` | int | 是 | - | 最大生成 token 数 | | `system` | string 或 list | 否 | null | 系统提示（字符串或 `{"type": "text", "text": "..."}` 块列表） | | `stream` | bool | 否 | false | 启用 SSE streaming | | `temperature` | float | 否 | 0.7 | 采样 temperature（0.0 为确定性，1.0 为创意性） | | `top_p` | float | 否 | 0.9 | nucleus sampling 阈值 | | `top_k` | int | 否 | null | top-k sampling | | `stop_sequences` | list | 否 | null | 触发停止生成的序列 | | `tools` | list | 否 | null | 包含 `name`、`description`、`input_schema` 的工具定义 | | `tool_choice` | dict | 否 | null | 工具选择模式（`auto`、`any`、`tool`、`none`） | | `metadata` | dict | 否 | null | 任意元数据（透传，服务器不使用） | #### 响应格式 非 streaming 响应： ```json { "id": "msg_abc123...", "type": "message", "role": "assistant", "model": "default", "content": [ {"type": "text", "text": "Hello! How can I help?"} ], "stop_reason": "end_turn", "stop_sequence": null, "usage": { "input_tokens": 12, "output_tokens": 8 } } ``` 调用工具时，`content` 包含 `tool_use` 块，且 `stop_reason` 为 `"tool_use"`： ```json { "content": [ {"type": "text", "text": "Let me check the weather."}, { "type": "tool_use", "id": "call_abc123", "name": "get_weather", "input": {"city": "Paris"} } ], "stop_reason": "tool_use" } ``` 停止原因： | `stop_reason` | 含义 | |---------------|------| | `end_turn` | 模型自然完成生成 | | `tool_use` | 模型需要调用工具 | | `max_tokens` | 达到 `max_tokens` 上限 | #### 与 Claude Code 配合使用 将 Claude Code 直接指向你的 vllm-mlx 服务器： ```bash # Start the server vllm-mlx serve mlx-community/Qwen3-Coder-Next-235B-A22B-4bit \ --continuous-batching \ --enable-auto-tool-choice \ --tool-call-parser hermes # In another terminal, configure Claude Code export ANTHROPIC_BASE_URL=http://localhost:8000 export ANTHROPIC_API_KEY=not-needed claude ``` ### 服务器状态 ```bash GET /v1/status ``` 实时监控端点，返回服务器全局统计信息及每个请求的详情。适用于调试性能、追踪缓存效率以及监控 Metal GPU 内存。 ```bash curl -s http://localhost:8000/v1/status | python -m json.tool ``` 示例响应： ```json { "status": "running", "model": "mlx-community/Qwen3-8B-4bit", "uptime_s": 342.5, "steps_executed": 1247, "num_running": 1, "num_waiting": 0, "total_requests_processed": 15, "total_prompt_tokens": 28450, "total_completion_tokens": 3200, "metal": { "active_memory_gb": 5.2, "peak_memory_gb": 8.1, "cache_memory_gb": 2.3 }, "cache": { "type": "memory_aware_cache", "entries": 5, "hit_rate": 0.87, "memory_mb": 2350 }, "requests": [ { "request_id": "req_abc123", "phase": "generation", "tokens_per_second": 45.2, "ttft_s": 0.8, "progress": 0.35, "cache_hit_type": "prefix", "cached_tokens": 1200, "generated_tokens": 85, "max_tokens": 256 } ] } ``` 响应字段： | 字段 | 说明 | |------|------| | `status` | 服务器状态：`running`、`stopped` 或 `not_loaded` | | `model` | 已加载模型的名称 | | `uptime_s` | 服务器启动后经过的秒数 | | `steps_executed` | 已执行的推理步骤总数 | | `num_running` | 当前正在生成 token 的请求数 | | `num_waiting` | 排队等待 prefill 的请求数 | | `total_requests_processed` | 启动以来已完成的请求总数 | | `total_prompt_tokens` | 启动以来处理的 prompt token 总数 | | `total_completion_tokens` | 启动以来生成的 completion token 总数 | | `metal.active_memory_gb` | 当前使用的 Metal GPU 内存（GB） | | `metal.peak_memory_gb` | Metal GPU 内存峰值用量（GB） | | `metal.cache_memory_gb` | Metal 缓存内存用量（GB） | | `cache` | 缓存统计信息（类型、条目数、命中率、内存用量） | | `requests` | 活跃请求列表，包含每个请求的详细信息 | `requests` 中的每请求字段： | 字段 | 说明 | |------|------| | `request_id` | 唯一请求标识符 | | `phase` | 当前阶段：`queued`、`prefill` 或 `generation` | | `tokens_per_second` | 该请求的生成吞吐量 | | `ttft_s` | 首 token 时间（秒），即 TTFT | | `progress` | 完成进度（0.0 到 1.0） | | `cache_hit_type` | 缓存匹配类型：`exact`、`prefix`、`supersequence`、`lcp` 或 `miss` | | `cached_tokens` | 从缓存中命中的 token 数 | | `generated_tokens` | 已生成的 token 数 | | `max_tokens` | 请求的最大 token 数 | ## Tool Calling 使用 `--enable-auto-tool-choice` 启用兼容 OpenAI 的 tool calling： ```bash vllm-mlx serve mlx-community/Devstral-Small-2507-4bit \ --enable-auto-tool-choice \ --tool-call-parser mistral ``` 使用 `--tool-call-parser` 选项为你的模型选择对应的解析器： | 解析器 | 适用模型 | |--------|----------| | `auto` | 自动检测（依次尝试所有解析器） | | `mistral` | Mistral、Devstral | | `qwen` | Qwen、Qwen3 | | `llama` | Llama 3.x、4.x | | `hermes` | Hermes、NousResearch | | `deepseek` | DeepSeek V3、R1 | | `kimi` | Kimi K2、Moonshot | | `granite` | IBM Granite 3.x、4.x | | `nemotron` | NVIDIA Nemotron | | `xlam` | Salesforce xLAM | | `functionary` | MeetKai Functionary | | `glm47` | GLM-4.7、GLM-4.7-Flash | ```python response = client.chat.completions.create( model="default", messages=[{"role": "user", "content": "What's the weather in Paris?"}], tools=[{ "type": "function", "function": { "name": "get_weather", "description": "Get weather for a city", "parameters": { "type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"] } } }] ) if response.choices[0].message.tool_calls: for tc in response.choices[0].message.tool_calls: print(f"{tc.function.name}: {tc.function.arguments}") ``` 完整文档参见 [Tool Calling 指南](tool-calling.md)。 ## Reasoning 模型 对于展示思考过程的模型（Qwen3、DeepSeek-R1），使用 `--reasoning-parser` 将 reasoning 内容与最终答案分离： ```bash # Qwen3 models vllm-mlx serve mlx-community/Qwen3-8B-4bit --reasoning-parser qwen3 # DeepSeek-R1 models vllm-mlx serve mlx-community/DeepSeek-R1-Distill-Qwen-7B-4bit --reasoning-parser deepseek_r1 ``` API 响应中包含 `reasoning` 字段，用于展示模型的思考过程： ```python response = client.chat.completions.create( model="default", messages=[{"role": "user", "content": "What is 17 × 23?"}] ) print(response.choices[0].message.reasoning) # Step-by-step thinking print(response.choices[0].message.content) # Final answer ``` streaming 时，reasoning chunk 先于 content chunk 到达： ```python for chunk in stream: delta = chunk.choices[0].delta if delta.reasoning: print(f"[Thinking] {delta.reasoning}") if delta.content: print(delta.content, end="") ``` 完整说明参见 [Reasoning 模型指南](reasoning.md)。 ## 结构化输出（JSON 模式） 使用 `response_format` 强制模型返回合法 JSON。 ### JSON Object 模式 返回任意合法 JSON： ```python response = client.chat.completions.create( model="default", messages=[{"role": "user", "content": "List 3 colors"}], response_format={"type": "json_object"} ) # Output: {"colors": ["red", "blue", "green"]} ``` ### JSON Schema 模式 返回符合指定 schema 的 JSON： ```python response = client.chat.completions.create( model="default", messages=[{"role": "user", "content": "List 3 colors"}], response_format={ "type": "json_schema", "json_schema": { "name": "colors", "schema": { "type": "object", "properties": { "colors": { "type": "array", "items": {"type": "string"} } }, "required": ["colors"] } } } ) # Output validated against schema data = json.loads(response.choices[0].message.content) assert "colors" in data ``` ### curl 示例 ```bash curl http://localhost:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "default", "messages": [{"role": "user", "content": "List 3 colors"}], "response_format": {"type": "json_object"} }' ``` ## curl 示例 ### Chat ```bash curl http://localhost:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "default", "messages": [{"role": "user", "content": "Hello!"}], "max_tokens": 100 }' ``` ### Streaming ```bash curl http://localhost:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "default", "messages": [{"role": "user", "content": "Hello!"}], "stream": true }' ``` ## Streaming 配置 使用 `--stream-interval` 控制 streaming 行为： | 值 | 行为 | |----|------| | `1`（默认） | 每个 token 立即发送 | | `2-5` | 积攒若干 token 后再发送 | | `10+` | 最大吞吐量，输出分块较大 | ```bash # Smooth streaming vllm-mlx serve model --continuous-batching --stream-interval 1 # Batched streaming (better for high-latency networks) vllm-mlx serve model --continuous-batching --stream-interval 5 ``` ## Open WebUI 集成 ```bash # 1. Start vllm-mlx server vllm-mlx serve mlx-community/Llama-3.2-3B-Instruct-4bit --port 8000 # 2. Start Open WebUI docker run -d -p 3000:8080 \ -e OPENAI_API_BASE_URL=http://host.docker.internal:8000/v1 \ -e OPENAI_API_KEY=not-needed \ --name open-webui \ ghcr.io/open-webui/open-webui:main # 3. Open http://localhost:3000 ``` ## 生产部署 ### 使用 systemd 创建 `/etc/systemd/system/vllm-mlx.service`： ```ini [Unit] Description=vLLM-MLX Server After=network.target [Service] Type=simple ExecStart=/usr/local/bin/vllm-mlx serve mlx-community/Qwen3-0.6B-8bit \ --continuous-batching --use-paged-cache --port 8000 Restart=always [Install] WantedBy=multi-user.target ``` ```bash sudo systemctl enable vllm-mlx sudo systemctl start vllm-mlx ``` ### 推荐配置 适用于 50 个以上并发用户的生产环境： ```bash vllm-mlx serve mlx-community/Qwen3-0.6B-8bit \ --continuous-batching \ --use-paged-cache \ --api-key your-secret-key \ --rate-limit 60 \ --timeout 120 \ --port 8000 ``` # Documentation page: `zh/guides/tool-calling.md` # Tool Calling vllm-mlx 支持与 OpenAI 兼容的 tool calling（function calling），并为多种主流模型系列提供自动解析。 ## 快速开始 启动服务器时添加 `--enable-auto-tool-choice` 标志即可启用 tool calling： ```bash vllm-mlx serve mlx-community/Devstral-Small-2507-4bit \ --enable-auto-tool-choice \ --tool-call-parser mistral ``` 然后通过标准 OpenAI API 使用工具： ```python from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed") response = client.chat.completions.create( model="default", messages=[{"role": "user", "content": "What's the weather in Paris?"}], tools=[{ "type": "function", "function": { "name": "get_weather", "description": "Get weather for a city", "parameters": { "type": "object", "properties": { "city": {"type": "string", "description": "City name"} }, "required": ["city"] } } }] ) # Check for tool calls if response.choices[0].message.tool_calls: for tc in response.choices[0].message.tool_calls: print(f"Function: {tc.function.name}") print(f"Arguments: {tc.function.arguments}") ``` ## 支持的 tool parser 使用 `--tool-call-parser` 为您的模型系列选择对应的 tool parser： | Parser | 别名 | 模型 | 格式 | |--------|------|------|------| | `auto` | | 任意模型 | 自动检测格式（依次尝试所有 parser） | | `mistral` | | Mistral、Devstral | `[TOOL_CALLS]` JSON 数组 | | `qwen` | `qwen3` | Qwen、Qwen3 | `` XML 或 `[Calling tool:]` | | `llama` | `llama3`、`llama4` | Llama 3.x、4.x | `` 标签 | | `hermes` | `nous` | Hermes、NousResearch | `` XML 包裹的 JSON | | `deepseek` | `deepseek_v3`、`deepseek_r1` | DeepSeek V3、R1 | Unicode 分隔符 | | `kimi` | `kimi_k2`、`moonshot` | Kimi K2、Moonshot | `<\|tool_call_begin\|>` 标记 | | `granite` | `granite3` | IBM Granite 3.x、4.x | `<\|tool_call\|>` 或 `` | | `nemotron` | `nemotron3` | NVIDIA Nemotron | `` | | `xlam` | | Salesforce xLAM | 含 `tool_calls` 数组的 JSON | | `functionary` | `meetkai` | MeetKai Functionary | 多个 function 块 | | `glm47` | `glm4` | GLM-4.7、GLM-4.7-Flash | `` 配合 ``/`` XML | ## 模型示例 ### Mistral / Devstral ```bash # Devstral Small（针对编程和 tool use 优化） vllm-mlx serve mlx-community/Devstral-Small-2507-4bit \ --enable-auto-tool-choice --tool-call-parser mistral # Mistral Instruct vllm-mlx serve mlx-community/Mistral-7B-Instruct-v0.3-4bit \ --enable-auto-tool-choice --tool-call-parser mistral ``` ### Qwen ```bash # Qwen3 vllm-mlx serve mlx-community/Qwen3-4B-4bit \ --enable-auto-tool-choice --tool-call-parser qwen ``` ### Llama ```bash # Llama 3.2 vllm-mlx serve mlx-community/Llama-3.2-3B-Instruct-4bit \ --enable-auto-tool-choice --tool-call-parser llama ``` ### DeepSeek ```bash # DeepSeek V3 vllm-mlx serve mlx-community/DeepSeek-V3-0324-4bit \ --enable-auto-tool-choice --tool-call-parser deepseek ``` ### IBM Granite ```bash # Granite 4.0 vllm-mlx serve mlx-community/granite-4.0-tiny-preview-4bit \ --enable-auto-tool-choice --tool-call-parser granite ``` ### NVIDIA Nemotron ```bash # Nemotron 3 Nano vllm-mlx serve mlx-community/NVIDIA-Nemotron-3-Nano-30B-A3B-MLX-6Bit \ --enable-auto-tool-choice --tool-call-parser nemotron ``` ### GLM-4.7 ```bash # GLM-4.7 Flash vllm-mlx serve lmstudio-community/GLM-4.7-Flash-MLX-8bit \ --enable-auto-tool-choice --tool-call-parser glm47 ``` ### Kimi K2 ```bash # Kimi K2 vllm-mlx serve mlx-community/Kimi-K2-Instruct-4bit \ --enable-auto-tool-choice --tool-call-parser kimi ``` ### Salesforce xLAM ```bash # xLAM vllm-mlx serve mlx-community/xLAM-2-fc-r-4bit \ --enable-auto-tool-choice --tool-call-parser xlam ``` ## Auto Parser 如果不确定使用哪个 tool parser，`auto` parser 会尝试自动检测格式： ```bash vllm-mlx serve mlx-community/Qwen3-4B-4bit \ --enable-auto-tool-choice --tool-call-parser auto ``` auto parser 按以下顺序依次尝试各种格式： 1. Mistral（`[TOOL_CALLS]`） 2. Qwen 括号格式（`[Calling tool:]`） 3. Nemotron（``） 4. Qwen/Hermes XML（`{...}`） 5. Llama（`{...}`） 6. 原始 JSON ## Streaming Tool Calls Tool calling 支持 streaming。模型生成完毕后发送 tool call 信息： ```python stream = client.chat.completions.create( model="default", messages=[{"role": "user", "content": "What's 25 * 17?"}], tools=[{ "type": "function", "function": { "name": "calculator", "description": "Calculate math expressions", "parameters": { "type": "object", "properties": { "expression": {"type": "string"} }, "required": ["expression"] } } }], stream=True ) for chunk in stream: if chunk.choices[0].delta.tool_calls: for tc in chunk.choices[0].delta.tool_calls: print(f"Tool call: {tc.function.name}({tc.function.arguments})") ``` ## 处理工具返回结果 收到 tool call 后，执行对应函数并将结果返回给模型： ```python import json # 第一次请求，模型决定调用工具 response = client.chat.completions.create( model="default", messages=[{"role": "user", "content": "What's the weather in Tokyo?"}], tools=[weather_tool] ) # 获取 tool call tool_call = response.choices[0].message.tool_calls[0] tool_call_id = tool_call.id function_name = tool_call.function.name arguments = json.loads(tool_call.function.arguments) # 执行函数（由您自行实现） result = get_weather(**arguments) # {"temperature": 22, "condition": "sunny"} # 将结果返回给模型 response = client.chat.completions.create( model="default", messages=[ {"role": "user", "content": "What's the weather in Tokyo?"}, {"role": "assistant", "tool_calls": [tool_call]}, {"role": "tool", "tool_call_id": tool_call_id, "content": json.dumps(result)} ], tools=[weather_tool] ) print(response.choices[0].message.content) # "The weather in Tokyo is sunny with a temperature of 22C." ``` ## Think 标签处理 会产生 `...` reasoning 标签的模型（如 DeepSeek-R1、Qwen3、GLM-4.7）均可自动处理。tool parser 会在提取 tool call 前剥离 thinking 内容，因此 reasoning 标签不会干扰 tool call 解析。 即使 `` 是通过提示词注入的（即仅有闭合标签 `` 的隐式 think 标签），也同样适用。 ## CLI 参数参考 | 选项 | 说明 | |------|------| | `--enable-auto-tool-choice` | 启用自动 tool calling | | `--tool-call-parser` | 选择 tool parser（见上表） | 完整选项请参阅 [CLI Reference](../reference/cli.md)。 # Documentation page: `zh/guides/warm-prompts.md` # Warm Prompts 在服务器启动时预先填充 prefix cache，使 agent 发送的**第一个**请求命中已预热的缓存，而无需为其数千字节的系统提示支付完整的 prefill 开销。 ## 适用场景 Agent 工作负载，如代理编码助手或推理助手的代理、MCP 服务器、多 agent 编排器，始终会发送相同的系统提示。在当前实现中，冷启动服务器收到的第一个请求需要为该系统提示支付完整的 prefill 代价。对于数十亿参数的模型，这意味着数秒的 TTFT，而此时用户正在等待其新 agent 首次响应。 如果您在部署时已知道各 agent 的系统提示，可将其写入一个 JSON 文件并通过 `--warm-prompts` 指向它。服务器会在启动时对每条提示执行一次 `max_tokens=1` 的聊天补全，KV cache 状态随即落入 prefix cache，后续真实请求即可通过严格前缀匹配命中缓存。 此功能需要 `--continuous-batching`（prefix cache 依赖该模式）。 ## 快速示例 ```bash # 一次性写入您关心的 agent cat > ~/.config/vllm-mlx/agents.json <<'JSON' [ [{"role": "system", "content": "You are a code assistant..."}] ] JSON # 将服务器指向该文件 vllm-mlx serve mlx-community/Qwen3-4B-4bit \ --continuous-batching \ --warm-prompts ~/.config/vllm-mlx/agents.json ``` 启动时您将看到： ``` [lifespan] Warm-up done (strict-prefix): 1 completed, 0 skipped, 1431 prompt tokens in 0.2s ``` 第一个共享已预热系统提示的真实请求将命中缓存，其 `tokens_saved` 接近预热提示的长度。 ## 文件格式 顶层为一个 JSON 列表，每个条目本身也是一个聊天消息列表，结构与 `/v1/chat/completions` 中的 `messages` 字段相同。 ```json [ [ {"role": "system", "content": "You are a code assistant..."} ], [ {"role": "system", "content": "You are a senior code reviewer..."} ], [ {"role": "system", "content": "You are a planner..."}, {"role": "user", "content": "hi"}, {"role": "assistant", "content": "Hello, what are we planning?"} ] ] ``` 单条系统提示是最常见的用法。多轮历史也受支持，适用于需要预热特定对话开头的场景（少样本示例、持续运行的助手角色等）。 ## 规模建议 预热提示通过 `asyncio.gather` **并发**处理，因此 N 条条目会在启动时触发 N 个并发 prefill，每个 prefill 会为其提示长度分配 KV cache。 **建议条目数为 1 至 3 条**，足以覆盖典型 agent 部署的热路径（每个角色一条）。在内存紧张的模型上，过大的 warm-prompts 文件可能在启动时耗尽可用空间。 如需预热数十个角色，请提交一个 issue 并说明您的工作负载，我们可以添加 `--warm-prompts-concurrency=N` 上限参数。 ## 基准测试 **测试环境：** M4 Max，128 GB 统一内存。每次测量使用两个独立服务器（冷启动与预热），隔离冷启动。`long` 提示集（约 2500 个用户 token）前置约 1700 token 的系统提示以匹配预热提示。`max_tokens=128`。bench-serve 使用 `--skip-preflight-token-count`，避免 count_prompt_tokens 预检污染缓存。 | 模型 | 并发 | 冷启动 TTFT | 预热 TTFT | 加速比 | |------|-----:|----------:|----------:|------:| | Qwen3-0.6B-8bit | 1 | 563 ms | 419 ms | 1.34x | | Qwen3-0.6B-8bit | 4 | 1 723 ms | 1 282 ms | 1.34x | | Qwen3-0.6B-8bit | 8 | 3 708 ms | 2 661 ms | 1.39x | | Llama-3.2-3B-Instruct-4bit | 1 | 1 754 ms | 1 060 ms | 1.65x | | Llama-3.2-3B-Instruct-4bit | 4 | 5 926 ms | 3 945 ms | 1.50x | | Llama-3.2-3B-Instruct-4bit | 8 | 15 161 ms | 9 820 ms | 1.54x | | Qwen3-4B-4bit | 1 | 4 937 ms | 2 191 ms | 2.25x | | Qwen3-4B-4bit | 4 | 12 535 ms | 9 623 ms | 1.30x | | Qwen3-4B-4bit | 8 | 38 148 ms | 23 878 ms | 1.60x | | Qwen3.6-35B-A3B-4bit (MoE/hybrid) | 1 | 2 400 ms | 1 603 ms | 1.50x | | Qwen3.6-35B-A3B-4bit | 4 | 8 735 ms | 6 054 ms | 1.44x | | Qwen3.6-35B-A3B-4bit | 8 | 22 419 ms | 14 409 ms | 1.56x | 全部 12 项配置均有提升。当提示占总长度比例最高时（并发=1，长系统提示），TTFT 节省最为显著，在并发负载下仍有实质性收益。 **生成 tok/s** 对于稠密模型基本持平（误差在 ±5% 以内）。Qwen3.6-35B-A3B（MoE）在并发数大于等于 4 时出现 20 至 35% 的解码速度下降，原因似乎是 MoE 路由与批量调度之间的交互。对于 agent 工作负载，TTFT 节省仍主导端到端延迟，但若您的工作流在高并发下以解码为瓶颈，请注意这一点。 ## 工作原理 朴素的预热方式，即用占位用户消息渲染聊天模板并缓存 token，对于混合 SSM+attention 模型（Qwen3.5-MoE、Qwen3.6-MoE）不适用。这类模型的缓存层包含无法裁剪的 SSM 状态，因此 `memory_cache.py` 禁用了 LCP 匹配。占位用户内容与真实用户内容不同，基于 token 的缓存条目不再是任何真实请求的严格前缀。 本预热器会用两个不同的用户内容（`"__PROBE_A__"` 和 `"__PROBE_B__"`）**两次**渲染聊天模板，找到两个字符串开始发散的字符位置，并在该边界处截断第一次渲染的结果。这段截断后的字符串，即用户内容被插入之前的全部内容，是发送给引擎的内容。 由于引擎的真实请求路径同样使用 `tokenize=False` 渲染模板，再由分词器对结果进行编码，因此预热生成的 token 保证是任何具有匹配系统提示且聊天历史为空的真实请求的严格前缀。严格前缀匹配适用于所有缓存层类型，包括禁用 LCP 的混合路径。 ## 管理操作 ### 清除内存中的 prefix cache ```bash curl -X DELETE http://localhost:8000/v1/cache/prefix ``` 若服务器以 `--warm-prompts` 启动，清除后会在后台重新执行预热。响应会立即返回，不等待重新预热完成。 响应： ```json {"status": "cleared", "rewarm_scheduled": true} ``` ### 查看缓存状态 ```bash curl http://localhost:8000/v1/status | jq '.cache' ``` 使用 warm-prompts 启动后，在第一个用户请求到来之前，您将看到 `entry_count > 0`。 ## 针对您自己的场景进行基准测试 如需测量对您的模型和提示的实际影响，请使用 `bench-serve`： ```bash # 冷启动：不使用 warm-prompts vllm-mlx serve MODEL --continuous-batching & vllm-mlx bench-serve --prompts long --concurrency 1,4 \ --system-prompt-file my-system.txt --tag cold \ --output cold.csv --format csv # 预热：相同服务器配置 + --warm-prompts vllm-mlx serve MODEL --continuous-batching \ --warm-prompts ~/.config/vllm-mlx/agents.json & vllm-mlx bench-serve --prompts long --concurrency 1,4 \ --system-prompt-file my-system.txt --tag warm \ --output warm.csv --format csv ``` 设置 `--system-prompt-file` 时会自动启用 `--skip-preflight-token-count`，防止 `count_prompt_tokens` 预检污染缓存。比较 `cold.csv` 与 `warm.csv` 即可评估您工作负载的实际效果。 # Documentation page: `zh/index.md` # vLLM-MLX 文档 **Apple Silicon 的 MLX 推理后端** - 在 Mac 上对文本、图像、视频和音频进行 GPU 加速推理 ## 什么是 vLLM-MLX？ vllm-mlx 通过集成以下组件，为 vLLM 带来原生 Apple Silicon GPU 加速： - **[MLX](https://github.com/ml-explore/mlx)**：Apple 的机器学习框架，具有统一内存和 Metal 内核 - **[mlx-lm](https://github.com/ml-explore/mlx-lm)**：经过优化的 LLM 推理，支持 KV cache 和量化 - **[mlx-vlm](https://github.com/Blaizzy/mlx-vlm)**：用于多模态推理的视觉语言模型 VLM - **[mlx-audio](https://github.com/Blaizzy/mlx-audio)**：基于原生语音的 TTS 和 STT - **[mlx-embeddings](https://github.com/Blaizzy/mlx-embeddings)**：用于语义搜索和 RAG 的文本 embeddings ## 主要特性 - **多模态** - 在同一平台上处理文本、图像、视频和音频 - **原生 GPU 加速**，支持 Apple Silicon（M1、M2、M3、M4、M5） - **原生 TTS 语音** - 支持西班牙语、法语、中文、日语及其他 5 种语言 - **OpenAI API 兼容** - 可直接替换 OpenAI 客户端 - **Embeddings** - 兼容 OpenAI 的 `/v1/embeddings` 端点 - **MCP Tool Calling** - 通过 Model Context Protocol 集成外部工具 - **Paged KV Cache** - 支持前缀共享的高效内存缓存 - **Continuous Batching** - 为多并发用户提供高吞吐量 ## 快速链接 ### 入门指南 - [安装](getting-started/installation.md) - [快速开始](getting-started/quickstart.md) ### 用户指南 - [兼容 OpenAI 的服务器](guides/server.md) - [Python API](guides/python-api.md) - [多模态（图像与视频）](guides/multimodal.md) - [音频（STT/TTS）](guides/audio.md) - [Embeddings](guides/embeddings.md) - [Reasoning 模型](guides/reasoning.md) - [Tool Calling](guides/tool-calling.md) - [MCP 与 Tool Calling](guides/mcp-tools.md) - [Continuous Batching](guides/continuous-batching.md) ### 参考文档 - [CLI 命令](reference/cli.md) - [支持的模型](reference/models.md) - [配置说明](reference/configuration.md) ### 基准测试 - [LLM 基准测试](benchmarks/llm.md) - [图像基准测试](benchmarks/image.md) - [视频基准测试](benchmarks/video.md) - [音频基准测试](benchmarks/audio.md) ### 开发者文档 - [架构设计（英文）](/development/architecture/) - [贡献指南（英文）](/development/contributing/) ## 环境要求 - 搭载 Apple Silicon 的 macOS（M1/M2/M3/M4/M5） - Python 3.10 及以上 - 推荐 8GB 及以上内存 ## 许可证 Apache 2.0。请参阅[仓库许可证](https://github.com/waybarrios/vllm-mlx/blob/main/LICENSE)。 # Documentation page: `zh/reference/cli.md` # CLI 参考 ## 命令概览 | 命令 | 说明 | |---------|-------------| | `vllm-mlx serve` | 启动兼容 OpenAI 的服务器 | | `vllm-mlx-bench` | 运行性能基准测试 | | `vllm-mlx-chat` | 启动 Gradio 对话界面 | ## `vllm-mlx serve` 启动兼容 OpenAI 的 API 服务器。 ### 用法 ```bash vllm-mlx serve [options] ``` ### 选项 | 选项 | 说明 | 默认值 | |--------|-------------|---------| | `--served-model-name` | 通过 OpenAI API 暴露的自定义模型名称。未设置时使用模型路径作为名称。 | None | | `--port` | 服务器端口 | 8000 | | `--host` | 服务器主机 | 127.0.0.1 | | `--api-key` | 用于身份验证的 API 密钥 | None | | `--rate-limit` | 每个客户端每分钟的请求数（0 表示禁用） | 0 | | `--timeout` | 请求超时时间，单位为秒 | 300 | | `--enable-metrics` | 在 `/metrics` 上暴露 Prometheus 指标 | False | | `--continuous-batching` | 为多用户启用 continuous batching | False | | `--cache-memory-mb` | 缓存内存上限，单位为 MB | Auto | | `--cache-memory-percent` | 用于缓存的内存占比 | 0.20 | | `--no-memory-aware-cache` | 使用旧版按条目数计数的缓存 | False | | `--use-paged-cache` | 启用 paged KV cache | False | | `--max-tokens` | 默认最大 token 数 | 32768 | | `--max-request-tokens` | API 客户端可传入的最大 `max_tokens` | 32768 | | `--stream-interval` | 每个 streaming 分块包含的 token 数 | 1 | | `--mcp-config` | MCP 配置文件路径 | None | | `--paged-cache-block-size` | 每个缓存块包含的 token 数 | 64 | | `--max-cache-blocks` | 最大缓存块数 | 1000 | | `--max-num-seqs` | 最大并发序列数 | 256 | | `--default-temperature` | 请求未指定时的默认 temperature | None | | `--default-top-p` | 请求未指定时的默认 top_p | None | | `--max-audio-upload-mb` | `/v1/audio/transcriptions` 接受的最大音频上传大小 | 25 | | `--max-tts-input-chars` | `/v1/audio/speech` 接受的最大文本长度 | 4096 | | `--reasoning-parser` | reasoning 模型的解析器（`qwen3`、`deepseek_r1`） | None | | `--embedding-model` | 启动时预加载 embeddings 模型 | None | | `--enable-auto-tool-choice` | 启用自动 tool calling | False | | `--tool-call-parser` | tool call 解析器（`auto`、`mistral`、`qwen`、`llama`、`hermes`、`deepseek`、`kimi`、`granite`、`nemotron`、`xlam`、`functionary`、`glm47`） | None | ### 示例 ```bash # Simple mode (single user, max throughput) # Model path is used as the model name in the OpenAI API (e.g. model="mlx-community/Llama-3.2-3B-Instruct-4bit") vllm-mlx serve mlx-community/Llama-3.2-3B-Instruct-4bit Model will show up as 'mlx-community/Llama-3.2-3B-Instruct-4bit' in the `/v1/models` API endpoint. View with `curl http://localhost:8000/v1/models` or similar. # With a custom API model name (model is accessed as "my-model" via the OpenAI API) # --served-model-name sets the name clients must use when calling the API (e.g. model="my-model") vllm-mlx serve --served-model-name my-model mlx-community/Llama-3.2-3B-Instruct-4bit # Note: Model will show up as 'my-model' in the `/v1/models` API endpoint. # Continuous batching (multiple users) vllm-mlx serve mlx-community/Llama-3.2-3B-Instruct-4bit --continuous-batching # With memory limit for large models vllm-mlx serve mlx-community/GLM-4.7-Flash-4bit \ --continuous-batching \ --cache-memory-mb 2048 # Production with paged cache vllm-mlx serve mlx-community/Qwen3-0.6B-8bit \ --continuous-batching \ --use-paged-cache \ --port 8000 # With MCP tools vllm-mlx serve mlx-community/Qwen3-4B-4bit --mcp-config mcp.json # Multimodal model vllm-mlx serve mlx-community/Qwen3-VL-4B-Instruct-3bit # Reasoning model (separates thinking from answer) vllm-mlx serve mlx-community/Qwen3-8B-4bit --reasoning-parser qwen3 # DeepSeek reasoning model vllm-mlx serve mlx-community/DeepSeek-R1-Distill-Qwen-7B-4bit --reasoning-parser deepseek_r1 # Tool calling with Mistral/Devstral vllm-mlx serve mlx-community/Devstral-Small-2507-4bit \ --enable-auto-tool-choice --tool-call-parser mistral # Tool calling with Granite vllm-mlx serve mlx-community/granite-4.0-tiny-preview-4bit \ --enable-auto-tool-choice --tool-call-parser granite # With API key authentication vllm-mlx serve mlx-community/Llama-3.2-3B-Instruct-4bit --api-key your-secret-key # Expose Prometheus metrics vllm-mlx serve mlx-community/Llama-3.2-3B-Instruct-4bit --enable-metrics # Production setup with security options vllm-mlx serve mlx-community/Qwen3-4B-4bit \ --api-key your-secret-key \ --rate-limit 60 \ --timeout 120 \ --continuous-batching ``` ### 安全 设置 `--api-key` 后，所有 API 请求都需要携带 `Authorization: Bearer ` 请求头： ```python from openai import OpenAI client = OpenAI( base_url="http://localhost:8000/v1", api_key="your-secret-key" # Must match --api-key ) ``` 或使用 curl： ```bash curl http://localhost:8000/v1/models \ -H "Authorization: Bearer your-secret-key" ``` ## `vllm-mlx-bench` 运行性能基准测试。 ### 用法 ```bash vllm-mlx-bench --model [options] ``` ### 选项 | 选项 | 说明 | 默认值 | |--------|-------------|---------| | `--model` | 模型名称 | 必填 | | `--prompts` | 提示词数量 | 5 | | `--max-tokens` | 每条提示词的最大 token 数 | 256 | | `--quick` | 快速基准测试模式 | False | | `--video` | 运行视频基准测试 | False | | `--video-url` | 自定义视频 URL | None | | `--video-path` | 自定义视频路径 | None | ### 示例 ```bash # LLM benchmark vllm-mlx-bench --model mlx-community/Llama-3.2-1B-Instruct-4bit # Quick benchmark vllm-mlx-bench --model mlx-community/Llama-3.2-1B-Instruct-4bit --quick # Image benchmark (auto-detected for VLM models) vllm-mlx-bench --model mlx-community/Qwen3-VL-8B-Instruct-4bit # Video benchmark vllm-mlx-bench --model mlx-community/Qwen3-VL-8B-Instruct-4bit --video # Custom video vllm-mlx-bench --model mlx-community/Qwen3-VL-8B-Instruct-4bit \ --video --video-url https://example.com/video.mp4 ``` ## `vllm-mlx-chat` 启动 Gradio 对话界面。 ### 用法 ```bash vllm-mlx-chat --served-model-name [options] ``` ### 选项 | 选项 | 说明 | 默认值 | |--------|-------------|---------| | `--model` | 模型名称 | 必填 | | `--port` | Gradio 端口 | 7860 | | `--text-only` | 禁用多模态功能 | False | ### 示例 ```bash # Multimodal chat (text + images + video) vllm-mlx-chat --served-model-name mlx-community/Qwen3-VL-4B-Instruct-3bit # Text-only chat vllm-mlx-chat --served-model-name mlx-community/Llama-3.2-3B-Instruct-4bit --text-only ``` ## 环境变量 | 变量 | 说明 | |----------|-------------| | `VLLM_MLX_TEST_MODEL` | 测试使用的模型 | | `HF_TOKEN` | HuggingFace token | # Documentation page: `zh/reference/configuration.md` # 配置参考 ## 服务器配置 ### 基本选项 | 选项 | 说明 | 默认值 | |--------|-------------|---------| | `--host` | 服务器主机地址 | `127.0.0.1` | | `--port` | 服务器端口 | `8000` | | `--max-tokens` | 默认最大 token 数 | `32768` | | `--max-request-tokens` | API 客户端可传入的最大 `max_tokens` 值 | `32768` | | `--default-temperature` | 请求未指定时使用的默认 temperature | None | | `--default-top-p` | 请求未指定时使用的默认 top_p | None | ### 安全选项 | 选项 | 说明 | 默认值 | |--------|-------------|---------| | `--api-key` | 用于身份验证的 API key | None | | `--rate-limit` | 每个客户端每分钟的请求数（0 表示禁用） | `0` | | `--timeout` | 请求超时时间（秒） | `300` | | `--enable-metrics` | 在 `/metrics` 上暴露 Prometheus 指标 | `false` | | `--max-audio-upload-mb` | `/v1/audio/transcriptions` 接受的最大音频上传大小 | `25` | | `--max-tts-input-chars` | `/v1/audio/speech` 接受的最大文本长度 | `4096` | ### 批处理选项 | 选项 | 说明 | 默认值 | |--------|-------------|---------| | `--continuous-batching` | 启用 continuous batching | `false` | | `--stream-interval` | 每个 streaming 分块包含的 token 数 | `1` | | `--max-num-seqs` | 最大并发序列数 | `256` | ### 缓存选项 | 选项 | 说明 | 默认值 | |--------|-------------|---------| | `--cache-memory-mb` | 缓存内存上限（MB） | 自动 | | `--cache-memory-percent` | 分配给缓存的内存比例 | `0.20` | | `--no-memory-aware-cache` | 使用旧版基于条目数量的缓存 | `false` | | `--use-paged-cache` | 启用 paged KV cache | `false` | | `--paged-cache-block-size` | 每个块包含的 token 数 | `64` | | `--max-cache-blocks` | 最大块数量 | `1000` | ### 工具调用选项 | 选项 | 说明 | 默认值 | |--------|-------------|---------| | `--enable-auto-tool-choice` | 启用自动工具调用 | `false` | | `--tool-call-parser` | 工具调用解析器（参见 [工具调用](../guides/tool-calling.md)） | None | ### 推理选项 | 选项 | 说明 | 默认值 | |--------|-------------|---------| | `--reasoning-parser` | 推理模型解析器（`qwen3`、`deepseek_r1`） | None | ### 嵌入选项 | 选项 | 说明 | 默认值 | |--------|-------------|---------| | `--embedding-model` | 启动时预加载嵌入模型 | None | ### MCP 选项 | 选项 | 说明 | 默认值 | |--------|-------------|---------| | `--mcp-config` | MCP 配置文件路径 | None | ## MCP 配置 创建 `mcp.json`： ```json { "mcpServers": { "server-name": { "command": "npx", "args": ["-y", "@modelcontextprotocol/server-name", "arg1"], "env": { "ENV_VAR": "value" } } } } ``` ### MCP 服务器字段 | 字段 | 说明 | 是否必填 | |-------|-------------|----------| | `command` | 可执行命令 | 是 | | `args` | 命令参数 | 是 | | `env` | 环境变量 | 否 | ## API 请求选项 ### 聊天补全 | 参数 | 说明 | 默认值 | |-----------|-------------|---------| | `model` | 模型名称 | 必填 | | `messages` | 聊天消息 | 必填 | | `max_tokens` | 最大生成 token 数 | 256 | | `temperature` | 采样 temperature | 模型默认值 | | `top_p` | Nucleus sampling | 模型默认值 | | `stream` | 启用 streaming | `true` | | `stop` | 停止序列 | None | | `tools` | 工具定义 | None | | `response_format` | 输出格式（`json_object`、`json_schema`） | None | ### 多模态选项 | 参数 | 说明 | 默认值 | |-----------|-------------|---------| | `video_fps` | 每秒帧数 | 2.0 | | `video_max_frames` | 最大帧数 | 32 | ## 环境变量 | 变量 | 说明 | |----------|-------------| | `VLLM_MLX_TEST_MODEL` | 测试使用的默认模型 | | `HF_TOKEN` | HuggingFace 身份验证 token | | `OPENAI_API_KEY` | 设为任意值以兼容 SDK | ## 配置示例 ### 开发环境（单用户） ```bash vllm-mlx serve mlx-community/Llama-3.2-3B-Instruct-4bit ``` ### 生产环境（多用户） ```bash vllm-mlx serve mlx-community/Qwen3-0.6B-8bit \ --continuous-batching \ --use-paged-cache \ --api-key your-secret-key \ --rate-limit 60 \ --port 8000 ``` ### 使用工具调用 ```bash vllm-mlx serve mlx-community/Devstral-Small-2507-4bit \ --enable-auto-tool-choice \ --tool-call-parser mistral \ --continuous-batching ``` ### 使用 MCP 工具 ```bash vllm-mlx serve mlx-community/Qwen3-4B-4bit \ --mcp-config mcp.json \ --enable-auto-tool-choice \ --tool-call-parser qwen \ --continuous-batching ``` ### 推理模型 ```bash vllm-mlx serve mlx-community/Qwen3-8B-4bit \ --reasoning-parser qwen3 \ --continuous-batching ``` ### 使用嵌入 ```bash vllm-mlx serve mlx-community/Qwen3-4B-4bit \ --embedding-model mlx-community/multilingual-e5-small-mlx \ --continuous-batching ``` ### 高吞吐量 ```bash vllm-mlx serve mlx-community/Qwen3-0.6B-8bit \ --continuous-batching \ --stream-interval 5 \ --max-num-seqs 256 ``` # Documentation page: `zh/reference/models.md` # 支持的模型 所有来自 [mlx-community on HuggingFace](https://huggingface.co/mlx-community/models) 的量化模型均兼容。 在以下地址浏览数千个预优化模型：**https://huggingface.co/mlx-community/models** ## 语言模型（通过 mlx-lm） | 模型系列 | 规格 | 量化方式 | |--------------|-------|--------------| | Llama 3.x, 4.x | 1B, 3B, 8B, 70B | 4-bit | | Mistral / Devstral | 7B, Mixtral 8x7B | 4-bit, 8-bit | | Qwen2/Qwen3 | 0.5B 至 72B | 多种 | | DeepSeek V3, R1 | 7B, 33B, 67B | 4-bit | | Gemma 2, 3, 4 | 2B, 9B, 27B | 4-bit | | GLM-4.7 | Flash, Base | 4-bit, 8-bit | | Kimi K2 | 多种 | 4-bit | | Phi-3 | 3.8B, 14B | 4-bit | | Granite 3.x, 4.x | 多种 | 4-bit | | Nemotron | 3 Nano 30B | 6-bit | ### 推荐模型 | 使用场景 | 模型 | 内存 | |----------|-------|--------| | 快速/轻量 | `mlx-community/Qwen3-0.6B-8bit` | ~0.7 GB | | 均衡 | `mlx-community/Llama-3.2-3B-Instruct-4bit` | ~1.8 GB | | 高质量 | `mlx-community/Llama-3.1-8B-Instruct-4bit` | ~4.5 GB | | 大型 | `mlx-community/Qwen3-30B-A3B-4bit` | ~16 GB | ## 多模态模型（通过 mlx-vlm） | 模型系列 | 示例模型 | |--------------|----------------| | **Qwen-VL** | `Qwen3-VL-4B-Instruct-3bit`, `Qwen3-VL-8B-Instruct-4bit`, `Qwen2-VL-2B/7B-Instruct-4bit` | | **LLaVA** | `llava-1.5-7b-4bit`, `llava-v1.6-mistral-7b-4bit`, `llava-llama-3-8b-v1_1-4bit` | | **Idefics** | `Idefics3-8B-Llama3-4bit`, `idefics2-8b-4bit` | | **Gemma 4** | `gemma-4-e2b-it-mxfp4`（视觉 + 音频） | | **PaliGemma** | `paligemma2-3b-mix-224-4bit`, `paligemma-3b-mix-224-8bit` | | **Pixtral** | `pixtral-12b-4bit`, `pixtral-12b-8bit` | | **Molmo** | `Molmo-7B-D-0924-4bit`, `Molmo-7B-D-0924-8bit` | | **Phi-3 Vision** | `Phi-3-vision-128k-instruct-4bit` | | **DeepSeek-VL** | `deepseek-vl-7b-chat-4bit`, `deepseek-vl2-small-4bit` | ### 推荐 VLM 模型 | 使用场景 | 模型 | 内存 | |----------|-------|--------| | 快速/轻量 | `mlx-community/Qwen3-VL-4B-Instruct-3bit` | ~3 GB | | 均衡 | `mlx-community/Qwen3-VL-8B-Instruct-4bit` | ~6 GB | | 高质量 | `mlx-community/Qwen3-VL-30B-A3B-Instruct-6bit` | ~20 GB | ## Embedding 模型（通过 mlx-embeddings） | 模型系列 | 示例模型 | |--------------|----------------| | **BERT** | `mlx-community/bert-base-uncased-mlx` | | **XLM-RoBERTa** | `mlx-community/multilingual-e5-small-mlx`, `mlx-community/multilingual-e5-large-mlx` | | **ModernBERT** | `mlx-community/ModernBERT-base-mlx` | ## 音频模型（通过 mlx-audio） | 类型 | 模型系列 | 示例模型 | |------|--------------|----------------| | **STT** | Whisper | `mlx-community/whisper-large-v3-turbo` | | **STT** | Parakeet | `mlx-community/parakeet-tdt-0.6b-v2` | | **TTS** | Kokoro | `prince-canuma/Kokoro-82M` | | **TTS** | Chatterbox | `chatterbox/chatterbox-tts-0.1` | ## 模型自动检测 vllm-mlx 通过名称模式自动检测多模态模型： - 包含 "VL"、"Vision"、"vision" - 包含 "llava"、"idefics"、"paligemma" - 包含 "pixtral"、"molmo"、"deepseek-vl" - 包含 "MedGemma"、"Gemma-3"、"Gemma-4"（多模态变体） ## 使用模型 ### 从 HuggingFace 加载 ```bash vllm-mlx serve mlx-community/Llama-3.2-3B-Instruct-4bit ``` ### 本地路径 ```bash vllm-mlx serve /path/to/local/model ``` ## 查找模型 按以下关键词筛选 mlx-community 模型： - **LLM**：`Llama`、`Qwen`、`Mistral`、`Phi`、`Gemma`、`DeepSeek`、`GLM`、`Kimi`、`Granite`、`Nemotron` - **VLM**：`-VL-`、`llava`、`paligemma`、`pixtral`、`molmo`、`idefics`、`deepseek-vl`、`MedGemma` - **Embedding**：`e5`、`bert`、`ModernBERT` - **规格**：`1B`、`3B`、`7B`、`8B`、`70B` - **量化方式**：`4bit`、`8bit`、`bf16` # Module `examples.audio_separation_example` Audio Separation Example - Isolate voice from background using SAM-Audio SAM-Audio uses text-guided source separation to isolate specific sounds. Usage: python examples/audio_separation_example.py input.mp3 python examples/audio_separation_example.py input.mp3 --description "music" python examples/audio_separation_example.py input.mp3 -o voice.wav Models: - mlx-community/sam-audio-large-fp16 (best quality, 3B params) - mlx-community/sam-audio-small-fp16 (faster, 0.6B params) Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/audio_separation_example.py#L1-L120 ## `examples.audio_separation_example.main` - Kind: function - Signature: `def main()` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/audio_separation_example.py#L25-L116 - Implementation: Function `main` calls `argparse.ArgumentParser`, `parser.add_argument`, `parser.parse_args`, `print`; returns `None`. Function `main` calls `argparse.ArgumentParser`, `parser.add_argument`, `parser.parse_args`, `print`; returns `None`. - Inputs: none - Return annotation: `not annotated` - Calls: argparse.ArgumentParser, parser.add_argument, parser.parse_args, print, os.path.exists, os.path.splitext, time.time, AudioProcessor, processor.load, processor.separate, processor.save, os.system - Return expressions: None # Module `examples.benchmark_all_models` Benchmark all text models for README. Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/benchmark_all_models.py#L1-L148 ## `examples.benchmark_all_models.benchmark_model` - Kind: function - Signature: `def benchmark_model(model_name: str)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/benchmark_all_models.py#L7-L107 - Implementation: Function `benchmark_model` calls `SamplingParams`, `print`, `load`, `format_prompt`; returns `{'model': model_name.split('/')[-1], 'single_tps': single_tps, 'batch_tps': batch_tps, 'speedup': speedup, 'ttft_ms': t…`. Benchmark a single model and return results. - Inputs: - `model_name` (str; required): Required positional or keyword input. - Return annotation: `not annotated` - Calls: SamplingParams, print, load, format_prompt, EngineConfig, SchedulerConfig, EngineCore, time.perf_counter, engine.generate_batch_sync, single_times.append, single_tokens.append, sum, engine.scheduler.reset, model_name.split, engine.close - Return expressions: {'model': model_name.split('/')[-1], 'single_tps': single_tps, 'batch_tps': batch_tps, 'speedup': speedup, 'ttft_ms': t… ## `examples.benchmark_all_models.benchmark_model.format_prompt` - Kind: nested function - Signature: `def format_prompt(p)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/benchmark_all_models.py#L29-L34 - Implementation: Nested Function `benchmark_model.format_prompt` calls `tokenizer.apply_chat_template`; returns `tokenizer.apply_chat_template([{'role': 'user', 'content': p}], tokenize=False, add_generation_prompt=True)`. Nested Function `benchmark_model.format_prompt` calls `tokenizer.apply_chat_template`; returns `tokenizer.apply_chat_template([{'role': 'user', 'content': p}], tokenize=False, add_generation_prompt=True)`. - Inputs: - `p` (not annotated; required): Required positional or keyword input. - Return annotation: `not annotated` - Calls: tokenizer.apply_chat_template - Return expressions: tokenizer.apply_chat_template([{'role': 'user', 'content': p}], tokenize=False, add_generation_prompt=True) ## `examples.benchmark_all_models.main` - Kind: function - Signature: `def main()` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/benchmark_all_models.py#L110-L144 - Implementation: Function `main` calls `benchmark_model`, `results.append`, `print`, `traceback.print_exc`. Function `main` calls `benchmark_model`, `results.append`, `print`, `traceback.print_exc`. - Inputs: none - Return annotation: `not annotated` - Calls: benchmark_model, results.append, print, traceback.print_exc # Module `examples.benchmark_audio` Audio benchmarks for vllm-mlx. Benchmarks STT (Speech-to-Text), TTS (Text-to-Speech), and audio processing. Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/benchmark_audio.py#L1-L332 ## `examples.benchmark_audio.generate_test_audio` - Kind: function - Signature: `def generate_test_audio(duration_seconds: float=5.0) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/benchmark_audio.py#L36-L61 - Implementation: Function `generate_test_audio` calls `np.linspace`, `int`, `np.sin`, `(audio * 32767).astype`; returns `path`. Generate a simple test audio file using TTS. - Inputs: - `duration_seconds` (float; optional; default `5.0`): Optional positional or keyword input; defaults to `5.0`. - Return annotation: `str` - Calls: np.linspace, int, np.sin, (audio * 32767).astype, tempfile.mkstemp, wave.open, f.setnchannels, f.setsampwidth, f.setframerate, f.writeframes, audio.tobytes, os.close - Return expressions: path ## `examples.benchmark_audio.benchmark_tts` - Kind: function - Signature: `def benchmark_tts(model_name: str, alias: str, texts: list[str], voice: str='af_heart')` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/benchmark_audio.py#L64-L124 - Implementation: Function `benchmark_tts` calls `print`, `time.time`, `TTSEngine`, `engine.load`; returns `{'model': alias, 'load_time': load_time, 'avg_chars_per_sec': avg_chars_per_sec, 'avg_rtf': avg_rtf}`. Benchmark TTS model. - Inputs: - `model_name` (str; required): Required positional or keyword input. - `alias` (str; required): Required positional or keyword input. - `texts` (list[str]; required): Required positional or keyword input. - `voice` (str; optional; default `'af_heart'`): Optional positional or keyword input; defaults to `'af_heart'`. - Return annotation: `not annotated` - Calls: print, time.time, TTSEngine, engine.load, enumerate, len, engine.generate, results.append, sum - Return expressions: {'model': alias, 'load_time': load_time, 'avg_chars_per_sec': avg_chars_per_sec, 'avg_rtf': avg_rtf} ## `examples.benchmark_audio.get_audio_duration` - Kind: function - Signature: `def get_audio_duration(audio_path: str) -> float` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/benchmark_audio.py#L127-L159 - Implementation: Function `get_audio_duration` calls `audio_path.endswith`, `contextlib.closing`, `wave.open`, `f.getnframes`; has 3 explicit return paths. Get audio duration in seconds. - Inputs: - `audio_path` (str; required): Required positional or keyword input. - Return annotation: `float` - Calls: audio_path.endswith, contextlib.closing, wave.open, f.getnframes, f.getframerate, float, subprocess.run, result.stdout.strip - Return expressions: frames / float(rate); float(result.stdout.strip()); 0.0 ## `examples.benchmark_audio.benchmark_stt` - Kind: function - Signature: `def benchmark_stt(model_name: str, alias: str, audio_path: str)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/benchmark_audio.py#L162-L212 - Implementation: Function `benchmark_stt` calls `print`, `get_audio_duration`, `time.time`, `STTEngine`; returns `{'model': alias, 'load_time': load_time, 'audio_duration': duration, 'trans_time': trans_time, 'rtf': rtf}`. Benchmark STT model. - Inputs: - `model_name` (str; required): Required positional or keyword input. - `alias` (str; required): Required positional or keyword input. - `audio_path` (str; required): Required positional or keyword input. - Return annotation: `not annotated` - Calls: print, get_audio_duration, time.time, STTEngine, engine.load, engine.transcribe, len - Return expressions: {'model': alias, 'load_time': load_time, 'audio_duration': duration, 'trans_time': trans_time, 'rtf': rtf} ## `examples.benchmark_audio.check_whisper_backend` - Kind: function - Signature: `def check_whisper_backend()` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/benchmark_audio.py#L215-L227 - Implementation: Function `check_whisper_backend` calls `str`; has 2 explicit return paths. Check whether the Whisper backend can be imported. Returns: (available: bool, reason: str) - Inputs: none - Return annotation: `not annotated` - Calls: str - Return expressions: (True, ''); (False, str(e)) ## `examples.benchmark_audio.run_tts_benchmarks` - Kind: function - Signature: `def run_tts_benchmarks()` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/benchmark_audio.py#L230-L257 - Implementation: Function `run_tts_benchmarks` calls `print`, `benchmark_tts`, `results.append`; returns `results`. Run all TTS benchmarks. - Inputs: none - Return annotation: `not annotated` - Calls: print, benchmark_tts, results.append - Return expressions: results ## `examples.benchmark_audio.run_stt_benchmarks` - Kind: function - Signature: `def run_stt_benchmarks(audio_path: str)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/benchmark_audio.py#L260-L295 - Implementation: Function `run_stt_benchmarks` calls `print`, `check_whisper_backend`, `alias.startswith`, `benchmark_stt`; returns `results`. Run all STT benchmarks. - Inputs: - `audio_path` (str; required): Required positional or keyword input. - Return annotation: `not annotated` - Calls: print, check_whisper_backend, alias.startswith, benchmark_stt, results.append - Return expressions: results ## `examples.benchmark_audio.main` - Kind: function - Signature: `def main()` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/benchmark_audio.py#L298-L328 - Implementation: Function `main` calls `argparse.ArgumentParser`, `parser.add_argument`, `parser.parse_args`, `print`. Function `main` calls `argparse.ArgumentParser`, `parser.add_argument`, `parser.parse_args`, `print`. - Inputs: none - Return annotation: `not annotated` - Calls: argparse.ArgumentParser, parser.add_argument, parser.parse_args, print, generate_test_audio, run_tts_benchmarks, run_stt_benchmarks, os.path.exists, os.unlink # Module `examples.benchmark_detokenizer` Benchmark: Streaming Detokenizer vs Naive Decode Compares performance of: 1. Old method: tokenizer.decode([token]) for each token 2. New method: StreamingDetokenizer.add_token() + last_segment Run: python examples/benchmark_detokenizer.py Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/benchmark_detokenizer.py#L1-L186 ## `examples.benchmark_detokenizer.benchmark_naive_decode` - Kind: function - Signature: `def benchmark_naive_decode(tokenizer, tokens, iterations=10)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/benchmark_detokenizer.py#L26-L48 - Implementation: Function `benchmark_naive_decode` calls `range`, `time.perf_counter`, `tokenizer.decode`, `texts.append`; returns `{'method': 'naive_decode', 'mean_ms': statistics.mean(times) * 1000, 'std_ms': statistics.stdev(times) * 1000 if len(ti…`. Benchmark naive decode approach (old method). - Inputs: - `tokenizer` (not annotated; required): Required positional or keyword input. - `tokens` (not annotated; required): Required positional or keyword input. - `iterations` (not annotated; optional; default `10`): Optional positional or keyword input; defaults to `10`. - Return annotation: `not annotated` - Calls: range, time.perf_counter, tokenizer.decode, texts.append, times.append, statistics.mean, len, statistics.stdev, min, max - Return expressions: {'method': 'naive_decode', 'mean_ms': statistics.mean(times) * 1000, 'std_ms': statistics.stdev(times) * 1000 if len(ti… ## `examples.benchmark_detokenizer.benchmark_streaming_detokenizer` - Kind: function - Signature: `def benchmark_streaming_detokenizer(tokenizer, tokens, detokenizer_class, iterations=10)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/benchmark_detokenizer.py#L51-L78 - Implementation: Function `benchmark_streaming_detokenizer` calls `range`, `detokenizer_class`, `detok.reset`, `time.perf_counter`; returns `{'method': detokenizer_class.__name__, 'mean_ms': statistics.mean(times) * 1000, 'std_ms': statistics.stdev(times) * 10…`. Benchmark streaming detokenizer approach (new method). - Inputs: - `tokenizer` (not annotated; required): Required positional or keyword input. - `tokens` (not annotated; required): Required positional or keyword input. - `detokenizer_class` (not annotated; required): Required positional or keyword input. - `iterations` (not annotated; optional; default `10`): Optional positional or keyword input; defaults to `10`. - Return annotation: `not annotated` - Calls: range, detokenizer_class, detok.reset, time.perf_counter, detok.add_token, detok.finalize, times.append, statistics.mean, len, statistics.stdev, min, max - Return expressions: {'method': detokenizer_class.__name__, 'mean_ms': statistics.mean(times) * 1000, 'std_ms': statistics.stdev(times) * 10… ## `examples.benchmark_detokenizer.main` - Kind: function - Signature: `def main()` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/benchmark_detokenizer.py#L81-L182 - Implementation: Function `main` calls `print`, `Path`, `snapshot_download`, `load_tokenizer`. Function `main` calls `print`, `Path`, `snapshot_download`, `load_tokenizer`. - Inputs: none - Return annotation: `not annotated` - Calls: print, Path, snapshot_download, load_tokenizer, AutoTokenizer.from_pretrained, type, raw_tokenizer.encode, len, benchmark_naive_decode, benchmark_streaming_detokenizer, float, results.append, statistics.mean, raw_tokenizer.decode, tokenizer_wrapper._detokenizer_class, detok.reset, detok.add_token, detok.finalize, repr # Module `examples.closed_captions` Closed Captions (CC) - Real-time Subtitles Ultra low-latency transcription for live subtitles/closed captions. Small chunks, fast processing, continuous output. Usage: python examples/closed_captions.py python examples/closed_captions.py --language es Requirements: pip install sounddevice soundfile numpy Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/closed_captions.py#L1-L166 ## `examples.closed_captions.ClosedCaptions` - Kind: class - Signature: `class ClosedCaptions` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/closed_captions.py#L40-L145 - Implementation: Class `ClosedCaptions` declares 7 direct member(s). Real-time closed captions. - Inputs: - `model_name` (str; required): Required positional or keyword input. - `language` (str; optional; default `None`): Optional positional or keyword input; defaults to `None`. - `chunk_sec` (float; optional; default `1.5`): Optional positional or keyword input; defaults to `1.5`. - Constructs: `examples.closed_captions.ClosedCaptions` ## `examples.closed_captions.ClosedCaptions.__init__` - Kind: method - Signature: `def __init__(self, model_name: str, language: str=None, chunk_sec: float=1.5)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/closed_captions.py#L43-L55 - Implementation: Method `ClosedCaptions.__init__` updates `self.model_name`, `self.language`, `self.chunk_sec`, `self.chunk_samples`; calls `int`, `queue.Queue`. Method `ClosedCaptions.__init__` updates `self.model_name`, `self.language`, `self.chunk_sec`, `self.chunk_samples`; calls `int`, `queue.Queue`. - Inputs: - `model_name` (str; required): Required positional or keyword input. - `language` (str; optional; default `None`): Optional positional or keyword input; defaults to `None`. - `chunk_sec` (float; optional; default `1.5`): Optional positional or keyword input; defaults to `1.5`. - Return annotation: `not annotated` - Calls: int, queue.Queue - State writes: self.model_name, self.language, self.chunk_sec, self.chunk_samples, self.audio_queue, self.running, self.engine, self.current_line, self.lines ## `examples.closed_captions.ClosedCaptions.load_model` - Kind: method - Signature: `def load_model(self)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/closed_captions.py#L57-L60 - Implementation: Method `ClosedCaptions.load_model` updates `self.engine`; calls `STTEngine`, `self.engine.load`. Method `ClosedCaptions.load_model` updates `self.engine`; calls `STTEngine`, `self.engine.load`. - Inputs: none - Return annotation: `not annotated` - Calls: STTEngine, self.engine.load - State reads: self.model_name, self.engine.load, self.engine - State writes: self.engine ## `examples.closed_captions.ClosedCaptions.audio_callback` - Kind: method - Signature: `def audio_callback(self, indata, frames, time_info, status)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/closed_captions.py#L62-L64 - Implementation: Method `ClosedCaptions.audio_callback` calls `self.audio_queue.put`, `indata.copy().flatten`, `indata.copy`. Method `ClosedCaptions.audio_callback` calls `self.audio_queue.put`, `indata.copy().flatten`, `indata.copy`. - Inputs: - `indata` (not annotated; required): Required positional or keyword input. - `frames` (not annotated; required): Required positional or keyword input. - `time_info` (not annotated; required): Required positional or keyword input. - `status` (not annotated; required): Required positional or keyword input. - Return annotation: `not annotated` - Calls: self.audio_queue.put, indata.copy().flatten, indata.copy - State reads: self.running, self.audio_queue.put, self.audio_queue ## `examples.closed_captions.ClosedCaptions.transcribe` - Kind: method - Signature: `def transcribe(self, audio)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/closed_captions.py#L66-L75 - Implementation: Method `ClosedCaptions.transcribe` calls `tempfile.NamedTemporaryFile`, `sf.write`, `self.engine.transcribe`, `result.text.strip`; returns `result.text.strip()`. Method `ClosedCaptions.transcribe` calls `tempfile.NamedTemporaryFile`, `sf.write`, `self.engine.transcribe`, `result.text.strip`; returns `result.text.strip()`. - Inputs: - `audio` (not annotated; required): Required positional or keyword input. - Return annotation: `not annotated` - Calls: tempfile.NamedTemporaryFile, sf.write, self.engine.transcribe, result.text.strip, os.unlink - State reads: self.engine.transcribe, self.engine, self.language - Return expressions: result.text.strip() ## `examples.closed_captions.ClosedCaptions.display_caption` - Kind: method - Signature: `def display_caption(self, text)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/closed_captions.py#L77-L83 - Implementation: Method `ClosedCaptions.display_caption` calls `print`; returns `None`. Display caption like subtitles. - Inputs: - `text` (not annotated; required): Required positional or keyword input. - Return annotation: `not annotated` - Calls: print - Return expressions: None ## `examples.closed_captions.ClosedCaptions.process_loop` - Kind: method - Signature: `def process_loop(self)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/closed_captions.py#L85-L109 - Implementation: Method `ClosedCaptions.process_loop` calls `np.array`, `self.audio_queue.get`, `np.concatenate`, `len`. Process audio continuously. - Inputs: none - Return annotation: `not annotated` - Calls: np.array, self.audio_queue.get, np.concatenate, len, np.sqrt, np.mean, self.transcribe, self.display_caption - State reads: self.running, self.audio_queue.get, self.audio_queue, self.chunk_samples, self.transcribe, self.display_caption ## `examples.closed_captions.ClosedCaptions.run` - Kind: method - Signature: `def run(self)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/closed_captions.py#L111-L145 - Implementation: Method `ClosedCaptions.run` updates `self.running`; calls `print`, `' 🎬 CLOSED CAPTIONS - Real-time Subtitles'.center`, `self.model_name.split`, `threading.Thread`. Method `ClosedCaptions.run` updates `self.running`; calls `print`, `' 🎬 CLOSED CAPTIONS - Real-time Subtitles'.center`, `self.model_name.split`, `threading.Thread`. - Inputs: none - Return annotation: `not annotated` - Calls: print, ' 🎬 CLOSED CAPTIONS - Real-time Subtitles'.center, self.model_name.split, threading.Thread, processor.start, sd.InputStream, int, time.sleep - State reads: self.chunk_sec, self.model_name.split, self.model_name, self.process_loop, self.audio_callback - State writes: self.running ## `examples.closed_captions.main` - Kind: function - Signature: `def main()` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/closed_captions.py#L148-L162 - Implementation: Function `main` calls `argparse.ArgumentParser`, `parser.add_argument`, `parser.parse_args`, `MODEL_ALIASES.get`. Function `main` calls `argparse.ArgumentParser`, `parser.add_argument`, `parser.parse_args`, `MODEL_ALIASES.get`. - Inputs: none - Return annotation: `not annotated` - Calls: argparse.ArgumentParser, parser.add_argument, parser.parse_args, MODEL_ALIASES.get, print, ClosedCaptions, cc.load_model, cc.run # Module `examples.demo_openai_image` Demo: OpenAI API - Image Analysis Shows how to use vllm-mlx with the OpenAI Python SDK for image understanding. Usage: 1. Start the server with a VLM model ("vision-model" is the name used in the OpenAI API): vllm-mlx serve --served-model-name vision-model mlx-community/Qwen3-VL-4B-Instruct-3bit --port 8000 2. Run this script: python examples/demo_openai_image.py Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/demo_openai_image.py#L1-L137 # Module `examples.demo_openai_text` Demo: OpenAI API - Text Chat Shows how to use vllm-mlx with the OpenAI Python SDK for text-only chat. Usage: 1. Start the server with any model (served model name is defaulted to "mlx-community/Llama-3.2-3B-Instruct-4bit"): vllm-mlx serve mlx-community/Llama-3.2-3B-Instruct-4bit --port 8000 2. Run this script: python examples/demo_openai_text.py Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/demo_openai_text.py#L1-L123 # Module `examples.demo_openai_video` Demo: OpenAI API - Video Analysis Shows how to use vllm-mlx with the OpenAI Python SDK for video understanding. Usage: 1. Start the server with a VLM model ("video-model" is the name used in the OpenAI API): vllm-mlx serve --served-model-name video-model mlx-community/Qwen3-VL-4B-Instruct-3bit --port 8000 2. Run this script: python examples/demo_openai_video.py Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/demo_openai_video.py#L1-L156 # Module `examples.mcp_chat` Interactive chat with MCP tools. The LLM can use MCP tools (filesystem, etc.) to perform actions. Usage: python examples/mcp_chat.py Example prompts: - "Create a file at /tmp/test.txt with content hello world" - "List files in /tmp" - "Read the file /tmp/test.txt" Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/mcp_chat.py#L1-L187 ## `examples.mcp_chat.get_mcp_tools` - Kind: function - Signature: `def get_mcp_tools()` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/mcp_chat.py#L22-L35 - Implementation: Function `get_mcp_tools` calls `requests.get(f'{BASE_URL}/v1/mcp/tools').json`, `requests.get`, `response.get`, `tools.append`; returns `tools`. Get MCP tools in OpenAI format. - Inputs: none - Return annotation: `not annotated` - Calls: requests.get(f'{BASE_URL}/v1/mcp/tools').json, requests.get, response.get, tools.append - Return expressions: tools ## `examples.mcp_chat.execute_tool` - Kind: function - Signature: `def execute_tool(tool_name: str, arguments: dict)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/mcp_chat.py#L38-L44 - Implementation: Function `execute_tool` calls `requests.post(f'{BASE_URL}/v1/mcp/execute', json={'tool_name': tool_name, 'arguments': arguments}).json`, `requests.post`; returns `response`. Execute an MCP tool. - Inputs: - `tool_name` (str; required): Required positional or keyword input. - `arguments` (dict; required): Required positional or keyword input. - Return annotation: `not annotated` - Calls: requests.post(f'{BASE_URL}/v1/mcp/execute', json={'tool_name': tool_name, 'arguments': arguments}).json, requests.post - Return expressions: response ## `examples.mcp_chat.chat` - Kind: function - Signature: `def chat(messages: list, tools: list)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/mcp_chat.py#L47-L66 - Implementation: Function `chat` calls `requests.post`, `response.json`; has 4 explicit return paths. Send message to LLM with tools. - Inputs: - `messages` (list; required): Required positional or keyword input. - `tools` (list; required): Required positional or keyword input. - Return annotation: `not annotated` - Calls: requests.post, response.json - Return expressions: {'error': f'HTTP {response.status_code}: {response.text[:200]}'}; response.json(); {'error': 'Request timed out'}; {'error': f'Invalid JSON response: {e}'} ## `examples.mcp_chat.main` - Kind: function - Signature: `def main()` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/mcp_chat.py#L69-L183 - Implementation: Function `main` calls `print`, `get_mcp_tools`, `len`, `'\n'.join`; returns `None`. Function `main` calls `print`, `get_mcp_tools`, `len`, `'\n'.join`; returns `None`. - Inputs: none - Return annotation: `not annotated` - Calls: print, get_mcp_tools, len, '\n'.join, input('\nYou: ').strip, input, user_input.lower, messages.append, chat, messages.pop, response.get, choice.get, assistant_message.get, json.loads, execute_tool, result.get, str - Return expressions: None # Module `examples.mcp_tool_use` Example: MCP Tool Use with vllm-mlx This example demonstrates how to use MCP (Model Context Protocol) tools with the vllm-mlx server. Prerequisites: 1. Install MCP support: pip install vllm-mlx[mcp] 2. Create mcp.json config (see example below) 3. Start server with MCP: vllm-mlx serve --mcp-config mcp.json Example mcp.json: { "servers": { "filesystem": { "transport": "stdio", "command": "npx", "args": ["-y", "@modelcontextprotocol/server-filesystem", "/tmp"] } } } Usage: python examples/mcp_tool_use.py Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/mcp_tool_use.py#L1-L176 ## `examples.mcp_tool_use.main` - Kind: function - Signature: `def main()` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/mcp_tool_use.py#L35-L159 - Implementation: Function `main` calls `OpenAI`, `print`, `requests.get(f'{base_url}/health').json`, `requests.get`; returns `None`. Function `main` calls `OpenAI`, `print`, `requests.get(f'{base_url}/health').json`, `requests.get`; returns `None`. - Inputs: none - Return annotation: `not annotated` - Calls: OpenAI, print, requests.get(f'{base_url}/health').json, requests.get, health.get, requests.get(f'{api_base}/mcp/tools').json, tools_response.get, client.chat.completions.create, len, requests.post(f'{api_base}/mcp/execute', json={'tool_name': tool_call.function.name, 'arguments': json.loads(tool_call.…, requests.post, json.loads, result.get, str, messages.append - Return expressions: None ## `examples.mcp_tool_use.list_mcp_servers` - Kind: function - Signature: `def list_mcp_servers()` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/mcp_tool_use.py#L162-L172 - Implementation: Function `list_mcp_servers` calls `requests.get(f'{base_url}/mcp/servers').json`, `requests.get`, `print`, `servers.get`. Helper to list MCP server status. - Inputs: none - Return annotation: `not annotated` - Calls: requests.get(f'{base_url}/mcp/servers').json, requests.get, print, servers.get, server.get # Module `examples.mic_live` Live Speech Transcription - Real-time with Voice Activity Detection Transcribes speech as you talk, detecting when you pause to process audio. Much more natural than fixed-chunk transcription. Usage: python examples/mic_live.py python examples/mic_live.py --model parakeet # Faster for English Requirements: pip install sounddevice soundfile numpy Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/mic_live.py#L1-L245 ## `examples.mic_live.LiveTranscriber` - Kind: class - Signature: `class LiveTranscriber` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/mic_live.py#L42-L201 - Implementation: Class `LiveTranscriber` declares 7 direct member(s). Live transcription with voice activity detection. - Inputs: - `model_name` (str; required): Required positional or keyword input. - `language` (str; optional; default `None`): Optional positional or keyword input; defaults to `None`. - Constructs: `examples.mic_live.LiveTranscriber` ## `examples.mic_live.LiveTranscriber.__init__` - Kind: method - Signature: `def __init__(self, model_name: str, language: str=None)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/mic_live.py#L45-L68 - Implementation: Method `LiveTranscriber.__init__` updates `self.model_name`, `self.language`, `self.silence_threshold`, `self.speech_pad_ms`; calls `deque`, `queue.Queue`. Method `LiveTranscriber.__init__` updates `self.model_name`, `self.language`, `self.silence_threshold`, `self.speech_pad_ms`; calls `deque`, `queue.Queue`. - Inputs: - `model_name` (str; required): Required positional or keyword input. - `language` (str; optional; default `None`): Optional positional or keyword input; defaults to `None`. - Return annotation: `not annotated` - Calls: deque, queue.Queue - State writes: self.model_name, self.language, self.silence_threshold, self.speech_pad_ms, self.min_speech_ms, self.silence_duration_ms, self.audio_buffer, self.is_speaking, self.speech_start, self.last_speech_time, self.pending_audio, self.audio_queue, self.result_queue, self.running, self.engine, self.full_transcript ## `examples.mic_live.LiveTranscriber.load_model` - Kind: method - Signature: `def load_model(self)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/mic_live.py#L70-L76 - Implementation: Method `LiveTranscriber.load_model` updates `self.engine`; calls `print`, `STTEngine`, `self.engine.load`. Load STT model. - Inputs: none - Return annotation: `not annotated` - Calls: print, STTEngine, self.engine.load - State reads: self.model_name, self.engine.load, self.engine - State writes: self.engine ## `examples.mic_live.LiveTranscriber.get_audio_level` - Kind: method - Signature: `def get_audio_level(self, audio)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/mic_live.py#L78-L80 - Implementation: Method `LiveTranscriber.get_audio_level` calls `np.sqrt`, `np.mean`; returns `np.sqrt(np.mean(audio ** 2))`. Get RMS audio level. - Inputs: - `audio` (not annotated; required): Required positional or keyword input. - Return annotation: `not annotated` - Calls: np.sqrt, np.mean - Return expressions: np.sqrt(np.mean(audio ** 2)) ## `examples.mic_live.LiveTranscriber.audio_callback` - Kind: method - Signature: `def audio_callback(self, indata, frames, time_info, status)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/mic_live.py#L82-L85 - Implementation: Method `LiveTranscriber.audio_callback` calls `self.audio_queue.put`, `time.time`, `indata.copy().flatten`, `indata.copy`. Audio input callback. - Inputs: - `indata` (not annotated; required): Required positional or keyword input. - `frames` (not annotated; required): Required positional or keyword input. - `time_info` (not annotated; required): Required positional or keyword input. - `status` (not annotated; required): Required positional or keyword input. - Return annotation: `not annotated` - Calls: self.audio_queue.put, time.time, indata.copy().flatten, indata.copy - State reads: self.running, self.audio_queue.put, self.audio_queue ## `examples.mic_live.LiveTranscriber.transcribe_audio` - Kind: method - Signature: `def transcribe_audio(self, audio)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/mic_live.py#L87-L99 - Implementation: Method `LiveTranscriber.transcribe_audio` calls `tempfile.NamedTemporaryFile`, `sf.write`, `self.engine.transcribe`, `result.text.strip`; returns `result.text.strip()`. Transcribe audio array. - Inputs: - `audio` (not annotated; required): Required positional or keyword input. - Return annotation: `not annotated` - Calls: tempfile.NamedTemporaryFile, sf.write, self.engine.transcribe, result.text.strip, os.unlink - State reads: self.engine.transcribe, self.engine, self.language - Return expressions: result.text.strip() ## `examples.mic_live.LiveTranscriber.process_audio_stream` - Kind: method - Signature: `def process_audio_stream(self)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/mic_live.py#L101-L160 - Implementation: Method `LiveTranscriber.process_audio_stream` updates `self.is_speaking`, `self.speech_start`, `self.last_speech_time`; calls `self.audio_queue.get`, `self.get_audio_level`, `print`, `speech_buffer.extend`. Process audio with VAD. - Inputs: none - Return annotation: `not annotated` - Calls: self.audio_queue.get, self.get_audio_level, print, speech_buffer.extend, np.array, self.transcribe_audio, len, self.full_transcript.append - State reads: self.running, self.audio_queue.get, self.audio_queue, self.get_audio_level, self.silence_threshold, self.is_speaking, self.last_speech_time, self.speech_start, self.silence_duration_ms, self.min_speech_ms, self.transcribe_audio, self.full_transcript.append, self.full_transcript - State writes: self.is_speaking, self.speech_start, self.last_speech_time ## `examples.mic_live.LiveTranscriber.run` - Kind: method - Signature: `def run(self)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/mic_live.py#L162-L201 - Implementation: Method `LiveTranscriber.run` updates `self.running`; calls `print`, `threading.Thread`, `process_thread.start`, `sd.InputStream`; returns `self.full_transcript`. Start live transcription. - Inputs: none - Return annotation: `not annotated` - Calls: print, threading.Thread, process_thread.start, sd.InputStream, int, time.sleep, process_thread.join - State reads: self.process_audio_stream, self.audio_callback, self.full_transcript - State writes: self.running - Return expressions: self.full_transcript ## `examples.mic_live.main` - Kind: function - Signature: `def main()` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/mic_live.py#L204-L241 - Implementation: Function `main` calls `argparse.ArgumentParser`, `parser.add_argument`, `parser.parse_args`, `print`. Function `main` calls `argparse.ArgumentParser`, `parser.add_argument`, `parser.parse_args`, `print`. - Inputs: none - Return annotation: `not annotated` - Calls: argparse.ArgumentParser, parser.add_argument, parser.parse_args, print, MODEL_ALIASES.get, LiveTranscriber, transcriber.load_model, transcriber.run, ' '.join # Module `examples.mic_realtime` Real-Time Microphone Transcription with Whisper - vllm-mlx Transcribes speech in real-time as you speak using your Mac's microphone. Usage: python examples/mic_realtime.py # Default (3s chunks) python examples/mic_realtime.py --chunk 5 # 5 second chunks python examples/mic_realtime.py --model parakeet # Faster model Requirements: pip install sounddevice soundfile numpy Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/mic_realtime.py#L1-L236 ## `examples.mic_realtime.RealtimeTranscriber` - Kind: class - Signature: `class RealtimeTranscriber` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/mic_realtime.py#L45-L171 - Implementation: Class `RealtimeTranscriber` declares 6 direct member(s). Real-time audio transcription using Whisper. - Inputs: - `model_name` (str; required): Required positional or keyword input. - `chunk_duration` (float; optional; default `3.0`): Optional positional or keyword input; defaults to `3.0`. - `language` (str; optional; default `None`): Optional positional or keyword input; defaults to `None`. - Constructs: `examples.mic_realtime.RealtimeTranscriber` ## `examples.mic_realtime.RealtimeTranscriber.__init__` - Kind: method - Signature: `def __init__(self, model_name: str, chunk_duration: float=3.0, language: str=None)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/mic_realtime.py#L48-L60 - Implementation: Method `RealtimeTranscriber.__init__` updates `self.model_name`, `self.chunk_duration`, `self.language`, `self.sample_rate`; calls `queue.Queue`. Method `RealtimeTranscriber.__init__` updates `self.model_name`, `self.chunk_duration`, `self.language`, `self.sample_rate`; calls `queue.Queue`. - Inputs: - `model_name` (str; required): Required positional or keyword input. - `chunk_duration` (float; optional; default `3.0`): Optional positional or keyword input; defaults to `3.0`. - `language` (str; optional; default `None`): Optional positional or keyword input; defaults to `None`. - Return annotation: `not annotated` - Calls: queue.Queue - State writes: self.model_name, self.chunk_duration, self.language, self.sample_rate, self.audio_queue, self.is_recording, self.engine, self.transcriptions ## `examples.mic_realtime.RealtimeTranscriber.load_model` - Kind: method - Signature: `def load_model(self)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/mic_realtime.py#L62-L68 - Implementation: Method `RealtimeTranscriber.load_model` updates `self.engine`; calls `print`, `STTEngine`, `self.engine.load`. Load the STT model. - Inputs: none - Return annotation: `not annotated` - Calls: print, STTEngine, self.engine.load - State reads: self.model_name, self.engine.load, self.engine - State writes: self.engine ## `examples.mic_realtime.RealtimeTranscriber.audio_callback` - Kind: method - Signature: `def audio_callback(self, indata, frames, time_info, status)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/mic_realtime.py#L70-L75 - Implementation: Method `RealtimeTranscriber.audio_callback` calls `print`, `self.audio_queue.put`, `indata.copy`. Callback for audio input stream. - Inputs: - `indata` (not annotated; required): Required positional or keyword input. - `frames` (not annotated; required): Required positional or keyword input. - `time_info` (not annotated; required): Required positional or keyword input. - `status` (not annotated; required): Required positional or keyword input. - Return annotation: `not annotated` - Calls: print, self.audio_queue.put, indata.copy - State reads: self.is_recording, self.audio_queue.put, self.audio_queue ## `examples.mic_realtime.RealtimeTranscriber.transcribe_chunk` - Kind: method - Signature: `def transcribe_chunk(self, audio_data)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/mic_realtime.py#L77-L90 - Implementation: Method `RealtimeTranscriber.transcribe_chunk` calls `tempfile.NamedTemporaryFile`, `sf.write`, `self.engine.transcribe`, `result.text.strip`; returns `result.text.strip()`. Transcribe a chunk of audio. - Inputs: - `audio_data` (not annotated; required): Required positional or keyword input. - Return annotation: `not annotated` - Calls: tempfile.NamedTemporaryFile, sf.write, self.engine.transcribe, result.text.strip, os.unlink - State reads: self.sample_rate, self.engine.transcribe, self.engine, self.language - Return expressions: result.text.strip() ## `examples.mic_realtime.RealtimeTranscriber.process_audio` - Kind: method - Signature: `def process_audio(self)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/mic_realtime.py#L92-L129 - Implementation: Method `RealtimeTranscriber.process_audio` calls `int`, `np.array`, `self.audio_queue.empty`, `self.audio_queue.get`. Process audio chunks in real-time. - Inputs: none - Return annotation: `not annotated` - Calls: int, np.array, self.audio_queue.empty, self.audio_queue.get, np.concatenate, data.flatten, len, np.abs(chunk).max, np.abs, self.transcribe_chunk, self.transcriptions.append, print, np.abs(buffer).max - State reads: self.chunk_duration, self.sample_rate, self.is_recording, self.audio_queue.empty, self.audio_queue, self.audio_queue.get, self.transcribe_chunk, self.transcriptions.append, self.transcriptions ## `examples.mic_realtime.RealtimeTranscriber.run` - Kind: method - Signature: `def run(self)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/mic_realtime.py#L131-L171 - Implementation: Method `RealtimeTranscriber.run` updates `self.is_recording`; calls `print`, `sd.InputStream`, `int`, `threading.Thread`; returns `self.transcriptions`. Start real-time transcription. - Inputs: none - Return annotation: `not annotated` - Calls: print, sd.InputStream, int, threading.Thread, process_thread.start, time.sleep, process_thread.join - State reads: self.chunk_duration, self.sample_rate, self.audio_callback, self.process_audio, self.transcriptions - State writes: self.is_recording - Return expressions: self.transcriptions ## `examples.mic_realtime.main` - Kind: function - Signature: `def main()` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/mic_realtime.py#L174-L232 - Implementation: Function `main` calls `argparse.ArgumentParser`, `parser.add_argument`, `parser.parse_args`, `print`; returns `None`. Function `main` calls `argparse.ArgumentParser`, `parser.add_argument`, `parser.parse_args`, `print`; returns `None`. - Inputs: none - Return annotation: `not annotated` - Calls: argparse.ArgumentParser, parser.add_argument, parser.parse_args, print, MODEL_ALIASES.items, MODEL_ALIASES.get, RealtimeTranscriber, transcriber.load_model, transcriber.run, ' '.join - Return expressions: None # Module `examples.mic_transcribe` Live Microphone Transcription with Whisper - vllm-mlx Records audio from your Mac's microphone and transcribes it using Whisper. Usage: python examples/mic_transcribe.py # Record until Enter python examples/mic_transcribe.py --duration 5 # Record for 5 seconds python examples/mic_transcribe.py --model whisper-small # Use smaller model python examples/mic_transcribe.py --continuous # Continuous mode Requirements: pip install sounddevice soundfile Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/mic_transcribe.py#L1-L221 ## `examples.mic_transcribe.record_audio` - Kind: function - Signature: `def record_audio(duration=None, sample_rate=16000)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/mic_transcribe.py#L39-L93 - Implementation: Function `record_audio` calls `print`, `sd.rec`, `int`, `sd.wait`; returns `(audio.flatten(), sample_rate)`. Record audio from microphone. Args: duration: Recording duration in seconds. If None, records until Enter. sample_rate: Audio sample rate (16000 Hz for Whisper) Returns: numpy array of audio data - Inputs: - `duration` (not annotated; optional; default `None`): Recording duration in seconds. If None, records until Enter. - `sample_rate` (not annotated; optional; default `16000`): Audio sample rate (16000 Hz for Whisper) - Return annotation: `not annotated` - Calls: print, sd.rec, int, sd.wait, threading.Event, threading.Thread, enter_thread.start, stop_recording.is_set, chunks.append, len, np.concatenate, np.array, audio.flatten - Return expressions: (audio.flatten(), sample_rate) ## `examples.mic_transcribe.record_audio.wait_for_enter` - Kind: nested function - Signature: `def wait_for_enter()` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/mic_transcribe.py#L71-L73 - Implementation: Nested Function `record_audio.wait_for_enter` calls `input`, `stop_recording.set`. Nested Function `record_audio.wait_for_enter` calls `input`, `stop_recording.set`. - Inputs: none - Return annotation: `not annotated` - Calls: input, stop_recording.set ## `examples.mic_transcribe.save_audio` - Kind: function - Signature: `def save_audio(audio, sample_rate, path)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/mic_transcribe.py#L96-L99 - Implementation: Function `save_audio` calls `sf.write`. Save audio to WAV file. - Inputs: - `audio` (not annotated; required): Required positional or keyword input. - `sample_rate` (not annotated; required): Required positional or keyword input. - `path` (not annotated; required): Required positional or keyword input. - Return annotation: `not annotated` - Calls: sf.write ## `examples.mic_transcribe.main` - Kind: function - Signature: `def main()` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/mic_transcribe.py#L102-L217 - Implementation: Function `main` calls `argparse.ArgumentParser`, `parser.add_argument`, `parser.parse_args`, `print`; returns `None`. Function `main` calls `argparse.ArgumentParser`, `parser.add_argument`, `parser.parse_args`, `print`; returns `None`. - Inputs: none - Return annotation: `not annotated` - Calls: argparse.ArgumentParser, parser.add_argument, parser.parse_args, print, sd.query_devices, MODEL_ALIASES.items, MODEL_ALIASES.get, STTEngine, engine.load, record_audio, len, tempfile.NamedTemporaryFile, save_audio, engine.transcribe, os.unlink - Return expressions: None # Module `examples.mllm_benchmark` MLLM Benchmark Script for vllm-mlx Tests Multimodal Language Models with real images of dogs from Wikimedia Commons at different resolutions and measures performance metrics. Usage: # Start the MLLM server first: python -m vllm_mlx.server --model mlx-community/Qwen3-VL-4B-Instruct-3bit --port 8000 # Run benchmark: python examples/mllm_benchmark.py # Or specify server URL: python examples/mllm_benchmark.py --server-url http://localhost:8000 Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/mllm_benchmark.py#L1-L443 ## `examples.mllm_benchmark.BenchmarkResult` - Kind: class - Signature: `class BenchmarkResult` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/mllm_benchmark.py#L65-L74 - Implementation: Class `BenchmarkResult` declares 0 direct member(s). Result from a single benchmark run. - Inputs: - `resolution` (str; required): Required constructor field. - `width` (int; required): Required constructor field. - `height` (int; required): Required constructor field. - `pixels` (int; required): Required constructor field. - `time_seconds` (float; required): Required constructor field. - `tokens_generated` (int; required): Required constructor field. - `tokens_per_second` (float; required): Required constructor field. - `response_preview` (str; required): Required constructor field. - Constructs: `examples.mllm_benchmark.BenchmarkResult` - Decorators: dataclass ## `examples.mllm_benchmark.download_image` - Kind: function - Signature: `def download_image(url: str, timeout: int=30) -> Image.Image` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/mllm_benchmark.py#L77-L84 - Implementation: Function `download_image` calls `requests.get`, `response.raise_for_status`, `Image.open`, `io.BytesIO`; returns `Image.open(io.BytesIO(response.content))`. Download image from URL and return PIL Image. - Inputs: - `url` (str; required): Required positional or keyword input. - `timeout` (int; optional; default `30`): Optional positional or keyword input; defaults to `30`. - Return annotation: `Image.Image` - Calls: requests.get, response.raise_for_status, Image.open, io.BytesIO - Return expressions: Image.open(io.BytesIO(response.content)) ## `examples.mllm_benchmark.resize_image` - Kind: function - Signature: `def resize_image(img: Image.Image, width: int, height: int) -> Image.Image` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/mllm_benchmark.py#L87-L89 - Implementation: Function `resize_image` calls `img.resize`; returns `img.resize((width, height), Image.Resampling.LANCZOS)`. Resize image to specified dimensions. - Inputs: - `img` (Image.Image; required): Required positional or keyword input. - `width` (int; required): Required positional or keyword input. - `height` (int; required): Required positional or keyword input. - Return annotation: `Image.Image` - Calls: img.resize - Return expressions: img.resize((width, height), Image.Resampling.LANCZOS) ## `examples.mllm_benchmark.image_to_base64` - Kind: function - Signature: `def image_to_base64(img: Image.Image, format: str='JPEG') -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/mllm_benchmark.py#L92-L106 - Implementation: Function `image_to_base64` calls `Image.new`, `background.paste`, `img.split`, `img.convert`; returns `f'data:{mime};base64,{b64}'`. Convert PIL Image to base64 data URL. - Inputs: - `img` (Image.Image; required): Required positional or keyword input. - `format` (str; optional; default `'JPEG'`): Optional positional or keyword input; defaults to `'JPEG'`. - Return annotation: `str` - Calls: Image.new, background.paste, img.split, img.convert, io.BytesIO, img.save, base64.b64encode(buffer.getvalue()).decode, base64.b64encode, buffer.getvalue - Return expressions: f'data:{mime};base64,{b64}' ## `examples.mllm_benchmark.run_mllm_request` - Kind: function - Signature: `def run_mllm_request(server_url: str, image_b64: str, prompt: str='Describe this image in detail. What do you see?', max_tokens: int=256, model: str='default') -> tuple[str, float, int]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/mllm_benchmark.py#L109-L153 - Implementation: Function `run_mllm_request` calls `time.perf_counter`, `requests.post`, `response.raise_for_status`, `response.json`; returns `(text, elapsed, tokens)`. Send an MLLM request to the server. Returns: (response_text, time_seconds, tokens_generated) - Inputs: - `server_url` (str; required): Required positional or keyword input. - `image_b64` (str; required): Required positional or keyword input. - `prompt` (str; optional; default `'Describe this image in detail. What do you see?'`): Optional positional or keyword input; defaults to `'Describe this image in detail. What do you see?'`. - `max_tokens` (int; optional; default `256`): Optional positional or keyword input; defaults to `256`. - `model` (str; optional; default `'default'`): Optional positional or keyword input; defaults to `'default'`. - Return annotation: `tuple[str, float, int]` - Calls: time.perf_counter, requests.post, response.raise_for_status, response.json, data.get('usage', {}).get, data.get, len, text.split - Return expressions: (text, elapsed, tokens) ## `examples.mllm_benchmark.benchmark_resolution` - Kind: function - Signature: `def benchmark_resolution(server_url: str, base_image: Image.Image, width: int, height: int, model: str, warmup: bool=False) -> BenchmarkResult` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/mllm_benchmark.py#L156-L198 - Implementation: Function `benchmark_resolution` calls `resize_image`, `image_to_base64`, `print`, `run_mllm_request`; returns `BenchmarkResult(resolution=resolution_name, width=width, height=height, pixels=pixels, time_seconds=elapsed, tokens_gen…`. Run benchmark for a specific resolution. - Inputs: - `server_url` (str; required): Required positional or keyword input. - `base_image` (Image.Image; required): Required positional or keyword input. - `width` (int; required): Required positional or keyword input. - `height` (int; required): Required positional or keyword input. - `model` (str; required): Required positional or keyword input. - `warmup` (bool; optional; default `False`): Optional positional or keyword input; defaults to `False`. - Return annotation: `BenchmarkResult` - Calls: resize_image, image_to_base64, print, run_mllm_request, BenchmarkResult, len - Return expressions: BenchmarkResult(resolution=resolution_name, width=width, height=height, pixels=pixels, time_seconds=elapsed, tokens_gen… ## `examples.mllm_benchmark.run_benchmark` - Kind: function - Signature: `def run_benchmark(server_url: str='http://localhost:8000', resolutions: list[tuple[int, int]]=None, warmup_runs: int=1, image_url: str=None) -> list[BenchmarkResult]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/mllm_benchmark.py#L201-L293 - Implementation: Function `run_benchmark` calls `print`, `requests.get`, `health.raise_for_status`, `health.json`; has 2 explicit return paths. Run full MLLM benchmark across multiple resolutions. Args: server_url: URL of the vllm-mlx server resolutions: List of (width, height) tuples to test warmup_runs: Number of warmup runs before measuring image_url: URL of image to use (default: dog from Wikimedia) Returns: List of BenchmarkResult objects - Inputs: - `server_url` (str; optional; default `'http://localhost:8000'`): URL of the vllm-mlx server - `resolutions` (list[tuple[int, int]]; optional; default `None`): List of (width, height) tuples to test - `warmup_runs` (int; optional; default `1`): Number of warmup runs before measuring - `image_url` (str; optional; default `None`): URL of image to use (default: dog from Wikimedia) - Return annotation: `list[BenchmarkResult]` - Calls: print, requests.get, health.raise_for_status, health.json, health_data.get, download_image, range, benchmark_resolution, len, results.append - Return expressions: []; results ## `examples.mllm_benchmark.print_results` - Kind: function - Signature: `def print_results(results: list[BenchmarkResult])` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/mllm_benchmark.py#L296-L338 - Implementation: Function `print_results` calls `print`, `table_data.append`, `tabulate`, `sum`; returns `None`. Print benchmark results in a nice table. - Inputs: - `results` (list[BenchmarkResult]; required): Required positional or keyword input. - Return annotation: `not annotated` - Calls: print, table_data.append, tabulate, sum, min, max - Return expressions: None ## `examples.mllm_benchmark.save_results` - Kind: function - Signature: `def save_results(results: list[BenchmarkResult], output_path: str)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/mllm_benchmark.py#L341-L364 - Implementation: Function `save_results` calls `time.strftime`, `open`, `json.dump`, `print`. Save benchmark results to JSON file. - Inputs: - `results` (list[BenchmarkResult]; required): Required positional or keyword input. - `output_path` (str; required): Required positional or keyword input. - Return annotation: `not annotated` - Calls: time.strftime, open, json.dump, print ## `examples.mllm_benchmark.main` - Kind: function - Signature: `def main()` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/mllm_benchmark.py#L367-L439 - Implementation: Function `main` calls `argparse.ArgumentParser`, `parser.add_argument`, `parser.parse_args`, `run_benchmark`. Function `main` calls `argparse.ArgumentParser`, `parser.add_argument`, `parser.parse_args`, `run_benchmark`. - Inputs: none - Return annotation: `not annotated` - Calls: argparse.ArgumentParser, parser.add_argument, parser.parse_args, run_benchmark, print_results, save_results # Module `examples.mllm_example` Multimodal Language Model (MLLM) example using vllm-mlx. This example demonstrates multimodal inference on Apple Silicon, including image understanding and visual question answering. Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/mllm_example.py#L1-L89 ## `examples.mllm_example.main` - Kind: function - Signature: `def main()` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/mllm_example.py#L16-L85 - Implementation: Function `main` calls `print`, `MLXMultimodalLM`, `mllm.load`, `len`; returns `None`. Function `main` calls `print`, `MLXMultimodalLM`, `mllm.load`, `len`; returns `None`. - Inputs: none - Return annotation: `not annotated` - Calls: print, MLXMultimodalLM, mllm.load, len, mllm.generate, Path(image_path).exists, Path, sys.exit, mllm.describe_image, mllm.answer_about_image - Return expressions: None # Module `examples.simple_generate` Simple text generation example using vllm-mlx. This example demonstrates basic LLM inference on Apple Silicon using the MLX backend. Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/simple_generate.py#L1-L71 ## `examples.simple_generate.main` - Kind: function - Signature: `def main()` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/simple_generate.py#L13-L67 - Implementation: Function `main` calls `print`, `MLXLanguageModel`, `model.load`, `model.generate`. Function `main` calls `print`, `MLXLanguageModel`, `model.load`, `model.generate`. - Inputs: none - Return annotation: `not annotated` - Calls: print, MLXLanguageModel, model.load, model.generate, model.stream_generate, model.chat # Module `examples.test_batch_sync` Test generate_batch_sync() performance. Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/test_batch_sync.py#L1-L105 ## `examples.test_batch_sync.main` - Kind: function - Signature: `def main()` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/test_batch_sync.py#L8-L101 - Implementation: Function `main` calls `print`, `load`, `SamplingParams`, `EngineConfig`. Function `main` calls `print`, `load`, `SamplingParams`, `EngineConfig`. - Inputs: none - Return annotation: `not annotated` - Calls: print, load, SamplingParams, EngineConfig, SchedulerConfig, EngineCore, format_prompt, time.perf_counter, engine.generate_batch_sync, sum, len, asyncio.run, run_async ## `examples.test_batch_sync.main.format_prompt` - Kind: nested function - Signature: `def format_prompt(p)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/test_batch_sync.py#L24-L29 - Implementation: Nested Function `main.format_prompt` calls `tokenizer.apply_chat_template`; returns `tokenizer.apply_chat_template([{'role': 'user', 'content': p}], tokenize=False, add_generation_prompt=True)`. Nested Function `main.format_prompt` calls `tokenizer.apply_chat_template`; returns `tokenizer.apply_chat_template([{'role': 'user', 'content': p}], tokenize=False, add_generation_prompt=True)`. - Inputs: - `p` (not annotated; required): Required positional or keyword input. - Return annotation: `not annotated` - Calls: tokenizer.apply_chat_template - Return expressions: tokenizer.apply_chat_template([{'role': 'user', 'content': p}], tokenize=False, add_generation_prompt=True) ## `examples.test_batch_sync.main.run_async` - Kind: nested function - Signature: `async def run_async()` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/test_batch_sync.py#L71-L93 - Implementation: Nested Function `main.run_async` calls `EngineConfig`, `SchedulerConfig`, `EngineCore`, `engine.start`; awaits asynchronous work; returns `(total_tokens, elapsed)`. Nested Function `main.run_async` calls `EngineConfig`, `SchedulerConfig`, `EngineCore`, `engine.start`; awaits asynchronous work; returns `(total_tokens, elapsed)`. - Inputs: none - Return annotation: `not annotated` - Calls: EngineConfig, SchedulerConfig, EngineCore, engine.start, format_prompt, time.perf_counter, engine.generate, asyncio.gather, sum, engine.stop - Return expressions: (total_tokens, elapsed) # Module `examples.test_batching` Example: Test continuous batching with vllm-mlx. This script demonstrates the continuous batching capability by sending multiple concurrent requests and measuring throughput. Usage: python examples/test_batching.py python examples/test_batching.py --model mlx-community/Qwen2.5-3B-Instruct-4bit python examples/test_batching.py --num-requests 10 --max-tokens 50 Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/test_batching.py#L1-L195 ## `examples.test_batching.run_single_request` - Kind: function - Signature: `async def run_single_request(engine: AsyncEngineCore, request_id: str, prompt: str, sampling_params: SamplingParams) -> dict` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/test_batching.py#L29-L65 - Implementation: Function `run_single_request` calls `time.perf_counter`, `engine.add_request`, `engine.stream_outputs`, `tokens.extend`; awaits asynchronous work; returns `{'request_id': request_id, 'prompt_length': len(prompt.split()), 'num_tokens': len(tokens), 'ttft': ttft, 'total_time':…`. Run a single request and collect timing. - Inputs: - `engine` (AsyncEngineCore; required): Required positional or keyword input. - `request_id` (str; required): Required positional or keyword input. - `prompt` (str; required): Required positional or keyword input. - `sampling_params` (SamplingParams; required): Required positional or keyword input. - Return annotation: `dict` - Calls: time.perf_counter, engine.add_request, engine.stream_outputs, tokens.extend, len, prompt.split - Return expressions: {'request_id': request_id, 'prompt_length': len(prompt.split()), 'num_tokens': len(tokens), 'ttft': ttft, 'total_time':… ## `examples.test_batching.run_concurrent_requests` - Kind: function - Signature: `async def run_concurrent_requests(engine: AsyncEngineCore, prompts: List[str], sampling_params: SamplingParams) -> List[dict]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/test_batching.py#L68-L79 - Implementation: Function `run_concurrent_requests` calls `enumerate`, `run_single_request`, `tasks.append`, `asyncio.gather`; awaits asynchronous work; returns `await asyncio.gather(*tasks)`. Run multiple requests concurrently. - Inputs: - `engine` (AsyncEngineCore; required): Required positional or keyword input. - `prompts` (List[str]; required): Required positional or keyword input. - `sampling_params` (SamplingParams; required): Required positional or keyword input. - Return annotation: `List[dict]` - Calls: enumerate, run_single_request, tasks.append, asyncio.gather - Return expressions: await asyncio.gather(*tasks) ## `examples.test_batching.print_results` - Kind: function - Signature: `def print_results(results: List[dict], total_time: float)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/test_batching.py#L82-L113 - Implementation: Function `print_results` calls `print`, `sum`, `len`. Print benchmark results. - Inputs: - `results` (List[dict]; required): Required positional or keyword input. - `total_time` (float; required): Required positional or keyword input. - Return annotation: `not annotated` - Calls: print, sum, len ## `examples.test_batching.main` - Kind: function - Signature: `async def main()` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/test_batching.py#L116-L191 - Implementation: Function `main` calls `argparse.ArgumentParser`, `parser.add_argument`, `parser.parse_args`, `print`; awaits asynchronous work. Function `main` calls `argparse.ArgumentParser`, `parser.add_argument`, `parser.parse_args`, `print`; awaits asynchronous work. - Inputs: none - Return annotation: `not annotated` - Calls: argparse.ArgumentParser, parser.add_argument, parser.parse_args, print, load, SchedulerConfig, EngineConfig, SamplingParams, len, AsyncEngineCore, asyncio.sleep, time.perf_counter, run_concurrent_requests, print_results # Module `examples.test_openai_compatibility` OpenAI API Compatibility Test Script for vllm-mlx. This script tests the OpenAI API compatibility of the vllm-mlx server. It tests both the direct HTTP API and the official OpenAI Python client. Usage: # First start the server: vllm-mlx serve --served-model-name default mlx-community/Qwen3-VL-4B-Instruct-3bit --port 8000 # Then run this script: python examples/test_openai_compatibility.py # With a different server URL: python examples/test_openai_compatibility.py --server-url http://localhost:9000 # Test only specific endpoints: python examples/test_openai_compatibility.py --test-image --test-video Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/test_openai_compatibility.py#L1-L739 ## `examples.test_openai_compatibility.print_header` - Kind: function - Signature: `def print_header(text: str)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/test_openai_compatibility.py#L37-L41 - Implementation: Function `print_header` calls `print`. Print a section header. - Inputs: - `text` (str; required): Required positional or keyword input. - Return annotation: `not annotated` - Calls: print ## `examples.test_openai_compatibility.print_test` - Kind: function - Signature: `def print_test(name: str, passed: bool, message: str='')` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/test_openai_compatibility.py#L44-L49 - Implementation: Function `print_test` calls `print`. Print test result. - Inputs: - `name` (str; required): Required positional or keyword input. - `passed` (bool; required): Required positional or keyword input. - `message` (str; optional; default `''`): Optional positional or keyword input; defaults to `''`. - Return annotation: `not annotated` - Calls: print ## `examples.test_openai_compatibility.print_warning` - Kind: function - Signature: `def print_warning(text: str)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/test_openai_compatibility.py#L52-L54 - Implementation: Function `print_warning` calls `print`. Print a warning message. - Inputs: - `text` (str; required): Required positional or keyword input. - Return annotation: `not annotated` - Calls: print ## `examples.test_openai_compatibility.create_test_image` - Kind: function - Signature: `def create_test_image() -> tuple[str, bytes]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/test_openai_compatibility.py#L57-L97 - Implementation: Function `create_test_image` calls `Image.new`, `io.BytesIO`, `img.save`, `buffer.getvalue`; has 2 explicit return paths. Create a simple test image and return (path, bytes). - Inputs: none - Return annotation: `tuple[str, bytes]` - Calls: Image.new, io.BytesIO, img.save, buffer.getvalue, tempfile.NamedTemporaryFile, temp_file.write, temp_file.close, print_warning, bytes - Return expressions: (temp_file.name, img_bytes); (temp_file.name, minimal_png) ## `examples.test_openai_compatibility.test_health_endpoint` - Kind: function - Signature: `def test_health_endpoint(server_url: str) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/test_openai_compatibility.py#L100-L109 - Implementation: Function `test_health_endpoint` calls `requests.get`, `print_warning`; has 2 explicit return paths. Test the /health endpoint. - Inputs: - `server_url` (str; required): Required positional or keyword input. - Return annotation: `bool` - Calls: requests.get, print_warning - Return expressions: response.status_code == 200; False ## `examples.test_openai_compatibility.test_models_endpoint` - Kind: function - Signature: `def test_models_endpoint(server_url: str) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/test_openai_compatibility.py#L112-L126 - Implementation: Function `test_models_endpoint` calls `requests.get`, `response.json`, `isinstance`, `print_warning`; has 2 explicit return paths. Test the /v1/models endpoint. - Inputs: - `server_url` (str; required): Required positional or keyword input. - Return annotation: `bool` - Calls: requests.get, response.json, isinstance, print_warning - Return expressions: False; 'data' in data and isinstance(data['data'], list) ## `examples.test_openai_compatibility.test_chat_completions_http` - Kind: function - Signature: `def test_chat_completions_http(server_url: str) -> tuple[bool, str]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/test_openai_compatibility.py#L129-L170 - Implementation: Function `test_chat_completions_http` calls `requests.post`, `response.json`, `len`, `str`; has 7 explicit return paths. Test /v1/chat/completions with direct HTTP. - Inputs: - `server_url` (str; required): Required positional or keyword input. - Return annotation: `tuple[bool, str]` - Calls: requests.post, response.json, len, str - Return expressions: (False, f'Status code: {response.status_code}'); (False, "Missing 'choices' in response"); (False, 'Empty choices array'); (False, "Missing 'message' in choice"); (False, "Missing 'content' in message"); (True, f'Response: {content[:50]}...'); (False, str(e)) ## `examples.test_openai_compatibility.test_chat_completions_openai` - Kind: function - Signature: `def test_chat_completions_openai(server_url: str) -> tuple[bool, str]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/test_openai_compatibility.py#L173-L199 - Implementation: Function `test_chat_completions_openai` calls `OpenAI`, `client.chat.completions.create`, `str`; has 3 explicit return paths. Test /v1/chat/completions with OpenAI Python client. - Inputs: - `server_url` (str; required): Required positional or keyword input. - Return annotation: `tuple[bool, str]` - Calls: OpenAI, client.chat.completions.create, str - Return expressions: (False, 'OpenAI package not installed. Run: pip install openai'); (True, f'Response: {content[:50]}...'); (False, str(e)) ## `examples.test_openai_compatibility.test_completions_endpoint` - Kind: function - Signature: `def test_completions_endpoint(server_url: str) -> tuple[bool, str]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/test_openai_compatibility.py#L202-L233 - Implementation: Function `test_completions_endpoint` calls `requests.post`, `response.json`, `len`, `data['choices'][0].get`; has 5 explicit return paths. Test /v1/completions endpoint (legacy). - Inputs: - `server_url` (str; required): Required positional or keyword input. - Return annotation: `tuple[bool, str]` - Calls: requests.post, response.json, len, data['choices'][0].get, str - Return expressions: (False, f'Status code: {response.status_code}'); (False, "Missing 'choices' in response"); (False, 'Empty choices array'); (True, f'Response: {text[:50]}...'); (False, str(e)) ## `examples.test_openai_compatibility.test_image_chat_http` - Kind: function - Signature: `def test_image_chat_http(server_url: str) -> tuple[bool, str]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/test_openai_compatibility.py#L236-L278 - Implementation: Function `test_image_chat_http` calls `create_test_image`, `base64.b64encode(image_bytes).decode`, `base64.b64encode`, `requests.post`; has 3 explicit return paths. Test multimodal image chat with direct HTTP. - Inputs: - `server_url` (str; required): Required positional or keyword input. - Return annotation: `tuple[bool, str]` - Calls: create_test_image, base64.b64encode(image_bytes).decode, base64.b64encode, requests.post, response.json, str, Path(image_path).unlink, Path - Return expressions: (False, f'Status code: {response.status_code}, Body: {response.text[:100]}'); (True, f'Response: {content[:50]}...'); (False, str(e)) ## `examples.test_openai_compatibility.test_image_chat_openai` - Kind: function - Signature: `def test_image_chat_openai(server_url: str) -> tuple[bool, str]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/test_openai_compatibility.py#L281-L322 - Implementation: Function `test_image_chat_openai` calls `create_test_image`, `base64.b64encode(image_bytes).decode`, `base64.b64encode`, `OpenAI`; has 3 explicit return paths. Test multimodal image chat with OpenAI client. - Inputs: - `server_url` (str; required): Required positional or keyword input. - Return annotation: `tuple[bool, str]` - Calls: create_test_image, base64.b64encode(image_bytes).decode, base64.b64encode, OpenAI, client.chat.completions.create, str, Path(image_path).unlink, Path - Return expressions: (False, 'OpenAI package not installed'); (True, f'Response: {content[:50]}...'); (False, str(e)) ## `examples.test_openai_compatibility.test_image_url_http` - Kind: function - Signature: `def test_image_url_http(server_url: str) -> tuple[bool, str]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/test_openai_compatibility.py#L325-L363 - Implementation: Function `test_image_url_http` calls `requests.post`, `response.json`, `str`; has 3 explicit return paths. Test image from URL. - Inputs: - `server_url` (str; required): Required positional or keyword input. - Return annotation: `tuple[bool, str]` - Calls: requests.post, response.json, str - Return expressions: (False, f'Status code: {response.status_code}'); (True, f'Response: {content[:80]}...'); (False, str(e)) ## `examples.test_openai_compatibility.test_streaming_chat` - Kind: function - Signature: `def test_streaming_chat(server_url: str) -> tuple[bool, str]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/test_openai_compatibility.py#L366-L405 - Implementation: Function `test_streaming_chat` calls `requests.post`, `response.iter_lines`, `line.decode`, `line.startswith`; has 4 explicit return paths. Test streaming chat completions. - Inputs: - `server_url` (str; required): Required positional or keyword input. - Return annotation: `tuple[bool, str]` - Calls: requests.post, response.iter_lines, line.decode, line.startswith, chunks.append, len, str - Return expressions: (False, f'Status code: {response.status_code}'); (False, 'No streaming chunks received'); (True, f'Received {len(chunks)} streaming chunks'); (False, str(e)) ## `examples.test_openai_compatibility.create_test_video` - Kind: function - Signature: `def create_test_video() -> tuple[str, bytes]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/test_openai_compatibility.py#L408-L449 - Implementation: Function `create_test_video` calls `tempfile.NamedTemporaryFile`, `temp_file.close`, `cv2.VideoWriter_fourcc`, `cv2.VideoWriter`; has 2 explicit return paths. Create a simple test video with colored frames. Returns (path, bytes) of a minimal MP4 video. - Inputs: none - Return annotation: `tuple[str, bytes]` - Calls: tempfile.NamedTemporaryFile, temp_file.close, cv2.VideoWriter_fourcc, cv2.VideoWriter, np.zeros, out.write, out.release, open, f.read, print_warning - Return expressions: (temp_path, video_bytes); (None, None) ## `examples.test_openai_compatibility.test_video_chat_http` - Kind: function - Signature: `def test_video_chat_http(server_url: str) -> tuple[bool, str]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/test_openai_compatibility.py#L452-L498 - Implementation: Function `test_video_chat_http` calls `create_test_video`, `base64.b64encode(video_bytes).decode`, `base64.b64encode`, `requests.post`; has 4 explicit return paths. Test multimodal video chat with direct HTTP. - Inputs: - `server_url` (str; required): Required positional or keyword input. - Return annotation: `tuple[bool, str]` - Calls: create_test_video, base64.b64encode(video_bytes).decode, base64.b64encode, requests.post, response.json, str, Path(video_path).unlink, Path - Return expressions: (False, 'Could not create test video (OpenCV required)'); (False, f'Status code: {response.status_code}, Body: {response.text[:100]}'); (True, f'Response: {content[:80]}...'); (False, str(e)) ## `examples.test_openai_compatibility.test_video_chat_openai` - Kind: function - Signature: `def test_video_chat_openai(server_url: str) -> tuple[bool, str]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/test_openai_compatibility.py#L501-L546 - Implementation: Function `test_video_chat_openai` calls `create_test_video`, `base64.b64encode(video_bytes).decode`, `base64.b64encode`, `OpenAI`; has 4 explicit return paths. Test multimodal video chat with OpenAI client. - Inputs: - `server_url` (str; required): Required positional or keyword input. - Return annotation: `tuple[bool, str]` - Calls: create_test_video, base64.b64encode(video_bytes).decode, base64.b64encode, OpenAI, client.chat.completions.create, str, Path(video_path).unlink, Path - Return expressions: (False, 'OpenAI package not installed'); (False, 'Could not create test video (OpenCV required)'); (True, f'Response: {content[:80]}...'); (False, str(e)) ## `examples.test_openai_compatibility.test_video_url_http` - Kind: function - Signature: `def test_video_url_http(server_url: str) -> tuple[bool, str]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/test_openai_compatibility.py#L549-L587 - Implementation: Function `test_video_url_http` calls `requests.post`, `response.json`, `str`; has 3 explicit return paths. Test video from URL. - Inputs: - `server_url` (str; required): Required positional or keyword input. - Return annotation: `tuple[bool, str]` - Calls: requests.post, response.json, str - Return expressions: (False, f'Status code: {response.status_code}'); (True, f'Response: {content[:80]}...'); (False, str(e)) ## `examples.test_openai_compatibility.run_all_tests` - Kind: function - Signature: `def run_all_tests(server_url: str, test_image: bool=True, test_video: bool=True)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/test_openai_compatibility.py#L590-L684 - Implementation: Function `run_all_tests` calls `print_header`, `print`, `test_health_endpoint`, `print_test`; has 2 explicit return paths. Run all compatibility tests. - Inputs: - `server_url` (str; required): Required positional or keyword input. - `test_image` (bool; optional; default `True`): Optional positional or keyword input; defaults to `True`. - `test_video` (bool; optional; default `True`): Optional positional or keyword input; defaults to `True`. - Return annotation: `not annotated` - Calls: print_header, print, test_health_endpoint, print_test, record, test_models_endpoint, test_chat_completions_http, test_chat_completions_openai, test_completions_endpoint, test_streaming_chat, test_image_chat_http, test_image_chat_openai, test_image_url_http, test_video_chat_http, test_video_chat_openai, test_video_url_http - Return expressions: 0; 1 ## `examples.test_openai_compatibility.run_all_tests.record` - Kind: nested function - Signature: `def record(passed: bool)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/test_openai_compatibility.py#L594-L598 - Implementation: Nested Function `run_all_tests.record` contains no state mutation, call, raise, return, await, or yield. Nested Function `run_all_tests.record` contains no state mutation, call, raise, return, await, or yield. - Inputs: - `passed` (bool; required): Required positional or keyword input. - Return annotation: `not annotated` ## `examples.test_openai_compatibility.main` - Kind: function - Signature: `def main()` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/test_openai_compatibility.py#L687-L735 - Implementation: Function `main` calls `argparse.ArgumentParser`, `parser.add_argument`, `parser.parse_args`, `print`; returns `run_all_tests(server_url=args.server_url, test_image=not args.no_image, test_video=not args.no_video)`. Function `main` calls `argparse.ArgumentParser`, `parser.add_argument`, `parser.parse_args`, `print`; returns `run_all_tests(server_url=args.server_url, test_image=not args.no_image, test_video=not args.no_video)`. - Inputs: none - Return annotation: `not annotated` - Calls: argparse.ArgumentParser, parser.add_argument, parser.parse_args, print, test_health_endpoint, sys.exit, run_all_tests - Return expressions: run_all_tests(server_url=args.server_url, test_image=not args.no_image, test_video=not args.no_video) # Module `examples.test_video` Test script for VLM video support in vllm-mlx. This script tests video understanding capabilities by: 1. Downloading a sample video (or using a local one) 2. Loading a VLM model that supports video 3. Running inference on the video Usage: python examples/test_video.py python examples/test_video.py --video /path/to/video.mp4 python examples/test_video.py --model mlx-community/Qwen3-VL-8B-Instruct-4bit Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/test_video.py#L1-L349 ## `examples.test_video.download_sample_video` - Kind: function - Signature: `def download_sample_video() -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/test_video.py#L27-L56 - Implementation: Function `download_sample_video` calls `logger.info`, `requests.get`, `response.raise_for_status`, `tempfile.NamedTemporaryFile`; returns `temp_file.name`. Download a sample video for testing. - Inputs: none - Return annotation: `str` - Calls: logger.info, requests.get, response.raise_for_status, tempfile.NamedTemporaryFile, response.iter_content, temp_file.write, temp_file.close, logger.error, sys.exit - Return expressions: temp_file.name ## `examples.test_video.create_test_video` - Kind: function - Signature: `def create_test_video() -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/test_video.py#L59-L105 - Implementation: Function `create_test_video` calls `logger.error`, `sys.exit`, `logger.info`, `tempfile.NamedTemporaryFile`; returns `temp_file.name`. Create a simple test video using OpenCV if download fails. - Inputs: none - Return annotation: `str` - Calls: logger.error, sys.exit, logger.info, tempfile.NamedTemporaryFile, temp_file.close, cv2.VideoWriter_fourcc, cv2.VideoWriter, range, np.zeros, len, cv2.putText, out.write, out.release - Return expressions: temp_file.name ## `examples.test_video.get_video_info` - Kind: function - Signature: `def get_video_info(video_path: str) -> dict` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/test_video.py#L108-L126 - Implementation: Function `get_video_info` calls `cv2.VideoCapture`, `cap.isOpened`, `int`, `cap.get`; has 2 explicit return paths. Get information about a video file. - Inputs: - `video_path` (str; required): Required positional or keyword input. - Return annotation: `dict` - Calls: cv2.VideoCapture, cap.isOpened, int, cap.get, cap.release - Return expressions: {'error': 'Cannot open video'}; info ## `examples.test_video.test_frame_extraction` - Kind: function - Signature: `def test_frame_extraction(video_path: str)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/test_video.py#L129-L145 - Implementation: Function `test_frame_extraction` calls `logger.info`, `get_video_info`, `time.time`, `extract_video_frames_smart`. Test video frame extraction. - Inputs: - `video_path` (str; required): Required positional or keyword input. - Return annotation: `not annotated` - Calls: logger.info, get_video_info, time.time, extract_video_frames_smart, len ## `examples.test_video.test_video_generation` - Kind: function - Signature: `def test_video_generation(video_path: str, model_name: str)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/test_video.py#L148-L219 - Implementation: Function `test_video_generation` calls `logger.info`, `time.time`, `MLXVisionLanguageModel`, `model.load`; returns `model`. Test video understanding with VLM. - Inputs: - `video_path` (str; required): Required positional or keyword input. - `model_name` (str; required): Required positional or keyword input. - Return annotation: `not annotated` - Calls: logger.info, time.time, MLXVisionLanguageModel, model.load, model.describe_video, model.generate, model.chat - Return expressions: model ## `examples.test_video.test_video_url` - Kind: function - Signature: `def test_video_url(model, video_url: str)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/test_video.py#L222-L266 - Implementation: Function `test_video_url` calls `logger.info`, `time.time`, `model.chat`, `model.generate`; returns `True`. Test video from URL. - Inputs: - `model` (not annotated; required): Required positional or keyword input. - `video_url` (str; required): Required positional or keyword input. - Return annotation: `not annotated` - Calls: logger.info, time.time, model.chat, model.generate - Return expressions: True ## `examples.test_video.main` - Kind: function - Signature: `def main()` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/test_video.py#L269-L345 - Implementation: Function `main` calls `argparse.ArgumentParser`, `parser.add_argument`, `parser.parse_args`, `Path(video_path).exists`. Function `main` calls `argparse.ArgumentParser`, `parser.add_argument`, `parser.parse_args`, `Path(video_path).exists`. - Inputs: none - Return annotation: `not annotated` - Calls: argparse.ArgumentParser, parser.add_argument, parser.parse_args, Path(video_path).exists, Path, logger.error, sys.exit, create_test_video, download_sample_video, logger.info, test_frame_extraction, test_video_generation, MLXVisionLanguageModel, model.load, test_video_url # Module `examples.tts_example` TTS Example - Text to Speech with vllm-mlx Usage: python examples/tts_example.py "Hello, how are you?" python examples/tts_example.py "Welcome!" --voice am_michael python examples/tts_example.py "Hola, como estas?" --lang es python examples/tts_example.py --list-voices python examples/tts_example.py --list-languages Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/tts_example.py#L1-L138 ## `examples.tts_example.main` - Kind: function - Signature: `def main()` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/tts_example.py#L46-L134 - Implementation: Function `main` calls `argparse.ArgumentParser`, `parser.add_argument`, `parser.parse_args`, `print`; returns `None`. Function `main` calls `argparse.ArgumentParser`, `parser.add_argument`, `parser.parse_args`, `print`; returns `None`. - Inputs: none - Return annotation: `not annotated` - Calls: argparse.ArgumentParser, parser.add_argument, parser.parse_args, print, LANGUAGES.items, sorted, LANG_ALIASES.items, args.lang.lower, LANG_ALIASES.get, LANGUAGES.get, TTSEngine, engine.load, engine.get_voices, len, engine.generate, engine.save, os.system - Return expressions: None # Module `examples.tts_multilingual` Multilingual TTS Example - Text to Speech with multiple models and languages Supported Models: - Kokoro: Fast, 82M params, 8 languages (en, es, fr, ja, zh, hi, it, pt) - Chatterbox: Expressive, voice cloning, 15+ languages - VibeVoice: Realtime, low latency, English - VoxCPM: High quality, Chinese/English - OuteTTS: Voice cloning, en/zh/ja/ko - Spark: Voice cloning, en/zh Usage: python examples/tts_multilingual.py "Hello world" python examples/tts_multilingual.py "Hola mundo" --lang es python examples/tts_multilingual.py "Bonjour le monde" --lang fr --model kokoro python examples/tts_multilingual.py --list-models python examples/tts_multilingual.py --list-languages Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/tts_multilingual.py#L1-L340 ## `examples.tts_multilingual.get_best_model_for_language` - Kind: function - Signature: `def get_best_model_for_language(lang: str) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/tts_multilingual.py#L109-L123 - Implementation: Function `get_best_model_for_language` calls `lang.lower`; has 3 explicit return paths. Get the best model for a given language. - Inputs: - `lang` (str; required): Required positional or keyword input. - Return annotation: `str` - Calls: lang.lower - Return expressions: 'kokoro'; 'voxcpm'; 'chatterbox' ## `examples.tts_multilingual.list_models` - Kind: function - Signature: `def list_models()` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/tts_multilingual.py#L126-L138 - Implementation: Function `list_models` calls `print`, `MODELS.items`, `', '.join`, `len`. Print available models. - Inputs: none - Return annotation: `not annotated` - Calls: print, MODELS.items, ', '.join, len ## `examples.tts_multilingual.list_languages` - Kind: function - Signature: `def list_languages()` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/tts_multilingual.py#L141-L152 - Implementation: Function `list_languages` calls `print`, `sorted`, `LANGUAGES.items`, `get_best_model_for_language`. Print available languages and best models. - Inputs: none - Return annotation: `not annotated` - Calls: print, sorted, LANGUAGES.items, get_best_model_for_language, MODELS.items, ', '.join ## `examples.tts_multilingual.generate_speech` - Kind: function - Signature: `def generate_speech(text: str, model_name: str, lang: str, voice: str, speed: float, output: str)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/tts_multilingual.py#L155-L244 - Implementation: Function `generate_speech` calls `print`, `LANGUAGES.get(lang, {}).get`, `LANGUAGES.get`, `time.time`; has 2 explicit return paths. Generate speech using the specified model. - Inputs: - `text` (str; required): Required positional or keyword input. - `model_name` (str; required): Required positional or keyword input. - `lang` (str; required): Required positional or keyword input. - `voice` (str; required): Required positional or keyword input. - `speed` (float; required): Required positional or keyword input. - `output` (str; required): Required positional or keyword input. - Return annotation: `not annotated` - Calls: print, LANGUAGES.get(lang, {}).get, LANGUAGES.get, time.time, load_model, lang_info.get, model.generate, hasattr, np.array, audio_data.tolist, audio_chunks.append, len, np.concatenate, (full_audio * 32767).astype, wave.open, wf.setnchannels, wf.setsampwidth, wf.setframerate, wf.writeframes, audio_int16.tobytes - Return expressions: None; output ## `examples.tts_multilingual.main` - Kind: function - Signature: `def main()` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/tts_multilingual.py#L247-L336 - Implementation: Function `main` calls `argparse.ArgumentParser`, `parser.add_argument`, `parser.parse_args`, `print`; returns `None`. Function `main` calls `argparse.ArgumentParser`, `parser.add_argument`, `parser.parse_args`, `print`; returns `None`. - Inputs: none - Return annotation: `not annotated` - Calls: argparse.ArgumentParser, parser.add_argument, parser.parse_args, print, list_models, list_languages, parser.print_help, get_best_model_for_language, ', '.join, MODELS.keys, generate_speech, os.system - Return expressions: None # Module `examples.video_benchmark` Video Benchmark Script for vllm-mlx Tests Vision-Language Models with video at different configurations (FPS, frame count, resolution) and measures performance metrics. Usage: # Direct API benchmark (no server needed): python examples/video_benchmark.py --model mlx-community/Qwen3-VL-4B-Instruct-3bit # With video URL: python examples/video_benchmark.py --video-url https://example.com/video.mp4 # Quick test: python examples/video_benchmark.py --quick Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/video_benchmark.py#L1-L563 ## `examples.video_benchmark.VideoBenchmarkResult` - Kind: class - Signature: `class VideoBenchmarkResult` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/video_benchmark.py#L80-L91 - Implementation: Class `VideoBenchmarkResult` declares 0 direct member(s). Result from a single video benchmark run. - Inputs: - `config_name` (str; required): Required constructor field. - `fps` (float; required): Required constructor field. - `max_frames` (int; required): Required constructor field. - `frames_extracted` (int; required): Required constructor field. - `video_duration` (float; required): Required constructor field. - `time_seconds` (float; required): Required constructor field. - `prompt_tokens` (int; required): Required constructor field. - `completion_tokens` (int; required): Required constructor field. - `tokens_per_second` (float; required): Required constructor field. - `response_preview` (str; required): Required constructor field. - Constructs: `examples.video_benchmark.VideoBenchmarkResult` - Decorators: dataclass ## `examples.video_benchmark.create_test_video` - Kind: function - Signature: `def create_test_video(duration: float=5.0, fps: float=30.0, width: int=640, height: int=480) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/video_benchmark.py#L94-L175 - Implementation: Function `create_test_video` calls `tempfile.NamedTemporaryFile`, `temp_file.close`, `cv2.VideoWriter_fourcc`, `cv2.VideoWriter`; returns `temp_file.name`. Create a synthetic test video with colored frames and text. Args: duration: Video duration in seconds fps: Frames per second width: Video width height: Video height Returns: Path to created video file - Inputs: - `duration` (float; optional; default `5.0`): Video duration in seconds - `fps` (float; optional; default `30.0`): Frames per second - `width` (int; optional; default `640`): Video width - `height` (int; optional; default `480`): Video height - Return annotation: `str` - Calls: tempfile.NamedTemporaryFile, temp_file.close, cv2.VideoWriter_fourcc, cv2.VideoWriter, int, len, range, np.zeros, min, cv2.putText, out.write, out.release - Return expressions: temp_file.name ## `examples.video_benchmark.download_video` - Kind: function - Signature: `def download_video(url: str, timeout: int=120) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/video_benchmark.py#L178-L198 - Implementation: Function `download_video` calls `logger.info`, `requests.get`, `response.raise_for_status`, `tempfile.NamedTemporaryFile`; returns `temp_file.name`. Download video from URL. - Inputs: - `url` (str; required): Required positional or keyword input. - `timeout` (int; optional; default `120`): Optional positional or keyword input; defaults to `120`. - Return annotation: `str` - Calls: logger.info, requests.get, response.raise_for_status, tempfile.NamedTemporaryFile, response.iter_content, temp_file.write, temp_file.close, Path(temp_file.name).stat, Path - Return expressions: temp_file.name ## `examples.video_benchmark.get_video_info` - Kind: function - Signature: `def get_video_info(video_path: str) -> dict` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/video_benchmark.py#L201-L217 - Implementation: Function `get_video_info` calls `cv2.VideoCapture`, `cap.isOpened`, `int`, `cap.get`; has 2 explicit return paths. Get information about a video file. - Inputs: - `video_path` (str; required): Required positional or keyword input. - Return annotation: `dict` - Calls: cv2.VideoCapture, cap.isOpened, int, cap.get, cap.release - Return expressions: {'error': 'Cannot open video'}; info ## `examples.video_benchmark.run_video_benchmark` - Kind: function - Signature: `def run_video_benchmark(model, video_path: str, fps: float, max_frames: int, config_name: str, warmup: bool=False) -> VideoBenchmarkResult` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/video_benchmark.py#L220-L271 - Implementation: Function `run_video_benchmark` calls `get_video_info`, `print`, `time.perf_counter`, `model.generate`; returns `VideoBenchmarkResult(config_name=config_name, fps=fps, max_frames=max_frames, frames_extracted=frames_extracted, video_…`. Run a single video benchmark configuration. - Inputs: - `model` (not annotated; required): Required positional or keyword input. - `video_path` (str; required): Required positional or keyword input. - `fps` (float; required): Required positional or keyword input. - `max_frames` (int; required): Required positional or keyword input. - `config_name` (str; required): Required positional or keyword input. - `warmup` (bool; optional; default `False`): Optional positional or keyword input; defaults to `False`. - Return annotation: `VideoBenchmarkResult` - Calls: get_video_info, print, time.perf_counter, model.generate, int, min, VideoBenchmarkResult, len - Return expressions: VideoBenchmarkResult(config_name=config_name, fps=fps, max_frames=max_frames, frames_extracted=frames_extracted, video_… ## `examples.video_benchmark.run_benchmark` - Kind: function - Signature: `def run_benchmark(model_name: str, video_path: str=None, video_url: str=None, video_duration: float=10.0, warmup_runs: int=1, quick: bool=False) -> list[VideoBenchmarkResult]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/video_benchmark.py#L274-L374 - Implementation: Function `run_benchmark` calls `print`, `time.time`, `MLXVisionLanguageModel`, `model.load`; returns `results`. Run full video benchmark across multiple configurations. Args: model_name: VLM model to use video_path: Local video file path video_url: URL to download video from video_duration: Duration for synthetic video warmup_runs: Number of warmup runs quick: Run quick benchmark with fewer configs Returns: List of VideoBenchmarkResult objects - Inputs: - `model_name` (str; required): VLM model to use - `video_path` (str; optional; default `None`): Local video file path - `video_url` (str; optional; default `None`): URL to download video from - `video_duration` (float; optional; default `10.0`): Duration for synthetic video - `warmup_runs` (int; optional; default `1`): Number of warmup runs - `quick` (bool; optional; default `False`): Run quick benchmark with fewer configs - Return annotation: `list[VideoBenchmarkResult]` - Calls: print, time.time, MLXVisionLanguageModel, model.load, Path(video_path).exists, Path, download_video, create_test_video, get_video_info, range, run_video_benchmark, results.append - Return expressions: results ## `examples.video_benchmark.print_results` - Kind: function - Signature: `def print_results(results: list[VideoBenchmarkResult])` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/video_benchmark.py#L377-L445 - Implementation: Function `print_results` calls `print`, `sorted`, `table_data.append`, `tabulate`; returns `None`. Print benchmark results in a nice table. - Inputs: - `results` (list[VideoBenchmarkResult]; required): Required positional or keyword input. - Return annotation: `not annotated` - Calls: print, sorted, table_data.append, tabulate, sum, min, max, frame_groups[key].append, frame_groups.keys, len, analysis_data.append - Return expressions: None ## `examples.video_benchmark.save_results` - Kind: function - Signature: `def save_results(results: list[VideoBenchmarkResult], output_path: str, model_name: str)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/video_benchmark.py#L448-L474 - Implementation: Function `save_results` calls `time.strftime`, `open`, `json.dump`, `print`. Save benchmark results to JSON file. - Inputs: - `results` (list[VideoBenchmarkResult]; required): Required positional or keyword input. - `output_path` (str; required): Required positional or keyword input. - `model_name` (str; required): Required positional or keyword input. - Return annotation: `not annotated` - Calls: time.strftime, open, json.dump, print ## `examples.video_benchmark.main` - Kind: function - Signature: `def main()` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/video_benchmark.py#L477-L559 - Implementation: Function `main` calls `argparse.ArgumentParser`, `parser.add_argument`, `parser.parse_args`, `run_benchmark`. Function `main` calls `argparse.ArgumentParser`, `parser.add_argument`, `parser.parse_args`, `run_benchmark`. - Inputs: none - Return annotation: `not annotated` - Calls: argparse.ArgumentParser, parser.add_argument, parser.parse_args, run_benchmark, print_results, save_results # Module `scripts.add_mtp_weights` Add MTP (Multi-Token Prediction) weights to an existing MLX Qwen3-Next model. This script: 1. Downloads the MTP shard from the original BF16 HuggingFace model 2. Extracts MTP weights (mtp.* keys) 3. Quantizes them to match the existing MLX model's quantization 4. Adds them to the MLX model's safetensors files 5. Updates config.json with num_nextn_predict_layers=1 Usage: ``python add_mtp_weights.py [--mlx-model-path PATH] [--source-model MODEL]`` Requirements: pip install mlx Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/scripts/add_mtp_weights.py#L1-L341 ## `scripts.add_mtp_weights.find_snapshot_dir` - Kind: function - Signature: `def find_snapshot_dir(model_path: str) -> Path` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/scripts/add_mtp_weights.py#L38-L51 - Implementation: Function `find_snapshot_dir` calls `Path`, `snapshots_dir.exists`, `(Path(model_path) / 'config.json').exists`, `FileNotFoundError`; can raise `FileNotFoundError`; has 2 explicit return paths. Find the latest snapshot directory in HF cache structure. - Inputs: - `model_path` (str; required): Required positional or keyword input. - Return annotation: `Path` - Calls: Path, snapshots_dir.exists, (Path(model_path) / 'config.json').exists, FileNotFoundError, sorted, snapshots_dir.iterdir - Raises directly: FileNotFoundError - Return expressions: Path(model_path); snapshots[-1] ## `scripts.add_mtp_weights.download_mtp_shard` - Kind: function - Signature: `def download_mtp_shard(dest_path: Path, source_model: str) -> Path` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/scripts/add_mtp_weights.py#L54-L82 - Implementation: Function `download_mtp_shard` calls `shard_path.exists`, `print`, `shard_path.stat`, `subprocess.run`; can raise `RuntimeError`; returns `shard_path`. Download the MTP shard using curl with resume support. - Inputs: - `dest_path` (Path; required): Required positional or keyword input. - `source_model` (str; required): Required positional or keyword input. - Return annotation: `Path` - Calls: shard_path.exists, print, shard_path.stat, subprocess.run, str, RuntimeError - Raises directly: RuntimeError - Return expressions: shard_path ## `scripts.add_mtp_weights.extract_and_quantize_mtp_weights` - Kind: function - Signature: `def extract_and_quantize_mtp_weights(shard_path: Path, snapshot_dir: Path, quantization_bits: int=6)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/scripts/add_mtp_weights.py#L85-L195 - Implementation: Function `extract_and_quantize_mtp_weights` calls `mx.set_default_device`, `print`, `mx.load`, `str`; returns `(mtp_output_file, list(quantized_weights.keys()))`. Extract MTP weights, quantize, and save to MLX model directory. - Inputs: - `shard_path` (Path; required): Required positional or keyword input. - `snapshot_dir` (Path; required): Required positional or keyword input. - `quantization_bits` (int; optional; default `6`): Optional positional or keyword input; defaults to `6`. - Return annotation: `not annotated` - Calls: mx.set_default_device, print, mx.load, str, all_weights.items, k.startswith, len, open, json.load, config.get, quant_config.get, range, all, mx.stack, mtp_weights.pop, mx.eval, quantized_weights.update, _quantize_one, list, mtp_weights.keys, mx.save_safetensors, sum, quantized_weights.values, quantized_weights.keys - Return expressions: (mtp_output_file, list(quantized_weights.keys())) ## `scripts.add_mtp_weights.extract_and_quantize_mtp_weights._quantize_one` - Kind: nested function - Signature: `def _quantize_one(key, weight)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/scripts/add_mtp_weights.py#L137-L160 - Implementation: Nested Function `extract_and_quantize_mtp_weights._quantize_one` calls `any`, `key.endswith`, `mx.eval`, `print`; has 2 explicit return paths. Quantize a single weight, apply norm adjustment, return dict entries. - Inputs: - `key` (not annotated; required): Required positional or keyword input. - `weight` (not annotated; required): Required positional or keyword input. - Return annotation: `not annotated` - Calls: any, key.endswith, mx.eval, print, mx.quantize, key.replace - Return expressions: {key: weight}; {key: q_w, key.replace('.weight', '.scales'): q_s, key.replace('.weight', '.biases'): q_b} ## `scripts.add_mtp_weights.update_model_index` - Kind: function - Signature: `def update_model_index(snapshot_dir: Path, mtp_keys: list)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/scripts/add_mtp_weights.py#L198-L221 - Implementation: Function `update_model_index` calls `index_path.exists`, `print`, `open`, `json.load`; returns `None`. Update model.safetensors.index.json to include MTP weight keys. - Inputs: - `snapshot_dir` (Path; required): Required positional or keyword input. - `mtp_keys` (list; required): Required positional or keyword input. - Return annotation: `not annotated` - Calls: index_path.exists, print, open, json.load, index.get, json.dump, len - Return expressions: None ## `scripts.add_mtp_weights.update_config` - Kind: function - Signature: `def update_config(snapshot_dir: Path)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/scripts/add_mtp_weights.py#L224-L236 - Implementation: Function `update_config` calls `open`, `json.load`, `json.dump`, `print`. Update config.json to enable MTP. - Inputs: - `snapshot_dir` (Path; required): Required positional or keyword input. - Return annotation: `not annotated` - Calls: open, json.load, json.dump, print ## `scripts.add_mtp_weights.main` - Kind: function - Signature: `def main()` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/scripts/add_mtp_weights.py#L239-L337 - Implementation: Function `main` calls `argparse.ArgumentParser`, `parser.add_argument`, `parser.parse_args`, `print`. Function `main` calls `argparse.ArgumentParser`, `parser.add_argument`, `parser.parse_args`, `print`. - Inputs: none - Return annotation: `not annotated` - Calls: argparse.ArgumentParser, parser.add_argument, parser.parse_args, print, find_snapshot_dir, config_path.exists, sys.exit, open, json.load, config.get, index_path.exists, index.get, k.startswith, len, Path, tempfile.mkdtemp, download_mtp_shard, shard_path.exists, extract_and_quantize_mtp_weights, update_model_index, update_config # Module `scripts.add_mtp_weights_qwen35` Add MTP (Multi-Token Prediction) weights to an existing MLX Qwen3.5 model. This script: 1. Fetches the safetensors index from the original BF16 HuggingFace model 2. Identifies shards containing MTP weights (mtp.* keys) 3. Downloads only those shards via curl -C - 4. Extracts MTP weights 5. For MoE models: stacks expert weights (256×) into switch_mlp format 6. Applies norm shift (HF weight → MLX weight+1.0) for RMSNorm keys 7. Quantizes to match the MLX model's quantization scheme 8. Saves as mtp/weights.safetensors (subdirectory avoids mlx_vlm glob) Supports both: - MoE models (Qwen3.5-122B-A10B, 35B-A3B): 256 experts, sparse MTP attention - Dense models (Qwen3.5-27B): full MTP with k/v projections and norms Usage: python add_mtp_weights_qwen35.py --mlx-model-path PATH --source-model MODEL Requirements: pip install mlx Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/scripts/add_mtp_weights_qwen35.py#L1-L470 ## `scripts.add_mtp_weights_qwen35.find_snapshot_dir` - Kind: function - Signature: `def find_snapshot_dir(model_path: str) -> Path` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/scripts/add_mtp_weights_qwen35.py#L53-L63 - Implementation: Function `find_snapshot_dir` calls `Path`, `snapshots_dir.exists`, `(Path(model_path) / 'config.json').exists`, `FileNotFoundError`; can raise `FileNotFoundError`; has 2 explicit return paths. Find the latest snapshot directory in HF cache structure. - Inputs: - `model_path` (str; required): Required positional or keyword input. - Return annotation: `Path` - Calls: Path, snapshots_dir.exists, (Path(model_path) / 'config.json').exists, FileNotFoundError, sorted, snapshots_dir.iterdir - Raises directly: FileNotFoundError - Return expressions: Path(model_path); snapshots[-1] ## `scripts.add_mtp_weights_qwen35.fetch_shard_index` - Kind: function - Signature: `def fetch_shard_index(source_model: str, download_dir: Path) -> dict` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/scripts/add_mtp_weights_qwen35.py#L66-L80 - Implementation: Function `fetch_shard_index` calls `print`, `subprocess.run`, `str`, `RuntimeError`; can raise `RuntimeError`; returns `json.load(f)`. Fetch model.safetensors.index.json from HuggingFace. - Inputs: - `source_model` (str; required): Required positional or keyword input. - `download_dir` (Path; required): Required positional or keyword input. - Return annotation: `dict` - Calls: print, subprocess.run, str, RuntimeError, open, json.load - Raises directly: RuntimeError - Return expressions: json.load(f) ## `scripts.add_mtp_weights_qwen35.identify_mtp_shards` - Kind: function - Signature: `def identify_mtp_shards(index: dict) -> tuple[dict[str, str], set[str]]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/scripts/add_mtp_weights_qwen35.py#L83-L98 - Implementation: Function `identify_mtp_shards` calls `index.get`, `set`, `weight_map.items`, `key.startswith`; returns `(mtp_keys, shards_needed)`. Identify which shards contain MTP weights. Returns: Tuple of (mtp_key_to_shard mapping, set of shard filenames to download) - Inputs: - `index` (dict; required): Required positional or keyword input. - Return annotation: `tuple[dict[str, str], set[str]]` - Calls: index.get, set, weight_map.items, key.startswith, shards_needed.add - Return expressions: (mtp_keys, shards_needed) ## `scripts.add_mtp_weights_qwen35.download_shards` - Kind: function - Signature: `def download_shards(shards: set[str], source_model: str, download_dir: Path) -> dict[str, Path]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/scripts/add_mtp_weights_qwen35.py#L101-L130 - Implementation: Function `download_shards` calls `sorted`, `shard_path.exists`, `shard_path.stat`, `print`; can raise `RuntimeError`; returns `shard_paths`. Download required shards using curl with resume support. - Inputs: - `shards` (set[str]; required): Required positional or keyword input. - `source_model` (str; required): Required positional or keyword input. - `download_dir` (Path; required): Required positional or keyword input. - Return annotation: `dict[str, Path]` - Calls: sorted, shard_path.exists, shard_path.stat, print, subprocess.run, str, RuntimeError - Raises directly: RuntimeError - Return expressions: shard_paths ## `scripts.add_mtp_weights_qwen35.extract_and_quantize_mtp_weights` - Kind: function - Signature: `def extract_and_quantize_mtp_weights(mtp_keys: dict[str, str], shard_paths: dict[str, Path], snapshot_dir: Path, is_moe: bool, num_experts: int, no_quantize: bool=False)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/scripts/add_mtp_weights_qwen35.py#L133-L272 - Implementation: Function `extract_and_quantize_mtp_weights` calls `mx.set_default_device`, `open`, `json.load`, `config.get`; returns `(mtp_output_file, list(quantized_weights.keys()))`. Extract MTP weights from BF16 shards, optionally quantize, and save. - Inputs: - `mtp_keys` (dict[str, str]; required): Required positional or keyword input. - `shard_paths` (dict[str, Path]; required): Required positional or keyword input. - `snapshot_dir` (Path; required): Required positional or keyword input. - `is_moe` (bool; required): Required positional or keyword input. - `num_experts` (int; required): Required positional or keyword input. - `no_quantize` (bool; optional; default `False`): Optional positional or keyword input; defaults to `False`. - Return annotation: `not annotated` - Calls: mx.set_default_device, open, json.load, config.get, text_config.get, quant_config.get, print, mtp_keys.items, shard_to_keys.setdefault(shard, []).append, shard_to_keys.setdefault, len, sorted, shard_to_keys.items, mx.load, str, range, all, mx.stack, all_mtp_weights.pop, mx.eval, quantized_weights.update, _quantize_one, sum, all_mtp_weights.keys, mtp_output_dir.mkdir, mx.save_safetensors, quantized_weights.values, list, quantized_weights.keys - Return expressions: (mtp_output_file, list(quantized_weights.keys())) ## `scripts.add_mtp_weights_qwen35.extract_and_quantize_mtp_weights._quantize_one` - Kind: nested function - Signature: `def _quantize_one(key: str, weight: mx.array) -> dict[str, mx.array]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/scripts/add_mtp_weights_qwen35.py#L204-L229 - Implementation: Nested Function `extract_and_quantize_mtp_weights._quantize_one` calls `any`, `key.endswith`, `mx.eval`, `print`; has 2 explicit return paths. Quantize a single weight, apply norm adjustment. - Inputs: - `key` (str; required): Required positional or keyword input. - `weight` (mx.array; required): Required positional or keyword input. - Return annotation: `dict[str, mx.array]` - Calls: any, key.endswith, mx.eval, print, mx.quantize, key.replace - Return expressions: {key: weight}; {key: q_w, key.replace('.weight', '.scales'): q_s, key.replace('.weight', '.biases'): q_b} ## `scripts.add_mtp_weights_qwen35.update_model_index` - Kind: function - Signature: `def update_model_index(snapshot_dir: Path, mtp_keys: list[str])` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/scripts/add_mtp_weights_qwen35.py#L275-L294 - Implementation: Function `update_model_index` calls `index_path.exists`, `print`, `open`, `json.load`; returns `None`. Update model.safetensors.index.json to include MTP weight keys. - Inputs: - `snapshot_dir` (Path; required): Required positional or keyword input. - `mtp_keys` (list[str]; required): Required positional or keyword input. - Return annotation: `not annotated` - Calls: index_path.exists, print, open, json.load, index.get, json.dump, len - Return expressions: None ## `scripts.add_mtp_weights_qwen35.update_config` - Kind: function - Signature: `def update_config(snapshot_dir: Path)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/scripts/add_mtp_weights_qwen35.py#L297-L322 - Implementation: Function `update_config` calls `open`, `json.load`, `config.get`, `text_config.get`. Update config.json to signal MTP availability. For Qwen3.5, mtp_num_hidden_layers already exists in text_config. We add num_nextn_predict_layers at top level for vllm-mlx compatibility. - Inputs: - `snapshot_dir` (Path; required): Required positional or keyword input. - Return annotation: `not annotated` - Calls: open, json.load, config.get, text_config.get, json.dump, print ## `scripts.add_mtp_weights_qwen35.main` - Kind: function - Signature: `def main()` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/scripts/add_mtp_weights_qwen35.py#L325-L466 - Implementation: Function `main` calls `argparse.ArgumentParser`, `parser.add_argument`, `parser.parse_args`, `print`. Function `main` calls `argparse.ArgumentParser`, `parser.add_argument`, `parser.parse_args`, `print`. - Inputs: none - Return annotation: `not annotated` - Calls: argparse.ArgumentParser, parser.add_argument, parser.parse_args, print, find_snapshot_dir, config_path.exists, sys.exit, open, json.load, config.get, text_config.get, mtp_file.exists, mtp_file.stat, Path, download_dir.mkdir, tempfile.mkdtemp, fetch_shard_index, identify_mtp_shards, len, sorted, sum, mtp_key_map.values, download_shards, p.exists, extract_and_quantize_mtp_weights, update_config, shard_paths.values, shard_path.unlink # Module `scripts.check_docs_coverage` Fail when source symbols or public explanations disappear from the docs. Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/scripts/check_docs_coverage.py#L1-L187 ## `scripts.check_docs_coverage.main` - Kind: function - Signature: `def main() -> int` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/scripts/check_docs_coverage.py#L17-L183 - Implementation: Function `main` calls `build_inventory`, `build_repository_inventory`, `build_cli_inventory`, `render_cli_reference`; has 2 explicit return paths. Validate module coverage, symbol coverage, and public docstring coverage. - Inputs: none - Return annotation: `int` - Calls: build_inventory, build_repository_inventory, build_cli_inventory, render_cli_reference, len, set, issues.append, (REPOSITORY_ROOT / 'mkdocs.yml').read_text, sum, sorted, docs_dir.rglob, page.read_text, content.startswith, hand_written_pages.append, any, line.startswith, content.splitlines, content.lower, page.relative_to, http_reference.read_text, next, ast.parse, (REPOSITORY_ROOT / server_module.path).read_text, ast.walk, isinstance, ast.literal_eval, decorator.func.attr.upper, path.exists, path.relative_to, (docs_dir / 'reference' / 'python-symbols.md').read_text, symbol_index.count, print, max - Return expressions: 1; 0 # Module `scripts.docs_inventory` Static source inventory shared by the documentation build tools. The inventory uses Python's AST instead of importing :mod:`vllm_mlx`. This is important because the documentation build runs on Linux while the runtime package depends on Apple Silicon and MLX. Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/scripts/docs_inventory.py#L1-L1182 ## `scripts.docs_inventory.Parameter` - Kind: class - Signature: `class Parameter` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/scripts/docs_inventory.py#L24-L37 - Implementation: Class `Parameter` declares 1 direct member(s). One explicit callable input reconstructed from the Python AST. - Inputs: - `name` (str; required): Required constructor field. - `kind` (str; required): Required constructor field. - `annotation` (str; required): Required constructor field. - `default` (str; required): Required constructor field. - `required` (bool; required): Required constructor field. - `description` (str; required): Required constructor field. - Constructs: `scripts.docs_inventory.Parameter` - Decorators: dataclass(frozen=True) ## `scripts.docs_inventory.Parameter.to_dict` - Kind: method - Signature: `def to_dict(self) -> dict[str, object]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/scripts/docs_inventory.py#L34-L37 - Implementation: Method `Parameter.to_dict` calls `asdict`; returns `asdict(self)`. Return a JSON-serializable representation of the parameter. - Inputs: none - Return annotation: `dict[str, object]` - Calls: asdict - Return expressions: asdict(self) ## `scripts.docs_inventory.Symbol` - Kind: class - Signature: `class Symbol` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/scripts/docs_inventory.py#L41-L72 - Implementation: Class `Symbol` declares 1 direct member(s). A class, function, method, or nested definition found in source code. - Inputs: - `name` (str; required): Required constructor field. - `qualname` (str; required): Required constructor field. - `full_name` (str; required): Required constructor field. - `kind` (str; required): Required constructor field. - `signature` (str; required): Required constructor field. - `parameters` (tuple[Parameter, ...]; required): Required constructor field. - `return_annotation` (str; required): Required constructor field. - `docstring` (str; required): Required constructor field. - `summary` (str; required): Required constructor field. - `implementation` (str; required): Required constructor field. - `documented` (bool; required): Required constructor field. - `public` (bool; required): Required constructor field. - `addressable` (bool; required): Required constructor field. - `line` (int; required): Required constructor field. - `end_line` (int; required): Required constructor field. - `source_url` (str; required): Required constructor field. - `decorators` (tuple[str, ...]; required): Required constructor field. - `calls` (tuple[str, ...]; required): Required constructor field. - `state_reads` (tuple[str, ...]; required): Required constructor field. - `state_writes` (tuple[str, ...]; required): Required constructor field. - `raises` (tuple[str, ...]; required): Required constructor field. - `return_expressions` (tuple[str, ...]; required): Required constructor field. - `awaits` (bool; required): Required constructor field. - `yields` (bool; required): Required constructor field. - Constructs: `scripts.docs_inventory.Symbol` - Decorators: dataclass(frozen=True) ## `scripts.docs_inventory.Symbol.to_dict` - Kind: method - Signature: `def to_dict(self) -> dict[str, object]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/scripts/docs_inventory.py#L69-L72 - Implementation: Method `Symbol.to_dict` calls `asdict`; returns `asdict(self)`. Return a JSON-serializable representation of the symbol. - Inputs: none - Return annotation: `dict[str, object]` - Calls: asdict - Return expressions: asdict(self) ## `scripts.docs_inventory.Module` - Kind: class - Signature: `class Module` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/scripts/docs_inventory.py#L76-L94 - Implementation: Class `Module` declares 1 direct member(s). Documentation metadata for one tracked Python module. - Inputs: - `name` (str; required): Required constructor field. - `path` (str; required): Required constructor field. - `page_path` (str; required): Required constructor field. - `docstring` (str; required): Required constructor field. - `summary` (str; required): Required constructor field. - `line_count` (int; required): Required constructor field. - `source_url` (str; required): Required constructor field. - `members` (tuple[str, ...]; required): Required constructor field. - `symbols` (tuple[Symbol, ...]; required): Required constructor field. - Constructs: `scripts.docs_inventory.Module` - Decorators: dataclass(frozen=True) ## `scripts.docs_inventory.Module.to_dict` - Kind: method - Signature: `def to_dict(self) -> dict[str, object]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/scripts/docs_inventory.py#L89-L94 - Implementation: Method `Module.to_dict` calls `asdict`, `symbol.to_dict`; returns `payload`. Return a JSON-serializable representation of the module. - Inputs: none - Return annotation: `dict[str, object]` - Calls: asdict, symbol.to_dict - State reads: self.symbols - Return expressions: payload ## `scripts.docs_inventory.CLIOption` - Kind: class - Signature: `class CLIOption` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/scripts/docs_inventory.py#L98-L118 - Implementation: Class `CLIOption` declares 1 direct member(s). One argparse option declaration found in executable source. - Inputs: - `context` (str; required): Required constructor field. - `receiver` (str; required): Required constructor field. - `flags` (tuple[str, ...]; required): Required constructor field. - `destination` (str; required): Required constructor field. - `description` (str; required): Required constructor field. - `default` (str; required): Required constructor field. - `required` (bool; required): Required constructor field. - `choices` (str; required): Required constructor field. - `action` (str; required): Required constructor field. - `path` (str; required): Required constructor field. - `line` (int; required): Required constructor field. - `end_line` (int; required): Required constructor field. - `source_url` (str; required): Required constructor field. - Constructs: `scripts.docs_inventory.CLIOption` - Decorators: dataclass(frozen=True) ## `scripts.docs_inventory.CLIOption.to_dict` - Kind: method - Signature: `def to_dict(self) -> dict[str, object]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/scripts/docs_inventory.py#L115-L118 - Implementation: Method `CLIOption.to_dict` calls `asdict`; returns `asdict(self)`. Return a JSON-serializable representation of the option. - Inputs: none - Return annotation: `dict[str, object]` - Calls: asdict - Return expressions: asdict(self) ## `scripts.docs_inventory._tracked_python_files` - Kind: function - Signature: `def _tracked_python_files(root: Path, source_roots: tuple[str, ...]=('vllm_mlx',)) -> list[Path]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/scripts/docs_inventory.py#L121-L154 - Implementation: Function `_tracked_python_files` calls `subprocess.run`, `completed.stdout.splitlines`, `path.relative_to`, `sorted`; has 2 explicit return paths. Return tracked modules under selected roots, with a filesystem fallback. - Inputs: - `root` (Path; required): Required positional or keyword input. - `source_roots` (tuple[str, ...]; optional; default `('vllm_mlx',)`): Optional positional or keyword input; defaults to `('vllm_mlx',)`. - Return annotation: `list[Path]` - Calls: subprocess.run, completed.stdout.splitlines, path.relative_to, sorted, (root / source_root).rglob - Return expressions: sorted(tracked); sorted((path for source_root in source_roots for path in (root / source_root).rglob('*.py'))) ## `scripts.docs_inventory.module_name_for_path` - Kind: function - Signature: `def module_name_for_path(path: Path, root: Path=REPOSITORY_ROOT) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/scripts/docs_inventory.py#L157-L164 - Implementation: Function `module_name_for_path` calls `path.relative_to(root).with_suffix`, `path.relative_to`, `list`, `parts.pop`; returns `'.'.join(parts)`. Convert a package source path into its importable dotted module name. - Inputs: - `path` (Path; required): Required positional or keyword input. - `root` (Path; optional; default `REPOSITORY_ROOT`): Optional positional or keyword input; defaults to `REPOSITORY_ROOT`. - Return annotation: `str` - Calls: path.relative_to(root).with_suffix, path.relative_to, list, parts.pop, '.'.join - Return expressions: '.'.join(parts) ## `scripts.docs_inventory.page_path_for_module` - Kind: function - Signature: `def page_path_for_module(path: Path, root: Path=REPOSITORY_ROOT) -> Path` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/scripts/docs_inventory.py#L167-L174 - Implementation: Function `page_path_for_module` calls `path.relative_to(root).with_suffix`, `path.relative_to`, `relative.with_name`, `Path`; returns `Path('reference') / section / relative`. Return the generated documentation path for a package source file. - Inputs: - `path` (Path; required): Required positional or keyword input. - `root` (Path; optional; default `REPOSITORY_ROOT`): Optional positional or keyword input; defaults to `REPOSITORY_ROOT`. - Return annotation: `Path` - Calls: path.relative_to(root).with_suffix, path.relative_to, relative.with_name, Path - Return expressions: Path('reference') / section / relative ## `scripts.docs_inventory._first_sentence` - Kind: function - Signature: `def _first_sentence(docstring: str) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/scripts/docs_inventory.py#L177-L184 - Implementation: Function `_first_sentence` calls `' '.join`, `docstring.strip().split`, `docstring.strip`, `re.search`; has 2 explicit return paths. Return a compact first sentence or line from a docstring. - Inputs: - `docstring` (str; required): Required positional or keyword input. - Return annotation: `str` - Calls: ' '.join, docstring.strip().split, docstring.strip, re.search, match.start - Return expressions: ''; text[:match.start()] if match else text ## `scripts.docs_inventory._parameter_descriptions` - Kind: function - Signature: `def _parameter_descriptions(docstring: str) -> dict[str, str]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/scripts/docs_inventory.py#L187-L266 - Implementation: Function `_parameter_descriptions` calls `docstring.splitlines`, `re.match`, `sphinx_match.group(1).lstrip`, `sphinx_match.group`; returns `descriptions`. Extract Google, NumPy, and Sphinx parameter descriptions. - Inputs: - `docstring` (str; required): Required positional or keyword input. - Return annotation: `dict[str, str]` - Calls: docstring.splitlines, re.match, sphinx_match.group(1).lstrip, sphinx_match.group, sphinx_match.group(2).strip, next, enumerate, line.strip, line[:1].isspace, stripped.endswith, name.strip().lstrip, name.strip, item.group(1).split, item.group, item.group(2).strip, ' '.join, set, lines[index + 1].strip, item_line.strip, item_line[:1].isspace, descriptions.setdefault - Return expressions: descriptions ## `scripts.docs_inventory._function_parameters` - Kind: function - Signature: `def _function_parameters(node: ast.FunctionDef | ast.AsyncFunctionDef, docstring: str) -> tuple[Parameter, ...]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/scripts/docs_inventory.py#L269-L353 - Implementation: Function `_function_parameters` calls `_parameter_descriptions`, `len`, `list`, `enumerate`; returns `tuple(parameters)`. Return the callable inputs, annotations, defaults, and descriptions. - Inputs: - `node` (ast.FunctionDef | ast.AsyncFunctionDef; required): Required positional or keyword input. - `docstring` (str; required): Required positional or keyword input. - Return annotation: `tuple[Parameter, ...]` - Calls: _parameter_descriptions, len, list, enumerate, zip, append_parameter, tuple - Return expressions: tuple(parameters) ## `scripts.docs_inventory._function_parameters.append_parameter` - Kind: nested function - Signature: `def append_parameter(argument: ast.arg, *, kind: str, default_node: ast.AST | None, required: bool, prefix: str='') -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/scripts/docs_inventory.py#L282-L314 - Implementation: Nested Function `_function_parameters.append_parameter` calls `_short_expression`, `descriptions.get`, `parameters.append`, `Parameter`; returns `None`. Nested Function `_function_parameters.append_parameter` calls `_short_expression`, `descriptions.get`, `parameters.append`, `Parameter`; returns `None`. - Inputs: - `argument` (ast.arg; required): Required positional or keyword input. - `kind` (str; required): Required keyword-only input. - `default_node` (ast.AST | None; required): Required keyword-only input. - `required` (bool; required): Required keyword-only input. - `prefix` (str; optional; default `''`): Optional keyword-only input; defaults to `''`. - Return annotation: `None` - Calls: _short_expression, descriptions.get, parameters.append, Parameter - Return expressions: None ## `scripts.docs_inventory._class_parameters` - Kind: function - Signature: `def _class_parameters(node: ast.ClassDef, docstring: str) -> tuple[Parameter, ...]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/scripts/docs_inventory.py#L356-L408 - Implementation: Function `_class_parameters` calls `next`, `isinstance`, `ast.get_docstring`, `_function_parameters`; has 3 explicit return paths. Return constructor inputs from ``__init__`` or declarative fields. - Inputs: - `node` (ast.ClassDef; required): Required positional or keyword input. - `docstring` (str; required): Required positional or keyword input. - Return annotation: `tuple[Parameter, ...]` - Calls: next, isinstance, ast.get_docstring, _function_parameters, '\n'.join, ast.unparse(item).split, ast.unparse, ast.unparse(base).rsplit, _parameter_descriptions, _short_expression, parameters.append, Parameter, descriptions.get, tuple - Return expressions: _function_parameters(initializer, '\n'.join((part for part in (docstring, initializer_docstring) if part))); (); tuple(parameters) ## `scripts.docs_inventory._owned_nodes` - Kind: function - Signature: `def _owned_nodes(node: ast.AST) -> Iterable[ast.AST]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/scripts/docs_inventory.py#L411-L423 - Implementation: Function `_owned_nodes` calls `getattr`, `list`, `reversed`, `stack.pop`; yields values incrementally. Walk implementation nodes without attributing nested bodies to parents. - Inputs: - `node` (ast.AST; required): Required positional or keyword input. - Return annotation: `Iterable[ast.AST]` - Calls: getattr, list, reversed, stack.pop, isinstance, stack.extend, ast.iter_child_nodes ## `scripts.docs_inventory._ordered_unique` - Kind: function - Signature: `def _ordered_unique(values: Iterable[str]) -> tuple[str, ...]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/scripts/docs_inventory.py#L426-L429 - Implementation: Function `_ordered_unique` calls `tuple`, `dict.fromkeys`; returns `tuple(dict.fromkeys((value for value in values if value)))`. Return non-empty strings once while preserving their source order. - Inputs: - `values` (Iterable[str]; required): Required positional or keyword input. - Return annotation: `tuple[str, ...]` - Calls: tuple, dict.fromkeys - Return expressions: tuple(dict.fromkeys((value for value in values if value))) ## `scripts.docs_inventory._short_expression` - Kind: function - Signature: `def _short_expression(node: ast.AST | None, *, limit: int=120) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/scripts/docs_inventory.py#L432-L438 - Implementation: Function `_short_expression` calls `' '.join`, `ast.unparse(node).split`, `ast.unparse`, `len`; has 2 explicit return paths. Render an AST expression without allowing one fact to dominate output. - Inputs: - `node` (ast.AST | None; required): Required positional or keyword input. - `limit` (int; optional; default `120`): Optional keyword-only input; defaults to `120`. - Return annotation: `str` - Calls: ' '.join, ast.unparse(node).split, ast.unparse, len - Return expressions: 'None'; value if len(value) <= limit else value[:limit - 1] + '…' ## `scripts.docs_inventory._attribute_name` - Kind: function - Signature: `def _attribute_name(node: ast.Attribute) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/scripts/docs_inventory.py#L441-L452 - Implementation: Function `_attribute_name` calls `isinstance`, `parts.append`, `'.'.join`, `reversed`; has 2 explicit return paths. Return tracked ``self`` or ``cls`` attribute access, if applicable. - Inputs: - `node` (ast.Attribute; required): Required positional or keyword input. - Return annotation: `str` - Calls: isinstance, parts.append, '.'.join, reversed - Return expressions: '.'.join(reversed(parts)); '' ## `scripts.docs_inventory._implementation_facts` - Kind: function - Signature: `def _implementation_facts(node: ast.ClassDef | ast.FunctionDef | ast.AsyncFunctionDef, *, kind: str, qualname: str) -> dict[str, object]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/scripts/docs_inventory.py#L455-L548 - Implementation: Function `_implementation_facts` calls `_ordered_unique`, `ast.unparse`, `isinstance`, `clauses.append`; has 2 explicit return paths. Extract conservative behavioral facts from one definition's own body. - Inputs: - `node` (ast.ClassDef | ast.FunctionDef | ast.AsyncFunctionDef; required): Required positional or keyword input. - `kind` (str; required): Required keyword-only input. - `qualname` (str; required): Required keyword-only input. - Return annotation: `dict[str, object]` - Calls: _ordered_unique, ast.unparse, isinstance, clauses.append, ', '.join, len, kind.title, ' and '.join, list, _owned_nodes, _short_expression, _attribute_name, any, '; '.join - Return expressions: {'implementation': implementation, 'decorators': decorators, 'calls': (), 'state_reads': (), 'state_writes': (), 'raise…; {'implementation': implementation, 'decorators': decorators, 'calls': calls, 'state_reads': state_reads, 'state_writes'… ## `scripts.docs_inventory._function_signature` - Kind: function - Signature: `def _function_signature(node: ast.FunctionDef | ast.AsyncFunctionDef) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/scripts/docs_inventory.py#L551-L556 - Implementation: Function `_function_signature` calls `isinstance`, `ast.unparse`; returns `f'{prefix} {node.name}({ast.unparse(node.args)}){returns}'`. Render a stable function signature without importing its module. - Inputs: - `node` (ast.FunctionDef | ast.AsyncFunctionDef; required): Required positional or keyword input. - Return annotation: `str` - Calls: isinstance, ast.unparse - Return expressions: f'{prefix} {node.name}({ast.unparse(node.args)}){returns}' ## `scripts.docs_inventory._class_signature` - Kind: function - Signature: `def _class_signature(node: ast.ClassDef) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/scripts/docs_inventory.py#L559-L565 - Implementation: Function `_class_signature` calls `ast.unparse`, `arguments.extend`, `', '.join`; returns `f'class {node.name}{suffix}'`. Render a class declaration and its bases from the AST. - Inputs: - `node` (ast.ClassDef; required): Required positional or keyword input. - Return annotation: `str` - Calls: ast.unparse, arguments.extend, ', '.join - Return expressions: f'class {node.name}{suffix}' ## `scripts.docs_inventory._defined_member_names` - Kind: function - Signature: `def _defined_member_names(tree: ast.Module) -> tuple[str, ...]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/scripts/docs_inventory.py#L568-L584 - Implementation: Function `_defined_member_names` calls `isinstance`, `names.append`, `tuple`, `dict.fromkeys`; returns `tuple(dict.fromkeys(names))`. Return names defined directly by a module in source order. - Inputs: - `tree` (ast.Module; required): Required positional or keyword input. - Return annotation: `tuple[str, ...]` - Calls: isinstance, names.append, tuple, dict.fromkeys - Return expressions: tuple(dict.fromkeys(names)) ## `scripts.docs_inventory.scan_python_file` - Kind: function - Signature: `def scan_python_file(path: Path, root: Path=REPOSITORY_ROOT) -> Module` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/scripts/docs_inventory.py#L587-L712 - Implementation: Function `scan_python_file` calls `path.read_text`, `ast.parse`, `str`, `ast.walk`; returns `Module(name=module_name, path=relative_path, page_path=page_path_for_module(path, root).as_posix(), docstring=module_do…`. Parse one Python file into module and symbol documentation metadata. - Inputs: - `path` (Path; required): Required positional or keyword input. - `root` (Path; optional; default `REPOSITORY_ROOT`): Optional positional or keyword input; defaults to `REPOSITORY_ROOT`. - Return annotation: `Module` - Calls: path.read_text, ast.parse, str, ast.walk, ast.iter_child_nodes, module_name_for_path, isinstance, parents.get, ancestors.append, ancestors.reverse, any, '.'.join, all, part.startswith, ast.get_docstring, _class_signature, _class_parameters, _function_signature, _function_parameters, _short_expression, getattr, path.relative_to(root).as_posix, path.relative_to, _implementation_facts, symbols.append, Symbol, _first_sentence, bool, symbols.sort, len, source.splitlines, Module, page_path_for_module(path, root).as_posix, page_path_for_module, max, _defined_member_names, tuple - Return expressions: Module(name=module_name, path=relative_path, page_path=page_path_for_module(path, root).as_posix(), docstring=module_do… ## `scripts.docs_inventory.build_inventory` - Kind: function - Signature: `def build_inventory(root: Path=REPOSITORY_ROOT, source_roots: tuple[str, ...]=('vllm_mlx',)) -> list[Module]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/scripts/docs_inventory.py#L715-L724 - Implementation: Function `build_inventory` calls `scan_python_file`, `_tracked_python_files`; returns `[scan_python_file(path, root) for path in _tracked_python_files(root, source_roots)]`. Build static source inventory for the selected repository roots. - Inputs: - `root` (Path; optional; default `REPOSITORY_ROOT`): Optional positional or keyword input; defaults to `REPOSITORY_ROOT`. - `source_roots` (tuple[str, ...]; optional; default `('vllm_mlx',)`): Optional positional or keyword input; defaults to `('vllm_mlx',)`. - Return annotation: `list[Module]` - Calls: scan_python_file, _tracked_python_files - Return expressions: [scan_python_file(path, root) for path in _tracked_python_files(root, source_roots)] ## `scripts.docs_inventory.build_repository_inventory` - Kind: function - Signature: `def build_repository_inventory(root: Path=REPOSITORY_ROOT) -> list[Module]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/scripts/docs_inventory.py#L727-L730 - Implementation: Function `build_repository_inventory` calls `build_inventory`; returns `build_inventory(root, ('vllm_mlx', 'scripts', 'examples'))`. Inventory runtime, documentation tools, and executable examples. - Inputs: - `root` (Path; optional; default `REPOSITORY_ROOT`): Optional positional or keyword input; defaults to `REPOSITORY_ROOT`. - Return annotation: `list[Module]` - Calls: build_inventory - Return expressions: build_inventory(root, ('vllm_mlx', 'scripts', 'examples')) ## `scripts.docs_inventory._literal_or_source` - Kind: function - Signature: `def _literal_or_source(node: ast.AST | None, default: str='') -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/scripts/docs_inventory.py#L733-L744 - Implementation: Function `_literal_or_source` calls `ast.literal_eval`, `' '.join`, `ast.unparse(node).split`, `ast.unparse`; has 4 explicit return paths. Render a simple literal cleanly and preserve expressions as source. - Inputs: - `node` (ast.AST | None; required): Required positional or keyword input. - `default` (str; optional; default `''`): Optional positional or keyword input; defaults to `''`. - Return annotation: `str` - Calls: ast.literal_eval, ' '.join, ast.unparse(node).split, ast.unparse, isinstance, value.split, repr - Return expressions: default; ' '.join(ast.unparse(node).split()); ' '.join(value.split()); repr(value) ## `scripts.docs_inventory.scan_cli_options` - Kind: function - Signature: `def scan_cli_options(paths: Iterable[Path], root: Path=REPOSITORY_ROOT) -> list[CLIOption]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/scripts/docs_inventory.py#L747-L817 - Implementation: Function `scan_cli_options` calls `sorted`, `path.read_text`, `ast.parse`, `str`; returns `sorted(options, key=lambda option: (option.path, option.line))`. Extract every argparse ``add_argument`` call from selected source files. - Inputs: - `paths` (Iterable[Path]; required): Required positional or keyword input. - `root` (Path; optional; default `REPOSITORY_ROOT`): Optional positional or keyword input; defaults to `REPOSITORY_ROOT`. - Return annotation: `list[CLIOption]` - Calls: sorted, path.read_text, ast.parse, str, ast.walk, ast.iter_child_nodes, module_name_for_path, path.relative_to(root).as_posix, path.relative_to, isinstance, tuple, _literal_or_source, keywords.get, next, flag.startswith, preferred.lstrip('-').replace, preferred.lstrip, parents.get, ancestors.append, ancestors.reverse, '.'.join, ' '.join, ast.unparse(node.func.value).split, ast.unparse, getattr, options.append, CLIOption, flags[0].startswith - Return expressions: sorted(options, key=lambda option: (option.path, option.line)) ## `scripts.docs_inventory.build_cli_inventory` - Kind: function - Signature: `def build_cli_inventory(root: Path=REPOSITORY_ROOT) -> list[CLIOption]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/scripts/docs_inventory.py#L820-L824 - Implementation: Function `build_cli_inventory` calls `_tracked_python_files`, `scan_cli_options`; returns `scan_cli_options(paths, root)`. Build the complete argparse option inventory for executable source. - Inputs: - `root` (Path; optional; default `REPOSITORY_ROOT`): Optional positional or keyword input; defaults to `REPOSITORY_ROOT`. - Return annotation: `list[CLIOption]` - Calls: _tracked_python_files, scan_cli_options - Return expressions: scan_cli_options(paths, root) ## `scripts.docs_inventory._escape_table_cell` - Kind: function - Signature: `def _escape_table_cell(value: str) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/scripts/docs_inventory.py#L827-L830 - Implementation: Function `_escape_table_cell` calls `' '.join`, `value.replace('|', '\\|').split`, `value.replace`; returns `' '.join(value.replace('|', '\\|').split())`. Escape text for a compact Markdown table cell. - Inputs: - `value` (str; required): Required positional or keyword input. - Return annotation: `str` - Calls: ' '.join, value.replace('|', '\\|').split, value.replace - Return expressions: ' '.join(value.replace('|', '\\|').split()) ## `scripts.docs_inventory.render_callable_signature` - Kind: function - Signature: `def render_callable_signature(symbol: Symbol, *, qualified: bool=False) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/scripts/docs_inventory.py#L833-L866 - Implementation: Function `render_callable_signature` calls `sum`, `any`, `pieces.append`, `symbol.signature.startswith`; returns `signature`. Render a reader-facing signature without implicit ``self`` or ``cls``. - Inputs: - `symbol` (Symbol; required): Required positional or keyword input. - `qualified` (bool; optional; default `False`): Optional keyword-only input; defaults to `False`. - Return annotation: `str` - Calls: sum, any, pieces.append, symbol.signature.startswith, ', '.join - Return expressions: signature ## `scripts.docs_inventory._symbol_details_url` - Kind: function - Signature: `def _symbol_details_url(module: Module, symbol: Symbol) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/scripts/docs_inventory.py#L869-L877 - Implementation: Function `_symbol_details_url` calls `Path(module.page_path).relative_to`, `Path`, `relative.parent.as_posix`, `relative.with_suffix('').as_posix`; returns `f'../{page_url}#contract-{symbol.full_name}'`. Return the generated details URL or source fallback for one symbol. - Inputs: - `module` (Module; required): Required positional or keyword input. - `symbol` (Symbol; required): Required positional or keyword input. - Return annotation: `str` - Calls: Path(module.page_path).relative_to, Path, relative.parent.as_posix, relative.with_suffix('').as_posix, relative.with_suffix - Return expressions: f'../{page_url}#contract-{symbol.full_name}' ## `scripts.docs_inventory.render_contract_details` - Kind: function - Signature: `def render_contract_details(module: Module) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/scripts/docs_inventory.py#L880-L959 - Implementation: Function `render_contract_details` calls `lines.extend`, `render_callable_signature`, `lines.append`, `_escape_table_cell`; has 2 explicit return paths. Render explicit inputs and behavior for every definition in a module. - Inputs: - `module` (Module; required): Required positional or keyword input. - Return annotation: `str` - Calls: lines.extend, render_callable_signature, lines.append, _escape_table_cell, '; '.join, ', '.join, '\n'.join - Return expressions: 'This module does not declare classes or functions.\n'; '\n'.join(lines) ## `scripts.docs_inventory.render_source_map` - Kind: function - Signature: `def render_source_map(module: Module) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/scripts/docs_inventory.py#L962-L981 - Implementation: Function `render_source_map` calls `lines.append`, `_escape_table_cell`, `render_callable_signature`, `'\n'.join`; has 2 explicit return paths. Render a line-precise source table for every definition in a module. - Inputs: - `module` (Module; required): Required positional or keyword input. - Return annotation: `str` - Calls: lines.append, _escape_table_cell, render_callable_signature, '\n'.join - Return expressions: 'This module does not declare classes or functions.\n'; '\n'.join(lines) + '\n' ## `scripts.docs_inventory.render_symbol_index` - Kind: function - Signature: `def render_symbol_index(modules: list[Module]) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/scripts/docs_inventory.py#L984-L1039 - Implementation: Function `render_symbol_index` calls `sorted`, `len`, `render_callable_signature`, `' '.join((symbol.full_name, symbol.kind, signature, symbol.summary)).casefold`; returns `'\n'.join(lines)`. Render a filterable index of every runtime class and callable. - Inputs: - `modules` (list[Module]; required): Required positional or keyword input. - Return annotation: `str` - Calls: sorted, len, render_callable_signature, ' '.join((symbol.full_name, symbol.kind, signature, symbol.summary)).casefold, ' '.join, _symbol_details_url, lines.extend, html.escape, '\n'.join - Return expressions: '\n'.join(lines) ## `scripts.docs_inventory.render_module_page` - Kind: function - Signature: `def render_module_page(module: Module) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/scripts/docs_inventory.py#L1042-L1086 - Implementation: Function `render_module_page` calls `lines.extend`, `lines.append`, `render_contract_details(module).rstrip`, `render_contract_details`; returns `'\n'.join(lines)`. Render the generated MkDocs page for one Python module. - Inputs: - `module` (Module; required): Required positional or keyword input. - Return annotation: `str` - Calls: lines.extend, lines.append, render_contract_details(module).rstrip, render_contract_details, render_source_map(module).rstrip, render_source_map, '\n'.join - Return expressions: '\n'.join(lines) ## `scripts.docs_inventory.render_module_for_llms` - Kind: function - Signature: `def render_module_for_llms(module: Module) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/scripts/docs_inventory.py#L1089-L1144 - Implementation: Function `render_module_for_llms` calls `lines.extend`, `lines.append`, `', '.join`, `'; '.join`; returns `'\n'.join(lines) + '\n'`. Render a self-contained plain-Markdown API record for language models. - Inputs: - `module` (Module; required): Required positional or keyword input. - Return annotation: `str` - Calls: lines.extend, lines.append, ', '.join, '; '.join, '\n'.join - Return expressions: '\n'.join(lines) + '\n' ## `scripts.docs_inventory.render_cli_reference` - Kind: function - Signature: `def render_cli_reference(options: list[CLIOption]) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/scripts/docs_inventory.py#L1147-L1182 - Implementation: Function `render_cli_reference` calls `lines.extend`, `', '.join`, `str(option.required).lower`, `str`; returns `'\n'.join(lines)`. Render every discovered argparse option as a line-precise reference. - Inputs: - `options` (list[CLIOption]; required): Required positional or keyword input. - Return annotation: `str` - Calls: lines.extend, ', '.join, str(option.required).lower, str, '\n'.join - Return expressions: '\n'.join(lines) # Module `scripts.gen_api_reference` Synchronize the exhaustive static API reference with the Python source tree. Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/scripts/gen_api_reference.py#L1-L103 ## `scripts.gen_api_reference.expected_pages` - Kind: function - Signature: `def expected_pages() -> dict[Path, str]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/scripts/gen_api_reference.py#L23-L38 - Implementation: Function `expected_pages` calls `build_inventory`, `render_module_page`, `build_repository_inventory`, `render_cli_reference`; returns `pages`. Return every generated reference path and its expected contents. - Inputs: none - Return annotation: `dict[Path, str]` - Calls: build_inventory, render_module_page, build_repository_inventory, render_cli_reference, build_cli_inventory, render_symbol_index - Return expressions: pages ## `scripts.gen_api_reference.generated_pages_on_disk` - Kind: function - Signature: `def generated_pages_on_disk() -> set[Path]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/scripts/gen_api_reference.py#L41-L49 - Implementation: Function `generated_pages_on_disk` calls `docs_dir.rglob`, `path.read_text(encoding='utf-8').startswith`, `path.read_text`; returns `{path for path in docs_dir.rglob('*.md') if path.read_text(encoding='utf-8').startswith(GENERATED_HEADER)}`. Return generated reference pages currently present in the docs tree. - Inputs: none - Return annotation: `set[Path]` - Calls: docs_dir.rglob, path.read_text(encoding='utf-8').startswith, path.read_text - Return expressions: {path for path in docs_dir.rglob('*.md') if path.read_text(encoding='utf-8').startswith(GENERATED_HEADER)} ## `scripts.gen_api_reference.check` - Kind: function - Signature: `def check() -> int` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/scripts/gen_api_reference.py#L52-L73 - Implementation: Function `check` calls `expected_pages`, `generated_pages_on_disk`, `expected.items`, `path.exists`; has 2 explicit return paths. Report missing, stale, or unexpected generated pages. - Inputs: none - Return annotation: `int` - Calls: expected_pages, generated_pages_on_disk, expected.items, path.exists, issues.append, path.relative_to, path.read_text, sorted, expected.keys, print, len - Return expressions: 1; 0 ## `scripts.gen_api_reference.write` - Kind: function - Signature: `def write() -> int` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/scripts/gen_api_reference.py#L76-L86 - Implementation: Function `write` calls `expected_pages`, `generated_pages_on_disk`, `expected.keys`, `path.unlink`; returns `0`. Write every reference page and remove obsolete generated pages. - Inputs: none - Return annotation: `int` - Calls: expected_pages, generated_pages_on_disk, expected.keys, path.unlink, expected.items, path.parent.mkdir, path.write_text, print, len - Return expressions: 0 ## `scripts.gen_api_reference.main` - Kind: function - Signature: `def main() -> int` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/scripts/gen_api_reference.py#L89-L99 - Implementation: Function `main` calls `argparse.ArgumentParser`, `parser.add_argument`, `parser.parse_args`, `check`; returns `check() if args.check else write()`. Parse the synchronization mode and update or verify generated pages. - Inputs: none - Return annotation: `int` - Calls: argparse.ArgumentParser, parser.add_argument, parser.parse_args, check, write - Return expressions: check() if args.check else write() # Module `scripts.mkdocs_hooks` MkDocs hooks that publish machine-readable documentation artifacts. Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/scripts/mkdocs_hooks.py#L1-L225 ## `scripts.mkdocs_hooks._source_revision` - Kind: function - Signature: `def _source_revision() -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/scripts/mkdocs_hooks.py#L27-L46 - Implementation: Function `_source_revision` calls `os.environ.get`, `re.fullmatch`, `candidate.lower`, `subprocess.run`; can raise `ValueError`; has 2 explicit return paths. Return the immutable commit represented by this documentation build. - Inputs: none - Return annotation: `str` - Decorators: lru_cache(maxsize=1) - Calls: os.environ.get, re.fullmatch, candidate.lower, subprocess.run, completed.stdout.strip, ValueError - Raises directly: ValueError - Return expressions: candidate.lower(); revision ## `scripts.mkdocs_hooks._pin_source_links` - Kind: function - Signature: `def _pin_source_links(text: str, revision: str) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/scripts/mkdocs_hooks.py#L49-L55 - Implementation: Function `_pin_source_links` calls `text.replace`; returns `text.replace(SOURCE_BRANCH_URL, f'https://github.com/waybarrios/vllm-mlx/blob/{revision}/')`. Replace mutable gh-pages source links with one commit permalink. - Inputs: - `text` (str; required): Required positional or keyword input. - `revision` (str; required): Required positional or keyword input. - Return annotation: `str` - Calls: text.replace - Return expressions: text.replace(SOURCE_BRANCH_URL, f'https://github.com/waybarrios/vllm-mlx/blob/{revision}/') ## `scripts.mkdocs_hooks.on_page_markdown` - Kind: function - Signature: `def on_page_markdown(markdown: str, **kwargs) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/scripts/mkdocs_hooks.py#L58-L62 - Implementation: Function `on_page_markdown` calls `_pin_source_links`, `_source_revision`; returns `_pin_source_links(markdown, _source_revision())`. Pin every rendered GitHub source link to the build commit. - Inputs: - `markdown` (str; required): Required positional or keyword input. - `**kwargs` (not annotated; optional): Additional variadic keyword inputs accepted by this callable. - Return annotation: `str` - Calls: _pin_source_links, _source_revision - Return expressions: _pin_source_links(markdown, _source_revision()) ## `scripts.mkdocs_hooks.on_post_page` - Kind: function - Signature: `def on_post_page(output: str, page=None, **kwargs) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/scripts/mkdocs_hooks.py#L65-L89 - Implementation: Function `on_post_page` calls `getattr`, `re.sub`; has 2 explicit return paths. Normalize search alternates and localized homepage presentation. - Inputs: - `output` (str; required): Required positional or keyword input. - `page` (not annotated; optional; default `None`): Optional positional or keyword input; defaults to `None`. - `**kwargs` (not annotated; optional): Additional variadic keyword inputs accepted by this callable. - Return annotation: `str` - Calls: getattr, re.sub - Return expressions: output; re.sub('\\s*]*\\brel=\\"edit\\")[^>]*>.*?', '', output, count=1, flags=re.DOTALL) ## `scripts.mkdocs_hooks._markdown_documents` - Kind: function - Signature: `def _markdown_documents() -> list[Path]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/scripts/mkdocs_hooks.py#L92-L101 - Implementation: Function `_markdown_documents` calls `sorted`, `(REPOSITORY_ROOT / 'docs').rglob`, `path.read_text(encoding='utf-8').startswith`, `path.read_text`; returns `sorted((path for path in (REPOSITORY_ROOT / 'docs').rglob('*.md') if not path.read_text(encoding='utf-8').startswith('<…`. Return tracked hand-written documentation pages in stable order. - Inputs: none - Return annotation: `list[Path]` - Calls: sorted, (REPOSITORY_ROOT / 'docs').rglob, path.read_text(encoding='utf-8').startswith, path.read_text - Return expressions: sorted((path for path in (REPOSITORY_ROOT / 'docs').rglob('*.md') if not path.read_text(encoding='utf-8').startswith('<… ## `scripts.mkdocs_hooks.on_post_build` - Kind: function - Signature: `def on_post_build(config, **kwargs) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/scripts/mkdocs_hooks.py#L104-L225 - Implementation: Function `on_post_build` calls `Path`, `build_inventory`, `build_repository_inventory`, `build_cli_inventory`. Write Markdown mirrors, the API inventory, and the full LLM corpus. - Inputs: - `config` (not annotated; required): Required positional or keyword input. - `**kwargs` (not annotated; optional): Additional variadic keyword inputs accepted by this callable. - Return annotation: `None` - Calls: Path, build_inventory, build_repository_inventory, build_cli_inventory, _source_revision, _markdown_documents, source_path.relative_to, _pin_source_links, source_path.read_text(encoding='utf-8').strip, source_path.read_text, mirror_path.parent.mkdir, mirror_path.write_text, full_parts.extend, relative.as_posix, render_module_page, render_module_for_llms, symbol_index_mirror.parent.mkdir, symbol_index_mirror.write_text, symbol_index_source.read_text, render_cli_reference, cli_mirror.parent.mkdir, cli_mirror.write_text, len, sum, module.to_dict, (site_dir / 'api-inventory.json').write_text, json.dumps, (site_dir / 'source-inventory.json').write_text, (site_dir / 'cli-inventory.json').write_text, option.to_dict, (site_dir / 'llms-full.txt').write_text, '\n'.join(full_parts).rstrip, '\n'.join # Module `vllm_mlx` vllm-mlx: Apple Silicon MLX backend for vLLM This package provides native Apple Silicon GPU acceleration for vLLM using Apple's MLX framework, mlx-lm for LLMs, and mlx-vlm for vision-language models. Features: - Continuous batching via vLLM-style scheduler - OpenAI-compatible API server - Support for LLM and multimodal models Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/__init__.py#L1-L132 ## `vllm_mlx.__getattr__` - Kind: function - Signature: `def __getattr__(name)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/__init__.py#L21-L90 - Implementation: Function `__getattr__` calls `getattr`, `name.startswith`, `name.replace`, `AttributeError`; can raise `AttributeError`; has 11 explicit return paths. Lazy load all components to avoid mlx_lm import on non-Apple platforms. - Inputs: - `name` (not annotated; required): Required positional or keyword input. - Return annotation: `not annotated` - Calls: getattr, name.startswith, name.replace, AttributeError - Raises directly: AttributeError - Return expressions: getattr(request, name); getattr(scheduler, name); getattr(engine_core, name); getattr(prefix_cache, name); getattr(paged_cache, name); getattr(mllm_cache, mllm_name); getattr(model_registry, name); MLXPlatform; MLXWorker; MLXModelRunner; MLXAttentionBackend # Module `vllm_mlx.api` API models, utilities, and tool calling support for vllm-mlx. This module provides shared components used by the server: - Pydantic models for OpenAI-compatible API - Utility functions for text processing and model detection - Tool calling parsing and conversion Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/__init__.py#L1-L167 # Module `vllm_mlx.api.anthropic_adapter` Adapter for converting between Anthropic Messages API and OpenAI Chat Completions API. Handles translation of: - Requests: Anthropic → OpenAI format - Responses: OpenAI → Anthropic format - Messages: Content blocks, tool calls, tool results Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/anthropic_adapter.py#L1-L321 ## `vllm_mlx.api.anthropic_adapter.anthropic_to_openai` - Kind: function - Signature: `def anthropic_to_openai(request: AnthropicRequest) -> ChatCompletionRequest` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/anthropic_adapter.py#L31-L99 - Implementation: Function `anthropic_to_openai` calls `isinstance`, `block.get`, `parts.append`, `'\n'.join`; returns `ChatCompletionRequest(model=request.model, messages=messages, max_tokens=request.max_tokens, temperature=request.temper…`. Convert an Anthropic Messages API request to OpenAI Chat Completions format. Handles: - system field → system message - Content blocks → OpenAI message format - tool_use/tool_result → OpenAI tool_calls/tool messages - Anthropic tools → OpenAI tools Args: request: Anthropic Messages API request Returns: OpenAI ChatCompletionRequest - Inputs: - `request` (AnthropicRequest; required): Anthropic Messages API request - Return annotation: `ChatCompletionRequest` - Calls: isinstance, block.get, parts.append, '\n'.join, str, re.sub, messages.append, Message, _convert_message, messages.extend, _convert_tool, _convert_tool_choice, ChatCompletionRequest - Return expressions: ChatCompletionRequest(model=request.model, messages=messages, max_tokens=request.max_tokens, temperature=request.temper… ## `vllm_mlx.api.anthropic_adapter.openai_to_anthropic` - Kind: function - Signature: `def openai_to_anthropic(response: ChatCompletionResponse, model: str) -> AnthropicResponse` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/anthropic_adapter.py#L102-L162 - Implementation: Function `openai_to_anthropic` calls `content.append`, `AnthropicResponseContentBlock`, `json.loads`, `_convert_stop_reason`; returns `AnthropicResponse(model=model, content=content, stop_reason=stop_reason, usage=AnthropicUsage(input_tokens=response.usa…`. Convert an OpenAI Chat Completions response to Anthropic Messages API format. Args: response: OpenAI ChatCompletionResponse model: Model name for the response Returns: Anthropic Messages API response - Inputs: - `response` (ChatCompletionResponse; required): OpenAI ChatCompletionResponse - `model` (str; required): Model name for the response - Return annotation: `AnthropicResponse` - Calls: content.append, AnthropicResponseContentBlock, json.loads, _convert_stop_reason, AnthropicResponse, AnthropicUsage - Return expressions: AnthropicResponse(model=model, content=content, stop_reason=stop_reason, usage=AnthropicUsage(input_tokens=response.usa… ## `vllm_mlx.api.anthropic_adapter._convert_message` - Kind: function - Signature: `def _convert_message(msg: AnthropicMessage) -> list[Message]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/anthropic_adapter.py#L165-L261 - Implementation: Function `_convert_message` calls `isinstance`, `Message`, `text_parts.append`, `tool_calls_for_assistant.append`; has 2 explicit return paths. Convert an Anthropic message to one or more OpenAI messages. Anthropic tool_result blocks (sent as user messages) need to be split into separate OpenAI tool messages. Args: msg: Anthropic message Returns: List of OpenAI messages - Inputs: - `msg` (AnthropicMessage; required): Anthropic message - Return annotation: `list[Message]` - Calls: isinstance, Message, text_parts.append, tool_calls_for_assistant.append, uuid.uuid4, json.dumps, item.get, parts.append, '\n'.join, tool_results.append, str, messages.append, messages.extend - Return expressions: [Message(role=msg.role, content=msg.content)]; messages ## `vllm_mlx.api.anthropic_adapter._convert_tool` - Kind: function - Signature: `def _convert_tool(tool: AnthropicToolDef) -> ToolDefinition` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/anthropic_adapter.py#L264-L278 - Implementation: Function `_convert_tool` calls `ToolDefinition`; returns `ToolDefinition(type='function', function={'name': tool.name, 'description': tool.description or '', 'parameters': tool.…`. Convert an Anthropic tool definition to OpenAI format. Anthropic: {"name": "...", "description": "...", "input_schema": {...}} OpenAI: {"type": "function", "function": {"name": "...", "description": "...", "parameters": {...}}} - Inputs: - `tool` (AnthropicToolDef; required): Required positional or keyword input. - Return annotation: `ToolDefinition` - Calls: ToolDefinition - Return expressions: ToolDefinition(type='function', function={'name': tool.name, 'description': tool.description or '', 'parameters': tool.… ## `vllm_mlx.api.anthropic_adapter._convert_tool_choice` - Kind: function - Signature: `def _convert_tool_choice(tool_choice: dict) -> str | dict | None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/anthropic_adapter.py#L281-L302 - Implementation: Function `_convert_tool_choice` calls `tool_choice.get`; has 4 explicit return paths. Convert Anthropic tool_choice to OpenAI format. Anthropic: {"type": "auto"} | {"type": "any"} | {"type": "tool", "name": "..."} OpenAI: "auto" | "none" | "required" | {"type": "function", "function": {"name": "..."}} - Inputs: - `tool_choice` (dict; required): Required positional or keyword input. - Return annotation: `str | dict | None` - Calls: tool_choice.get - Return expressions: 'auto'; 'required'; {'type': 'function', 'function': {'name': tool_choice.get('name', '')}}; 'none' ## `vllm_mlx.api.anthropic_adapter._convert_stop_reason` - Kind: function - Signature: `def _convert_stop_reason(openai_reason: str | None) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/anthropic_adapter.py#L305-L321 - Implementation: Function `_convert_stop_reason` calls `mapping.get`; has 2 explicit return paths. Convert OpenAI finish_reason to Anthropic stop_reason. OpenAI: "stop" | "tool_calls" | "length" | "content_filter" Anthropic: "end_turn" | "tool_use" | "max_tokens" | "stop_sequence" - Inputs: - `openai_reason` (str | None; required): Required positional or keyword input. - Return annotation: `str` - Calls: mapping.get - Return expressions: 'end_turn'; mapping.get(openai_reason, 'end_turn') # Module `vllm_mlx.api.anthropic_models` Pydantic models for Anthropic Messages API. These models define the request and response schemas for the Anthropic-compatible /v1/messages endpoint, enabling clients like Claude Code to communicate with vllm-mlx. Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/anthropic_models.py#L1-L113 ## `vllm_mlx.api.anthropic_models.AnthropicContentBlock` - Kind: class - Signature: `class AnthropicContentBlock(BaseModel)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/anthropic_models.py#L20-L35 - Implementation: Class `AnthropicContentBlock` derives from `BaseModel` and declares 0 direct member(s). A content block in an Anthropic message. - Inputs: - `type` (str; required): Required constructor field. - `text` (str | None; optional; default `None`): Optional constructor field; defaults to `None`. - `id` (str | None; optional; default `None`): Optional constructor field; defaults to `None`. - `name` (str | None; optional; default `None`): Optional constructor field; defaults to `None`. - `input` (dict | None; optional; default `None`): Optional constructor field; defaults to `None`. - `tool_use_id` (str | None; optional; default `None`): Optional constructor field; defaults to `None`. - `content` (str | list | None; optional; default `None`): Optional constructor field; defaults to `None`. - `is_error` (bool | None; optional; default `None`): Optional constructor field; defaults to `None`. - `source` (dict | None; optional; default `None`): Optional constructor field; defaults to `None`. - Constructs: `vllm_mlx.api.anthropic_models.AnthropicContentBlock` ## `vllm_mlx.api.anthropic_models.AnthropicMessage` - Kind: class - Signature: `class AnthropicMessage(BaseModel)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/anthropic_models.py#L38-L42 - Implementation: Class `AnthropicMessage` derives from `BaseModel` and declares 0 direct member(s). A message in an Anthropic conversation. - Inputs: - `role` (str; required): Required constructor field. - `content` (str | list[AnthropicContentBlock]; required): Required constructor field. - Constructs: `vllm_mlx.api.anthropic_models.AnthropicMessage` ## `vllm_mlx.api.anthropic_models.AnthropicToolDef` - Kind: class - Signature: `class AnthropicToolDef(BaseModel)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/anthropic_models.py#L45-L50 - Implementation: Class `AnthropicToolDef` derives from `BaseModel` and declares 0 direct member(s). Definition of a tool in Anthropic format. - Inputs: - `name` (str; required): Required constructor field. - `description` (str | None; optional; default `None`): Optional constructor field; defaults to `None`. - `input_schema` (dict | None; optional; default `None`): Optional constructor field; defaults to `None`. - Constructs: `vllm_mlx.api.anthropic_models.AnthropicToolDef` ## `vllm_mlx.api.anthropic_models.AnthropicRequest` - Kind: class - Signature: `class AnthropicRequest(BaseModel)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/anthropic_models.py#L53-L73 - Implementation: Class `AnthropicRequest` derives from `BaseModel` and declares 0 direct member(s). Request for Anthropic Messages API. - Inputs: - `model` (str; required): Required constructor field. - `messages` (list[AnthropicMessage]; required): Required constructor field. - `system` (str | list[dict] | None; optional; default `None`): Optional constructor field; defaults to `None`. - `max_tokens` (int; optional; default `Field(gt=0)`): Optional constructor field; defaults to `Field(gt=0)`. - `temperature` (float | None; optional; default `None`): Optional constructor field; defaults to `None`. - `top_p` (float | None; optional; default `None`): Optional constructor field; defaults to `None`. - `stream` (bool; optional; default `False`): Optional constructor field; defaults to `False`. - `stop_sequences` (list[str] | None; optional; default `None`): Optional constructor field; defaults to `None`. - `tools` (list[AnthropicToolDef] | None; optional; default `None`): Optional constructor field; defaults to `None`. - `tool_choice` (dict | None; optional; default `None`): Optional constructor field; defaults to `None`. - `metadata` (dict | None; optional; default `None`): Optional constructor field; defaults to `None`. - `top_k` (int | None; optional; default `None`): Optional constructor field; defaults to `None`. - `response_format` (dict | None; optional; default `None`): Optional constructor field; defaults to `None`. - `chat_template_kwargs` (dict[str, Any] | None; optional; default `None`): Optional constructor field; defaults to `None`. - Constructs: `vllm_mlx.api.anthropic_models.AnthropicRequest` ## `vllm_mlx.api.anthropic_models.AnthropicUsage` - Kind: class - Signature: `class AnthropicUsage(BaseModel)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/anthropic_models.py#L81-L87 - Implementation: Class `AnthropicUsage` derives from `BaseModel` and declares 0 direct member(s). Token usage for Anthropic response. - Inputs: - `input_tokens` (int; optional; default `0`): Optional constructor field; defaults to `0`. - `output_tokens` (int; optional; default `0`): Optional constructor field; defaults to `0`. - `cache_creation_input_tokens` (int | None; optional; default `None`): Optional constructor field; defaults to `None`. - `cache_read_input_tokens` (int | None; optional; default `None`): Optional constructor field; defaults to `None`. - Constructs: `vllm_mlx.api.anthropic_models.AnthropicUsage` ## `vllm_mlx.api.anthropic_models.AnthropicResponseContentBlock` - Kind: class - Signature: `class AnthropicResponseContentBlock(BaseModel)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/anthropic_models.py#L90-L100 - Implementation: Class `AnthropicResponseContentBlock` derives from `BaseModel` and declares 0 direct member(s). A content block in the Anthropic response. - Inputs: - `type` (str; required): Required constructor field. - `text` (str | None; optional; default `None`): Optional constructor field; defaults to `None`. - `thinking` (str | None; optional; default `None`): Optional constructor field; defaults to `None`. - `id` (str | None; optional; default `None`): Optional constructor field; defaults to `None`. - `name` (str | None; optional; default `None`): Optional constructor field; defaults to `None`. - `input` (Any | None; optional; default `None`): Optional constructor field; defaults to `None`. - Constructs: `vllm_mlx.api.anthropic_models.AnthropicResponseContentBlock` ## `vllm_mlx.api.anthropic_models.AnthropicResponse` - Kind: class - Signature: `class AnthropicResponse(BaseModel)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/anthropic_models.py#L103-L113 - Implementation: Class `AnthropicResponse` derives from `BaseModel` and declares 0 direct member(s). Response for Anthropic Messages API. - Inputs: - `id` (str; optional; default `Field(default_factory=lambda: f'msg_{uuid.uuid4().hex[:24]}')`): Optional constructor field; defaults to `Field(default_factory=lambda: f'msg_{uuid.uuid4().hex[:24]}')`. - `type` (str; optional; default `'message'`): Optional constructor field; defaults to `'message'`. - `role` (str; optional; default `'assistant'`): Optional constructor field; defaults to `'assistant'`. - `model` (str; required): Required constructor field. - `content` (list[AnthropicResponseContentBlock]; required): Required constructor field. - `stop_reason` (str | None; optional; default `None`): Optional constructor field; defaults to `None`. - `stop_sequence` (str | None; optional; default `None`): Optional constructor field; defaults to `None`. - `usage` (AnthropicUsage; optional; default `Field(default_factory=AnthropicUsage)`): Optional constructor field; defaults to `Field(default_factory=AnthropicUsage)`. - Constructs: `vllm_mlx.api.anthropic_models.AnthropicResponse` # Module `vllm_mlx.api.harmony_tools` TypeScript-style tool definition converter for Harmony/GPT-OSS models. Harmony models expect tool definitions in TypeScript namespace format: namespace functions { // Get weather for a location type get_weather = (_: { location: string, unit?: "celsius" | "fahrenheit" }) => any; } This module converts OpenAI JSON Schema tool definitions to that format. Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/harmony_tools.py#L1-L109 ## `vllm_mlx.api.harmony_tools._convert_type` - Kind: function - Signature: `def _convert_type(prop: dict[str, Any]) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/harmony_tools.py#L31-L54 - Implementation: Function `_convert_type` calls `' | '.join`, `prop.get`, `_convert_type`, `_TYPE_MAP.get`; has 3 explicit return paths. Convert a JSON Schema property to a TypeScript type string. Args: prop: JSON Schema property definition. Returns: TypeScript type string. - Inputs: - `prop` (dict[str, Any]; required): JSON Schema property definition. - Return annotation: `str` - Calls: ' | '.join, prop.get, _convert_type, _TYPE_MAP.get - Return expressions: ' | '.join(literals); f'Array<{item_type}>'; _TYPE_MAP.get(schema_type, 'any') ## `vllm_mlx.api.harmony_tools.convert_tools_to_typescript` - Kind: function - Signature: `def convert_tools_to_typescript(tools: list[dict[str, Any]] | None) -> str | None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/harmony_tools.py#L57-L109 - Implementation: Function `convert_tools_to_typescript` calls `tool.get`, `func.get`, `parameters.get`, `set`; has 2 explicit return paths. Convert OpenAI JSON Schema tool definitions to TypeScript namespace format. Args: tools: List of tool definitions in OpenAI format, e.g.: [{"type": "function", "function": {"name": "...", ...}}] Returns: TypeScript namespace string, or None if no tools. - Inputs: - `tools` (list[dict[str, Any]] | None; required): List of tool definitions in OpenAI format, e.g.: [{"type": "function", "function": {"name": "...", ...}}] - Return annotation: `str | None` - Calls: tool.get, func.get, parameters.get, set, properties.items, _convert_type, params.append, lines.append, '\n'.join, functions.append, '\n\n'.join - Return expressions: None; f'namespace functions {{\n{body}\n}}' # Module `vllm_mlx.api.models` Pydantic models for OpenAI-compatible API. These models define the request and response schemas for: - Chat completions - Text completions - Tool calling - MCP (Model Context Protocol) integration Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/models.py#L1-L581 ## `vllm_mlx.api.models.ImageUrl` - Kind: class - Signature: `class ImageUrl(BaseModel)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/models.py#L24-L28 - Implementation: Class `ImageUrl` derives from `BaseModel` and declares 0 direct member(s). Image URL with optional detail level. - Inputs: - `url` (str; required): Required constructor field. - `detail` (str | None; optional; default `None`): Optional constructor field; defaults to `None`. - Constructs: `vllm_mlx.api.models.ImageUrl` ## `vllm_mlx.api.models.VideoUrl` - Kind: class - Signature: `class VideoUrl(BaseModel)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/models.py#L31-L34 - Implementation: Class `VideoUrl` derives from `BaseModel` and declares 0 direct member(s). Video URL. - Inputs: - `url` (str; required): Required constructor field. - Constructs: `vllm_mlx.api.models.VideoUrl` ## `vllm_mlx.api.models.AudioUrl` - Kind: class - Signature: `class AudioUrl(BaseModel)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/models.py#L37-L40 - Implementation: Class `AudioUrl` derives from `BaseModel` and declares 0 direct member(s). Audio URL for audio content. - Inputs: - `url` (str; required): Required constructor field. - Constructs: `vllm_mlx.api.models.AudioUrl` ## `vllm_mlx.api.models.ContentPart` - Kind: class - Signature: `class ContentPart(BaseModel)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/models.py#L43-L60 - Implementation: Class `ContentPart` derives from `BaseModel` and declares 0 direct member(s). A part of a multimodal message content. Supports: - text: Plain text content - image_url: Image from URL or base64 - video: Video from local path - video_url: Video from URL or base64 - audio_url: Audio from URL or base64 - Inputs: - `type` (str; required): Required constructor field. - `text` (str | None; optional; default `None`): Optional constructor field; defaults to `None`. - `image_url` (ImageUrl | dict | str | None; optional; default `None`): Optional constructor field; defaults to `None`. - `video` (str | None; optional; default `None`): Optional constructor field; defaults to `None`. - `video_url` (VideoUrl | dict | str | None; optional; default `None`): Optional constructor field; defaults to `None`. - `audio_url` (AudioUrl | dict | str | None; optional; default `None`): Optional constructor field; defaults to `None`. - Constructs: `vllm_mlx.api.models.ContentPart` ## `vllm_mlx.api.models.Message` - Kind: class - Signature: `class Message(BaseModel)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/models.py#L68-L84 - Implementation: Class `Message` derives from `BaseModel` and declares 0 direct member(s). A message in a chat conversation. Supports: - Simple text messages (role + content string) - Multimodal messages (role + content list with text/images/videos) - Tool call messages (assistant with tool_calls) - Tool response messages (role="tool" with tool_call_id) - Inputs: - `role` (str; required): Required constructor field. - `content` (str | list[ContentPart] | list[dict] | None; optional; default `None`): Optional constructor field; defaults to `None`. - `tool_calls` (list[dict] | None; optional; default `None`): Optional constructor field; defaults to `None`. - `tool_call_id` (str | None; optional; default `None`): Optional constructor field; defaults to `None`. - Constructs: `vllm_mlx.api.models.Message` ## `vllm_mlx.api.models.FunctionCall` - Kind: class - Signature: `class FunctionCall(BaseModel)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/models.py#L95-L99 - Implementation: Class `FunctionCall` derives from `BaseModel` and declares 0 direct member(s). A function call with name and arguments. - Inputs: - `name` (str; required): Required constructor field. - `arguments` (str; required): Required constructor field. - Constructs: `vllm_mlx.api.models.FunctionCall` ## `vllm_mlx.api.models.ToolCall` - Kind: class - Signature: `class ToolCall(BaseModel)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/models.py#L102-L107 - Implementation: Class `ToolCall` derives from `BaseModel` and declares 0 direct member(s). A tool call from the model. - Inputs: - `id` (str; required): Required constructor field. - `type` (str; optional; default `'function'`): Optional constructor field; defaults to `'function'`. - `function` (FunctionCall; required): Required constructor field. - Constructs: `vllm_mlx.api.models.ToolCall` ## `vllm_mlx.api.models.ToolDefinition` - Kind: class - Signature: `class ToolDefinition(BaseModel)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/models.py#L110-L123 - Implementation: Class `ToolDefinition` derives from `BaseModel` and declares 1 direct member(s). Definition of a tool that can be called by the model. - Inputs: - `type` (str; optional; default `'function'`): Optional constructor field; defaults to `'function'`. - `function` (dict; required): Required constructor field. - Constructs: `vllm_mlx.api.models.ToolDefinition` ## `vllm_mlx.api.models.ToolDefinition._validate_openai_function_name` - Kind: method - Signature: `def _validate_openai_function_name(self)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/models.py#L117-L123 - Implementation: Method `ToolDefinition._validate_openai_function_name` calls `self.function.get`, `isinstance`, `_OPENAI_FUNCTION_NAME_RE.fullmatch`, `ValueError`; can raise `ValueError`; returns `self`. Method `ToolDefinition._validate_openai_function_name` calls `self.function.get`, `isinstance`, `_OPENAI_FUNCTION_NAME_RE.fullmatch`, `ValueError`; can raise `ValueError`; returns `self`. - Inputs: none - Return annotation: `not annotated` - Decorators: model_validator(mode='after') - Calls: self.function.get, isinstance, _OPENAI_FUNCTION_NAME_RE.fullmatch, ValueError - State reads: self.type, self.function.get, self.function - Raises directly: ValueError - Return expressions: self ## `vllm_mlx.api.models.ResponseFormatJsonSchema` - Kind: class - Signature: `class ResponseFormatJsonSchema(BaseModel)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/models.py#L131-L142 - Implementation: Class `ResponseFormatJsonSchema` derives from `BaseModel` and declares 1 direct member(s). JSON Schema definition for structured output. - Inputs: - `name` (str; required): Required constructor field. - `description` (str | None; optional; default `None`): Optional constructor field; defaults to `None`. - `schema_` (dict; optional; default `Field(alias='schema')`): Optional constructor field; defaults to `Field(alias='schema')`. - `strict` (bool | None; optional; default `False`): Optional constructor field; defaults to `False`. - Constructs: `vllm_mlx.api.models.ResponseFormatJsonSchema` ## `vllm_mlx.api.models.ResponseFormatJsonSchema.Config` - Kind: class - Signature: `class Config` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/models.py#L139-L142 - Implementation: Class `ResponseFormatJsonSchema.Config` declares 0 direct member(s). Allow callers to populate the aliased ``schema`` field by name. - Inputs: none - Constructs: `vllm_mlx.api.models.ResponseFormatJsonSchema.Config` ## `vllm_mlx.api.models.ResponseFormat` - Kind: class - Signature: `class ResponseFormat(BaseModel)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/models.py#L145-L156 - Implementation: Class `ResponseFormat` derives from `BaseModel` and declares 0 direct member(s). Response format specification for structured output. Supports: - "text": Default text output (no structure enforcement) - "json_object": Forces valid JSON output - "json_schema": Forces JSON matching a specific schema - Inputs: - `type` (str; optional; default `'text'`): Optional constructor field; defaults to `'text'`. - `json_schema` (ResponseFormatJsonSchema | None; optional; default `None`): Optional constructor field; defaults to `None`. - Constructs: `vllm_mlx.api.models.ResponseFormat` ## `vllm_mlx.api.models.StreamOptions` - Kind: class - Signature: `class StreamOptions(BaseModel)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/models.py#L164-L167 - Implementation: Class `StreamOptions` derives from `BaseModel` and declares 0 direct member(s). Options for streaming responses. - Inputs: - `include_usage` (bool; optional; default `False`): Optional constructor field; defaults to `False`. - Constructs: `vllm_mlx.api.models.StreamOptions` ## `vllm_mlx.api.models.ChatCompletionRequest` - Kind: class - Signature: `class ChatCompletionRequest(BaseModel)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/models.py#L170-L216 - Implementation: Class `ChatCompletionRequest` derives from `BaseModel` and declares 0 direct member(s). Request for chat completion. - Inputs: - `model` (str; required): Required constructor field. - `messages` (list[Message]; required): Required constructor field. - `temperature` (float | None; optional; default `None`): Optional constructor field; defaults to `None`. - `top_p` (float | None; optional; default `None`): Optional constructor field; defaults to `None`. - `top_k` (int | None; optional; default `None`): Optional constructor field; defaults to `None`. - `min_p` (float | None; optional; default `None`): Optional constructor field; defaults to `None`. - `presence_penalty` (float | None; optional; default `None`): Optional constructor field; defaults to `None`. - `max_tokens` (int | None; optional; default `Field(default=None, gt=0)`): Optional constructor field; defaults to `Field(default=None, gt=0)`. - `stream` (bool; optional; default `False`): Optional constructor field; defaults to `False`. - `stream_options` (StreamOptions | None; optional; default `None`): Optional constructor field; defaults to `None`. - `stop` (list[str] | None; optional; default `None`): Optional constructor field; defaults to `None`. - `tools` (list[ToolDefinition] | None; optional; default `None`): Optional constructor field; defaults to `None`. - `tool_choice` (str | dict | None; optional; default `None`): Optional constructor field; defaults to `None`. - `response_format` (ResponseFormat | dict | None; optional; default `None`): Optional constructor field; defaults to `None`. - `logit_bias` (dict[str, float] | None; optional; default `None`): Optional constructor field; defaults to `None`. - `chat_template_kwargs` (dict[str, Any] | None; optional; default `None`): Optional constructor field; defaults to `None`. - `video_fps` (float | None; optional; default `None`): Optional constructor field; defaults to `None`. - `video_max_frames` (int | None; optional; default `None`): Optional constructor field; defaults to `None`. - `repetition_penalty` (float | None; optional; default `None`): Optional constructor field; defaults to `None`. - `timeout` (float | None; optional; default `None`): Optional constructor field; defaults to `None`. - `specprefill` (bool | None; optional; default `None`): Optional constructor field; defaults to `None`. - `specprefill_keep_pct` (float | None; optional; default `None`): Optional constructor field; defaults to `None`. - `specprefill_backbone_pct` (float | None; optional; default `None`): Optional constructor field; defaults to `None`. - `enable_thinking` (bool | None; optional; default `None`): Optional constructor field; defaults to `None`. - `mllm_draft` (bool | None; optional; default `None`): Optional constructor field; defaults to `None`. - `thinking_token_budget` (int | None; optional; default `Field(default=None, gt=0)`): Optional constructor field; defaults to `Field(default=None, gt=0)`. - Constructs: `vllm_mlx.api.models.ChatCompletionRequest` ## `vllm_mlx.api.models.AssistantMessage` - Kind: class - Signature: `class AssistantMessage(BaseModel)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/models.py#L219-L248 - Implementation: Class `AssistantMessage` derives from `BaseModel` and declares 2 direct member(s). Response message from the assistant. - Inputs: - `role` (str; optional; default `'assistant'`): Optional constructor field; defaults to `'assistant'`. - `content` (str | None; optional; default `None`): Optional constructor field; defaults to `None`. - `reasoning_content` (str | None; optional; default `Field(default=None, validation_alias=AliasChoices('reasoning_content', 'reasoning'))`): Optional constructor field; defaults to `Field(default=None, validation_alias=AliasChoices('reasoning_content', 'reasoning'))`. - `tool_calls` (list[ToolCall] | None; optional; default `None`): Optional constructor field; defaults to `None`. - Constructs: `vllm_mlx.api.models.AssistantMessage` ## `vllm_mlx.api.models.AssistantMessage.reasoning` - Kind: method - Signature: `def reasoning(self) -> str | None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/models.py#L231-L234 - Implementation: Method `AssistantMessage.reasoning` returns `self.reasoning_content`. Return reasoning content through the legacy compatibility alias. - Inputs: none - Return annotation: `str | None` - Decorators: property - State reads: self.reasoning_content - Return expressions: self.reasoning_content ## `vllm_mlx.api.models.AssistantMessage._serialize` - Kind: method - Signature: `def _serialize(self) -> dict` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/models.py#L237-L248 - Implementation: Method `AssistantMessage._serialize` calls `tc.model_dump`; returns `d`. Serialize with OpenAI-compatible schema. - ``tool_calls`` and ``reasoning_content`` are omitted when None. - ``content`` is always included (even as null) per OpenAI spec. - Inputs: none - Return annotation: `dict` - Decorators: model_serializer - Calls: tc.model_dump - State reads: self.role, self.content, self.reasoning_content, self.tool_calls - Return expressions: d ## `vllm_mlx.api.models.ChatCompletionChoice` - Kind: class - Signature: `class ChatCompletionChoice(BaseModel)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/models.py#L251-L256 - Implementation: Class `ChatCompletionChoice` derives from `BaseModel` and declares 0 direct member(s). A single choice in chat completion response. - Inputs: - `index` (int; optional; default `0`): Optional constructor field; defaults to `0`. - `message` (AssistantMessage; required): Required constructor field. - `finish_reason` (str | None; optional; default `'stop'`): Optional constructor field; defaults to `'stop'`. - Constructs: `vllm_mlx.api.models.ChatCompletionChoice` ## `vllm_mlx.api.models.Usage` - Kind: class - Signature: `class Usage(BaseModel)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/models.py#L259-L264 - Implementation: Class `Usage` derives from `BaseModel` and declares 0 direct member(s). Token usage statistics. - Inputs: - `prompt_tokens` (int; optional; default `0`): Optional constructor field; defaults to `0`. - `completion_tokens` (int; optional; default `0`): Optional constructor field; defaults to `0`. - `total_tokens` (int; optional; default `0`): Optional constructor field; defaults to `0`. - Constructs: `vllm_mlx.api.models.Usage` ## `vllm_mlx.api.models.GenerationMetadata` - Kind: class - Signature: `class GenerationMetadata(BaseModel)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/models.py#L267-L271 - Implementation: Class `GenerationMetadata` derives from `BaseModel` and declares 0 direct member(s). Optional generation diagnostics emitted for feature-bearing requests. - Inputs: - `no_final_content_watchdog_tokens` (int | None; optional; default `None`): Optional constructor field; defaults to `None`. - `no_final_content_watchdog_enforced` (bool; optional; default `False`): Optional constructor field; defaults to `False`. - Constructs: `vllm_mlx.api.models.GenerationMetadata` ## `vllm_mlx.api.models.ChatCompletionResponse` - Kind: class - Signature: `class ChatCompletionResponse(BaseModel)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/models.py#L274-L283 - Implementation: Class `ChatCompletionResponse` derives from `BaseModel` and declares 0 direct member(s). Response for chat completion. - Inputs: - `id` (str; optional; default `Field(default_factory=lambda: f'chatcmpl-{uuid.uuid4().hex[:8]}')`): Optional constructor field; defaults to `Field(default_factory=lambda: f'chatcmpl-{uuid.uuid4().hex[:8]}')`. - `object` (str; optional; default `'chat.completion'`): Optional constructor field; defaults to `'chat.completion'`. - `created` (int; optional; default `Field(default_factory=lambda: int(time.time()))`): Optional constructor field; defaults to `Field(default_factory=lambda: int(time.time()))`. - `model` (str; required): Required constructor field. - `choices` (list[ChatCompletionChoice]; required): Required constructor field. - `usage` (Usage; optional; default `Field(default_factory=Usage)`): Optional constructor field; defaults to `Field(default_factory=Usage)`. - `generation_metadata` (GenerationMetadata | None; optional; default `None`): Optional constructor field; defaults to `None`. - Constructs: `vllm_mlx.api.models.ChatCompletionResponse` ## `vllm_mlx.api.models.CompletionRequest` - Kind: class - Signature: `class CompletionRequest(BaseModel)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/models.py#L291-L313 - Implementation: Class `CompletionRequest` derives from `BaseModel` and declares 0 direct member(s). Request for text completion. - Inputs: - `model` (str; required): Required constructor field. - `prompt` (str | list[str]; required): Required constructor field. - `temperature` (float | None; optional; default `None`): Optional constructor field; defaults to `None`. - `top_p` (float | None; optional; default `None`): Optional constructor field; defaults to `None`. - `top_k` (int | None; optional; default `None`): Optional constructor field; defaults to `None`. - `min_p` (float | None; optional; default `None`): Optional constructor field; defaults to `None`. - `presence_penalty` (float | None; optional; default `None`): Optional constructor field; defaults to `None`. - `max_tokens` (int | None; optional; default `Field(default=None, gt=0)`): Optional constructor field; defaults to `Field(default=None, gt=0)`. - `stream` (bool; optional; default `False`): Optional constructor field; defaults to `False`. - `stop` (list[str] | None; optional; default `None`): Optional constructor field; defaults to `None`. - `repetition_penalty` (float | None; optional; default `None`): Optional constructor field; defaults to `None`. - `timeout` (float | None; optional; default `None`): Optional constructor field; defaults to `None`. - `specprefill` (bool | None; optional; default `None`): Optional constructor field; defaults to `None`. - `specprefill_keep_pct` (float | None; optional; default `None`): Optional constructor field; defaults to `None`. - `specprefill_backbone_pct` (float | None; optional; default `None`): Optional constructor field; defaults to `None`. - Constructs: `vllm_mlx.api.models.CompletionRequest` ## `vllm_mlx.api.models.CompletionChoice` - Kind: class - Signature: `class CompletionChoice(BaseModel)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/models.py#L316-L321 - Implementation: Class `CompletionChoice` derives from `BaseModel` and declares 0 direct member(s). A single choice in text completion response. - Inputs: - `index` (int; optional; default `0`): Optional constructor field; defaults to `0`. - `text` (str; required): Required constructor field. - `finish_reason` (str | None; optional; default `'stop'`): Optional constructor field; defaults to `'stop'`. - Constructs: `vllm_mlx.api.models.CompletionChoice` ## `vllm_mlx.api.models.CompletionResponse` - Kind: class - Signature: `class CompletionResponse(BaseModel)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/models.py#L324-L332 - Implementation: Class `CompletionResponse` derives from `BaseModel` and declares 0 direct member(s). Response for text completion. - Inputs: - `id` (str; optional; default `Field(default_factory=lambda: f'cmpl-{uuid.uuid4().hex[:8]}')`): Optional constructor field; defaults to `Field(default_factory=lambda: f'cmpl-{uuid.uuid4().hex[:8]}')`. - `object` (str; optional; default `'text_completion'`): Optional constructor field; defaults to `'text_completion'`. - `created` (int; optional; default `Field(default_factory=lambda: int(time.time()))`): Optional constructor field; defaults to `Field(default_factory=lambda: int(time.time()))`. - `model` (str; required): Required constructor field. - `choices` (list[CompletionChoice]; required): Required constructor field. - `usage` (Usage; optional; default `Field(default_factory=Usage)`): Optional constructor field; defaults to `Field(default_factory=Usage)`. - Constructs: `vllm_mlx.api.models.CompletionResponse` ## `vllm_mlx.api.models.ModelInfo` - Kind: class - Signature: `class ModelInfo(BaseModel)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/models.py#L340-L346 - Implementation: Class `ModelInfo` derives from `BaseModel` and declares 0 direct member(s). Information about an available model. - Inputs: - `id` (str; required): Required constructor field. - `object` (str; optional; default `'model'`): Optional constructor field; defaults to `'model'`. - `created` (int; optional; default `Field(default_factory=lambda: int(time.time()))`): Optional constructor field; defaults to `Field(default_factory=lambda: int(time.time()))`. - `owned_by` (str; optional; default `'vllm-mlx'`): Optional constructor field; defaults to `'vllm-mlx'`. - Constructs: `vllm_mlx.api.models.ModelInfo` ## `vllm_mlx.api.models.ModelsResponse` - Kind: class - Signature: `class ModelsResponse(BaseModel)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/models.py#L349-L353 - Implementation: Class `ModelsResponse` derives from `BaseModel` and declares 0 direct member(s). Response for listing models. - Inputs: - `object` (str; optional; default `'list'`): Optional constructor field; defaults to `'list'`. - `data` (list[ModelInfo]; required): Required constructor field. - Constructs: `vllm_mlx.api.models.ModelsResponse` ## `vllm_mlx.api.models.MCPToolInfo` - Kind: class - Signature: `class MCPToolInfo(BaseModel)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/models.py#L361-L367 - Implementation: Class `MCPToolInfo` derives from `BaseModel` and declares 0 direct member(s). Information about an MCP tool. - Inputs: - `name` (str; required): Required constructor field. - `description` (str; required): Required constructor field. - `server` (str; required): Required constructor field. - `parameters` (dict; optional; default `Field(default_factory=dict)`): Optional constructor field; defaults to `Field(default_factory=dict)`. - Constructs: `vllm_mlx.api.models.MCPToolInfo` ## `vllm_mlx.api.models.MCPToolsResponse` - Kind: class - Signature: `class MCPToolsResponse(BaseModel)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/models.py#L370-L374 - Implementation: Class `MCPToolsResponse` derives from `BaseModel` and declares 0 direct member(s). Response for listing MCP tools. - Inputs: - `tools` (list[MCPToolInfo]; required): Required constructor field. - `count` (int; required): Required constructor field. - Constructs: `vllm_mlx.api.models.MCPToolsResponse` ## `vllm_mlx.api.models.MCPServerInfo` - Kind: class - Signature: `class MCPServerInfo(BaseModel)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/models.py#L377-L384 - Implementation: Class `MCPServerInfo` derives from `BaseModel` and declares 0 direct member(s). Information about an MCP server. - Inputs: - `name` (str; required): Required constructor field. - `state` (str; required): Required constructor field. - `transport` (str; required): Required constructor field. - `tools_count` (int; required): Required constructor field. - `error` (str | None; optional; default `None`): Optional constructor field; defaults to `None`. - Constructs: `vllm_mlx.api.models.MCPServerInfo` ## `vllm_mlx.api.models.MCPServersResponse` - Kind: class - Signature: `class MCPServersResponse(BaseModel)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/models.py#L387-L390 - Implementation: Class `MCPServersResponse` derives from `BaseModel` and declares 0 direct member(s). Response for listing MCP servers. - Inputs: - `servers` (list[MCPServerInfo]; required): Required constructor field. - Constructs: `vllm_mlx.api.models.MCPServersResponse` ## `vllm_mlx.api.models.MCPExecuteRequest` - Kind: class - Signature: `class MCPExecuteRequest(BaseModel)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/models.py#L393-L397 - Implementation: Class `MCPExecuteRequest` derives from `BaseModel` and declares 0 direct member(s). Request to execute an MCP tool. - Inputs: - `tool_name` (str; required): Required constructor field. - `arguments` (dict; optional; default `Field(default_factory=dict)`): Optional constructor field; defaults to `Field(default_factory=dict)`. - Constructs: `vllm_mlx.api.models.MCPExecuteRequest` ## `vllm_mlx.api.models.MCPExecuteResponse` - Kind: class - Signature: `class MCPExecuteResponse(BaseModel)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/models.py#L400-L406 - Implementation: Class `MCPExecuteResponse` derives from `BaseModel` and declares 0 direct member(s). Response from executing an MCP tool. - Inputs: - `tool_name` (str; required): Required constructor field. - `content` (str | list | dict | None; optional; default `None`): Optional constructor field; defaults to `None`. - `is_error` (bool; optional; default `False`): Optional constructor field; defaults to `False`. - `error_message` (str | None; optional; default `None`): Optional constructor field; defaults to `None`. - Constructs: `vllm_mlx.api.models.MCPExecuteResponse` ## `vllm_mlx.api.models.AudioTranscriptionRequest` - Kind: class - Signature: `class AudioTranscriptionRequest(BaseModel)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/models.py#L414-L421 - Implementation: Class `AudioTranscriptionRequest` derives from `BaseModel` and declares 0 direct member(s). Request for audio transcription (STT). - Inputs: - `model` (str; optional; default `'whisper-large-v3'`): Optional constructor field; defaults to `'whisper-large-v3'`. - `language` (str | None; optional; default `None`): Optional constructor field; defaults to `None`. - `response_format` (str; optional; default `'json'`): Optional constructor field; defaults to `'json'`. - `temperature` (float; optional; default `0.0`): Optional constructor field; defaults to `0.0`. - `timestamp_granularities` (list[str] | None; optional; default `None`): Optional constructor field; defaults to `None`. - Constructs: `vllm_mlx.api.models.AudioTranscriptionRequest` ## `vllm_mlx.api.models.AudioTranscriptionResponse` - Kind: class - Signature: `class AudioTranscriptionResponse(BaseModel)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/models.py#L424-L430 - Implementation: Class `AudioTranscriptionResponse` derives from `BaseModel` and declares 0 direct member(s). Response from audio transcription. - Inputs: - `text` (str; required): Required constructor field. - `language` (str | None; optional; default `None`): Optional constructor field; defaults to `None`. - `duration` (float | None; optional; default `None`): Optional constructor field; defaults to `None`. - `segments` (list[dict] | None; optional; default `None`): Optional constructor field; defaults to `None`. - Constructs: `vllm_mlx.api.models.AudioTranscriptionResponse` ## `vllm_mlx.api.models.AudioSpeechRequest` - Kind: class - Signature: `class AudioSpeechRequest(BaseModel)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/models.py#L433-L440 - Implementation: Class `AudioSpeechRequest` derives from `BaseModel` and declares 0 direct member(s). Request for text-to-speech. - Inputs: - `model` (str; optional; default `'kokoro'`): Optional constructor field; defaults to `'kokoro'`. - `input` (str; required): Required constructor field. - `voice` (str; optional; default `'af_heart'`): Optional constructor field; defaults to `'af_heart'`. - `speed` (float; optional; default `1.0`): Optional constructor field; defaults to `1.0`. - `response_format` (str; optional; default `'wav'`): Optional constructor field; defaults to `'wav'`. - Constructs: `vllm_mlx.api.models.AudioSpeechRequest` ## `vllm_mlx.api.models.AudioSeparationRequest` - Kind: class - Signature: `class AudioSeparationRequest(BaseModel)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/models.py#L443-L447 - Implementation: Class `AudioSeparationRequest` derives from `BaseModel` and declares 0 direct member(s). Request for audio source separation. - Inputs: - `model` (str; optional; default `'htdemucs'`): Optional constructor field; defaults to `'htdemucs'`. - `stems` (list[str]; optional; default `Field(default_factory=lambda: ['vocals', 'accompaniment'])`): Optional constructor field; defaults to `Field(default_factory=lambda: ['vocals', 'accompaniment'])`. - Constructs: `vllm_mlx.api.models.AudioSeparationRequest` ## `vllm_mlx.api.models.EmbeddingRequest` - Kind: class - Signature: `class EmbeddingRequest(BaseModel)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/models.py#L455-L460 - Implementation: Class `EmbeddingRequest` derives from `BaseModel` and declares 0 direct member(s). Request for text embeddings (OpenAI compatible). - Inputs: - `input` (str | list[str]; required): Required constructor field. - `model` (str; required): Required constructor field. - `encoding_format` (str | None; optional; default `'float'`): Optional constructor field; defaults to `'float'`. - Constructs: `vllm_mlx.api.models.EmbeddingRequest` ## `vllm_mlx.api.models.EmbeddingData` - Kind: class - Signature: `class EmbeddingData(BaseModel)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/models.py#L463-L468 - Implementation: Class `EmbeddingData` derives from `BaseModel` and declares 0 direct member(s). A single embedding result. - Inputs: - `object` (str; optional; default `'embedding'`): Optional constructor field; defaults to `'embedding'`. - `index` (int; required): Required constructor field. - `embedding` (list[float]; required): Required constructor field. - Constructs: `vllm_mlx.api.models.EmbeddingData` ## `vllm_mlx.api.models.EmbeddingUsage` - Kind: class - Signature: `class EmbeddingUsage(BaseModel)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/models.py#L471-L475 - Implementation: Class `EmbeddingUsage` derives from `BaseModel` and declares 0 direct member(s). Token usage for embedding requests. - Inputs: - `prompt_tokens` (int; optional; default `0`): Optional constructor field; defaults to `0`. - `total_tokens` (int; optional; default `0`): Optional constructor field; defaults to `0`. - Constructs: `vllm_mlx.api.models.EmbeddingUsage` ## `vllm_mlx.api.models.EmbeddingResponse` - Kind: class - Signature: `class EmbeddingResponse(BaseModel)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/models.py#L478-L484 - Implementation: Class `EmbeddingResponse` derives from `BaseModel` and declares 0 direct member(s). Response for embeddings endpoint (OpenAI compatible). - Inputs: - `object` (str; optional; default `'list'`): Optional constructor field; defaults to `'list'`. - `data` (list[EmbeddingData]; required): Required constructor field. - `model` (str; required): Required constructor field. - `usage` (EmbeddingUsage; optional; default `Field(default_factory=EmbeddingUsage)`): Optional constructor field; defaults to `Field(default_factory=EmbeddingUsage)`. - Constructs: `vllm_mlx.api.models.EmbeddingResponse` ## `vllm_mlx.api.models.RerankRequest` - Kind: class - Signature: `class RerankRequest(BaseModel)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/models.py#L492-L499 - Implementation: Class `RerankRequest` derives from `BaseModel` and declares 0 direct member(s). Request for reranking documents against a query (Jina/Cohere convention). - Inputs: - `model` (str; required): Required constructor field. - `query` (str; required): Required constructor field. - `documents` (list[str | dict]; required): Required constructor field. - `top_n` (int | None; optional; default `None`): Optional constructor field; defaults to `None`. - `return_documents` (bool; optional; default `True`): Optional constructor field; defaults to `True`. - Constructs: `vllm_mlx.api.models.RerankRequest` ## `vllm_mlx.api.models.RerankResult` - Kind: class - Signature: `class RerankResult(BaseModel)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/models.py#L502-L507 - Implementation: Class `RerankResult` derives from `BaseModel` and declares 0 direct member(s). A single reranked document result. - Inputs: - `index` (int; required): Required constructor field. - `relevance_score` (float; required): Required constructor field. - `document` (dict | None; optional; default `None`): Optional constructor field; defaults to `None`. - Constructs: `vllm_mlx.api.models.RerankResult` ## `vllm_mlx.api.models.RerankUsage` - Kind: class - Signature: `class RerankUsage(BaseModel)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/models.py#L510-L513 - Implementation: Class `RerankUsage` derives from `BaseModel` and declares 0 direct member(s). Token usage for rerank requests. - Inputs: - `total_tokens` (int; optional; default `0`): Optional constructor field; defaults to `0`. - Constructs: `vllm_mlx.api.models.RerankUsage` ## `vllm_mlx.api.models.RerankResponse` - Kind: class - Signature: `class RerankResponse(BaseModel)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/models.py#L516-L521 - Implementation: Class `RerankResponse` derives from `BaseModel` and declares 0 direct member(s). Response for reranking endpoint (Jina/Cohere convention). - Inputs: - `model` (str; required): Required constructor field. - `results` (list[RerankResult]; required): Required constructor field. - `usage` (RerankUsage; optional; default `Field(default_factory=RerankUsage)`): Optional constructor field; defaults to `Field(default_factory=RerankUsage)`. - Constructs: `vllm_mlx.api.models.RerankResponse` ## `vllm_mlx.api.models.ChatCompletionChunkDelta` - Kind: class - Signature: `class ChatCompletionChunkDelta(BaseModel)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/models.py#L529-L562 - Implementation: Class `ChatCompletionChunkDelta` derives from `BaseModel` and declares 2 direct member(s). Delta content in a streaming chunk. - Inputs: - `role` (str | None; optional; default `None`): Optional constructor field; defaults to `None`. - `content` (str | None; optional; default `None`): Optional constructor field; defaults to `None`. - `reasoning_content` (str | None; optional; default `Field(default=None, validation_alias=AliasChoices('reasoning_content', 'reasoning'))`): Optional constructor field; defaults to `Field(default=None, validation_alias=AliasChoices('reasoning_content', 'reasoning'))`. - `tool_calls` (list[dict] | None; optional; default `None`): Optional constructor field; defaults to `None`. - Constructs: `vllm_mlx.api.models.ChatCompletionChunkDelta` ## `vllm_mlx.api.models.ChatCompletionChunkDelta.reasoning` - Kind: method - Signature: `def reasoning(self) -> str | None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/models.py#L541-L544 - Implementation: Method `ChatCompletionChunkDelta.reasoning` returns `self.reasoning_content`. Return incremental reasoning through the compatibility alias. - Inputs: none - Return annotation: `str | None` - Decorators: property - State reads: self.reasoning_content - Return expressions: self.reasoning_content ## `vllm_mlx.api.models.ChatCompletionChunkDelta._serialize` - Kind: method - Signature: `def _serialize(self) -> dict` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/models.py#L547-L562 - Implementation: Method `ChatCompletionChunkDelta._serialize` returns `d`. Serialize delta with only non-None fields. Per OpenAI streaming spec, delta objects only include fields that carry new content. - Inputs: none - Return annotation: `dict` - Decorators: model_serializer - State reads: self.role, self.content, self.reasoning_content, self.tool_calls - Return expressions: d ## `vllm_mlx.api.models.ChatCompletionChunkChoice` - Kind: class - Signature: `class ChatCompletionChunkChoice(BaseModel)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/models.py#L565-L570 - Implementation: Class `ChatCompletionChunkChoice` derives from `BaseModel` and declares 0 direct member(s). A single choice in a streaming chunk. - Inputs: - `index` (int; optional; default `0`): Optional constructor field; defaults to `0`. - `delta` (ChatCompletionChunkDelta; required): Required constructor field. - `finish_reason` (str | None; optional; default `None`): Optional constructor field; defaults to `None`. - Constructs: `vllm_mlx.api.models.ChatCompletionChunkChoice` ## `vllm_mlx.api.models.ChatCompletionChunk` - Kind: class - Signature: `class ChatCompletionChunk(BaseModel)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/models.py#L573-L581 - Implementation: Class `ChatCompletionChunk` derives from `BaseModel` and declares 0 direct member(s). A streaming chunk for chat completion. - Inputs: - `id` (str; optional; default `Field(default_factory=lambda: f'chatcmpl-{uuid.uuid4().hex[:8]}')`): Optional constructor field; defaults to `Field(default_factory=lambda: f'chatcmpl-{uuid.uuid4().hex[:8]}')`. - `object` (str; optional; default `'chat.completion.chunk'`): Optional constructor field; defaults to `'chat.completion.chunk'`. - `created` (int; optional; default `Field(default_factory=lambda: int(time.time()))`): Optional constructor field; defaults to `Field(default_factory=lambda: int(time.time()))`. - `model` (str; required): Required constructor field. - `choices` (list[ChatCompletionChunkChoice]; required): Required constructor field. - `usage` (Usage | None; optional; default `None`): Optional constructor field; defaults to `None`. - Constructs: `vllm_mlx.api.models.ChatCompletionChunk` # Module `vllm_mlx.api.prompt_canonicalize` System-prompt canonicalization helpers. Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/prompt_canonicalize.py#L1-L51 ## `vllm_mlx.api.prompt_canonicalize.canonicalize_system_prompt` - Kind: function - Signature: `def canonicalize_system_prompt(text: str | None) -> str | None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/prompt_canonicalize.py#L17-L24 - Implementation: Function `canonicalize_system_prompt` calls `pattern.sub`; has 2 explicit return paths. Remove known non-semantic volatile lines from system prompt text. - Inputs: - `text` (str | None; required): Required positional or keyword input. - Return annotation: `str | None` - Calls: pattern.sub - Return expressions: None; text ## `vllm_mlx.api.prompt_canonicalize.canonicalize_system_messages` - Kind: function - Signature: `def canonicalize_system_messages(messages: list[dict]) -> list[dict]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/prompt_canonicalize.py#L27-L51 - Implementation: Function `canonicalize_system_messages` calls `message.get`, `canonicalized.append`, `isinstance`, `canonicalize_system_prompt`; returns `canonicalized if changed else messages`. Canonicalize string content on system-role messages without mutation. - Inputs: - `messages` (list[dict]; required): Required positional or keyword input. - Return annotation: `list[dict]` - Calls: message.get, canonicalized.append, isinstance, canonicalize_system_prompt, message.copy - Return expressions: canonicalized if changed else messages # Module `vllm_mlx.api.responses_models` Pydantic models for the OpenAI-compatible Responses API. This intentionally implements the subset needed for local coding-agent workflows: text messages, function tools, function call outputs, and SSE streaming events. The object and event shapes follow the conventions used by OpenAI's gpt-oss reference server and llama.cpp's OpenAI-compatible server. Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/responses_models.py#L1-L342 ## `vllm_mlx.api.responses_models.ResponseTextFormat` - Kind: class - Signature: `class ResponseTextFormat(BaseModel)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/responses_models.py#L19-L22 - Implementation: Class `ResponseTextFormat` derives from `BaseModel` and declares 0 direct member(s). Output text format configuration. - Inputs: - `type` (Literal['text', 'json_object']; optional; default `'text'`): Optional constructor field; defaults to `'text'`. - Constructs: `vllm_mlx.api.responses_models.ResponseTextFormat` ## `vllm_mlx.api.responses_models.ResponseTextConfig` - Kind: class - Signature: `class ResponseTextConfig(BaseModel)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/responses_models.py#L25-L28 - Implementation: Class `ResponseTextConfig` derives from `BaseModel` and declares 0 direct member(s). Text output configuration. - Inputs: - `format` (ResponseTextFormat; optional; default `Field(default_factory=ResponseTextFormat)`): Optional constructor field; defaults to `Field(default_factory=ResponseTextFormat)`. - Constructs: `vllm_mlx.api.responses_models.ResponseTextConfig` ## `vllm_mlx.api.responses_models.ResponseReasoningConfig` - Kind: class - Signature: `class ResponseReasoningConfig(BaseModel)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/responses_models.py#L31-L34 - Implementation: Class `ResponseReasoningConfig` derives from `BaseModel` and declares 0 direct member(s). Reasoning configuration. - Inputs: - `effort` (Literal['none', 'minimal', 'low', 'medium', 'high', 'xhigh'] | None; optional; default `None`): Optional constructor field; defaults to `None`. - Constructs: `vllm_mlx.api.responses_models.ResponseReasoningConfig` ## `vllm_mlx.api.responses_models.ResponseTextContentPart` - Kind: class - Signature: `class ResponseTextContentPart(BaseModel)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/responses_models.py#L37-L43 - Implementation: Class `ResponseTextContentPart` derives from `BaseModel` and declares 0 direct member(s). A text content part for message items. - Inputs: - `type` (Literal['text', 'input_text', 'output_text']; optional; default `'output_text'`): Optional constructor field; defaults to `'output_text'`. - `text` (str; required): Required constructor field. - `annotations` (list[dict]; optional; default `Field(default_factory=list)`): Optional constructor field; defaults to `Field(default_factory=list)`. - `logprobs` (list[dict]; optional; default `Field(default_factory=list)`): Optional constructor field; defaults to `Field(default_factory=list)`. - Constructs: `vllm_mlx.api.responses_models.ResponseTextContentPart` ## `vllm_mlx.api.responses_models.ResponseReasoningTextPart` - Kind: class - Signature: `class ResponseReasoningTextPart(BaseModel)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/responses_models.py#L46-L50 - Implementation: Class `ResponseReasoningTextPart` derives from `BaseModel` and declares 0 direct member(s). A reasoning text content part. - Inputs: - `type` (Literal['reasoning_text']; optional; default `'reasoning_text'`): Optional constructor field; defaults to `'reasoning_text'`. - `text` (str; required): Required constructor field. - Constructs: `vllm_mlx.api.responses_models.ResponseReasoningTextPart` ## `vllm_mlx.api.responses_models.ResponseReasoningSummaryTextPart` - Kind: class - Signature: `class ResponseReasoningSummaryTextPart(BaseModel)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/responses_models.py#L53-L57 - Implementation: Class `ResponseReasoningSummaryTextPart` derives from `BaseModel` and declares 0 direct member(s). A reasoning summary item. - Inputs: - `type` (Literal['summary_text']; optional; default `'summary_text'`): Optional constructor field; defaults to `'summary_text'`. - `text` (str; required): Required constructor field. - Constructs: `vllm_mlx.api.responses_models.ResponseReasoningSummaryTextPart` ## `vllm_mlx.api.responses_models.ResponseMessageItem` - Kind: class - Signature: `class ResponseMessageItem(BaseModel)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/responses_models.py#L60-L67 - Implementation: Class `ResponseMessageItem` derives from `BaseModel` and declares 0 direct member(s). A Responses API message item. - Inputs: - `id` (str | None; optional; default `None`): Optional constructor field; defaults to `None`. - `type` (Literal['message']; optional; default `'message'`): Optional constructor field; defaults to `'message'`. - `role` (Literal['system', 'user', 'assistant', 'developer']; optional; default `'assistant'`): Optional constructor field; defaults to `'assistant'`. - `content` (str | list[ResponseTextContentPart]; optional; default `Field(default_factory=list)`): Optional constructor field; defaults to `Field(default_factory=list)`. - `status` (Literal['in_progress', 'completed', 'incomplete'] | None; optional; default `'completed'`): Optional constructor field; defaults to `'completed'`. - Constructs: `vllm_mlx.api.responses_models.ResponseMessageItem` ## `vllm_mlx.api.responses_models.ResponseReasoningItem` - Kind: class - Signature: `class ResponseReasoningItem(BaseModel)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/responses_models.py#L70-L77 - Implementation: Class `ResponseReasoningItem` derives from `BaseModel` and declares 0 direct member(s). A reasoning output item. - Inputs: - `id` (str | None; optional; default `None`): Optional constructor field; defaults to `None`. - `type` (Literal['reasoning']; optional; default `'reasoning'`): Optional constructor field; defaults to `'reasoning'`. - `summary` (list[ResponseReasoningSummaryTextPart]; optional; default `Field(default_factory=list)`): Optional constructor field; defaults to `Field(default_factory=list)`. - `content` (list[ResponseReasoningTextPart]; optional; default `Field(default_factory=list)`): Optional constructor field; defaults to `Field(default_factory=list)`. - `status` (Literal['in_progress', 'completed', 'incomplete'] | None; optional; default `'completed'`): Optional constructor field; defaults to `'completed'`. - Constructs: `vllm_mlx.api.responses_models.ResponseReasoningItem` ## `vllm_mlx.api.responses_models.ResponseFunctionCallItem` - Kind: class - Signature: `class ResponseFunctionCallItem(BaseModel)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/responses_models.py#L80-L88 - Implementation: Class `ResponseFunctionCallItem` derives from `BaseModel` and declares 0 direct member(s). A function call output item. - Inputs: - `id` (str | None; optional; default `None`): Optional constructor field; defaults to `None`. - `type` (Literal['function_call']; optional; default `'function_call'`): Optional constructor field; defaults to `'function_call'`. - `call_id` (str; required): Required constructor field. - `name` (str; required): Required constructor field. - `arguments` (str; required): Required constructor field. - `status` (Literal['in_progress', 'completed', 'incomplete']; optional; default `'completed'`): Optional constructor field; defaults to `'completed'`. - Constructs: `vllm_mlx.api.responses_models.ResponseFunctionCallItem` ## `vllm_mlx.api.responses_models.ResponseFunctionCallOutputItem` - Kind: class - Signature: `class ResponseFunctionCallOutputItem(BaseModel)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/responses_models.py#L91-L96 - Implementation: Class `ResponseFunctionCallOutputItem` derives from `BaseModel` and declares 0 direct member(s). A tool result item passed back into a later request. - Inputs: - `type` (Literal['function_call_output']; optional; default `'function_call_output'`): Optional constructor field; defaults to `'function_call_output'`. - `call_id` (str; required): Required constructor field. - `output` (str; required): Required constructor field. - Constructs: `vllm_mlx.api.responses_models.ResponseFunctionCallOutputItem` ## `vllm_mlx.api.responses_models.ResponseFunctionTool` - Kind: class - Signature: `class ResponseFunctionTool(BaseModel)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/responses_models.py#L99-L108 - Implementation: Class `ResponseFunctionTool` derives from `BaseModel` and declares 0 direct member(s). A function tool definition. - Inputs: - `type` (Literal['function']; optional; default `'function'`): Optional constructor field; defaults to `'function'`. - `name` (str; required): Required constructor field. - `description` (str | None; optional; default `''`): Optional constructor field; defaults to `''`. - `parameters` (dict; optional; default `Field(default_factory=lambda: {'type': 'object', 'properties': {}})`): Optional constructor field; defaults to `Field(default_factory=lambda: {'type': 'object', 'properties': {}})`. - `strict` (bool; optional; default `False`): Optional constructor field; defaults to `False`. - Constructs: `vllm_mlx.api.responses_models.ResponseFunctionTool` ## `vllm_mlx.api.responses_models.ResponsesInputTokenDetails` - Kind: class - Signature: `class ResponsesInputTokenDetails(BaseModel)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/responses_models.py#L111-L114 - Implementation: Class `ResponsesInputTokenDetails` derives from `BaseModel` and declares 0 direct member(s). Input token breakdown. - Inputs: - `cached_tokens` (int; optional; default `0`): Optional constructor field; defaults to `0`. - Constructs: `vllm_mlx.api.responses_models.ResponsesInputTokenDetails` ## `vllm_mlx.api.responses_models.ResponsesOutputTokenDetails` - Kind: class - Signature: `class ResponsesOutputTokenDetails(BaseModel)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/responses_models.py#L117-L120 - Implementation: Class `ResponsesOutputTokenDetails` derives from `BaseModel` and declares 0 direct member(s). Output token breakdown. - Inputs: - `reasoning_tokens` (int; optional; default `0`): Optional constructor field; defaults to `0`. - Constructs: `vllm_mlx.api.responses_models.ResponsesOutputTokenDetails` ## `vllm_mlx.api.responses_models.ResponsesUsage` - Kind: class - Signature: `class ResponsesUsage(BaseModel)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/responses_models.py#L123-L134 - Implementation: Class `ResponsesUsage` derives from `BaseModel` and declares 0 direct member(s). Responses API token usage. - Inputs: - `input_tokens` (int; required): Required constructor field. - `output_tokens` (int; required): Required constructor field. - `total_tokens` (int; required): Required constructor field. - `input_tokens_details` (ResponsesInputTokenDetails; optional; default `Field(default_factory=ResponsesInputTokenDetails)`): Optional constructor field; defaults to `Field(default_factory=ResponsesInputTokenDetails)`. - `output_tokens_details` (ResponsesOutputTokenDetails; optional; default `Field(default_factory=ResponsesOutputTokenDetails)`): Optional constructor field; defaults to `Field(default_factory=ResponsesOutputTokenDetails)`. - Constructs: `vllm_mlx.api.responses_models.ResponsesUsage` ## `vllm_mlx.api.responses_models.ResponseError` - Kind: class - Signature: `class ResponseError(BaseModel)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/responses_models.py#L137-L141 - Implementation: Class `ResponseError` derives from `BaseModel` and declares 0 direct member(s). Error payload. - Inputs: - `code` (str; required): Required constructor field. - `message` (str; required): Required constructor field. - Constructs: `vllm_mlx.api.responses_models.ResponseError` ## `vllm_mlx.api.responses_models.ResponseIncompleteDetails` - Kind: class - Signature: `class ResponseIncompleteDetails(BaseModel)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/responses_models.py#L144-L147 - Implementation: Class `ResponseIncompleteDetails` derives from `BaseModel` and declares 0 direct member(s). Incomplete response details. - Inputs: - `reason` (str; required): Required constructor field. - Constructs: `vllm_mlx.api.responses_models.ResponseIncompleteDetails` ## `vllm_mlx.api.responses_models.ResponsesRequest` - Kind: class - Signature: `class ResponsesRequest(BaseModel)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/responses_models.py#L150-L179 - Implementation: Class `ResponsesRequest` derives from `BaseModel` and declares 0 direct member(s). Request payload for /v1/responses. - Inputs: - `model` (str; required): Required constructor field. - `input` (str | list[ResponseMessageItem | ResponseReasoningItem | ResponseFunctionCallItem | ResponseFunctionCallOutputItem | di…; required): Required constructor field. - `instructions` (str | None; optional; default `None`): Optional constructor field; defaults to `None`. - `max_output_tokens` (int | None; optional; default `None`): Optional constructor field; defaults to `None`. - `stream` (bool; optional; default `False`): Optional constructor field; defaults to `False`. - `tools` (list[ResponseFunctionTool | dict]; optional; default `Field(default_factory=list)`): Optional constructor field; defaults to `Field(default_factory=list)`. - `tool_choice` (str | dict | None; optional; default `'auto'`): Optional constructor field; defaults to `'auto'`. - `parallel_tool_calls` (bool; optional; default `True`): Optional constructor field; defaults to `True`. - `previous_response_id` (str | None; optional; default `None`): Optional constructor field; defaults to `None`. - `temperature` (float | None; optional; default `None`): Optional constructor field; defaults to `None`. - `top_p` (float | None; optional; default `None`): Optional constructor field; defaults to `None`. - `chat_template_kwargs` (dict[str, Any] | None; optional; default `None`): Optional constructor field; defaults to `None`. - `metadata` (dict; optional; default `Field(default_factory=dict)`): Optional constructor field; defaults to `Field(default_factory=dict)`. - `text` (ResponseTextConfig; optional; default `Field(default_factory=ResponseTextConfig)`): Optional constructor field; defaults to `Field(default_factory=ResponseTextConfig)`. - `reasoning` (ResponseReasoningConfig | None; optional; default `None`): Optional constructor field; defaults to `None`. - `store` (bool; optional; default `True`): Optional constructor field; defaults to `True`. - `truncation` (str; optional; default `'disabled'`): Optional constructor field; defaults to `'disabled'`. - `user` (str | None; optional; default `None`): Optional constructor field; defaults to `None`. - Constructs: `vllm_mlx.api.responses_models.ResponsesRequest` ## `vllm_mlx.api.responses_models.ResponseObject` - Kind: class - Signature: `class ResponseObject(BaseModel)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/responses_models.py#L182-L226 - Implementation: Class `ResponseObject` derives from `BaseModel` and declares 1 direct member(s). Response object for /v1/responses. - Inputs: - `id` (str; optional; default `Field(default_factory=lambda: f'resp_{uuid.uuid4().hex}')`): Optional constructor field; defaults to `Field(default_factory=lambda: f'resp_{uuid.uuid4().hex}')`. - `object` (Literal['response']; optional; default `'response'`): Optional constructor field; defaults to `'response'`. - `created_at` (int; optional; default `Field(default_factory=lambda: int(time.time()))`): Optional constructor field; defaults to `Field(default_factory=lambda: int(time.time()))`. - `status` (Literal['completed', 'failed', 'incomplete', 'in_progress']; optional; default `'completed'`): Optional constructor field; defaults to `'completed'`. - `background` (bool; optional; default `False`): Optional constructor field; defaults to `False`. - `error` (ResponseError | None; optional; default `None`): Optional constructor field; defaults to `None`. - `incomplete_details` (ResponseIncompleteDetails | None; optional; default `None`): Optional constructor field; defaults to `None`. - `instructions` (str | None; optional; default `None`): Optional constructor field; defaults to `None`. - `max_output_tokens` (int | None; optional; default `None`): Optional constructor field; defaults to `None`. - `max_tool_calls` (int | None; optional; default `None`): Optional constructor field; defaults to `None`. - `metadata` (dict; optional; default `Field(default_factory=dict)`): Optional constructor field; defaults to `Field(default_factory=dict)`. - `model` (str; required): Required constructor field. - `output` (list[ResponseMessageItem | ResponseReasoningItem | ResponseFunctionCallItem]; optional; default `Field(default_factory=list)`): Optional constructor field; defaults to `Field(default_factory=list)`. - `parallel_tool_calls` (bool; optional; default `True`): Optional constructor field; defaults to `True`. - `previous_response_id` (str | None; optional; default `None`): Optional constructor field; defaults to `None`. - `text` (ResponseTextConfig; optional; default `Field(default_factory=ResponseTextConfig)`): Optional constructor field; defaults to `Field(default_factory=ResponseTextConfig)`. - `tool_choice` (str | dict | None; optional; default `'auto'`): Optional constructor field; defaults to `'auto'`. - `tools` (list[ResponseFunctionTool | dict]; optional; default `Field(default_factory=list)`): Optional constructor field; defaults to `Field(default_factory=list)`. - `top_p` (float; optional; default `1.0`): Optional constructor field; defaults to `1.0`. - `temperature` (float | None; optional; default `None`): Optional constructor field; defaults to `None`. - `truncation` (str; optional; default `'disabled'`): Optional constructor field; defaults to `'disabled'`. - `usage` (ResponsesUsage | None; optional; default `None`): Optional constructor field; defaults to `None`. - `user` (str | None; optional; default `None`): Optional constructor field; defaults to `None`. - `store` (bool; optional; default `True`): Optional constructor field; defaults to `True`. - Constructs: `vllm_mlx.api.responses_models.ResponseObject` ## `vllm_mlx.api.responses_models.ResponseObject.output_text` - Kind: method - Signature: `def output_text(self) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/responses_models.py#L214-L226 - Implementation: Method `ResponseObject.output_text` calls `isinstance`, `text_parts.append`, `''.join`; returns `''.join(text_parts)`. Concatenate assistant text content into the convenience field. - Inputs: none - Return annotation: `str` - Decorators: computed_field, property - Calls: isinstance, text_parts.append, ''.join - State reads: self.output - Return expressions: ''.join(text_parts) ## `vllm_mlx.api.responses_models.ResponsesEventBase` - Kind: class - Signature: `class ResponsesEventBase(BaseModel)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/responses_models.py#L229-L232 - Implementation: Class `ResponsesEventBase` derives from `BaseModel` and declares 0 direct member(s). Base event fields. - Inputs: - `sequence_number` (int; required): Required constructor field. - Constructs: `vllm_mlx.api.responses_models.ResponsesEventBase` ## `vllm_mlx.api.responses_models.ResponseCreatedEvent` - Kind: class - Signature: `class ResponseCreatedEvent(ResponsesEventBase)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/responses_models.py#L235-L239 - Implementation: Class `ResponseCreatedEvent` derives from `ResponsesEventBase` and declares 0 direct member(s). Signal that a response object has been created. - Inputs: none - Constructs: `vllm_mlx.api.responses_models.ResponseCreatedEvent` ## `vllm_mlx.api.responses_models.ResponseInProgressEvent` - Kind: class - Signature: `class ResponseInProgressEvent(ResponsesEventBase)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/responses_models.py#L242-L246 - Implementation: Class `ResponseInProgressEvent` derives from `ResponsesEventBase` and declares 0 direct member(s). Signal that response generation is in progress. - Inputs: none - Constructs: `vllm_mlx.api.responses_models.ResponseInProgressEvent` ## `vllm_mlx.api.responses_models.ResponseCompletedEvent` - Kind: class - Signature: `class ResponseCompletedEvent(ResponsesEventBase)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/responses_models.py#L249-L253 - Implementation: Class `ResponseCompletedEvent` derives from `ResponsesEventBase` and declares 0 direct member(s). Carry the terminal completed response object. - Inputs: none - Constructs: `vllm_mlx.api.responses_models.ResponseCompletedEvent` ## `vllm_mlx.api.responses_models.ResponseOutputItemAddedEvent` - Kind: class - Signature: `class ResponseOutputItemAddedEvent(ResponsesEventBase)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/responses_models.py#L256-L261 - Implementation: Class `ResponseOutputItemAddedEvent` derives from `ResponsesEventBase` and declares 0 direct member(s). Announce a newly added response output item. - Inputs: none - Constructs: `vllm_mlx.api.responses_models.ResponseOutputItemAddedEvent` ## `vllm_mlx.api.responses_models.ResponseOutputItemDoneEvent` - Kind: class - Signature: `class ResponseOutputItemDoneEvent(ResponsesEventBase)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/responses_models.py#L264-L269 - Implementation: Class `ResponseOutputItemDoneEvent` derives from `ResponsesEventBase` and declares 0 direct member(s). Signal that a response output item is complete. - Inputs: none - Constructs: `vllm_mlx.api.responses_models.ResponseOutputItemDoneEvent` ## `vllm_mlx.api.responses_models.ResponseContentPartAddedEvent` - Kind: class - Signature: `class ResponseContentPartAddedEvent(ResponsesEventBase)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/responses_models.py#L272-L279 - Implementation: Class `ResponseContentPartAddedEvent` derives from `ResponsesEventBase` and declares 0 direct member(s). Announce a content part attached to an output item. - Inputs: none - Constructs: `vllm_mlx.api.responses_models.ResponseContentPartAddedEvent` ## `vllm_mlx.api.responses_models.ResponseContentPartDoneEvent` - Kind: class - Signature: `class ResponseContentPartDoneEvent(ResponsesEventBase)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/responses_models.py#L282-L289 - Implementation: Class `ResponseContentPartDoneEvent` derives from `ResponsesEventBase` and declares 0 direct member(s). Signal that an output item's content part is complete. - Inputs: none - Constructs: `vllm_mlx.api.responses_models.ResponseContentPartDoneEvent` ## `vllm_mlx.api.responses_models.ResponseOutputTextDeltaEvent` - Kind: class - Signature: `class ResponseOutputTextDeltaEvent(ResponsesEventBase)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/responses_models.py#L292-L300 - Implementation: Class `ResponseOutputTextDeltaEvent` derives from `ResponsesEventBase` and declares 0 direct member(s). Carry an incremental final-answer text fragment. - Inputs: none - Constructs: `vllm_mlx.api.responses_models.ResponseOutputTextDeltaEvent` ## `vllm_mlx.api.responses_models.ResponseOutputTextDoneEvent` - Kind: class - Signature: `class ResponseOutputTextDoneEvent(ResponsesEventBase)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/responses_models.py#L303-L311 - Implementation: Class `ResponseOutputTextDoneEvent` derives from `ResponsesEventBase` and declares 0 direct member(s). Carry the complete final-answer text for one content part. - Inputs: none - Constructs: `vllm_mlx.api.responses_models.ResponseOutputTextDoneEvent` ## `vllm_mlx.api.responses_models.ResponseReasoningTextDeltaEvent` - Kind: class - Signature: `class ResponseReasoningTextDeltaEvent(ResponsesEventBase)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/responses_models.py#L314-L321 - Implementation: Class `ResponseReasoningTextDeltaEvent` derives from `ResponsesEventBase` and declares 0 direct member(s). Carry an incremental reasoning text fragment. - Inputs: none - Constructs: `vllm_mlx.api.responses_models.ResponseReasoningTextDeltaEvent` ## `vllm_mlx.api.responses_models.ResponseReasoningTextDoneEvent` - Kind: class - Signature: `class ResponseReasoningTextDoneEvent(ResponsesEventBase)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/responses_models.py#L324-L331 - Implementation: Class `ResponseReasoningTextDoneEvent` derives from `ResponsesEventBase` and declares 0 direct member(s). Carry the complete reasoning text for one content part. - Inputs: none - Constructs: `vllm_mlx.api.responses_models.ResponseReasoningTextDoneEvent` ## `vllm_mlx.api.responses_models.ResponseFunctionCallArgumentsDeltaEvent` - Kind: class - Signature: `class ResponseFunctionCallArgumentsDeltaEvent(ResponsesEventBase)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/responses_models.py#L334-L342 - Implementation: Class `ResponseFunctionCallArgumentsDeltaEvent` derives from `ResponsesEventBase` and declares 0 direct member(s). Carry an incremental fragment of function-call arguments. - Inputs: none - Constructs: `vllm_mlx.api.responses_models.ResponseFunctionCallArgumentsDeltaEvent` # Module `vllm_mlx.api.streaming` Optimized streaming JSON encoder for SSE responses. This module provides a pre-computed template-based JSON encoder that reduces CPU overhead during streaming by avoiding repeated json.dumps() calls for static parts of the response. Performance improvement: ~20-30% reduction in server CPU overhead for streaming. Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/streaming.py#L1-L210 ## `vllm_mlx.api.streaming._escape_json_string` - Kind: function - Signature: `def _escape_json_string(s: str) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/streaming.py#L16-L24 - Implementation: Function `_escape_json_string` calls `json.dumps`; returns `json.dumps(s)[1:-1]`. Escape a string for JSON without the surrounding quotes. Uses json.dumps for correctness then strips the quotes. This handles all special characters: quotes, backslashes, newlines, tabs, unicode. - Inputs: - `s` (str; required): Required positional or keyword input. - Return annotation: `str` - Calls: json.dumps - Return expressions: json.dumps(s)[1:-1] ## `vllm_mlx.api.streaming.StreamingJSONEncoder` - Kind: class - Signature: `class StreamingJSONEncoder` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/streaming.py#L27-L210 - Implementation: Class `StreamingJSONEncoder` declares 4 direct member(s). Optimized JSON encoder for OpenAI-compatible streaming responses. Pre-computes static parts of the JSON response at initialization time, then only inserts dynamic content (text/content, finish_reason) per token. The main optimization is pre-building the static JSON prefix and suffix that don't change between tokens. Only the dynamic parts (content, finish_reason) are escaped and inserted per token. Example usage: encoder = StreamingJSONEncoder( response_id="chatcmpl-123", model="gpt-4", object_type="chat.completion.chunk" ) # Encode each token for token in tokens: yield encoder.encode_chat_chunk(content=token) # Final chunk with finish_reason yield encoder.encode_chat_chunk(finish_reason="stop") yield encoder.encode_done() - Inputs: - `response_id` (str; required): Unique response ID (e.g., "chatcmpl-abc123") - `model` (str; required): Model name (e.g., "mlx-community/Llama-3.2-3B-Instruct-4bit") - `object_type` (str; required): Response object type ("text_completion" or "chat.completion.chunk") - `created` (int | None; optional; default `None`): Unix timestamp (defaults to current time) - Constructs: `vllm_mlx.api.streaming.StreamingJSONEncoder` ## `vllm_mlx.api.streaming.StreamingJSONEncoder.__init__` - Kind: method - Signature: `def __init__(self, response_id: str, model: str, object_type: str, created: int | None=None)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/streaming.py#L57-L102 - Implementation: Method `StreamingJSONEncoder.__init__` updates `self.response_id`, `self.model`, `self.object_type`, `self.created`; calls `int`, `time.time`, `_escape_json_string`. Initialize the encoder with static response metadata. Pre-computes template parts that don't change between tokens. Args: response_id: Unique response ID (e.g., "chatcmpl-abc123") model: Model name (e.g., "mlx-community/Llama-3.2-3B-Instruct-4bit") object_type: Response object type ("text_completion" or "chat.completion.chunk") created: Unix timestamp (defaults to current time) - Inputs: - `response_id` (str; required): Unique response ID (e.g., "chatcmpl-abc123") - `model` (str; required): Model name (e.g., "mlx-community/Llama-3.2-3B-Instruct-4bit") - `object_type` (str; required): Response object type ("text_completion" or "chat.completion.chunk") - `created` (int | None; optional; default `None`): Unix timestamp (defaults to current time) - Return annotation: `not annotated` - Calls: int, time.time, _escape_json_string - State reads: self.created - State writes: self.response_id, self.model, self.object_type, self.created, self._prefix, self._completion_choices_prefix, self._completion_text_prefix, self._completion_text_suffix, self._chat_choices_prefix, self._chat_finish_prefix ## `vllm_mlx.api.streaming.StreamingJSONEncoder.encode_completion_chunk` - Kind: method - Signature: `def encode_completion_chunk(self, text: str, index: int=0, finish_reason: str | None=None, usage: dict[str, int] | None=None) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/streaming.py#L104-L149 - Implementation: Method `StreamingJSONEncoder.encode_completion_chunk` calls `_escape_json_string`, `json.dumps`; returns `result`. Encode a text completion chunk using pre-computed templates. Args: text: The generated text for this chunk index: Choice index (usually 0) finish_reason: "stop", "length", or None if not finished usage: Optional usage stats (prompt_tokens, completion_tokens, total_tokens) Returns: SSE-formatted string: "data: {json} " - Inputs: - `text` (str; required): The generated text for this chunk - `index` (int; optional; default `0`): Choice index (usually 0) - `finish_reason` (str | None; optional; default `None`): "stop", "length", or None if not finished - `usage` (dict[str, int] | None; optional; default `None`): Optional usage stats (prompt_tokens, completion_tokens, total_tokens) - Return annotation: `str` - Calls: _escape_json_string, json.dumps - State reads: self._completion_choices_prefix, self._completion_text_prefix, self._completion_text_suffix, self._prefix - Return expressions: result ## `vllm_mlx.api.streaming.StreamingJSONEncoder.encode_chat_chunk` - Kind: method - Signature: `def encode_chat_chunk(self, role: str | None=None, content: str | None=None, finish_reason: str | None=None, usage: dict[str, int] | None=None) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/streaming.py#L151-L201 - Implementation: Method `StreamingJSONEncoder.encode_chat_chunk` calls `delta_parts.append`, `_escape_json_string`, `','.join`, `json.dumps`; returns `result`. Encode a chat completion chunk using pre-computed templates. Args: role: Assistant role (only for first chunk) content: Generated content for this chunk finish_reason: "stop", "length", or None if not finished usage: Optional usage stats Returns: SSE-formatted string: "data: {json} " - Inputs: - `role` (str | None; optional; default `None`): Assistant role (only for first chunk) - `content` (str | None; optional; default `None`): Generated content for this chunk - `finish_reason` (str | None; optional; default `None`): "stop", "length", or None if not finished - `usage` (dict[str, int] | None; optional; default `None`): Optional usage stats - Return annotation: `str` - Calls: delta_parts.append, _escape_json_string, ','.join, json.dumps - State reads: self._chat_choices_prefix, self._chat_finish_prefix, self._prefix - Return expressions: result ## `vllm_mlx.api.streaming.StreamingJSONEncoder.encode_done` - Kind: method - Signature: `def encode_done(self) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/streaming.py#L203-L210 - Implementation: Method `StreamingJSONEncoder.encode_done` returns `self._DONE_MSG`. Encode the [DONE] message that signals end of stream. Returns: SSE-formatted done message: "data: [DONE] " - Inputs: none - Return annotation: `str` - State reads: self._DONE_MSG - Return expressions: self._DONE_MSG # Module `vllm_mlx.api.tool_calling` Tool calling parsing and conversion utilities. Supports parsing tool calls from multiple model formats: - Qwen: {"name": "...", "arguments": {...}} - Llama: {"arg": "value"} Also includes structured output (JSON Schema) utilities: - parse_json_output: Extract JSON from model output - validate_json_schema: Validate JSON against a schema Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/tool_calling.py#L1-L1035 ## `vllm_mlx.api.tool_calling.InvalidResponseFormatOutput` - Kind: class - Signature: `class InvalidResponseFormatOutput(ValueError)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/tool_calling.py#L24-L29 - Implementation: Class `InvalidResponseFormatOutput` derives from `ValueError` and declares 1 direct member(s). Raised when generated content does not satisfy response_format. - Inputs: - `message` (str; required): Required positional or keyword input. - Constructs: `vllm_mlx.api.tool_calling.InvalidResponseFormatOutput` ## `vllm_mlx.api.tool_calling.InvalidResponseFormatOutput.__init__` - Kind: method - Signature: `def __init__(self, message: str)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/tool_calling.py#L27-L29 - Implementation: Method `InvalidResponseFormatOutput.__init__` updates `self.message`; calls `super().__init__`, `super`. Method `InvalidResponseFormatOutput.__init__` updates `self.message`; calls `super().__init__`, `super`. - Inputs: - `message` (str; required): Required positional or keyword input. - Return annotation: `not annotated` - Calls: super().__init__, super - State writes: self.message ## `vllm_mlx.api.tool_calling._looks_like_tool_call` - Kind: function - Signature: `def _looks_like_tool_call(obj: Any) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/tool_calling.py#L32-L57 - Implementation: Function `_looks_like_tool_call` calls `isinstance`; has 2 explicit return paths. Heuristic: decide whether a parsed JSON object really represents a tool call as opposed to user data that happens to carry a ``"name"`` field. The OpenAI tool-call wire format ALWAYS has both ``"name"`` and ``"arguments"``. Accepting bare ``{"name": ...}`` (previous behaviour) caused ``response_format={"type": "json_schema"}`` payloads with a ``name`` field to be hijacked as fake tool calls (observed on MiniMax-M2: ``{"name": "John", "age": 25}`` -> ``function.name="John"``). Args: obj: Parsed JSON object. Returns: True if obj looks like a tool call, False otherwise. - Inputs: - `obj` (Any; required): Parsed JSON object. - Return annotation: `bool` - Calls: isinstance - Return expressions: False; isinstance(args, (dict, str)) ## `vllm_mlx.api.tool_calling._parse_raw_json_tool_calls` - Kind: function - Signature: `def _parse_raw_json_tool_calls(text: str) -> Optional[List[dict]]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/tool_calling.py#L60-L123 - Implementation: Function `_parse_raw_json_tool_calls` calls `text.strip`, `text.startswith`, `json.loads`, `isinstance`; has 3 explicit return paths. Parse raw JSON tool calls from model output. Handles: - Single JSON object: {"name": "func", "arguments": {...}} - Multiple objects separated by commas: {...}, {...} - JSON array: [{...}, {...}] Only accepts objects that carry both ``name`` AND ``arguments`` fields to avoid hijacking user data emitted via ``response_format``. Args: text: Raw model output text Returns: List of tool call dicts with 'name' and 'arguments', or None if no valid tool calls found - Inputs: - `text` (str; required): Raw model output text - Return annotation: `Optional[List[dict]]` - Calls: text.strip, text.startswith, json.loads, isinstance, all, _looks_like_tool_call, enumerate, tool_calls.append - Return expressions: None; [{'name': item['name'], 'arguments': item['arguments']} for item in parsed]; tool_calls if tool_calls else None ## `vllm_mlx.api.tool_calling.parse_tool_calls` - Kind: function - Signature: `def parse_tool_calls(text: str, request: dict[str, Any] | None=None) -> Tuple[str, Optional[List[ToolCall]]]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/tool_calling.py#L126-L351 - Implementation: Function `parse_tool_calls` calls `re.findall`, `json.loads`, `tool_calls.append`, `ToolCall`; returns `(cleaned_text, tool_calls if tool_calls else None)`. Parse tool calls from model output. Supports multiple formats: - MiniMax: v - Qwen3 bracket: [Calling tool: function_name({"arg": "value"})] - Qwen: {"name": "...", "arguments": {...}} - Llama: {"arg": "value"} - Nemotron: v - Raw JSON: {"name": "...", "arguments": {...}} (single or multiple) Args: text: Raw model output text Returns: Tuple of (cleaned_text, tool_calls or None) - cleaned_text: Text with tool call tags removed - tool_calls: List of ToolCall objects, or None if no tool calls found - Inputs: - `text` (str; required): Raw model output text - `request` (dict[str, Any] | None; optional; default `None`): Optional positional or keyword input; defaults to `None`. - Return annotation: `Tuple[str, Optional[List[ToolCall]]]` - Calls: re.findall, json.loads, tool_calls.append, ToolCall, uuid.uuid4, FunctionCall, name.strip, json.dumps, re.sub('\\s*.*?\\s*', '', cleaned_text, flags=re.DOTALL).strip, re.sub, isinstance, str, re.sub('\\[Calling tool:\\s*\\w+\\(\\{.*?\\}\\)\\]', '', cleaned_text, flags=re.DOTALL).strip, p_value.strip, p_name.strip, re.sub('\\s*]+>.*?\\s*', '', text, flags=re.DOTALL).strip, data.get, re.sub('\\s*\\{.*?\\}\\s*', '', cleaned_text, flags=re.DOTALL).strip, re.sub(']+>\\{.*?\\}', '', cleaned_text, flags=re.DOTALL).strip, bool, request.get, _parse_raw_json_tool_calls - Return expressions: (cleaned_text, tool_calls if tool_calls else None) ## `vllm_mlx.api.tool_calling.convert_tools_for_template` - Kind: function - Signature: `def convert_tools_for_template(tools: Optional[List]) -> Optional[List[dict]]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/tool_calling.py#L354-L409 - Implementation: Function `convert_tools_for_template` calls `isinstance`, `tool.get`, `getattr`, `tool_func.get`; has 2 explicit return paths. Convert OpenAI tools format to format expected by tokenizer.apply_chat_template. OpenAI format: [{"type": "function", "function": {"name": "...", "description": "...", "parameters": {...}}}] Template format (commonly used by models): [{"type": "function", "function": {"name": "...", "description": "...", "parameters": {...}}}] Args: tools: List of ToolDefinition objects or dicts in OpenAI format Returns: List of tool definitions in template format, or None if no tools - Inputs: - `tools` (Optional[List]; required): List of ToolDefinition objects or dicts in OpenAI format - Return annotation: `Optional[List[dict]]` - Calls: isinstance, tool.get, getattr, tool_func.get, converted.append - Return expressions: None; converted if converted else None ## `vllm_mlx.api.tool_calling.format_tool_call_for_message` - Kind: function - Signature: `def format_tool_call_for_message(tool_call: ToolCall) -> dict` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/tool_calling.py#L412-L429 - Implementation: Function `format_tool_call_for_message` returns `{'id': tool_call.id, 'type': tool_call.type, 'function': {'name': tool_call.function.name, 'arguments': tool_call.funct…`. Format a ToolCall object for inclusion in a message. Args: tool_call: ToolCall object Returns: Dict representation suitable for message content - Inputs: - `tool_call` (ToolCall; required): ToolCall object - Return annotation: `dict` - Return expressions: {'id': tool_call.id, 'type': tool_call.type, 'function': {'name': tool_call.function.name, 'arguments': tool_call.funct… ## `vllm_mlx.api.tool_calling.validate_json_schema` - Kind: function - Signature: `def validate_json_schema(data: Any, schema: Dict[str, Any]) -> Tuple[bool, Optional[str]]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/tool_calling.py#L437-L456 - Implementation: Function `validate_json_schema` calls `validate`, `str`; has 2 explicit return paths. Validate JSON data against a JSON Schema. Args: data: The JSON data to validate (dict, list, etc.) schema: JSON Schema specification Returns: Tuple of (is_valid, error_message) - is_valid: True if data matches schema - error_message: Error description if invalid, None if valid - Inputs: - `data` (Any; required): The JSON data to validate (dict, list, etc.) - `schema` (Dict[str, Any]; required): JSON Schema specification - Return annotation: `Tuple[bool, Optional[str]]` - Calls: validate, str - Return expressions: (True, None); (False, str(e.message)) ## `vllm_mlx.api.tool_calling._scan_balanced_json` - Kind: function - Signature: `def _scan_balanced_json(text: str, start: int) -> Optional[str]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/tool_calling.py#L459-L494 - Implementation: Function `_scan_balanced_json` calls `len`, `range`; has 2 explicit return paths. Walk forward from ``start`` (which must point at ``{`` or ``[``) and return the substring that represents the first balanced JSON value, respecting strings and escapes. Returns ``None`` if the opening bracket is never closed (truncated output). - Inputs: - `text` (str; required): Required positional or keyword input. - `start` (int; required): Required positional or keyword input. - Return annotation: `Optional[str]` - Calls: len, range - Return expressions: None; text[start:i + 1] ## `vllm_mlx.api.tool_calling._repair_truncated_json` - Kind: function - Signature: `def _repair_truncated_json(fragment: str) -> Optional[Dict[str, Any]]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/tool_calling.py#L497-L582 - Implementation: Function `_repair_truncated_json` calls `stack.append`, `stack.pop`, `candidates.append`, `_close`; has 2 explicit return paths. Attempt to parse a JSON fragment whose closing brackets were cut off (e.g. because the model hit ``max_tokens`` mid-object). Strategy: scan once to determine the open-bracket stack and whether we ended mid-string, then try a handful of repair candidates in order of likelihood: 1. Close unterminated string, close brackets. 2. Also strip a dangling ``,`` / ``:`` before closing. 3. Also drop a dangling key (``"k":`` or bare ``"k"``) before closing. 4. Drop a dangling partial token (number / true / fals / nul) before closing. Returns the first candidate that ``json.loads`` accepts, or ``None``. - Inputs: - `fragment` (str; required): Required positional or keyword input. - Return annotation: `Optional[Dict[str, Any]]` - Calls: stack.append, stack.pop, candidates.append, _close, re.sub, json.loads - Return expressions: None; json.loads(candidate) ## `vllm_mlx.api.tool_calling._repair_truncated_json._close` - Kind: nested function - Signature: `def _close(text: str) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/tool_calling.py#L540-L543 - Implementation: Nested Function `_repair_truncated_json._close` calls `reversed`; returns `text`. Nested Function `_repair_truncated_json._close` calls `reversed`; returns `text`. - Inputs: - `text` (str; required): Required positional or keyword input. - Return annotation: `str` - Calls: reversed - Return expressions: text ## `vllm_mlx.api.tool_calling.extract_json_from_text` - Kind: function - Signature: `def extract_json_from_text(text: str) -> Optional[Dict[str, Any]]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/tool_calling.py#L585-L666 - Implementation: Function `extract_json_from_text` calls `text.strip`, `json.loads`, `re.findall`, `match.strip`; has 6 explicit return paths. Extract JSON from model output text. Tries multiple strategies, in order of specificity: 1. Parse entire text as JSON 2. Extract JSON from complete markdown code blocks (``` ... ```) 3. Extract JSON from an unterminated markdown code block (``` json\n{ ... ) — handles the common "chatty + truncation" failure mode where the model starts a ```json fence, never closes it, then hits max_tokens. 4. Balanced-brace scan for the first ``{`` or ``[`` in the text 5. Repair truncated JSON by closing unclosed brackets/strings Args: text: Raw model output text Returns: Parsed JSON data, or None if no valid JSON found - Inputs: - `text` (str; required): Raw model output text - Return annotation: `Optional[Dict[str, Any]]` - Calls: text.strip, json.loads, re.findall, match.strip, re.search, unterminated_fence.group(1).strip, unterminated_fence.group, fenced_candidate.endswith, fenced_candidate[:-3].strip, text.find, _scan_balanced_json, candidates.append, _repair_truncated_json - Return expressions: json.loads(text); json.loads(match.strip()); json.loads(fenced_candidate); json.loads(candidate); repaired; None ## `vllm_mlx.api.tool_calling.StreamingJsonFenceStripper` - Kind: class - Signature: `class StreamingJsonFenceStripper` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/tool_calling.py#L669-L787 - Implementation: Class `StreamingJsonFenceStripper` declares 3 direct member(s). Strip markdown code fences from streamed content when response_format is set. Without guided decoding, chat models often wrap their JSON output in markdown fences (```json ... ```) even when the system prompt says not to. The non- streaming path strips those via ``extract_json_from_text`` / ``parse_json_output``, but the streaming path used to emit the raw deltas, so clients got ``"```json{...}```"`` instead of ``"{...}"``. This filter buffers just enough text to detect: * a leading fence like ``"```"``, ``"```json"``, ``"```\n"`` or ``"```json\n"`` (with optional leading whitespace), possibly split across SSE deltas, and * a trailing fence like ``"```"`` or ``"\n```\n"`` on stream end. Leading-whitespace and leading fences are consumed; trailing fences are dropped in :meth:`finalize`. Non-fenced content passes through with at most a ``_TAIL_HOLDBACK``-char delay. - Inputs: none - Constructs: `vllm_mlx.api.tool_calling.StreamingJsonFenceStripper` ## `vllm_mlx.api.tool_calling.StreamingJsonFenceStripper.__init__` - Kind: method - Signature: `def __init__(self) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/tool_calling.py#L695-L697 - Implementation: Method `StreamingJsonFenceStripper.__init__` updates `self._buf`, `self._past_opening`. Method `StreamingJsonFenceStripper.__init__` updates `self._buf`, `self._past_opening`. - Inputs: none - Return annotation: `None` - State writes: self._buf, self._past_opening ## `vllm_mlx.api.tool_calling.StreamingJsonFenceStripper.feed` - Kind: method - Signature: `def feed(self, delta: str) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/tool_calling.py#L699-L748 - Implementation: Method `StreamingJsonFenceStripper.feed` updates `self._buf`, `self._past_opening`; calls `self._buf.lstrip`, `len`, `opening.startswith`, `ls.startswith`; has 2 explicit return paths. Append a content delta and return the portion safe to emit now. - Inputs: - `delta` (str; required): Required positional or keyword input. - Return annotation: `str` - Calls: self._buf.lstrip, len, opening.startswith, ls.startswith, ls[len(matched):].lstrip, buf[i - 1].isspace, min - State reads: self._past_opening, self._buf.lstrip, self._buf, self._OPENINGS, self._TAIL_HOLDBACK - State writes: self._buf, self._past_opening - Return expressions: ''; to_emit ## `vllm_mlx.api.tool_calling.StreamingJsonFenceStripper.finalize` - Kind: method - Signature: `def finalize(self) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/tool_calling.py#L750-L787 - Implementation: Method `StreamingJsonFenceStripper.finalize` updates `self._buf`, `self._past_opening`; calls `tail.lstrip`, `len`, `opening.startswith`, `ls.startswith`; has 3 explicit return paths. Flush the remaining buffer, dropping any trailing fence. - Inputs: none - Return annotation: `str` - Calls: tail.lstrip, len, opening.startswith, ls.startswith, ls[len(matched):].lstrip, tail.rstrip, stripped.endswith, stripped[:-len(closing)].rstrip - State reads: self._buf, self._past_opening, self._OPENINGS - State writes: self._buf, self._past_opening - Return expressions: ''; stripped[:-len(closing)].rstrip(); tail ## `vllm_mlx.api.tool_calling.parse_json_output` - Kind: function - Signature: `def parse_json_output(text: str, response_format: Optional[Union[ResponseFormat, Dict[str, Any]]]=None) -> Tuple[str, Optional[Dict[str, Any]], bool, Optional[str]]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/tool_calling.py#L790-L855 - Implementation: Function `parse_json_output` calls `isinstance`, `rf_dict.get`, `extract_json_from_text`, `json_schema_spec.get`; has 4 explicit return paths. Parse JSON from model output when response_format is set. Args: text: Raw model output text response_format: ResponseFormat specification (optional) - If type="json_object", extracts any valid JSON - If type="json_schema", extracts and validates against schema Returns: Tuple of (cleaned_text, parsed_json, is_valid, error_message) - cleaned_text: Original text (preserved for reference) - parsed_json: Extracted JSON data, or None if extraction failed - is_valid: True if JSON is valid (and matches schema if specified) - error_message: Error description if invalid, None if valid - Inputs: - `text` (str; required): Raw model output text - `response_format` (Optional[Union[ResponseFormat, Dict[str, Any]]]; optional; default `None`): ResponseFormat specification (optional) - If type="json_object", extracts any valid JSON - If type="json_schema", extracts and validates against schema - Return annotation: `Tuple[str, Optional[Dict[str, Any]], bool, Optional[str]]` - Calls: isinstance, rf_dict.get, extract_json_from_text, json_schema_spec.get, validate_json_schema - Return expressions: (text, None, True, None); (text, None, False, 'Failed to extract valid JSON from output'); (text, parsed, True, None); (text, parsed, False, f'JSON Schema validation failed: {error}') ## `vllm_mlx.api.tool_calling.apply_response_format_or_error` - Kind: function - Signature: `def apply_response_format_or_error(text: str, response_format: object, *, ensure_ascii: bool=False) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/tool_calling.py#L858-L873 - Implementation: Function `apply_response_format_or_error` calls `parse_json_output`, `json.dumps`, `InvalidResponseFormatOutput`; can raise `InvalidResponseFormatOutput`; has 2 explicit return paths. Return canonical JSON content or raise for invalid response_format output. - Inputs: - `text` (str; required): Required positional or keyword input. - `response_format` (object; required): Required positional or keyword input. - `ensure_ascii` (bool; optional; default `False`): Optional keyword-only input; defaults to `False`. - Return annotation: `str` - Calls: parse_json_output, json.dumps, InvalidResponseFormatOutput - Raises directly: InvalidResponseFormatOutput - Return expressions: json.dumps(parsed_json, ensure_ascii=ensure_ascii); text ## `vllm_mlx.api.tool_calling.build_json_system_prompt` - Kind: function - Signature: `def build_json_system_prompt(response_format: Optional[Union[ResponseFormat, Dict[str, Any]]]=None, *, thinking_model: bool=False) -> Optional[str]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/tool_calling.py#L876-L953 - Implementation: Function `build_json_system_prompt` calls `isinstance`, `rf_dict.get`, `json_schema_spec.get`, `json.dumps`; has 3 explicit return paths. Build a system prompt instruction for JSON output. For models without native JSON mode support, this adds instructions to the prompt to encourage proper JSON formatting. Args: response_format: ResponseFormat specification thinking_model: When ``True`` use softer output rules that allow the model to reason (think) before emitting JSON. Strict rules that demand ``{`` as the very first character conflict with ```` blocks and cause degenerated output. Returns: System prompt instruction string, or None if not needed - Inputs: - `response_format` (Optional[Union[ResponseFormat, Dict[str, Any]]]; optional; default `None`): ResponseFormat specification - `thinking_model` (bool; optional; default `False`): When ``True`` use softer output rules that allow the model to reason (think) before emitting JSON. Strict rules that demand ``{`` as the very first character conflict with ```` blocks and cause degenerated output. - Return annotation: `Optional[str]` - Calls: isinstance, rf_dict.get, json_schema_spec.get, json.dumps - Return expressions: None; 'You must respond with a single valid JSON value only.\n\n' + strict_rules; prompt ## `vllm_mlx.api.tool_calling.build_json_logits_processor` - Kind: function - Signature: `def build_json_logits_processor(response_format: ResponseFormat | dict[str, Any] | None, tokenizer: Any)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/tool_calling.py#L956-L1035 - Implementation: Function `build_json_logits_processor` calls `isinstance`, `response_format.get`, `json_schema_spec.get`, `getattr`; has 2 explicit return paths. Build a logits processor that constrains generation to valid JSON matching ``response_format``. Unlike :func:`build_json_system_prompt` which nudges the model via the system prompt, this processor masks logits at every generation step so the model *cannot* emit invalid JSON (grammar-guided decoding). Args: response_format: ``ResponseFormat`` specification (or dict). tokenizer: The tokenizer used by the engine. May be a HF tokenizer, a ``mlx_lm.TokenizerWrapper``, or a VLM ``processor``; the underlying tokenizer is resolved automatically. Returns: A callable ``(tokens, logits) -> logits`` suitable for passing to ``mlx_lm.stream_generate`` via ``logits_processors``. ``None`` when no constraint is needed (e.g. ``type=text``) or when constrained decoding cannot be enabled (missing optional dependency, tokenizer incompatibility) — in that case the caller should fall back to the system-prompt path. - Inputs: - `response_format` (ResponseFormat | dict[str, Any] | None; required): ``ResponseFormat`` specification (or dict). - `tokenizer` (Any; required): The tokenizer used by the engine. May be a HF tokenizer, a ``mlx_lm.TokenizerWrapper``, or a VLM ``processor``; the underlying tokenizer is resolved automatically. - Return annotation: `not annotated` - Calls: isinstance, response_format.get, json_schema_spec.get, getattr, is_available, JSONSchemaLogitsProcessor - Return expressions: None; JSONSchemaLogitsProcessor(schema=schema, tokenizer=tokenizer) # Module `vllm_mlx.api.utils` Utility functions for text processing and model detection. Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/utils.py#L1-L747 ## `vllm_mlx.api.utils._clean_gpt_oss_output` - Kind: function - Signature: `def _clean_gpt_oss_output(text: str) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/utils.py#L39-L73 - Implementation: Function `_clean_gpt_oss_output` calls `_FINAL_CHANNEL_RE.search`, `match.end`, `re.sub`, `content.strip`; has 2 explicit return paths. Extract final channel content from GPT-OSS channel-based output. When reasoning parser is not enabled, this provides a fallback that extracts the 'final' channel content so the API response is usable. Handles both standard and extended format with constrain token: <|channel|>final<|message|>... <|channel|>final <|constrain|>JSON<|message|>... Args: text: Raw model output containing channel tokens. Returns: Extracted final content, or text with channel tokens stripped. - Inputs: - `text` (str; required): Raw model output containing channel tokens. - Return annotation: `str` - Calls: _FINAL_CHANNEL_RE.search, match.end, re.sub, content.strip, cleaned.strip - Return expressions: content.strip(); cleaned.strip() ## `vllm_mlx.api.utils.clean_output_text` - Kind: function - Signature: `def clean_output_text(text: str) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/utils.py#L76-L108 - Implementation: Function `clean_output_text` calls `_clean_gpt_oss_output`, `SPECIAL_TOKENS_PATTERN.sub`, `text.strip`, `text.lstrip().startswith`; returns `text`. Clean model output by removing special tokens. Keeps ... blocks intact for reasoning models. Adds opening tag if missing (happens when thinking is enabled in the prompt template but the tag is part of the prompt, not output). Handles GPT-OSS channel-based format as fallback when reasoning parser is not enabled. Args: text: Raw model output Returns: Cleaned text with special tokens removed - Inputs: - `text` (str; required): Raw model output - Return annotation: `str` - Calls: _clean_gpt_oss_output, SPECIAL_TOKENS_PATTERN.sub, text.strip, text.lstrip().startswith, text.lstrip - Return expressions: text ## `vllm_mlx.api.utils.StreamingToolCallFilter` - Kind: class - Signature: `class StreamingToolCallFilter` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/utils.py#L134-L229 - Implementation: Class `StreamingToolCallFilter` declares 5 direct member(s). Buffer streaming text to suppress tool call markup. Tool call XML (e.g. ...) arrives split across multiple streaming deltas. This filter detects entry into a tool call block, suppresses all output until the block closes, and emits only non-tool-call text. The full unfiltered text is still accumulated separately for tool call parsing at stream end. - Inputs: none - Constructs: `vllm_mlx.api.utils.StreamingToolCallFilter` ## `vllm_mlx.api.utils.StreamingToolCallFilter.__init__` - Kind: method - Signature: `def __init__(self)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/utils.py#L146-L151 - Implementation: Method `StreamingToolCallFilter.__init__` updates `self._buffer`, `self._in_block`, `self._close_tag`, `self._max_open_len`; calls `max`, `len`. Method `StreamingToolCallFilter.__init__` updates `self._buffer`, `self._in_block`, `self._close_tag`, `self._max_open_len`; calls `max`, `len`. - Inputs: none - Return annotation: `not annotated` - Calls: max, len - State writes: self._buffer, self._in_block, self._close_tag, self._max_open_len ## `vllm_mlx.api.utils.StreamingToolCallFilter.process` - Kind: method - Signature: `def process(self, delta: str) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/utils.py#L153-L160 - Implementation: Method `StreamingToolCallFilter.process` updates `self._buffer`; calls `self._consume_block`, `self._scan_for_open`; has 2 explicit return paths. Process a streaming delta. Returns text to emit (may be empty). - Inputs: - `delta` (str; required): Required positional or keyword input. - Return annotation: `str` - Calls: self._consume_block, self._scan_for_open - State reads: self._in_block, self._consume_block, self._scan_for_open - State writes: self._buffer - Return expressions: self._consume_block(); self._scan_for_open() ## `vllm_mlx.api.utils.StreamingToolCallFilter._scan_for_open` - Kind: method - Signature: `def _scan_for_open(self) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/utils.py#L162-L194 - Implementation: Method `StreamingToolCallFilter._scan_for_open` updates `self._buffer`, `self._in_block`, `self._close_tag`; calls `self._buffer.find`, `len`, `self._consume_block`, `range`; has 2 explicit return paths. Scan buffer for tool call open tags. Emit safe text. - Inputs: none - Return annotation: `str` - Calls: self._buffer.find, len, self._consume_block, range, min, self._buffer.endswith, max - State reads: self._buffer.find, self._buffer, self._consume_block, self._buffer.endswith - State writes: self._buffer, self._in_block, self._close_tag - Return expressions: emit + after; emit ## `vllm_mlx.api.utils.StreamingToolCallFilter._consume_block` - Kind: method - Signature: `def _consume_block(self) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/utils.py#L196-L218 - Implementation: Method `StreamingToolCallFilter._consume_block` updates `self._buffer`, `self._in_block`, `self._close_tag`; calls `self._buffer.find`, `len`, `self._scan_for_open`, `logger.warning`; has 2 explicit return paths. Consume content inside a tool call block. Returns empty string unless the block closes and there's text after it. - Inputs: none - Return annotation: `str` - Calls: self._buffer.find, len, self._scan_for_open, logger.warning - State reads: self._buffer.find, self._buffer, self._close_tag, self._scan_for_open - State writes: self._buffer, self._in_block, self._close_tag - Return expressions: self._scan_for_open(); '' ## `vllm_mlx.api.utils.StreamingToolCallFilter.flush` - Kind: method - Signature: `def flush(self) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/utils.py#L220-L229 - Implementation: Method `StreamingToolCallFilter.flush` updates `self._buffer`, `self._in_block`; has 2 explicit return paths. Flush remaining buffer at end of stream. - Inputs: none - Return annotation: `str` - State reads: self._in_block, self._buffer - State writes: self._buffer, self._in_block - Return expressions: ''; emit ## `vllm_mlx.api.utils.StreamingThinkRouter` - Kind: class - Signature: `class StreamingThinkRouter` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/utils.py#L237-L327 - Implementation: Class `StreamingThinkRouter` declares 4 direct member(s). Route ... content to separate Anthropic thinking blocks. Instead of emitting thinking content as plain text (where it's indistinguishable from the response), this router yields tagged pieces that the streaming handler can emit as proper Anthropic content block types. Each call to process() returns a list of (block_type, text) tuples: - ("thinking", text) for content inside ... - ("text", text) for content outside think blocks Args: start_in_thinking: If True, assume the model starts in thinking mode (e.g. MiniMax adds to the generation prompt, so the tag never appears in the output stream). - Inputs: - `start_in_thinking` (bool; optional; default `False`): If True, assume the model starts in thinking mode (e.g. MiniMax adds to the generation prompt, so the tag never appears in the output stream). - Constructs: `vllm_mlx.api.utils.StreamingThinkRouter` ## `vllm_mlx.api.utils.StreamingThinkRouter.__init__` - Kind: method - Signature: `def __init__(self, start_in_thinking: bool=False)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/utils.py#L255-L257 - Implementation: Method `StreamingThinkRouter.__init__` updates `self._buffer`, `self._in_think`. Method `StreamingThinkRouter.__init__` updates `self._buffer`, `self._in_think`. - Inputs: - `start_in_thinking` (bool; optional; default `False`): Optional positional or keyword input; defaults to `False`. - Return annotation: `not annotated` - State writes: self._buffer, self._in_think ## `vllm_mlx.api.utils.StreamingThinkRouter.process` - Kind: method - Signature: `def process(self, delta: str) -> list[tuple[str, str]]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/utils.py#L259-L264 - Implementation: Method `StreamingThinkRouter.process` updates `self._buffer`; calls `self._extract_pieces`; returns `pieces`. Process a delta. Returns list of (block_type, text) pieces. - Inputs: - `delta` (str; required): Required positional or keyword input. - Return annotation: `list[tuple[str, str]]` - Calls: self._extract_pieces - State reads: self._extract_pieces - State writes: self._buffer - Return expressions: pieces ## `vllm_mlx.api.utils.StreamingThinkRouter._extract_pieces` - Kind: method - Signature: `def _extract_pieces(self, pieces: list[tuple[str, str]]) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/utils.py#L266-L317 - Implementation: Method `StreamingThinkRouter._extract_pieces` updates `self._buffer`, `self._in_think`; calls `self._buffer.find`, `len`, `pieces.append`, `range`; returns `None`. Extract all complete pieces from the buffer. - Inputs: - `pieces` (list[tuple[str, str]]; required): Required positional or keyword input. - Return annotation: `None` - Calls: self._buffer.find, len, pieces.append, range, min, self._buffer.endswith - State reads: self._in_think, self._buffer.find, self._buffer, self._buffer.endswith - State writes: self._buffer, self._in_think - Return expressions: None ## `vllm_mlx.api.utils.StreamingThinkRouter.flush` - Kind: method - Signature: `def flush(self) -> list[tuple[str, str]]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/utils.py#L319-L327 - Implementation: Method `StreamingThinkRouter.flush` updates `self._buffer`, `self._in_think`; calls `pieces.append`; returns `pieces`. Flush remaining buffer at end of stream. - Inputs: none - Return annotation: `list[tuple[str, str]]` - Calls: pieces.append - State reads: self._buffer, self._in_think - State writes: self._buffer, self._in_think - Return expressions: pieces ## `vllm_mlx.api.utils._try_read_config_json` - Kind: function - Signature: `def _try_read_config_json(name_or_path: str) -> dict | None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/utils.py#L408-L434 - Implementation: Function `_try_read_config_json` calls `Path`, `candidate.is_dir`, `config_path.is_file`, `config_path.stat`; has 2 explicit return paths. Read config.json from a local model directory. Returns None when the input is not a local directory, the directory has no config.json, the file is too large, or it cannot be parsed. - Inputs: - `name_or_path` (str; required): Required positional or keyword input. - Return annotation: `dict | None` - Calls: Path, candidate.is_dir, config_path.is_file, config_path.stat, config_path.open, json.load, isinstance - Return expressions: None; data if isinstance(data, dict) else None ## `vllm_mlx.api.utils._config_indicates_vlm` - Kind: function - Signature: `def _config_indicates_vlm(config: dict) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/utils.py#L437-L453 - Implementation: Function `_config_indicates_vlm` calls `config.get`, `isinstance`, `arch.lower`, `keyword.lower`; has 2 explicit return paths. Inspect a parsed config.json dict for multimodal markers. - Inputs: - `config` (dict; required): Required positional or keyword input. - Return annotation: `bool` - Calls: config.get, isinstance, arch.lower, keyword.lower - Return expressions: True; False ## `vllm_mlx.api.utils._check_legacy_string_patterns` - Kind: function - Signature: `def _check_legacy_string_patterns(model_name: str) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/utils.py#L456-L466 - Implementation: Function `_check_legacy_string_patterns` calls `model_name.lower`, `pattern.lower`; has 2 explicit return paths. Validation 1: substring match of MLLM_PATTERNS against the input string. Kept for HF repo IDs (where no local config.json is reachable) and as a fallback when config.json cannot be read. - Inputs: - `model_name` (str; required): Required positional or keyword input. - Return annotation: `bool` - Calls: model_name.lower, pattern.lower - Return expressions: True; False ## `vllm_mlx.api.utils.is_mllm_model` - Kind: function - Signature: `def is_mllm_model(model_name: str) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/utils.py#L469-L493 - Implementation: Function `is_mllm_model` calls `_try_read_config_json`, `_config_indicates_vlm`, `_check_legacy_string_patterns`; has 2 explicit return paths. Check if a model name or path indicates a multimodal language model. Two complementary validations are run: 1. config.json inspection: when ``model_name`` resolves to a local directory containing a readable config.json, inspect the model's own metadata (``architectures`` field, ``vision_config``, ``audio_config``, etc.). Authoritative when available because it reflects what the model actually is, not how it is named on disk. 2. Legacy substring match against ``MLLM_PATTERNS``: applied when no config.json is reachable (e.g., a HuggingFace repo ID before the weights are downloaded). Preserves the historical behaviour. Args: model_name: HuggingFace repo ID or local filesystem path. Returns: True if the model is detected as multimodal (MLLM/VLM). - Inputs: - `model_name` (str; required): HuggingFace repo ID or local filesystem path. - Return annotation: `bool` - Calls: _try_read_config_json, _config_indicates_vlm, _check_legacy_string_patterns - Return expressions: _config_indicates_vlm(config); _check_legacy_string_patterns(model_name) ## `vllm_mlx.api.utils.has_media_content` - Kind: function - Signature: `def has_media_content(messages: list) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/utils.py#L516-L536 - Implementation: Function `has_media_content` calls `isinstance`, `msg.get`, `getattr`, `part.get`; has 2 explicit return paths. Check if any message contains media content (images, video, audio). Handles both plain dicts (``msg.get("content")``) and Pydantic-style objects (``msg.content``) so it works in both engine and server contexts. - Inputs: - `messages` (list; required): Required positional or keyword input. - Return annotation: `bool` - Calls: isinstance, msg.get, getattr, part.get - Return expressions: True; False ## `vllm_mlx.api.utils._content_to_text` - Kind: function - Signature: `def _content_to_text(content) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/utils.py#L544-L560 - Implementation: Function `_content_to_text` calls `isinstance`, `hasattr`, `item.model_dump`, `item.dict().items`; has 4 explicit return paths. Extract text from content that can be str, list[ContentPart], or None. - Inputs: - `content` (not annotated; required): Required positional or keyword input. - Return annotation: `str` - Calls: isinstance, hasattr, item.model_dump, item.dict().items, item.dict, item.get, parts.append, '\n'.join, str - Return expressions: ''; content; '\n'.join(parts); str(content) ## `vllm_mlx.api.utils.extract_multimodal_content` - Kind: function - Signature: `def extract_multimodal_content(messages: list[Message], preserve_native_format: bool=False) -> tuple[list[dict], list[str], list[str], list[str]]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/api/utils.py#L563-L747 - Implementation: Function `extract_multimodal_content` calls `isinstance`, `msg.get`, `getattr`, `processed_messages.append`; returns `(processed_messages, images, videos, audios)`. Extract text content, images, videos, and audio from OpenAI-format messages. Handles: - Simple text messages - Multimodal messages with images/videos/audio - Tool call messages (assistant with tool_calls) - Tool response messages (role="tool") Args: messages: List of Message objects preserve_native_format: If True, preserve native tool message format (role="tool", tool_calls field) instead of converting to text. Required for models with native tool support in chat templates (e.g., Mistral, Llama 3+, DeepSeek V3). Returns: Tuple of (processed_messages, images, videos, audios) - processed_messages: List of {"role": str, "content": str} - images: List of image URLs/paths/base64 - videos: List of video URLs/paths/base64 - audios: List of audio URLs/paths/base64 - Inputs: - `messages` (list[Message]; required): List of Message objects - `preserve_native_format` (bool; optional; default `False`): If True, preserve native tool message format (role="tool", tool_calls field) instead of converting to text. Required for models with native tool support in chat templates (e.g., Mistral, Llama 3+, DeepSeek V3). - Return annotation: `tuple[list[dict], list[str], list[str], list[str]]` - Calls: isinstance, msg.get, getattr, processed_messages.append, hasattr, tc.model_dump, tc.dict, tc_copy.get, func.get, json.loads, tool_calls_list.append, _content_to_text, tc.get, tool_calls_text.append, '\n'.join, item.model_dump, item.dict().items, item.dict, item.get, text_parts.append, images.append, img_url.get, videos.append, vid_url.get, audios.append, audio_url.get, str - Return expressions: (processed_messages, images, videos, audios) # Module `vllm_mlx.attention` MLX Attention Backend for vLLM. This module provides an attention backend that uses MLX's native attention implementation, optimized for Apple Silicon. Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/attention.py#L1-L245 ## `vllm_mlx.attention.MLXAttentionMetadata` - Kind: class - Signature: `class MLXAttentionMetadata` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/attention.py#L20-L39 - Implementation: Class `MLXAttentionMetadata` declares 0 direct member(s). Metadata for MLX attention computation. - Inputs: - `seq_lens` (list[int]; required): Required constructor field. - `max_seq_len` (int; required): Required constructor field. - `num_prefill_tokens` (int; optional; default `0`): Optional constructor field; defaults to `0`. - `num_decode_tokens` (int; optional; default `0`): Optional constructor field; defaults to `0`. - `block_tables` (Any | None; optional; default `None`): Optional constructor field; defaults to `None`. - `slot_mapping` (Any | None; optional; default `None`): Optional constructor field; defaults to `None`. - Constructs: `vllm_mlx.attention.MLXAttentionMetadata` - Decorators: dataclass ## `vllm_mlx.attention.MLXAttentionBackend` - Kind: class - Signature: `class MLXAttentionBackend` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/attention.py#L42-L135 - Implementation: Class `MLXAttentionBackend` declares 9 direct member(s). Attention backend using MLX's native attention. MLX provides optimized attention implementations that run on Apple Silicon's GPU via Metal. This backend wraps those implementations for use with vLLM. Note: mlx-lm handles attention internally, so this backend primarily serves as a compatibility layer. - Inputs: none - Constructs: `vllm_mlx.attention.MLXAttentionBackend` ## `vllm_mlx.attention.MLXAttentionBackend.get_name` - Kind: method - Signature: `def get_name() -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/attention.py#L55-L57 - Implementation: Method `MLXAttentionBackend.get_name` returns `'MLX'`. Return backend name. - Inputs: none - Return annotation: `str` - Decorators: staticmethod - Return expressions: 'MLX' ## `vllm_mlx.attention.MLXAttentionBackend.get_impl_cls` - Kind: method - Signature: `def get_impl_cls() -> type` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/attention.py#L60-L62 - Implementation: Method `MLXAttentionBackend.get_impl_cls` returns `MLXAttentionImpl`. Return the implementation class. - Inputs: none - Return annotation: `type` - Decorators: staticmethod - Return expressions: MLXAttentionImpl ## `vllm_mlx.attention.MLXAttentionBackend.get_metadata_cls` - Kind: method - Signature: `def get_metadata_cls() -> type` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/attention.py#L65-L67 - Implementation: Method `MLXAttentionBackend.get_metadata_cls` returns `MLXAttentionMetadata`. Return the metadata class. - Inputs: none - Return annotation: `type` - Decorators: staticmethod - Return expressions: MLXAttentionMetadata ## `vllm_mlx.attention.MLXAttentionBackend.get_kv_cache_shape` - Kind: method - Signature: `def get_kv_cache_shape(num_blocks: int, block_size: int, num_kv_heads: int, head_size: int) -> tuple[int, ...]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/attention.py#L70-L89 - Implementation: Method `MLXAttentionBackend.get_kv_cache_shape` returns `(num_blocks, block_size, num_kv_heads, head_size)`. Get the shape of KV cache. Args: num_blocks: Number of cache blocks block_size: Tokens per block num_kv_heads: Number of KV attention heads head_size: Size of each attention head Returns: Shape tuple for KV cache tensor - Inputs: - `num_blocks` (int; required): Number of cache blocks - `block_size` (int; required): Tokens per block - `num_kv_heads` (int; required): Number of KV attention heads - `head_size` (int; required): Size of each attention head - Return annotation: `tuple[int, ...]` - Decorators: staticmethod - Return expressions: (num_blocks, block_size, num_kv_heads, head_size) ## `vllm_mlx.attention.MLXAttentionBackend.get_supported_head_sizes` - Kind: method - Signature: `def get_supported_head_sizes() -> list[int]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/attention.py#L92-L94 - Implementation: Method `MLXAttentionBackend.get_supported_head_sizes` returns `[64, 80, 96, 112, 128, 256]`. Return supported attention head sizes. - Inputs: none - Return annotation: `list[int]` - Decorators: staticmethod - Return expressions: [64, 80, 96, 112, 128, 256] ## `vllm_mlx.attention.MLXAttentionBackend.validate_configuration` - Kind: method - Signature: `def validate_configuration(num_heads: int, head_size: int, num_kv_heads: int, dtype: 'torch.dtype', block_size: int, **kwargs) -> list[str]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/attention.py#L97-L118 - Implementation: Method `MLXAttentionBackend.validate_configuration` calls `MLXAttentionBackend.get_supported_head_sizes`, `errors.append`; returns `errors`. Validate attention configuration. Returns list of error messages (empty if valid). - Inputs: - `num_heads` (int; required): Required positional or keyword input. - `head_size` (int; required): Required positional or keyword input. - `num_kv_heads` (int; required): Required positional or keyword input. - `dtype` ('torch.dtype'; required): Required positional or keyword input. - `block_size` (int; required): Required positional or keyword input. - `**kwargs` (not annotated; optional): Additional variadic keyword inputs accepted by this callable. - Return annotation: `list[str]` - Decorators: staticmethod - Calls: MLXAttentionBackend.get_supported_head_sizes, errors.append - Return expressions: errors ## `vllm_mlx.attention.MLXAttentionBackend.supports_dtype` - Kind: method - Signature: `def supports_dtype(dtype: 'torch.dtype') -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/attention.py#L121-L125 - Implementation: Method `MLXAttentionBackend.supports_dtype` returns `dtype in [torch.float16, torch.bfloat16, torch.float32]`. Check if dtype is supported. - Inputs: - `dtype` ('torch.dtype'; required): Required positional or keyword input. - Return annotation: `bool` - Decorators: staticmethod - Return expressions: dtype in [torch.float16, torch.bfloat16, torch.float32] ## `vllm_mlx.attention.MLXAttentionBackend.supports_block_size` - Kind: method - Signature: `def supports_block_size(block_size: int) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/attention.py#L128-L130 - Implementation: Method `MLXAttentionBackend.supports_block_size` returns `block_size in [8, 16, 32]`. Check if block size is supported. - Inputs: - `block_size` (int; required): Required positional or keyword input. - Return annotation: `bool` - Decorators: staticmethod - Return expressions: block_size in [8, 16, 32] ## `vllm_mlx.attention.MLXAttentionBackend.supports_attn_type` - Kind: method - Signature: `def supports_attn_type(attn_type: str) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/attention.py#L133-L135 - Implementation: Method `MLXAttentionBackend.supports_attn_type` returns `attn_type in ['decoder', 'encoder', 'encoder_decoder']`. Check if attention type is supported. - Inputs: - `attn_type` (str; required): Required positional or keyword input. - Return annotation: `bool` - Decorators: staticmethod - Return expressions: attn_type in ['decoder', 'encoder', 'encoder_decoder'] ## `vllm_mlx.attention.MLXAttentionImpl` - Kind: class - Signature: `class MLXAttentionImpl` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/attention.py#L138-L240 - Implementation: Class `MLXAttentionImpl` declares 2 direct member(s). MLX attention implementation. This class provides the actual attention computation using MLX. Since mlx-lm handles attention internally during generation, this serves as a compatibility interface. - Inputs: - `num_heads` (int; required): Number of attention heads - `head_size` (int; required): Size of each head - `scale` (float; required): Attention scale factor - `num_kv_heads` (int | None; optional; default `None`): Number of KV heads (for GQA/MQA) - `alibi_slopes` (list[float] | None; optional; default `None`): ALiBi position encoding slopes - `sliding_window` (int | None; optional; default `None`): Sliding window attention size - `kv_cache_dtype` (str; optional; default `'auto'`): KV cache data type - `blocksparse_params` (dict | None; optional; default `None`): Block-sparse attention params - `logits_soft_cap` (float | None; optional; default `None`): Soft cap for logits - `**kwargs` (not annotated; optional): Additional variadic keyword inputs accepted by this callable. - Constructs: `vllm_mlx.attention.MLXAttentionImpl` ## `vllm_mlx.attention.MLXAttentionImpl.__init__` - Kind: method - Signature: `def __init__(self, num_heads: int, head_size: int, scale: float, num_kv_heads: int | None=None, alibi_slopes: list[float] | None=None, sliding_window: int | None=None, kv_cache_dtype: str='auto', blocksparse_params: dict | None=None, logits_soft_cap: float | None=None, **kwargs)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/attention.py#L147-L186 - Implementation: Method `MLXAttentionImpl.__init__` updates `self.num_heads`, `self.head_size`, `self.scale`, `self.num_kv_heads`; calls `logger.debug`. Initialize MLX attention. Args: num_heads: Number of attention heads head_size: Size of each head scale: Attention scale factor num_kv_heads: Number of KV heads (for GQA/MQA) alibi_slopes: ALiBi position encoding slopes sliding_window: Sliding window attention size kv_cache_dtype: KV cache data type blocksparse_params: Block-sparse attention params logits_soft_cap: Soft cap for logits - Inputs: - `num_heads` (int; required): Number of attention heads - `head_size` (int; required): Size of each head - `scale` (float; required): Attention scale factor - `num_kv_heads` (int | None; optional; default `None`): Number of KV heads (for GQA/MQA) - `alibi_slopes` (list[float] | None; optional; default `None`): ALiBi position encoding slopes - `sliding_window` (int | None; optional; default `None`): Sliding window attention size - `kv_cache_dtype` (str; optional; default `'auto'`): KV cache data type - `blocksparse_params` (dict | None; optional; default `None`): Block-sparse attention params - `logits_soft_cap` (float | None; optional; default `None`): Soft cap for logits - `**kwargs` (not annotated; optional): Additional variadic keyword inputs accepted by this callable. - Return annotation: `not annotated` - Calls: logger.debug - State reads: self.num_kv_heads - State writes: self.num_heads, self.head_size, self.scale, self.num_kv_heads, self.alibi_slopes, self.sliding_window, self.kv_cache_dtype, self.logits_soft_cap ## `vllm_mlx.attention.MLXAttentionImpl.forward` - Kind: method - Signature: `def forward(self, query: Any, key: Any, value: Any, kv_cache: Any | None=None, attn_metadata: MLXAttentionMetadata | None=None, output: Any | None=None, **kwargs) -> Any` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/attention.py#L188-L240 - Implementation: Method `MLXAttentionImpl.forward` calls `isinstance`, `mx.array`, `hasattr`, `query.numpy`; returns `attn_output`. Compute attention. Note: In the MLX backend, attention is handled internally by mlx-lm during the generation process. This method is provided for compatibility but may not be called directly. Args: query: Query tensor key: Key tensor value: Value tensor kv_cache: Optional KV cache attn_metadata: Attention metadata output: Optional output buffer Returns: Attention output tensor - Inputs: - `query` (Any; required): Query tensor - `key` (Any; required): Key tensor - `value` (Any; required): Value tensor - `kv_cache` (Any | None; optional; default `None`): Optional KV cache - `attn_metadata` (MLXAttentionMetadata | None; optional; default `None`): Attention metadata - `output` (Any | None; optional; default `None`): Optional output buffer - `**kwargs` (not annotated; optional): Additional variadic keyword inputs accepted by this callable. - Return annotation: `Any` - Calls: isinstance, mx.array, hasattr, query.numpy, key.numpy, value.numpy, mx.fast.scaled_dot_product_attention, logger.error - State reads: self.scale - Return expressions: attn_output ## `vllm_mlx.attention.create_mlx_attention_backend` - Kind: function - Signature: `def create_mlx_attention_backend() -> type` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/attention.py#L243-L245 - Implementation: Function `create_mlx_attention_backend` returns `MLXAttentionBackend`. Factory function to create MLX attention backend. - Inputs: none - Return annotation: `type` - Return expressions: MLXAttentionBackend # Module `vllm_mlx.audio` Audio support for vllm-mlx using mlx-audio. Provides: - STT (Speech-to-Text): Whisper, Parakeet - TTS (Text-to-Speech): Kokoro, Chatterbox, VibeVoice, VoxCPM - Audio Processing: SAM-Audio (voice separation) Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/audio/__init__.py#L1-L25 # Module `vllm_mlx.audio.processor` Audio processing using mlx-audio. Supports: - SAM-Audio: Text-guided source separation (isolate voice from background) - MossFormer2: Speech enhancement (noise removal) Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/audio/processor.py#L1-L214 ## `vllm_mlx.audio.processor.SeparationResult` - Kind: class - Signature: `class SeparationResult` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/audio/processor.py#L24-L30 - Implementation: Class `SeparationResult` declares 0 direct member(s). Result from audio separation. - Inputs: - `target` (np.ndarray; required): Required constructor field. - `residual` (np.ndarray; required): Required constructor field. - `sample_rate` (int; required): Required constructor field. - `peak_memory` (float; required): Required constructor field. - Constructs: `vllm_mlx.audio.processor.SeparationResult` - Decorators: dataclass ## `vllm_mlx.audio.processor.AudioProcessor` - Kind: class - Signature: `class AudioProcessor` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/audio/processor.py#L33-L192 - Implementation: Class `AudioProcessor` declares 6 direct member(s). Audio processor for voice separation and enhancement. Uses SAM-Audio for text-guided source separation: - Isolate speech from music/noise - Extract specific sounds by description Usage: processor = AudioProcessor() processor.load() result = processor.separate("meeting.mp3", description="speech") processor.save(result.target, "voice_only.wav") - Inputs: - `model_name` (str; optional; default `DEFAULT_SAM_MODEL`): HuggingFace model name. Supported: - mlx-community/sam-audio-large-fp16 (best quality) - mlx-community/sam-audio-large - mlx-community/sam-audio-small-fp16 (faster) - mlx-community/sam-audio-small - Constructs: `vllm_mlx.audio.processor.AudioProcessor` ## `vllm_mlx.audio.processor.AudioProcessor.__init__` - Kind: method - Signature: `def __init__(self, model_name: str=DEFAULT_SAM_MODEL)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/audio/processor.py#L48-L66 - Implementation: Method `AudioProcessor.__init__` updates `self.model_name`, `self.model`, `self.processor`, `self._loaded`. Initialize audio processor. Args: model_name: HuggingFace model name. Supported: - mlx-community/sam-audio-large-fp16 (best quality) - mlx-community/sam-audio-large - mlx-community/sam-audio-small-fp16 (faster) - mlx-community/sam-audio-small - Inputs: - `model_name` (str; optional; default `DEFAULT_SAM_MODEL`): HuggingFace model name. Supported: - mlx-community/sam-audio-large-fp16 (best quality) - mlx-community/sam-audio-large - mlx-community/sam-audio-small-fp16 (faster) - mlx-community/sam-audio-small - Return annotation: `not annotated` - State writes: self.model_name, self.model, self.processor, self._loaded, self.sample_rate ## `vllm_mlx.audio.processor.AudioProcessor.load` - Kind: method - Signature: `def load(self) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/audio/processor.py#L68-L88 - Implementation: Method `AudioProcessor.load` updates `self.model`, `self.processor`, `self.sample_rate`, `self._loaded`; calls `SAMAudio.from_pretrained`, `SAMAudioProcessor.from_pretrained`, `hasattr`, `logger.info`; can raise `ImportError`; returns `None`. Load the SAM-Audio model. - Inputs: none - Return annotation: `None` - Calls: SAMAudio.from_pretrained, SAMAudioProcessor.from_pretrained, hasattr, logger.info, logger.error, ImportError - State reads: self._loaded, self.model_name, self.model, self.model.sample_rate - State writes: self.model, self.processor, self.sample_rate, self._loaded - Raises directly: ImportError - Return expressions: None ## `vllm_mlx.audio.processor.AudioProcessor.separate` - Kind: method - Signature: `def separate(self, audio_path: Union[str, Path], description: str='speech', chunk_seconds: Optional[float]=None) -> SeparationResult` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/audio/processor.py#L90-L151 - Implementation: Method `AudioProcessor.separate` calls `self.load`, `str`, `self.processor`, `self.model.separate_long`; returns `SeparationResult(target=target, residual=residual, sample_rate=self.sample_rate, peak_memory=getattr(result, 'peak_memo…`. Separate audio based on text description. Args: audio_path: Path to audio file description: What to isolate (e.g., "speech", "music", "a person speaking") chunk_seconds: Process in chunks for long audio (memory efficient) Returns: SeparationResult with target (isolated) and residual (background) audio - Inputs: - `audio_path` (Union[str, Path]; required): Path to audio file - `description` (str; optional; default `'speech'`): What to isolate (e.g., "speech", "music", "a person speaking") - `chunk_seconds` (Optional[float]; optional; default `None`): Process in chunks for long audio (memory efficient) - Return annotation: `SeparationResult` - Calls: self.load, str, self.processor, self.model.separate_long, getattr, self.model.separate, self._to_numpy, SeparationResult, logger.error - State reads: self._loaded, self.load, self.processor, self.model.separate_long, self.model, self.model.separate, self._to_numpy, self.sample_rate - Return expressions: SeparationResult(target=target, residual=residual, sample_rate=self.sample_rate, peak_memory=getattr(result, 'peak_memo… ## `vllm_mlx.audio.processor.AudioProcessor._to_numpy` - Kind: method - Signature: `def _to_numpy(self, audio) -> np.ndarray` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/audio/processor.py#L153-L157 - Implementation: Method `AudioProcessor._to_numpy` calls `hasattr`, `np.array`, `audio.tolist`; has 2 explicit return paths. Convert audio to numpy array. - Inputs: - `audio` (not annotated; required): Required positional or keyword input. - Return annotation: `np.ndarray` - Calls: hasattr, np.array, audio.tolist - Return expressions: np.array(audio.tolist(), dtype=np.float32); np.array(audio, dtype=np.float32) ## `vllm_mlx.audio.processor.AudioProcessor.save` - Kind: method - Signature: `def save(self, audio: np.ndarray, path: Union[str, Path], sample_rate: Optional[int]=None) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/audio/processor.py#L159-L185 - Implementation: Method `AudioProcessor.save` calls `save_audio`, `str`, `(audio * 32767).astype`, `wav.write`. Save audio to file. Args: audio: Audio data as numpy array path: Output file path sample_rate: Sample rate (uses model default if None) - Inputs: - `audio` (np.ndarray; required): Audio data as numpy array - `path` (Union[str, Path]; required): Output file path - `sample_rate` (Optional[int]; optional; default `None`): Sample rate (uses model default if None) - Return annotation: `None` - Calls: save_audio, str, (audio * 32767).astype, wav.write, logger.info - State reads: self.sample_rate ## `vllm_mlx.audio.processor.AudioProcessor.unload` - Kind: method - Signature: `def unload(self) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/audio/processor.py#L187-L192 - Implementation: Method `AudioProcessor.unload` updates `self.model`, `self.processor`, `self._loaded`; calls `logger.info`. Unload model to free memory. - Inputs: none - Return annotation: `None` - Calls: logger.info - State writes: self.model, self.processor, self._loaded ## `vllm_mlx.audio.processor.separate_voice` - Kind: function - Signature: `def separate_voice(audio_path: Union[str, Path], model_name: str=DEFAULT_SAM_MODEL, description: str='speech') -> Tuple[np.ndarray, np.ndarray]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/audio/processor.py#L195-L214 - Implementation: Function `separate_voice` calls `AudioProcessor`, `processor.load`, `processor.separate`; returns `(result.target, result.residual)`. Convenience function to separate voice from audio. Args: audio_path: Path to audio file model_name: Model to use description: What to isolate Returns: Tuple of (voice_audio, background_audio) as numpy arrays - Inputs: - `audio_path` (Union[str, Path]; required): Path to audio file - `model_name` (str; optional; default `DEFAULT_SAM_MODEL`): Model to use - `description` (str; optional; default `'speech'`): What to isolate - Return annotation: `Tuple[np.ndarray, np.ndarray]` - Calls: AudioProcessor, processor.load, processor.separate - Return expressions: (result.target, result.residual) # Module `vllm_mlx.audio.stt` Speech-to-Text (STT) engine using mlx-audio. Supports: - Whisper (multilingual, 99+ languages) - Parakeet (English-focused, fast) Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/audio/stt.py#L1-L160 ## `vllm_mlx.audio.stt.TranscriptionResult` - Kind: class - Signature: `class TranscriptionResult` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/audio/stt.py#L23-L29 - Implementation: Class `TranscriptionResult` declares 0 direct member(s). Result from audio transcription. - Inputs: - `text` (str; required): Required constructor field. - `language` (Optional[str]; optional; default `None`): Optional constructor field; defaults to `None`. - `duration` (Optional[float]; optional; default `None`): Optional constructor field; defaults to `None`. - `segments` (Optional[list]; optional; default `None`): Optional constructor field; defaults to `None`. - Constructs: `vllm_mlx.audio.stt.TranscriptionResult` - Decorators: dataclass ## `vllm_mlx.audio.stt.STTEngine` - Kind: class - Signature: `class STTEngine` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/audio/stt.py#L32-L139 - Implementation: Class `STTEngine` declares 4 direct member(s). Speech-to-Text engine supporting Whisper and Parakeet models. Usage: engine = STTEngine("mlx-community/whisper-large-v3-mlx") engine.load() result = engine.transcribe("audio.mp3") print(result.text) - Inputs: - `model_name` (str; optional; default `DEFAULT_WHISPER_MODEL`): HuggingFace model name. Supported: - mlx-community/whisper-large-v3-mlx (multilingual) - mlx-community/whisper-large-v3-turbo (fast) - mlx-community/whisper-medium-mlx - mlx-community/whisper-small-mlx - mlx-community/parakeet-tdt-0.6b-v2 (English, fastest) - mlx-community/parakeet-tdt-0.6b-v3 - Constructs: `vllm_mlx.audio.stt.STTEngine` ## `vllm_mlx.audio.stt.STTEngine.__init__` - Kind: method - Signature: `def __init__(self, model_name: str=DEFAULT_WHISPER_MODEL)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/audio/stt.py#L43-L62 - Implementation: Method `STTEngine.__init__` updates `self.model_name`, `self.model`, `self._loaded`, `self._is_parakeet`; calls `model_name.lower`. Initialize STT engine. Args: model_name: HuggingFace model name. Supported: - mlx-community/whisper-large-v3-mlx (multilingual) - mlx-community/whisper-large-v3-turbo (fast) - mlx-community/whisper-medium-mlx - mlx-community/whisper-small-mlx - mlx-community/parakeet-tdt-0.6b-v2 (English, fastest) - mlx-community/parakeet-tdt-0.6b-v3 - Inputs: - `model_name` (str; optional; default `DEFAULT_WHISPER_MODEL`): HuggingFace model name. Supported: - mlx-community/whisper-large-v3-mlx (multilingual) - mlx-community/whisper-large-v3-turbo (fast) - mlx-community/whisper-medium-mlx - mlx-community/whisper-small-mlx - mlx-community/parakeet-tdt-0.6b-v2 (English, fastest) - mlx-community/parakeet-tdt-0.6b-v3 - Return annotation: `not annotated` - Calls: model_name.lower - State writes: self.model_name, self.model, self._loaded, self._is_parakeet ## `vllm_mlx.audio.stt.STTEngine.load` - Kind: method - Signature: `def load(self) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/audio/stt.py#L64-L79 - Implementation: Method `STTEngine.load` updates `self.model`, `self._loaded`; calls `load_model`, `logger.info`, `logger.error`, `ImportError`; can raise `ImportError`; returns `None`. Load the STT model. - Inputs: none - Return annotation: `None` - Calls: load_model, logger.info, logger.error, ImportError - State reads: self._loaded, self.model_name - State writes: self.model, self._loaded - Raises directly: ImportError - Return expressions: None ## `vllm_mlx.audio.stt.STTEngine.transcribe` - Kind: method - Signature: `def transcribe(self, audio_path: Union[str, Path], language: Optional[str]=None, task: str='transcribe') -> TranscriptionResult` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/audio/stt.py#L81-L133 - Implementation: Method `STTEngine.transcribe` calls `self.load`, `str`, `self.model.generate`, `getattr`; returns `TranscriptionResult(text=text.strip() if isinstance(text, str) else str(text), language=detected_lang, duration=duratio…`. Transcribe audio file to text. Args: audio_path: Path to audio file (mp3, wav, m4a, etc.) language: Language code (e.g., "en", "es"). Auto-detected if None. task: "transcribe" or "translate" (translate to English) Returns: TranscriptionResult with text and metadata - Inputs: - `audio_path` (Union[str, Path]; required): Path to audio file (mp3, wav, m4a, etc.) - `language` (Optional[str]; optional; default `None`): Language code (e.g., "en", "es"). Auto-detected if None. - `task` (str; optional; default `'transcribe'`): "transcribe" or "translate" (translate to English) - Return annotation: `TranscriptionResult` - Calls: self.load, str, self.model.generate, getattr, hasattr, TranscriptionResult, isinstance, text.strip, logger.error - State reads: self._loaded, self.load, self._is_parakeet, self.model.generate, self.model - Return expressions: TranscriptionResult(text=text.strip() if isinstance(text, str) else str(text), language=detected_lang, duration=duratio… ## `vllm_mlx.audio.stt.STTEngine.unload` - Kind: method - Signature: `def unload(self) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/audio/stt.py#L135-L139 - Implementation: Method `STTEngine.unload` updates `self.model`, `self._loaded`; calls `logger.info`. Unload model to free memory. - Inputs: none - Return annotation: `None` - Calls: logger.info - State writes: self.model, self._loaded ## `vllm_mlx.audio.stt.transcribe_audio` - Kind: function - Signature: `def transcribe_audio(audio_path: Union[str, Path], model_name: str=DEFAULT_WHISPER_MODEL, language: Optional[str]=None) -> TranscriptionResult` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/audio/stt.py#L142-L160 - Implementation: Function `transcribe_audio` calls `STTEngine`, `engine.load`, `engine.transcribe`; returns `engine.transcribe(audio_path, language=language)`. Convenience function to transcribe audio without managing engine. Args: audio_path: Path to audio file model_name: Model to use language: Language code (optional) Returns: TranscriptionResult - Inputs: - `audio_path` (Union[str, Path]; required): Path to audio file - `model_name` (str; optional; default `DEFAULT_WHISPER_MODEL`): Model to use - `language` (Optional[str]; optional; default `None`): Language code (optional) - Return annotation: `TranscriptionResult` - Calls: STTEngine, engine.load, engine.transcribe - Return expressions: engine.transcribe(audio_path, language=language) # Module `vllm_mlx.audio.tts` Text-to-Speech (TTS) engine using mlx-audio. Supports: - Kokoro (fast, lightweight) - Chatterbox (multilingual, expressive) - VibeVoice (realtime, low latency) - VoxCPM (Chinese/English, high quality) Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/audio/tts.py#L1-L315 ## `vllm_mlx.audio.tts.AudioOutput` - Kind: class - Signature: `class AudioOutput` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/audio/tts.py#L44-L49 - Implementation: Class `AudioOutput` declares 0 direct member(s). Output from TTS generation. - Inputs: - `audio` (np.ndarray; required): Required constructor field. - `sample_rate` (int; required): Required constructor field. - `duration` (float; required): Required constructor field. - Constructs: `vllm_mlx.audio.tts.AudioOutput` - Decorators: dataclass ## `vllm_mlx.audio.tts.TTSEngine` - Kind: class - Signature: `class TTSEngine` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/audio/tts.py#L52-L292 - Implementation: Class `TTSEngine` declares 9 direct member(s). Text-to-Speech engine supporting multiple model families. Usage: engine = TTSEngine("mlx-community/Kokoro-82M-bf16") engine.load() audio = engine.generate("Hello world!", voice="af_heart") engine.save(audio, "output.wav") - Inputs: - `model_name` (str; optional; default `DEFAULT_TTS_MODEL`): HuggingFace model name. Supported families: - Kokoro: mlx-community/Kokoro-82M-bf16, Kokoro-82M-4bit - Chatterbox: mlx-community/chatterbox-turbo-fp16 - VibeVoice: mlx-community/VibeVoice-Realtime-0.5B-4bit - VoxCPM: mlx-community/VoxCPM1.5 - Constructs: `vllm_mlx.audio.tts.TTSEngine` ## `vllm_mlx.audio.tts.TTSEngine.__init__` - Kind: method - Signature: `def __init__(self, model_name: str=DEFAULT_TTS_MODEL)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/audio/tts.py#L63-L80 - Implementation: Method `TTSEngine.__init__` updates `self.model_name`, `self.model`, `self._loaded`, `self._model_family`; calls `self._detect_family`. Initialize TTS engine. Args: model_name: HuggingFace model name. Supported families: - Kokoro: mlx-community/Kokoro-82M-bf16, Kokoro-82M-4bit - Chatterbox: mlx-community/chatterbox-turbo-fp16 - VibeVoice: mlx-community/VibeVoice-Realtime-0.5B-4bit - VoxCPM: mlx-community/VoxCPM1.5 - Inputs: - `model_name` (str; optional; default `DEFAULT_TTS_MODEL`): HuggingFace model name. Supported families: - Kokoro: mlx-community/Kokoro-82M-bf16, Kokoro-82M-4bit - Chatterbox: mlx-community/chatterbox-turbo-fp16 - VibeVoice: mlx-community/VibeVoice-Realtime-0.5B-4bit - VoxCPM: mlx-community/VoxCPM1.5 - Return annotation: `not annotated` - Calls: self._detect_family - State reads: self._detect_family - State writes: self.model_name, self.model, self._loaded, self._model_family ## `vllm_mlx.audio.tts.TTSEngine._detect_family` - Kind: method - Signature: `def _detect_family(self, model_name: str) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/audio/tts.py#L82-L98 - Implementation: Method `TTSEngine._detect_family` calls `model_name.lower`; has 6 explicit return paths. Detect model family from name. - Inputs: - `model_name` (str; required): Required positional or keyword input. - Return annotation: `str` - Calls: model_name.lower - Return expressions: 'kokoro'; 'chatterbox'; 'vibevoice'; 'voxcpm'; 'csm'; 'cosyvoice' ## `vllm_mlx.audio.tts.TTSEngine.load` - Kind: method - Signature: `def load(self) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/audio/tts.py#L100-L117 - Implementation: Method `TTSEngine.load` updates `self.model`, `self._loaded`; calls `load_model`, `logger.info`, `logger.error`, `ImportError`; can raise `ImportError`; returns `None`. Load the TTS model. - Inputs: none - Return annotation: `None` - Calls: load_model, logger.info, logger.error, ImportError - State reads: self._loaded, self.model_name, self._model_family - State writes: self.model, self._loaded - Raises directly: ImportError - Return expressions: None ## `vllm_mlx.audio.tts.TTSEngine.generate` - Kind: method - Signature: `def generate(self, text: str, voice: str='af_heart', speed: float=1.0, lang_code: str='a') -> AudioOutput` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/audio/tts.py#L119-L185 - Implementation: Method `TTSEngine.generate` calls `self.load`, `self.model.generate`, `hasattr`, `isinstance`; can raise `RuntimeError`; returns `AudioOutput(audio=full_audio, sample_rate=sample_rate, duration=duration)`. Generate speech from text. Args: text: Text to synthesize voice: Voice ID (model-specific) speed: Speech speed (0.5 to 2.0) lang_code: Language code (a=English, e=Spanish, f=French, etc.) Returns: AudioOutput with audio data and metadata - Inputs: - `text` (str; required): Text to synthesize - `voice` (str; optional; default `'af_heart'`): Voice ID (model-specific) - `speed` (float; optional; default `1.0`): Speech speed (0.5 to 2.0) - `lang_code` (str; optional; default `'a'`): Language code (a=English, e=Spanish, f=French, etc.) - Return annotation: `AudioOutput` - Calls: self.load, self.model.generate, hasattr, isinstance, np.array, audio_data.tolist, audio_chunks.append, RuntimeError, len, np.concatenate, AudioOutput, logger.error - State reads: self._loaded, self.load, self.model.generate, self.model - Raises directly: RuntimeError - Return expressions: AudioOutput(audio=full_audio, sample_rate=sample_rate, duration=duration) ## `vllm_mlx.audio.tts.TTSEngine.stream_generate` - Kind: method - Signature: `def stream_generate(self, text: str, voice: str='af_heart', speed: float=1.0) -> Iterator[AudioOutput]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/audio/tts.py#L187-L227 - Implementation: Method `TTSEngine.stream_generate` calls `self.load`, `self.model.generate`, `hasattr`, `np.array`; yields values incrementally. Stream speech generation chunk by chunk. Args: text: Text to synthesize voice: Voice ID speed: Speech speed Yields: AudioOutput chunks - Inputs: - `text` (str; required): Text to synthesize - `voice` (str; optional; default `'af_heart'`): Voice ID - `speed` (float; optional; default `1.0`): Speech speed - Return annotation: `Iterator[AudioOutput]` - Calls: self.load, self.model.generate, hasattr, np.array, audio_data.tolist, AudioOutput, len - State reads: self._loaded, self.load, self.model.generate, self.model ## `vllm_mlx.audio.tts.TTSEngine.save` - Kind: method - Signature: `def save(self, audio: AudioOutput, path: Union[str, Path], format: str='wav') -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/audio/tts.py#L229-L255 - Implementation: Method `TTSEngine.save` calls `save_audio`, `str`, `logger.info`, `(audio.audio * 32767).astype`. Save audio to file. Args: audio: AudioOutput to save path: Output file path format: Output format (wav, mp3) - Inputs: - `audio` (AudioOutput; required): AudioOutput to save - `path` (Union[str, Path]; required): Output file path - `format` (str; optional; default `'wav'`): Output format (wav, mp3) - Return annotation: `None` - Calls: save_audio, str, logger.info, (audio.audio * 32767).astype, wav.write ## `vllm_mlx.audio.tts.TTSEngine.to_bytes` - Kind: method - Signature: `def to_bytes(self, audio: AudioOutput, format: str='wav') -> bytes` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/audio/tts.py#L257-L277 - Implementation: Method `TTSEngine.to_bytes` calls `io.BytesIO`, `(audio.audio * 32767).astype`, `wav.write`, `buffer.getvalue`; returns `buffer.getvalue()`. Convert audio to bytes. Args: audio: AudioOutput to convert format: Output format (wav, mp3) Returns: Audio data as bytes - Inputs: - `audio` (AudioOutput; required): AudioOutput to convert - `format` (str; optional; default `'wav'`): Output format (wav, mp3) - Return annotation: `bytes` - Calls: io.BytesIO, (audio.audio * 32767).astype, wav.write, buffer.getvalue - Return expressions: buffer.getvalue() ## `vllm_mlx.audio.tts.TTSEngine.get_voices` - Kind: method - Signature: `def get_voices(self) -> list` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/audio/tts.py#L279-L286 - Implementation: Method `TTSEngine.get_voices` has 3 explicit return paths. Get available voices for current model. - Inputs: none - Return annotation: `list` - State reads: self._model_family - Return expressions: KOKORO_VOICES; CHATTERBOX_VOICES; ['default'] ## `vllm_mlx.audio.tts.TTSEngine.unload` - Kind: method - Signature: `def unload(self) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/audio/tts.py#L288-L292 - Implementation: Method `TTSEngine.unload` updates `self.model`, `self._loaded`; calls `logger.info`. Unload model to free memory. - Inputs: none - Return annotation: `None` - Calls: logger.info - State writes: self.model, self._loaded ## `vllm_mlx.audio.tts.generate_speech` - Kind: function - Signature: `def generate_speech(text: str, model_name: str=DEFAULT_TTS_MODEL, voice: str='af_heart', speed: float=1.0) -> AudioOutput` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/audio/tts.py#L295-L315 - Implementation: Function `generate_speech` calls `TTSEngine`, `engine.load`, `engine.generate`; returns `engine.generate(text, voice=voice, speed=speed)`. Convenience function to generate speech without managing engine. Args: text: Text to synthesize model_name: Model to use voice: Voice ID speed: Speech speed Returns: AudioOutput - Inputs: - `text` (str; required): Text to synthesize - `model_name` (str; optional; default `DEFAULT_TTS_MODEL`): Model to use - `voice` (str; optional; default `'af_heart'`): Voice ID - `speed` (float; optional; default `1.0`): Speech speed - Return annotation: `AudioOutput` - Calls: TTSEngine, engine.load, engine.generate - Return expressions: engine.generate(text, voice=voice, speed=speed) # Module `vllm_mlx.audio_limits` Resource limits for optional audio endpoints. Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/audio_limits.py#L1-L77 ## `vllm_mlx.audio_limits.AsyncReadableUpload` - Kind: class - Signature: `class AsyncReadableUpload(Protocol)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/audio_limits.py#L17-L25 - Implementation: Class `AsyncReadableUpload` derives from `Protocol` and declares 1 direct member(s). Structural type for an asynchronously readable uploaded file. - Inputs: none - Constructs: `vllm_mlx.audio_limits.AsyncReadableUpload` ## `vllm_mlx.audio_limits.AsyncReadableUpload.read` - Kind: method - Signature: `async def read(self, size: int=-1) -> bytes` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/audio_limits.py#L22-L25 - Implementation: Method `AsyncReadableUpload.read` contains no state mutation, call, raise, return, await, or yield. Read at most ``size`` bytes, or all remaining bytes when negative. - Inputs: - `size` (int; optional; default `-1`): Optional positional or keyword input; defaults to `-1`. - Return annotation: `bytes` ## `vllm_mlx.audio_limits.save_upload_with_limit` - Kind: function - Signature: `async def save_upload_with_limit(file: AsyncReadableUpload, *, max_bytes: int, default_suffix: str='.wav', chunk_size: int=UPLOAD_CHUNK_SIZE) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/audio_limits.py#L28-L65 - Implementation: Function `save_upload_with_limit` calls `Path`, `tempfile.NamedTemporaryFile`, `file.read`, `len`; awaits asynchronous work; can raise `HTTPException`; returns `tmp_path`. Stream an uploaded file to disk while enforcing a hard byte limit. This prevents large audio uploads from being buffered entirely in memory. - Inputs: - `file` (AsyncReadableUpload; required): Required positional or keyword input. - `max_bytes` (int; required): Required keyword-only input. - `default_suffix` (str; optional; default `'.wav'`): Optional keyword-only input; defaults to `'.wav'`. - `chunk_size` (int; optional; default `UPLOAD_CHUNK_SIZE`): Optional keyword-only input; defaults to `UPLOAD_CHUNK_SIZE`. - Return annotation: `str` - Calls: Path, tempfile.NamedTemporaryFile, file.read, len, HTTPException, tmp.write, os.path.exists, os.unlink - Raises directly: HTTPException - Return expressions: tmp_path ## `vllm_mlx.audio_limits.validate_tts_input_length` - Kind: function - Signature: `def validate_tts_input_length(text: str, *, max_chars: int) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/audio_limits.py#L68-L77 - Implementation: Function `validate_tts_input_length` calls `len`, `HTTPException`; can raise `HTTPException`. Reject oversized TTS requests before synthesis starts. - Inputs: - `text` (str; required): Required positional or keyword input. - `max_chars` (int; required): Required keyword-only input. - Return annotation: `None` - Calls: len, HTTPException - Raises directly: HTTPException # Module `vllm_mlx.bench_serve` Serving benchmark for vllm-mlx. Measures end-to-end HTTP performance of a running vllm-mlx server: - Time to First Token (TTFT) - Time Per Output Token (TPOT) - End-to-end latency - Generation and prompt throughput - Concurrent request handling - KV cache hit rates - Metal memory utilization This module has no MLX dependency and can be imported on any platform. It is a pure HTTP client that talks to a running OpenAI-compatible server. Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/bench_serve.py#L1-L2638 ## `vllm_mlx.bench_serve.WorkloadCase` - Kind: class - Signature: `class WorkloadCase` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/bench_serve.py#L51-L62 - Implementation: Class `WorkloadCase` declares 0 direct member(s). One declarative benchmark case for contract-style serving tests. - Inputs: - `case_id` (str; required): Required constructor field. - `messages` (list[dict]; required): Required constructor field. - `request_path` (Optional[str]; optional; default `None`): Optional constructor field; defaults to `None`. - `max_tokens` (Optional[int]; optional; default `None`): Optional constructor field; defaults to `None`. - `enable_thinking` (Optional[bool]; optional; default `None`): Optional constructor field; defaults to `None`. - `extra_body` (Optional[dict]; optional; default `None`): Optional constructor field; defaults to `None`. - `policy_timeout_ms` (Optional[int]; optional; default `None`): Optional constructor field; defaults to `None`. - `checks` (Optional[dict]; optional; default `None`): Optional constructor field; defaults to `None`. - `tags` (tuple[str, ...]; optional; default `()`): Optional constructor field; defaults to `()`. - Constructs: `vllm_mlx.bench_serve.WorkloadCase` - Decorators: dataclass ## `vllm_mlx.bench_serve.Workload` - Kind: class - Signature: `class Workload` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/bench_serve.py#L66-L72 - Implementation: Class `Workload` declares 0 direct member(s). Normalized bench-serve workload manifest. - Inputs: - `name` (str; required): Required constructor field. - `description` (str; required): Required constructor field. - `defaults` (dict; required): Required constructor field. - `cases` (list[WorkloadCase]; required): Required constructor field. - Constructs: `vllm_mlx.bench_serve.Workload` - Decorators: dataclass ## `vllm_mlx.bench_serve.load_prompt_set` - Kind: function - Signature: `def load_prompt_set(name_or_path: str) -> list[list[dict]]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/bench_serve.py#L75-L137 - Implementation: Function `load_prompt_set` calls `target.exists`, `FileNotFoundError`, `target.open`, `json.load`; can raise `FileNotFoundError`, `ValueError`; has 2 explicit return paths. Load a prompt set by builtin name or file path. Builtin sets (``short``, ``medium``, ``long``, ``thinking``) are loaded from the ``bench_serve_prompts/`` directory next to this module. Any other value is treated as a filesystem path and loaded directly. Two file formats are accepted (detected automatically): 1. **Flat** — list of single message dicts. Each dict becomes a single-message prompt. Backwards-compatible with the original format. ``[{"role": "user", "content": "..."}, ...]`` 2. **Multi-message** — list of message-dict lists. Each inner list is a full chat history (e.g. ``[system, user]``). Use this format when you want to benchmark with system prompts that match an ``--warm-prompts`` warm-up, or to simulate multi-turn conversation. ``[[{"role":"system","content":"..."}, {"role":"user","content":"..."}], ...]`` Returns: A list of message-dict lists, i.e. every entry is a full chat history. Flat-format files are normalized to single-element lists. Raises: FileNotFoundError: If ``name_or_path`` is not a known builtin name and the path does not exist, or if a builtin name is requested but its JSON file is missing from the package. ValueError: If the file shape is not recognised. - Inputs: - `name_or_path` (str; required): Required positional or keyword input. - Return annotation: `list[list[dict]]` - Calls: target.exists, FileNotFoundError, target.open, json.load, Path(name_or_path).expanduser, Path, path.exists, sorted, path.open, isinstance, ValueError, type - Raises directly: FileNotFoundError, ValueError - Return expressions: [[msg] for msg in raw]; raw ## `vllm_mlx.bench_serve._require_message_list` - Kind: function - Signature: `def _require_message_list(value: Any, *, label: str) -> list[dict]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/bench_serve.py#L140-L148 - Implementation: Function `_require_message_list` calls `isinstance`, `ValueError`, `enumerate`; can raise `ValueError`; returns `value`. Function `_require_message_list` calls `isinstance`, `ValueError`, `enumerate`; can raise `ValueError`; returns `value`. - Inputs: - `value` (Any; required): Required positional or keyword input. - `label` (str; required): Required keyword-only input. - Return annotation: `list[dict]` - Calls: isinstance, ValueError, enumerate - Raises directly: ValueError - Return expressions: value ## `vllm_mlx.bench_serve._load_case_request` - Kind: function - Signature: `def _load_case_request(path: str, *, workload_path: Path, case_id: str) -> dict` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/bench_serve.py#L151-L159 - Implementation: Function `_load_case_request` calls `Path(path).expanduser`, `Path`, `request_path.is_absolute`, `request_path.open`; can raise `ValueError`; returns `request`. Function `_load_case_request` calls `Path(path).expanduser`, `Path`, `request_path.is_absolute`, `request_path.open`; can raise `ValueError`; returns `request`. - Inputs: - `path` (str; required): Required positional or keyword input. - `workload_path` (Path; required): Required keyword-only input. - `case_id` (str; required): Required keyword-only input. - Return annotation: `dict` - Calls: Path(path).expanduser, Path, request_path.is_absolute, request_path.open, json.load, isinstance, ValueError - Raises directly: ValueError - Return expressions: request ## `vllm_mlx.bench_serve._request_extra_body` - Kind: function - Signature: `def _request_extra_body(request: dict) -> dict` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/bench_serve.py#L162-L171 - Implementation: Function `_request_extra_body` calls `request.items`; returns `{key: value for key, value in request.items() if key not in reserved}`. Function `_request_extra_body` calls `request.items`; returns `{key: value for key, value in request.items() if key not in reserved}`. - Inputs: - `request` (dict; required): Required positional or keyword input. - Return annotation: `dict` - Calls: request.items - Return expressions: {key: value for key, value in request.items() if key not in reserved} ## `vllm_mlx.bench_serve._first_not_none` - Kind: function - Signature: `def _first_not_none(*values: Any) -> Any` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/bench_serve.py#L174-L178 - Implementation: Function `_first_not_none` has 2 explicit return paths. Function `_first_not_none` has 2 explicit return paths. - Inputs: - `*values` (Any; optional): Additional variadic positional inputs accepted by this callable. - Return annotation: `Any` - Return expressions: value; None ## `vllm_mlx.bench_serve._normalize_tags` - Kind: function - Signature: `def _normalize_tags(tags: Any, *, case_id: str) -> tuple[str, ...]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/bench_serve.py#L181-L191 - Implementation: Function `_normalize_tags` calls `isinstance`, `ValueError`, `tuple`, `str`; can raise `ValueError`; returns `tuple((str(tag) for tag in tags))`. Coerce a workload case's ``tags`` field to a tuple of strings. Accepts either a single string (treated as a one-element list) or a list. Any other type is rejected with a case-scoped ``ValueError``. - Inputs: - `tags` (Any; required): Required positional or keyword input. - `case_id` (str; required): Required keyword-only input. - Return annotation: `tuple[str, ...]` - Calls: isinstance, ValueError, tuple, str - Raises directly: ValueError - Return expressions: tuple((str(tag) for tag in tags)) ## `vllm_mlx.bench_serve._merge_case_checks` - Kind: function - Signature: `def _merge_case_checks(default_checks: Any, case_checks: Any, *, case_id: str) -> Optional[dict]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/bench_serve.py#L194-L227 - Implementation: Function `_merge_case_checks` calls `dict`, `isinstance`, `ValueError`, `case_checks.items`; can raise `ValueError`; returns `merged or None`. Merge a case's ``checks`` over the workload defaults. Most keys are overridden by the case-level value. The two regex list keys (``required_regex``, ``forbidden_regex``) are list-concatenated with default patterns first and case patterns appended, so case-level patterns extend defaults rather than replace them. Returns ``None`` when neither source contributes any checks. Rejects a non-dict ``case_checks`` with a ``ValueError`` named after the case so the operator gets a clear message instead of the ``AttributeError`` that the previous inline code raised on ``case_checks.items()``. - Inputs: - `default_checks` (Any; required): Required positional or keyword input. - `case_checks` (Any; required): Required positional or keyword input. - `case_id` (str; required): Required keyword-only input. - Return annotation: `Optional[dict]` - Calls: dict, isinstance, ValueError, case_checks.items, merged.get - Raises directly: ValueError - Return expressions: merged or None ## `vllm_mlx.bench_serve._build_workload_case` - Kind: function - Signature: `def _build_workload_case(item: Any, idx: int, *, defaults: dict, workload_path: Path) -> WorkloadCase` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/bench_serve.py#L230-L300 - Implementation: Function `_build_workload_case` calls `isinstance`, `ValueError`, `str`, `item.get`; can raise `ValueError`; returns `WorkloadCase(case_id=case_id, messages=messages, request_path=str(request_path) if request_path is not None else None, …`. Construct one ``WorkloadCase`` from a raw workload entry. Validates the entry shape, loads request defaults from a sibling JSON file when ``request_path`` is provided, merges ``extra_body`` and ``checks`` against the workload defaults, and resolves scalar fields (``max_tokens``, ``enable_thinking``, ``policy_timeout_ms``) via ``_first_not_none`` priority: case-level beats request_path defaults beats workload defaults. ``extra_body`` follows a different merge: it composes the ``request_path`` extras (base) with either the case-level ``extra_body`` if present, otherwise the workload-default ``extra_body``. The case-vs-default fallback is a get-with-default, not a three-way merge. - Inputs: - `item` (Any; required): Required positional or keyword input. - `idx` (int; required): Required positional or keyword input. - `defaults` (dict; required): Required keyword-only input. - `workload_path` (Path; required): Required keyword-only input. - Return annotation: `WorkloadCase` - Calls: isinstance, ValueError, str, item.get, _load_case_request, _require_message_list, request_defaults.get, defaults.get, _request_extra_body, request_extra.update, _merge_case_checks, WorkloadCase, _first_not_none, _normalize_tags - Raises directly: ValueError - Return expressions: WorkloadCase(case_id=case_id, messages=messages, request_path=str(request_path) if request_path is not None else None, … ## `vllm_mlx.bench_serve.load_workload` - Kind: function - Signature: `def load_workload(path: str | Path) -> Workload` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/bench_serve.py#L303-L335 - Implementation: Function `load_workload` calls `Path(path).expanduser`, `Path`, `workload_path.open`, `json.load`; can raise `ValueError`; returns `Workload(name=str(raw.get('name') or workload_path.stem), description=str(raw.get('description') or ''), defaults=defau…`. Load a declarative serving benchmark workload. Workloads are for product-like qualification where each case can carry request settings, comparison-only policy timeouts, and quality checks. Timeout fields are metadata unless the runner explicitly uses them as a transport limit; they are not treated as hardware capability claims. - Inputs: - `path` (str | Path; required): Required positional or keyword input. - Return annotation: `Workload` - Calls: Path(path).expanduser, Path, workload_path.open, json.load, isinstance, ValueError, raw.get, _build_workload_case, enumerate, Workload, str - Raises directly: ValueError - Return expressions: Workload(name=str(raw.get('name') or workload_path.stem), description=str(raw.get('description') or ''), defaults=defau… ## `vllm_mlx.bench_serve.BenchServeResult` - Kind: class - Signature: `class BenchServeResult` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/bench_serve.py#L344-L400 - Implementation: Class `BenchServeResult` declares 0 direct member(s). Aggregated results from a single bench-serve run configuration. - Inputs: - `run_id` (str; optional; default `''`): Optional constructor field; defaults to `''`. - `timestamp` (str; optional; default `''`): Optional constructor field; defaults to `''`. - `tag` (str; optional; default `''`): Optional constructor field; defaults to `''`. - `chip` (str; optional; default `''`): Optional constructor field; defaults to `''`. - `gpu_cores` (int; optional; default `0`): Optional constructor field; defaults to `0`. - `memory_gb` (float; optional; default `0.0`): Optional constructor field; defaults to `0.0`. - `bandwidth_gbs` (float; optional; default `0.0`): Optional constructor field; defaults to `0.0`. - `os_version` (str; optional; default `''`): Optional constructor field; defaults to `''`. - `model_id` (str; optional; default `''`): Optional constructor field; defaults to `''`. - `model_type` (str; optional; default `''`): Optional constructor field; defaults to `''`. - `engine_type` (str; optional; default `''`): Optional constructor field; defaults to `''`. - `mtp_enabled` (bool; optional; default `False`): Optional constructor field; defaults to `False`. - `specprefill` (bool; optional; default `False`): Optional constructor field; defaults to `False`. - `kv_quant` (str; optional; default `''`): Optional constructor field; defaults to `''`. - `cache_type` (str; optional; default `''`): Optional constructor field; defaults to `''`. - `prompt_set` (str; optional; default `''`): Optional constructor field; defaults to `''`. - `concurrency` (int; optional; default `1`): Optional constructor field; defaults to `1`. - `max_tokens` (int; optional; default `256`): Optional constructor field; defaults to `256`. - `enable_thinking` (Optional[bool]; optional; default `None`): Optional constructor field; defaults to `None`. - `extra_body` (str; optional; default `''`): Optional constructor field; defaults to `''`. - `repetition` (int; optional; default `0`): Optional constructor field; defaults to `0`. - `prompt_tokens` (int; optional; default `0`): Optional constructor field; defaults to `0`. - `ttft_ms` (float; optional; default `0.0`): Optional constructor field; defaults to `0.0`. - `tpot_ms` (float; optional; default `0.0`): Optional constructor field; defaults to `0.0`. - `e2e_latency_ms` (float; optional; default `0.0`): Optional constructor field; defaults to `0.0`. - `gen_tps` (float; optional; default `0.0`): Optional constructor field; defaults to `0.0`. - `prompt_tps` (float; optional; default `0.0`): Optional constructor field; defaults to `0.0`. - `throughput_tps` (float; optional; default `0.0`): Optional constructor field; defaults to `0.0`. - `requests_per_s` (float; optional; default `0.0`): Optional constructor field; defaults to `0.0`. - `metal_active_gb` (float; optional; default `0.0`): Optional constructor field; defaults to `0.0`. - `metal_peak_gb` (float; optional; default `0.0`): Optional constructor field; defaults to `0.0`. - `metal_cache_gb` (float; optional; default `0.0`): Optional constructor field; defaults to `0.0`. - `cache_hits` (int; optional; default `0`): Optional constructor field; defaults to `0`. - `cache_misses` (int; optional; default `0`): Optional constructor field; defaults to `0`. - `cache_hit_rate` (float; optional; default `0.0`): Optional constructor field; defaults to `0.0`. - `tokens_saved` (int; optional; default `0`): Optional constructor field; defaults to `0`. - `validated` (bool; optional; default `True`): Optional constructor field; defaults to `True`. - Constructs: `vllm_mlx.bench_serve.BenchServeResult` - Decorators: dataclass ## `vllm_mlx.bench_serve.expand_sweep` - Kind: function - Signature: `def expand_sweep(prompt_sets: list[str], concurrencies: list[int], thinking_values: list[Optional[bool]], extra_bodies: list[str], repetitions: int) -> list[SweepConfig]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/bench_serve.py#L411-L444 - Implementation: Function `expand_sweep` calls `itertools.product`, `range`, `configs.append`; returns `configs`. Expand sweep parameters into a flat list of configurations. Performs the full Cartesian product of all input dimensions and then unfolds each combination across ``repetitions`` repetition indices (0-based). Args: prompt_sets: Names or paths of prompt sets to include. concurrencies: Concurrency levels to test (e.g. ``[1, 4, 16]``). thinking_values: Values for ``enable_thinking`` (e.g. ``[None, True, False]``). extra_bodies: JSON strings (or empty string) to pass as extra body parameters on each request. repetitions: Number of times to repeat each unique combination. Each repeat gets a distinct 0-based repetition index. Returns: A list of :data:`SweepConfig` tuples in the order:: (prompt_set, concurrency, thinking, extra_body, repetition_index) - Inputs: - `prompt_sets` (list[str]; required): Names or paths of prompt sets to include. - `concurrencies` (list[int]; required): Concurrency levels to test (e.g. ``[1, 4, 16]``). - `thinking_values` (list[Optional[bool]]; required): Values for ``enable_thinking`` (e.g. ``[None, True, False]``). - `extra_bodies` (list[str]; required): JSON strings (or empty string) to pass as extra body parameters on each request. - `repetitions` (int; required): Number of times to repeat each unique combination. Each repeat gets a distinct 0-based repetition index. - Return annotation: `list[SweepConfig]` - Calls: itertools.product, range, configs.append - Return expressions: configs ## `vllm_mlx.bench_serve.parse_health_response` - Kind: function - Signature: `def parse_health_response(data: dict) -> dict` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/bench_serve.py#L452-L467 - Implementation: Function `parse_health_response` calls `data.get`; returns `{'model_name': data.get('model_name', ''), 'model_type': data.get('model_type', '')}`. Extract model identity fields from a GET /health response. Args: data: Parsed JSON body from the /health endpoint. Expected shape:: {"status": "healthy", "model_loaded": True, "model_name": "...", "model_type": "llm"|"mllm"} Returns: ``{"model_name": str, "model_type": str}`` - Inputs: - `data` (dict; required): Parsed JSON body from the /health endpoint. Expected shape:: {"status": "healthy", "model_loaded": True, "model_name": "...", "model_type": "llm"|"mllm"} - Return annotation: `dict` - Calls: data.get - Return expressions: {'model_name': data.get('model_name', ''), 'model_type': data.get('model_type', '')} ## `vllm_mlx.bench_serve.parse_status_response` - Kind: function - Signature: `def parse_status_response(data: dict) -> dict` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/bench_serve.py#L470-L496 - Implementation: Function `parse_status_response` calls `data.get`, `float`, `metal.get`, `cache.get`; returns `{'model': data.get('model', ''), 'metal_active_gb': float(metal.get('active_memory_gb') or metal.get('active_gb') or 0.…`. Extract metal and cache info from a GET /v1/status response. Args: data: Parsed JSON body from the /v1/status endpoint. Metal info is expected under ``data["metal"]`` and cache info under ``data["cache"]``. Missing keys are handled gracefully. Returns: ``{"model": str, "metal_active_gb": float, "metal_peak_gb": float, "metal_cache_gb": float, "cache_type": str}`` - Inputs: - `data` (dict; required): Parsed JSON body from the /v1/status endpoint. Metal info is expected under ``data["metal"]`` and cache info under ``data["cache"]``. Missing keys are handled gracefully. - Return annotation: `dict` - Calls: data.get, float, metal.get, cache.get - Return expressions: {'model': data.get('model', ''), 'metal_active_gb': float(metal.get('active_memory_gb') or metal.get('active_gb') or 0.… ## `vllm_mlx.bench_serve.parse_metrics_text` - Kind: function - Signature: `def parse_metrics_text(text: str) -> dict` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/bench_serve.py#L499-L521 - Implementation: Function `parse_metrics_text` calls `_extract`; returns `{'cache_hits': _extract('vllm_prefix_cache_hits_total'), 'cache_misses': _extract('vllm_prefix_cache_misses_total'), 't…`. Parse Prometheus text exposition format from GET /metrics. Extracts the three prefix-cache counters used for bench reporting. Args: text: Raw response body from the /metrics endpoint. Returns: ``{"cache_hits": int, "cache_misses": int, "tokens_saved": int}`` — each value defaults to ``0`` when the metric line is absent. - Inputs: - `text` (str; required): Raw response body from the /metrics endpoint. - Return annotation: `dict` - Calls: _extract - Return expressions: {'cache_hits': _extract('vllm_prefix_cache_hits_total'), 'cache_misses': _extract('vllm_prefix_cache_misses_total'), 't… ## `vllm_mlx.bench_serve.parse_metrics_text._extract` - Kind: nested function - Signature: `def _extract(metric_name: str) -> int` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/bench_serve.py#L512-L515 - Implementation: Nested Function `parse_metrics_text._extract` calls `re.escape`, `re.search`, `int`, `m.group`; returns `int(m.group(1)) if m else 0`. Nested Function `parse_metrics_text._extract` calls `re.escape`, `re.search`, `int`, `m.group`; returns `int(m.group(1)) if m else 0`. - Inputs: - `metric_name` (str; required): Required positional or keyword input. - Return annotation: `int` - Calls: re.escape, re.search, int, m.group - Return expressions: int(m.group(1)) if m else 0 ## `vllm_mlx.bench_serve.detect_hardware_fingerprint` - Kind: function - Signature: `def detect_hardware_fingerprint() -> dict` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/bench_serve.py#L524-L573 - Implementation: Function `detect_hardware_fingerprint` calls `platform.platform`, `detect_hardware`, `subprocess.run`, `int`; has 2 explicit return paths. Return a hardware fingerprint dict for the current machine. Tries to import :func:`vllm_mlx.optimizations.detect_hardware` (which requires MLX). Falls back to reading ``hw.memsize`` via ``sysctl`` when MLX is unavailable. ``os_version`` is always obtained from :func:`platform.platform`. Returns: ``{"chip": str, "gpu_cores": int, "memory_gb": float, "bandwidth_gbs": float, "os_version": str}`` - Inputs: none - Return annotation: `dict` - Calls: platform.platform, detect_hardware, subprocess.run, int, result.stdout.strip - Return expressions: {'chip': hw.chip_name, 'gpu_cores': hw.gpu_cores, 'memory_gb': hw.total_memory_gb, 'bandwidth_gbs': hw.memory_bandwidth…; {'chip': '', 'gpu_cores': 0, 'memory_gb': memory_gb, 'bandwidth_gbs': 0.0, 'os_version': os_version} ## `vllm_mlx.bench_serve.auto_detect_runtime` - Kind: function - Signature: `async def auto_detect_runtime(client: httpx.AsyncClient, base_url: str) -> dict` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/bench_serve.py#L576-L642 - Implementation: Function `auto_detect_runtime` calls `client.get`, `resp.raise_for_status`, `parse_health_response`, `resp.json`; awaits asynchronous work; returns `result`. Query the running server and return a runtime descriptor dict. Hits ``/health``, ``/v1/models``, and ``/v1/status`` in sequence. Each call is wrapped in an :exc:`httpx.HTTPError` guard so a missing endpoint does not abort the whole detection. Args: client: An open :class:`httpx.AsyncClient`. base_url: Base URL of the server (e.g. ``"http://localhost:8080"``). Returns: Dict with keys: ``model_id``, ``model_type``, ``engine_type``, ``mtp_enabled``, ``specprefill``, ``kv_quant``, ``cache_type``, ``metal_active_gb``, ``metal_peak_gb``, ``metal_cache_gb``. - Inputs: - `client` (httpx.AsyncClient; required): An open :class:`httpx.AsyncClient`. - `base_url` (str; required): Base URL of the server (e.g. ``"http://localhost:8080"``). - Return annotation: `dict` - Calls: client.get, resp.raise_for_status, parse_health_response, resp.json, health.get, models_data.get, models[0].get, parse_status_response, status.get, raw.get, bool - Return expressions: result ## `vllm_mlx.bench_serve.scrape_metrics` - Kind: function - Signature: `async def scrape_metrics(client: httpx.AsyncClient, base_url: str) -> dict` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/bench_serve.py#L645-L661 - Implementation: Function `scrape_metrics` calls `client.get`, `resp.raise_for_status`, `parse_metrics_text`; awaits asynchronous work; has 2 explicit return paths. Scrape Prometheus metrics from the server. Args: client: An open :class:`httpx.AsyncClient`. base_url: Base URL of the server. Returns: Parsed metrics dict (see :func:`parse_metrics_text`), or an empty dict if the endpoint is unreachable. - Inputs: - `client` (httpx.AsyncClient; required): An open :class:`httpx.AsyncClient`. - `base_url` (str; required): Base URL of the server. - Return annotation: `dict` - Calls: client.get, resp.raise_for_status, parse_metrics_text - Return expressions: parse_metrics_text(resp.text); {} ## `vllm_mlx.bench_serve.clear_runtime_cache` - Kind: function - Signature: `async def clear_runtime_cache(client: httpx.AsyncClient, base_url: str) -> dict` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/bench_serve.py#L664-L684 - Implementation: Function `clear_runtime_cache` calls `client.delete`, `resp.json`, `resp.raise_for_status`, `str`; awaits asynchronous work; returns `event`. Clear server-side runtime caches and return a JSON-serializable event. - Inputs: - `client` (httpx.AsyncClient; required): Required positional or keyword input. - `base_url` (str; required): Required positional or keyword input. - Return annotation: `dict` - Calls: client.delete, resp.json, resp.raise_for_status, str - Return expressions: event ## `vllm_mlx.bench_serve._normalize_cache_policy` - Kind: function - Signature: `def _normalize_cache_policy(value: Optional[str]) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/bench_serve.py#L687-L698 - Implementation: Function `_normalize_cache_policy` calls `(value or 'preserve').strip().lower().replace`, `(value or 'preserve').strip().lower`, `(value or 'preserve').strip`, `ValueError`; can raise `ValueError`; returns `policy`. Normalize cache-policy spelling from CLI or workload JSON. CLI choices are hyphenated, but workload JSON may use underscores when it follows common Python/YAML identifier style. - Inputs: - `value` (Optional[str]; required): Required positional or keyword input. - Return annotation: `str` - Calls: (value or 'preserve').strip().lower().replace, (value or 'preserve').strip().lower, (value or 'preserve').strip, ValueError - Raises directly: ValueError - Return expressions: policy ## `vllm_mlx.bench_serve.parse_sse_line` - Kind: function - Signature: `def parse_sse_line(line: str) -> Optional[dict]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/bench_serve.py#L706-L754 - Implementation: Function `parse_sse_line` calls `line.strip`, `line.startswith`, `len`, `json.loads`; has 2 explicit return paths. Parse one Server-Sent Events line from a streaming chat completion. Args: line: A single raw line from the SSE stream (may or may not include a trailing newline — it is stripped before processing). Returns: ``None`` for blank lines, comment lines (starting with ``:``) and the ``data: [DONE]`` sentinel. For all other ``data:`` lines the JSON is parsed and a dict is returned:: {"id": Optional[str], "content": str, "finish_reason": Optional[str], "usage": Optional[dict], "tool_calls_delta": Optional[list]} Missing keys (``choices``, ``delta``, ``content``) are handled gracefully and default to empty string / ``None``. - Inputs: - `line` (str; required): A single raw line from the SSE stream (may or may not include a trailing newline — it is stripped before processing). - Return annotation: `Optional[dict]` - Calls: line.strip, line.startswith, len, json.loads, chunk.get, choices[0].get, delta.get - Return expressions: None; {'id': chunk.get('id'), 'content': content, 'finish_reason': finish_reason, 'usage': usage, 'tool_calls_delta': tool_ca… ## `vllm_mlx.bench_serve._cancel_server_request` - Kind: function - Signature: `async def _cancel_server_request(client: httpx.AsyncClient, base_url: str, request_id: Optional[str]) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/bench_serve.py#L757-L770 - Implementation: Function `_cancel_server_request` calls `client.post`; awaits asynchronous work; returns `None`. Best-effort server-side cancellation for timed-out workload streams. - Inputs: - `client` (httpx.AsyncClient; required): Required positional or keyword input. - `base_url` (str; required): Required positional or keyword input. - `request_id` (Optional[str]; required): Required positional or keyword input. - Return annotation: `None` - Calls: client.post - Return expressions: None ## `vllm_mlx.bench_serve.accumulate_tool_calls` - Kind: function - Signature: `def accumulate_tool_calls(acc: dict[int, dict], delta_list: list[dict]) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/bench_serve.py#L773-L792 - Implementation: Function `accumulate_tool_calls` calls `int`, `tc_delta.get`, `function_delta.get`. Merge streamed OpenAI tool-call deltas into *acc* by index. - Inputs: - `acc` (dict[int, dict]; required): Required positional or keyword input. - `delta_list` (list[dict]; required): Required positional or keyword input. - Return annotation: `None` - Calls: int, tc_delta.get, function_delta.get ## `vllm_mlx.bench_serve.finalize_tool_calls` - Kind: function - Signature: `def finalize_tool_calls(acc: dict[int, dict]) -> list[dict]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/bench_serve.py#L795-L797 - Implementation: Function `finalize_tool_calls` calls `sorted`; returns `[acc[idx] for idx in sorted(acc)]`. Return accumulated tool calls in stream index order. - Inputs: - `acc` (dict[int, dict]; required): Required positional or keyword input. - Return annotation: `list[dict]` - Calls: sorted - Return expressions: [acc[idx] for idx in sorted(acc)] ## `vllm_mlx.bench_serve.compute_request_metrics` - Kind: function - Signature: `def compute_request_metrics(t_start: float, t_first_token: float, token_times: list, t_end: float, prompt_tokens: int, completion_tokens: int) -> dict` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/bench_serve.py#L800-L852 - Implementation: Function `compute_request_metrics` calls `len`, `range`, `statistics.mean`; returns `{'ttft_ms': ttft_ms, 'tpot_ms': tpot_ms, 'e2e_latency_ms': e2e_latency_ms, 'gen_tps': gen_tps, 'prompt_tps': prompt_tps}`. Compute standard latency and throughput metrics for a single request. All time arguments are :func:`time.perf_counter` values (seconds as floats). Args: t_start: Timestamp immediately before the request was sent. t_first_token: Timestamp when the first content token was received. token_times: List of timestamps, one per content token (including the first). When there is only one token ``tpot_ms`` is ``0.0``. t_end: Timestamp after the final SSE chunk was consumed. prompt_tokens: Number of prompt tokens reported by the server. completion_tokens: Number of completion tokens generated. Returns: Dict with keys ``ttft_ms``, ``tpot_ms``, ``e2e_latency_ms``, ``gen_tps``, ``prompt_tps`` — all floats. - Inputs: - `t_start` (float; required): Timestamp immediately before the request was sent. - `t_first_token` (float; required): Timestamp when the first content token was received. - `token_times` (list; required): List of timestamps, one per content token (including the first). When there is only one token ``tpot_ms`` is ``0.0``. - `t_end` (float; required): Timestamp after the final SSE chunk was consumed. - `prompt_tokens` (int; required): Number of prompt tokens reported by the server. - `completion_tokens` (int; required): Number of completion tokens generated. - Return annotation: `dict` - Calls: len, range, statistics.mean - Return expressions: {'ttft_ms': ttft_ms, 'tpot_ms': tpot_ms, 'e2e_latency_ms': e2e_latency_ms, 'gen_tps': gen_tps, 'prompt_tps': prompt_tps} ## `vllm_mlx.bench_serve.count_prompt_tokens` - Kind: function - Signature: `async def count_prompt_tokens(client: httpx.AsyncClient, base_url: str, messages: list[dict], model: str) -> int` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/bench_serve.py#L855-L889 - Implementation: Function `count_prompt_tokens` calls `client.post`, `resp.raise_for_status`, `resp.json`, `int`; awaits asynchronous work; has 2 explicit return paths. Count prompt tokens for a message list by sending a 1-token request. Sends a non-streaming chat completion with ``max_tokens=1`` and reads ``usage.prompt_tokens`` from the response. Args: client: An open :class:`httpx.AsyncClient`. base_url: Base URL of the server. messages: The message list to send. model: Model ID to target. Returns: Number of prompt tokens, or ``0`` on error. - Inputs: - `client` (httpx.AsyncClient; required): An open :class:`httpx.AsyncClient`. - `base_url` (str; required): Base URL of the server. - `messages` (list[dict]; required): The message list to send. - `model` (str; required): Model ID to target. - Return annotation: `int` - Calls: client.post, resp.raise_for_status, resp.json, int, (data.get('usage') or {}).get, data.get - Return expressions: int((data.get('usage') or {}).get('prompt_tokens', 0)); 0 ## `vllm_mlx.bench_serve.stream_chat_completion` - Kind: function - Signature: `async def stream_chat_completion(client: httpx.AsyncClient, base_url: str, messages: list[dict], model: str, max_tokens: int=256, enable_thinking: Optional[bool]=None, extra_body: Optional[dict]=None, timeout_s: Optional[float]=None) -> dict` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/bench_serve.py#L892-L1012 - Implementation: Function `stream_chat_completion` calls `body.update`, `time.perf_counter`, `asyncio.timeout`, `_consume_stream`; awaits asynchronous work; can raise `TimeoutError`; returns `{**metrics, 'completion_tokens': completion_tokens, 'prompt_tokens': prompt_tokens, 'finish_reason': finish_reason, 'co…`. Send a streaming chat completion and collect per-token timing data. Tracks TTFT, per-token timestamps, accumulated content, finish reason, and usage (via ``stream_options: {"include_usage": True}``). Args: client: An open :class:`httpx.AsyncClient`. base_url: Base URL of the server. messages: The message list to send. model: Model ID to target. max_tokens: Maximum tokens to generate (default ``256``). enable_thinking: If not ``None``, passed as ``enable_thinking`` in the request body. extra_body: Optional extra keys merged into the request body. timeout_s: Optional case-level timeout. When set, the stream is closed and best-effort server cancellation is attempted before raising :class:`TimeoutError`. Returns: Dict with all :func:`compute_request_metrics` fields plus ``completion_tokens``, ``prompt_tokens``, ``finish_reason``, ``content``. - Inputs: - `client` (httpx.AsyncClient; required): An open :class:`httpx.AsyncClient`. - `base_url` (str; required): Base URL of the server. - `messages` (list[dict]; required): The message list to send. - `model` (str; required): Model ID to target. - `max_tokens` (int; optional; default `256`): Maximum tokens to generate (default ``256``). - `enable_thinking` (Optional[bool]; optional; default `None`): If not ``None``, passed as ``enable_thinking`` in the request body. - `extra_body` (Optional[dict]; optional; default `None`): Optional extra keys merged into the request body. - `timeout_s` (Optional[float]; optional; default `None`): Optional case-level timeout. When set, the stream is closed and best-effort server cancellation is attempted before raising :class:`TimeoutError`. - Return annotation: `dict` - Calls: body.update, time.perf_counter, asyncio.timeout, _consume_stream, _cancel_server_request, TimeoutError, int, (usage or {}).get, compute_request_metrics, ''.join, finalize_tool_calls - Raises directly: TimeoutError - Return expressions: {**metrics, 'completion_tokens': completion_tokens, 'prompt_tokens': prompt_tokens, 'finish_reason': finish_reason, 'co… ## `vllm_mlx.bench_serve.stream_chat_completion._consume_stream` - Kind: nested function - Signature: `async def _consume_stream() -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/bench_serve.py#L946-L975 - Implementation: Nested Function `stream_chat_completion._consume_stream` calls `client.stream`, `response.raise_for_status`, `response.aiter_lines`, `parse_sse_line`. Nested Function `stream_chat_completion._consume_stream` calls `client.stream`, `response.raise_for_status`, `response.aiter_lines`, `parse_sse_line`. - Inputs: none - Return annotation: `None` - Calls: client.stream, response.raise_for_status, response.aiter_lines, parse_sse_line, parsed.get, time.perf_counter, token_times.append, accumulate_tool_calls, content_parts.append ## `vllm_mlx.bench_serve.validate_response` - Kind: function - Signature: `def validate_response(finish_reason: Optional[str], content: str, status_code: int, *, tool_calls: Optional[list[dict]]=None) -> tuple[bool, str]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/bench_serve.py#L1020-L1049 - Implementation: Function `validate_response` has 5 explicit return paths. Validate a single streaming response result. Args: finish_reason: The ``finish_reason`` from the final SSE chunk, or ``None`` if not received. content: The accumulated text content of the response. status_code: The HTTP status code of the response (use ``200`` for successful streaming requests). Returns: ``(is_valid, message)`` — ``is_valid`` is ``True`` when the response passes all checks; ``message`` is an empty string on success or a human-readable description of the first failure. - Inputs: - `finish_reason` (Optional[str]; required): The ``finish_reason`` from the final SSE chunk, or ``None`` if not received. - `content` (str; required): The accumulated text content of the response. - `status_code` (int; required): The HTTP status code of the response (use ``200`` for successful streaming requests). - `tool_calls` (Optional[list[dict]]; optional; default `None`): Optional keyword-only input; defaults to `None`. - Return annotation: `tuple[bool, str]` - Return expressions: (False, f'HTTP error {status_code}'); (False, 'Missing finish_reason'); (False, 'Truncated (finish_reason=length)'); (False, 'Empty response content'); (True, '') ## `vllm_mlx.bench_serve._check_finish_reason` - Kind: function - Signature: `def _check_finish_reason(allowed: Any, finish_reason: Optional[str]) -> list[str]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/bench_serve.py#L1052-L1059 - Implementation: Function `_check_finish_reason` calls `isinstance`, `list`; has 2 explicit return paths. Verify ``finish_reason`` is in the allowed set, if one is configured. - Inputs: - `allowed` (Any; required): Required positional or keyword input. - `finish_reason` (Optional[str]; required): Required positional or keyword input. - Return annotation: `list[str]` - Calls: isinstance, list - Return expressions: []; [f'finish_reason {finish_reason!r} not in allowed set {allowed_list!r}'] ## `vllm_mlx.bench_serve._check_length_bounds` - Kind: function - Signature: `def _check_length_bounds(min_chars: Any, max_chars: Any, content: str) -> list[str]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/bench_serve.py#L1062-L1069 - Implementation: Function `_check_length_bounds` calls `len`, `int`, `issues.append`; returns `issues`. Apply ``min_chars`` / ``max_chars`` content-length bounds. - Inputs: - `min_chars` (Any; required): Required positional or keyword input. - `max_chars` (Any; required): Required positional or keyword input. - `content` (str; required): Required positional or keyword input. - Return annotation: `list[str]` - Calls: len, int, issues.append - Return expressions: issues ## `vllm_mlx.bench_serve._check_regex_patterns` - Kind: function - Signature: `def _check_regex_patterns(patterns: Any, content: str, *, kind: str, expect_match: bool) -> list[str]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/bench_serve.py#L1072-L1096 - Implementation: Function `_check_regex_patterns` calls `bool`, `re.search`, `str`, `issues.append`; returns `issues`. Validate that each pattern either matches or does not, per ``expect_match``. ``kind`` is the diagnostic name (``"required_regex"`` or ``"forbidden_regex"``) and is reused across the resulting issue strings so operators can grep for the failing check. - Inputs: - `patterns` (Any; required): Required positional or keyword input. - `content` (str; required): Required positional or keyword input. - `kind` (str; required): Required keyword-only input. - `expect_match` (bool; required): Required keyword-only input. - Return annotation: `list[str]` - Calls: bool, re.search, str, issues.append - Return expressions: issues ## `vllm_mlx.bench_serve._check_json_content` - Kind: function - Signature: `def _check_json_content(should_be_json: Any, content: str) -> list[str]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/bench_serve.py#L1099-L1107 - Implementation: Function `_check_json_content` calls `json.loads`; has 2 explicit return paths. Verify ``content`` parses as JSON when ``checks['json']`` is truthy. - Inputs: - `should_be_json` (Any; required): Required positional or keyword input. - `content` (str; required): Required positional or keyword input. - Return annotation: `list[str]` - Calls: json.loads - Return expressions: []; [f'content is not valid JSON: {exc}'] ## `vllm_mlx.bench_serve._check_tool_call_count_and_names` - Kind: function - Signature: `def _check_tool_call_count_and_names(checks: dict, tool_calls: list[dict]) -> list[str]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/bench_serve.py#L1110-L1132 - Implementation: Function `_check_tool_call_count_and_names` calls `checks.get`, `issues.append`, `len`, `int`; returns `issues`. Apply ``no_tool_calls`` / ``tool_call_count`` / ``tool_call_names``. - Inputs: - `checks` (dict; required): Required positional or keyword input. - `tool_calls` (list[dict]; required): Required positional or keyword input. - Return annotation: `list[str]` - Calls: checks.get, issues.append, len, int, sorted, tc.get('function', {}).get, tc.get, str - Return expressions: issues ## `vllm_mlx.bench_serve._check_tool_call_args` - Kind: function - Signature: `def _check_tool_call_args(required_args: Any, tool_calls: list[dict]) -> list[str]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/bench_serve.py#L1135-L1174 - Implementation: Function `_check_tool_call_args` calls `tc.get('function', {}).get`, `tc.get`, `by_name.setdefault(name, []).append`, `by_name.setdefault`; has 2 explicit return paths. Validate parsed JSON arguments include the required keys per function. For each named function, looks up matching tool calls, parses their ``arguments`` as JSON, and reports issues for: missing tool call, invalid JSON, non-object arguments, or missing required keys. - Inputs: - `required_args` (Any; required): Required positional or keyword input. - `tool_calls` (list[dict]; required): Required positional or keyword input. - Return annotation: `list[str]` - Calls: tc.get('function', {}).get, tc.get, by_name.setdefault(name, []).append, by_name.setdefault, required_args.items, by_name.get, str, issues.append, json.loads, isinstance - Return expressions: []; issues ## `vllm_mlx.bench_serve.validate_quality_checks` - Kind: function - Signature: `def validate_quality_checks(finish_reason: Optional[str], content: str, checks: Optional[dict], *, status_code: int=200, tool_calls: Optional[list[dict]]=None) -> tuple[bool, list[str]]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/bench_serve.py#L1177-L1231 - Implementation: Function `validate_quality_checks` calls `validate_response`, `issues.extend`, `_check_finish_reason`, `checks.get`; returns `(not issues, issues)`. Validate content against generic workload quality checks. Supported checks: - ``finish_reason``: string or list of allowed finish reasons - ``required_regex``: list of regex patterns that must match - ``forbidden_regex``: list of regex patterns that must not match - ``min_chars`` / ``max_chars``: length bounds - ``json``: when true, content must parse as JSON - ``tool_call_count``: exact number of streamed tool calls - ``tool_call_names``: expected function names, order-independent - ``tool_call_args_required_keys``: required JSON argument keys by function - ``no_tool_calls``: assert that no tool calls were emitted - Inputs: - `finish_reason` (Optional[str]; required): Required positional or keyword input. - `content` (str; required): Required positional or keyword input. - `checks` (Optional[dict]; required): Required positional or keyword input. - `status_code` (int; optional; default `200`): Optional keyword-only input; defaults to `200`. - `tool_calls` (Optional[list[dict]]; optional; default `None`): Optional keyword-only input; defaults to `None`. - Return annotation: `tuple[bool, list[str]]` - Calls: validate_response, issues.extend, _check_finish_reason, checks.get, _check_length_bounds, _check_regex_patterns, _check_json_content, _check_tool_call_count_and_names, _check_tool_call_args - Return expressions: (not issues, issues) ## `vllm_mlx.bench_serve.compute_summary_stats` - Kind: function - Signature: `def compute_summary_stats(values: list[float]) -> dict` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/bench_serve.py#L1234-L1276 - Implementation: Function `compute_summary_stats` calls `ValueError`, `len`, `statistics.mean`, `statistics.stdev`; can raise `ValueError`; returns `{'mean': mean, 'stddev': stddev, 'min': sorted_vals[0], 'max': sorted_vals[-1], 'p50': _percentile(50), 'p95': _percent…`. Compute summary statistics over a list of floats. Args: values: Non-empty list of floats to summarise. Returns: Dict with keys ``mean``, ``stddev``, ``min``, ``max``, ``p50``, ``p95``, ``p99``. Percentiles use linear interpolation on sorted values. Raises: ValueError: If ``values`` is empty. - Inputs: - `values` (list[float]; required): Non-empty list of floats to summarise. - Return annotation: `dict` - Calls: ValueError, len, statistics.mean, statistics.stdev, sorted, _percentile - Raises directly: ValueError - Return expressions: {'mean': mean, 'stddev': stddev, 'min': sorted_vals[0], 'max': sorted_vals[-1], 'p50': _percentile(50), 'p95': _percent… ## `vllm_mlx.bench_serve.compute_summary_stats._percentile` - Kind: nested function - Signature: `def _percentile(p: float) -> float` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/bench_serve.py#L1256-L1266 - Implementation: Nested Function `compute_summary_stats._percentile` calls `int`; has 3 explicit return paths. Nested Function `compute_summary_stats._percentile` calls `int`; has 3 explicit return paths. - Inputs: - `p` (float; required): Required positional or keyword input. - Return annotation: `float` - Calls: int - Return expressions: sorted_vals[0]; sorted_vals[-1]; sorted_vals[lo] + frac * (sorted_vals[hi] - sorted_vals[lo]) ## `vllm_mlx.bench_serve.run_concurrent_requests` - Kind: function - Signature: `async def run_concurrent_requests(client: httpx.AsyncClient, base_url: str, prompts: list[list[dict]], model: str, concurrency: int, max_tokens: int=256, enable_thinking: Optional[bool]=None, extra_body: Optional[dict]=None, do_validate: bool=True) -> list[dict]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/bench_serve.py#L1279-L1342 - Implementation: Function `run_concurrent_requests` calls `itertools.cycle`, `next`, `range`, `asyncio.gather`; awaits asynchronous work; returns `list(results)`. Fire ``concurrency`` concurrent streaming requests and collect results. Prompts are selected round-robin from ``prompts``. All requests are launched simultaneously with :func:`asyncio.gather`. Exceptions are caught per-task and wrapped in an error dict rather than propagated. Args: client: An open :class:`httpx.AsyncClient`. base_url: Base URL of the server. prompts: List of message dicts to cycle through. model: Model ID to target. concurrency: Number of simultaneous requests to fire. max_tokens: Maximum tokens to generate per request (default ``256``). enable_thinking: Passed through to :func:`stream_chat_completion`. extra_body: Passed through to :func:`stream_chat_completion`. do_validate: When ``True``, call :func:`validate_response` on each result and add a ``"validated"`` key. Returns: List of result dicts (one per request). Each dict contains at minimum a ``"validated"`` key when ``do_validate`` is ``True``. - Inputs: - `client` (httpx.AsyncClient; required): An open :class:`httpx.AsyncClient`. - `base_url` (str; required): Base URL of the server. - `prompts` (list[list[dict]]; required): List of message dicts to cycle through. - `model` (str; required): Model ID to target. - `concurrency` (int; required): Number of simultaneous requests to fire. - `max_tokens` (int; optional; default `256`): Maximum tokens to generate per request (default ``256``). - `enable_thinking` (Optional[bool]; optional; default `None`): Passed through to :func:`stream_chat_completion`. - `extra_body` (Optional[dict]; optional; default `None`): Passed through to :func:`stream_chat_completion`. - `do_validate` (bool; optional; default `True`): When ``True``, call :func:`validate_response` on each result and add a ``"validated"`` key. - Return annotation: `list[dict]` - Calls: itertools.cycle, next, range, asyncio.gather, _single, list - Return expressions: list(results) ## `vllm_mlx.bench_serve.run_concurrent_requests._single` - Kind: nested function - Signature: `async def _single(messages: list[dict]) -> dict` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/bench_serve.py#L1315-L1339 - Implementation: Nested Function `run_concurrent_requests._single` calls `stream_chat_completion`, `validate_response`, `result.get`, `str`; awaits asynchronous work; has 2 explicit return paths. Nested Function `run_concurrent_requests._single` calls `stream_chat_completion`, `validate_response`, `result.get`, `str`; awaits asynchronous work; has 2 explicit return paths. - Inputs: - `messages` (list[dict]; required): Required positional or keyword input. - Return annotation: `dict` - Calls: stream_chat_completion, validate_response, result.get, str - Return expressions: result; err ## `vllm_mlx.bench_serve._summary_or_empty` - Kind: function - Signature: `def _summary_or_empty(values: list[float]) -> dict` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/bench_serve.py#L1345-L1346 - Implementation: Function `_summary_or_empty` calls `compute_summary_stats`; returns `compute_summary_stats(values) if values else {}`. Function `_summary_or_empty` calls `compute_summary_stats`; returns `compute_summary_stats(values) if values else {}`. - Inputs: - `values` (list[float]; required): Required positional or keyword input. - Return annotation: `dict` - Calls: compute_summary_stats - Return expressions: compute_summary_stats(values) if values else {} ## `vllm_mlx.bench_serve._resolve_max_tokens` - Kind: function - Signature: `def _resolve_max_tokens(case: WorkloadCase, workload: Workload) -> int` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/bench_serve.py#L1349-L1352 - Implementation: Function `_resolve_max_tokens` calls `int`, `workload.defaults.get`; returns `int(case.max_tokens or workload.defaults.get('max_tokens', 256))`. Return the effective ``max_tokens`` for a case, falling back to workload defaults and finally to 256. - Inputs: - `case` (WorkloadCase; required): Required positional or keyword input. - `workload` (Workload; required): Required positional or keyword input. - Return annotation: `int` - Calls: int, workload.defaults.get - Return expressions: int(case.max_tokens or workload.defaults.get('max_tokens', 256)) ## `vllm_mlx.bench_serve._assemble_case_request_kwargs` - Kind: function - Signature: `def _assemble_case_request_kwargs(case: WorkloadCase, workload: Workload, model: str) -> dict` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/bench_serve.py#L1355-L1372 - Implementation: Function `_assemble_case_request_kwargs` calls `_resolve_max_tokens`; returns `{'messages': case.messages, 'model': model, 'max_tokens': _resolve_max_tokens(case, workload), 'enable_thinking': case.…`. Build the keyword-arguments dict passed to ``stream_chat_completion`` for one case, applying max_tokens fallback and converting ``policy_timeout_ms`` to seconds. - Inputs: - `case` (WorkloadCase; required): Required positional or keyword input. - `workload` (Workload; required): Required positional or keyword input. - `model` (str; required): Required positional or keyword input. - Return annotation: `dict` - Calls: _resolve_max_tokens - Return expressions: {'messages': case.messages, 'model': model, 'max_tokens': _resolve_max_tokens(case, workload), 'enable_thinking': case.… ## `vllm_mlx.bench_serve._empty_completion_result` - Kind: function - Signature: `def _empty_completion_result() -> dict` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/bench_serve.py#L1375-L1390 - Implementation: Function `_empty_completion_result` returns `{'ttft_ms': 0.0, 'tpot_ms': 0.0, 'e2e_latency_ms': 0.0, 'gen_tps': 0.0, 'prompt_tps': 0.0, 'prompt_tokens': 0, 'complet…`. Zero-valued completion result used when ``stream_chat_completion`` raises. The structure matches a real successful response so downstream code can read ``result.get(...)`` without branching on the failure. - Inputs: none - Return annotation: `dict` - Return expressions: {'ttft_ms': 0.0, 'tpot_ms': 0.0, 'e2e_latency_ms': 0.0, 'gen_tps': 0.0, 'prompt_tps': 0.0, 'prompt_tokens': 0, 'complet… ## `vllm_mlx.bench_serve._fetch_post_run_status` - Kind: function - Signature: `async def _fetch_post_run_status(client: httpx.AsyncClient, base_url: str) -> dict` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/bench_serve.py#L1393-L1402 - Implementation: Function `_fetch_post_run_status` calls `client.get`, `resp.raise_for_status`, `resp.json`; awaits asynchronous work; has 2 explicit return paths. GET ``/v1/status`` after a case run, swallowing transport errors so a missing or temporarily-unavailable status endpoint does not fail the case record. - Inputs: - `client` (httpx.AsyncClient; required): Required positional or keyword input. - `base_url` (str; required): Required positional or keyword input. - Return annotation: `dict` - Calls: client.get, resp.raise_for_status, resp.json - Return expressions: resp.json(); {} ## `vllm_mlx.bench_serve._compute_within_policy_timeout` - Kind: function - Signature: `def _compute_within_policy_timeout(timeout_ms: Optional[int], *, error_present: bool, e2e_latency_ms: float) -> Optional[bool]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/bench_serve.py#L1405-L1418 - Implementation: Function `_compute_within_policy_timeout` has 3 explicit return paths. Resolve the ``policy.within_timeout`` field. ``None`` when the case did not configure a policy timeout, ``False`` when the request errored (any latency claim would be misleading), and otherwise the latency comparison result. - Inputs: - `timeout_ms` (Optional[int]; required): Required positional or keyword input. - `error_present` (bool; required): Required keyword-only input. - `e2e_latency_ms` (float; required): Required keyword-only input. - Return annotation: `Optional[bool]` - Return expressions: None; False; e2e_latency_ms <= timeout_ms ## `vllm_mlx.bench_serve._build_tool_calls_summary` - Kind: function - Signature: `def _build_tool_calls_summary(tool_calls: Any) -> Optional[dict]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/bench_serve.py#L1421-L1434 - Implementation: Function `_build_tool_calls_summary` calls `len`, `sorted`, `tc.get('function', {}).get`, `tc.get`; has 2 explicit return paths. Compact summary of streamed tool calls for the case record. Returns ``None`` when no tool calls were emitted so consumers can distinguish "feature not exercised" from "feature exercised, zero calls" if that ever matters. - Inputs: - `tool_calls` (Any; required): Required positional or keyword input. - Return annotation: `Optional[dict]` - Calls: len, sorted, tc.get('function', {}).get, tc.get - Return expressions: None; {'count': len(tool_calls), 'names': sorted((tc.get('function', {}).get('name', '') for tc in tool_calls)), 'raw': tool_… ## `vllm_mlx.bench_serve._build_workload_record` - Kind: function - Signature: `def _build_workload_record(*, case: WorkloadCase, workload: Workload, model: str, runtime: dict, hardware: dict, run_id: str, timestamp: str, started_wall: str, repetition: int, result: dict, error: str, quality_ok: bool, quality_issues: list[str], content: str, cache_hits_delta: int, cache_misses_delta: int, tokens_saved_delta: int, status_after: dict, cache_reset: Optional[dict], include_content: bool) -> dict` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/bench_serve.py#L1437-L1515 - Implementation: Function `_build_workload_record` calls `list`, `_resolve_max_tokens`, `len`, `_compute_within_policy_timeout`; returns `record`. Assemble the JSON-serializable workload-case record from the raw inputs and the completion result. Pure function: no I/O, deterministic given its arguments. - Inputs: - `case` (WorkloadCase; required): Required keyword-only input. - `workload` (Workload; required): Required keyword-only input. - `model` (str; required): Required keyword-only input. - `runtime` (dict; required): Required keyword-only input. - `hardware` (dict; required): Required keyword-only input. - `run_id` (str; required): Required keyword-only input. - `timestamp` (str; required): Required keyword-only input. - `started_wall` (str; required): Required keyword-only input. - `repetition` (int; required): Required keyword-only input. - `result` (dict; required): Required keyword-only input. - `error` (str; required): Required keyword-only input. - `quality_ok` (bool; required): Required keyword-only input. - `quality_issues` (list[str]; required): Required keyword-only input. - `content` (str; required): Required keyword-only input. - `cache_hits_delta` (int; required): Required keyword-only input. - `cache_misses_delta` (int; required): Required keyword-only input. - `tokens_saved_delta` (int; required): Required keyword-only input. - `status_after` (dict; required): Required keyword-only input. - `cache_reset` (Optional[dict]; required): Required keyword-only input. - `include_content` (bool; required): Required keyword-only input. - Return annotation: `dict` - Calls: list, _resolve_max_tokens, len, _compute_within_policy_timeout, bool, parse_status_response, result.get, _build_tool_calls_summary - Return expressions: record ## `vllm_mlx.bench_serve.run_workload_case` - Kind: function - Signature: `async def run_workload_case(client: httpx.AsyncClient, base_url: str, *, workload: Workload, case: WorkloadCase, model: str, runtime: dict, hardware: dict, run_id: str, timestamp: str, repetition: int=0, scrape: bool=True, include_content: bool=False, cache_reset: Optional[dict]=None) -> dict` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/bench_serve.py#L1518-L1593 - Implementation: Function `run_workload_case` calls `scrape_metrics`, `datetime.now(timezone.utc).isoformat`, `datetime.now`, `_assemble_case_request_kwargs`; awaits asynchronous work; returns `_build_workload_record(case=case, workload=workload, model=model, runtime=runtime, hardware=hardware, run_id=run_id, ti…`. Run one workload case and return a JSON-serializable result. - Inputs: - `client` (httpx.AsyncClient; required): Required positional or keyword input. - `base_url` (str; required): Required positional or keyword input. - `workload` (Workload; required): Required keyword-only input. - `case` (WorkloadCase; required): Required keyword-only input. - `model` (str; required): Required keyword-only input. - `runtime` (dict; required): Required keyword-only input. - `hardware` (dict; required): Required keyword-only input. - `run_id` (str; required): Required keyword-only input. - `timestamp` (str; required): Required keyword-only input. - `repetition` (int; optional; default `0`): Optional keyword-only input; defaults to `0`. - `scrape` (bool; optional; default `True`): Optional keyword-only input; defaults to `True`. - `include_content` (bool; optional; default `False`): Optional keyword-only input; defaults to `False`. - `cache_reset` (Optional[dict]; optional; default `None`): Optional keyword-only input; defaults to `None`. - Return annotation: `dict` - Calls: scrape_metrics, datetime.now(timezone.utc).isoformat, datetime.now, _assemble_case_request_kwargs, stream_chat_completion, _empty_completion_result, str, _fetch_post_run_status, metrics_after.get, metrics_before.get, result.get, validate_quality_checks, quality_issues.append, _build_workload_record - Return expressions: _build_workload_record(case=case, workload=workload, model=model, runtime=runtime, hardware=hardware, run_id=run_id, ti… ## `vllm_mlx.bench_serve._group_results_by_case_id` - Kind: function - Signature: `def _group_results_by_case_id(results: list[dict]) -> dict[str, list[dict]]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/bench_serve.py#L1596-L1602 - Implementation: Function `_group_results_by_case_id` calls `cases.setdefault(str(result.get('case_id', '')), []).append`, `cases.setdefault`, `str`, `result.get`; returns `cases`. Bucket workload case records by their ``case_id`` field, defaulting a missing ``case_id`` to the empty string so the grouping is stable. - Inputs: - `results` (list[dict]; required): Required positional or keyword input. - Return annotation: `dict[str, list[dict]]` - Calls: cases.setdefault(str(result.get('case_id', '')), []).append, cases.setdefault, str, result.get - Return expressions: cases ## `vllm_mlx.bench_serve._summarize_case` - Kind: function - Signature: `def _summarize_case(case_results: list[dict]) -> dict` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/bench_serve.py#L1605-L1648 - Implementation: Function `_summarize_case` calls `r['quality'].get`, `r['policy'].get`, `len`, `sorted`; returns `{'sample_count': len(case_results), 'repetitions': sorted({int(r.get('repetition', 0)) for r in case_results if r.get('…`. Build the per-case summary block. Mirrors the shape used at the run-level (sample counts, pass/fail rates, policy-timeout outcome, latency / ttft / gen_tps summaries) and adds two case-only fields: ``sample_count`` and ``repetitions`` (the sorted set of repetition indices the case was run under), plus ``content_chars`` since content length is more useful per-case than per-run. - Inputs: - `case_results` (list[dict]; required): Required positional or keyword input. - Return annotation: `dict` - Calls: r['quality'].get, r['policy'].get, len, sorted, int, r.get, round, _summary_or_empty - Return expressions: {'sample_count': len(case_results), 'repetitions': sorted({int(r.get('repetition', 0)) for r in case_results if r.get('… ## `vllm_mlx.bench_serve.summarize_workload_results` - Kind: function - Signature: `def summarize_workload_results(results: list[dict]) -> dict` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/bench_serve.py#L1651-L1689 - Implementation: Function `summarize_workload_results` calls `r['policy'].get`, `_group_results_by_case_id`, `_summarize_case`, `sorted`; returns `{'case_count': len(results), 'unique_case_count': len(cases), 'repetition_count': max((len(summary['repetitions']) for …`. Aggregate workload case records into stable qualification summary stats. - Inputs: - `results` (list[dict]; required): Required positional or keyword input. - Return annotation: `dict` - Calls: r['policy'].get, _group_results_by_case_id, _summarize_case, sorted, cases.items, len, max, case_summaries.values, round, _summary_or_empty - Return expressions: {'case_count': len(results), 'unique_case_count': len(cases), 'repetition_count': max((len(summary['repetitions']) for … ## `vllm_mlx.bench_serve.run_bench_serve_workload` - Kind: function - Signature: `async def run_bench_serve_workload(*, url: str, workload_path: str, model: Optional[str]=None, output_path: Optional[str]=None, output_format: str='json', scrape: bool=True, include_content: bool=False, request_timeout_s: Optional[float]=300.0, repetitions: int=1, cache_policy: Optional[str]=None) -> dict` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/bench_serve.py#L1692-L1818 - Implementation: Function `run_bench_serve_workload` calls `ValueError`, `load_workload`, `_normalize_cache_policy`, `workload.defaults.get`; awaits asynchronous work; can raise `ValueError`; returns `payload`. Run a declarative workload against a running server. This is the contract-style counterpart to prompt sweeps: it keeps product policy knobs in the manifest, records them as evidence, and measures what the server actually does before anyone promotes a model or feature stack. - Inputs: - `url` (str; required): Required keyword-only input. - `workload_path` (str; required): Required keyword-only input. - `model` (Optional[str]; optional; default `None`): Optional keyword-only input; defaults to `None`. - `output_path` (Optional[str]; optional; default `None`): Optional keyword-only input; defaults to `None`. - `output_format` (str; optional; default `'json'`): Optional keyword-only input; defaults to `'json'`. - `scrape` (bool; optional; default `True`): Optional keyword-only input; defaults to `True`. - `include_content` (bool; optional; default `False`): Optional keyword-only input; defaults to `False`. - `request_timeout_s` (Optional[float]; optional; default `300.0`): Optional keyword-only input; defaults to `300.0`. - `repetitions` (int; optional; default `1`): Optional keyword-only input; defaults to `1`. - `cache_policy` (Optional[str]; optional; default `None`): Optional keyword-only input; defaults to `None`. - Return annotation: `dict` - Calls: ValueError, load_workload, _normalize_cache_policy, workload.defaults.get, str, uuid.uuid4, datetime.now(timezone.utc).isoformat, datetime.now, httpx.Timeout, httpx.AsyncClient, auto_detect_runtime, detect_hardware_fingerprint, runtime.get, cache_events.append, clear_runtime_cache, len, range, print, run_workload_case, records.append, record.get('metrics', {}).get, record.get, Path(workload_path).expanduser, Path, summarize_workload_results, write_workload_sqlite, format_workload_payload, Path(output_path).expanduser().write_text, Path(output_path).expanduser - Raises directly: ValueError - Return expressions: payload ## `vllm_mlx.bench_serve._result_to_dict` - Kind: function - Signature: `def _result_to_dict(r: BenchServeResult) -> dict` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/bench_serve.py#L1840-L1846 - Implementation: Function `_result_to_dict` calls `getattr`, `_dataclasses.fields`; returns `{f.name: getattr(r, f.name) for f in _dataclasses.fields(r)}`. Convert a :class:`BenchServeResult` to an ordered dict. Returns an ``OrderedDict``-style plain ``dict`` whose keys follow the dataclass field declaration order (as listed in :data:`RESULT_COLUMNS`). - Inputs: - `r` (BenchServeResult; required): Required positional or keyword input. - Return annotation: `dict` - Calls: getattr, _dataclasses.fields - Return expressions: {f.name: getattr(r, f.name) for f in _dataclasses.fields(r)} ## `vllm_mlx.bench_serve.format_table` - Kind: function - Signature: `def format_table(results: list[BenchServeResult]) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/bench_serve.py#L1849-L1871 - Implementation: Function `format_table` calls `_result_to_dict`, `d.get`, `isinstance`, `round`; returns `_tabulate(rows, headers=_TABLE_COLUMNS, tablefmt='simple')`. Render a human-readable terminal table of benchmark results. Only the columns in :data:`_TABLE_COLUMNS` are shown. Float values are rounded to one decimal place. Args: results: List of :class:`BenchServeResult` instances. Returns: Formatted string using ``tabulate`` with ``tablefmt="simple"``. - Inputs: - `results` (list[BenchServeResult]; required): List of :class:`BenchServeResult` instances. - Return annotation: `str` - Calls: _result_to_dict, d.get, isinstance, round, row.append, rows.append, _tabulate - Return expressions: _tabulate(rows, headers=_TABLE_COLUMNS, tablefmt='simple') ## `vllm_mlx.bench_serve.format_json` - Kind: function - Signature: `def format_json(results: list[BenchServeResult]) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/bench_serve.py#L1874-L1885 - Implementation: Function `format_json` calls `json.dumps`, `_result_to_dict`; returns `json.dumps([_result_to_dict(r) for r in results], indent=2)`. Serialize benchmark results as a JSON array. All fields from :data:`RESULT_COLUMNS` are included. Args: results: List of :class:`BenchServeResult` instances. Returns: JSON string with ``indent=2``. - Inputs: - `results` (list[BenchServeResult]; required): List of :class:`BenchServeResult` instances. - Return annotation: `str` - Calls: json.dumps, _result_to_dict - Return expressions: json.dumps([_result_to_dict(r) for r in results], indent=2) ## `vllm_mlx.bench_serve.format_csv` - Kind: function - Signature: `def format_csv(results: list[BenchServeResult]) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/bench_serve.py#L1888-L1904 - Implementation: Function `format_csv` calls `io.StringIO`, `csv_mod.DictWriter`, `writer.writeheader`, `writer.writerow`; returns `buf.getvalue()`. Serialize benchmark results as CSV with a header row. All columns are included. Args: results: List of :class:`BenchServeResult` instances. Returns: CSV string (header + one row per result). - Inputs: - `results` (list[BenchServeResult]; required): List of :class:`BenchServeResult` instances. - Return annotation: `str` - Calls: io.StringIO, csv_mod.DictWriter, writer.writeheader, writer.writerow, _result_to_dict, buf.getvalue - Return expressions: buf.getvalue() ## `vllm_mlx.bench_serve._sql_escape` - Kind: function - Signature: `def _sql_escape(value) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/bench_serve.py#L1907-L1927 - Implementation: Function `_sql_escape` calls `isinstance`, `math.isnan`, `math.isinf`, `str`; has 4 explicit return paths. Escape a Python value for use as a SQL literal. - ``None`` -> ``"NULL"`` - ``bool`` -> ``"1"`` or ``"0"`` - ``int`` / ``float`` -> string representation - ``str`` -> single-quoted with internal single-quotes doubled - Inputs: - `value` (not annotated; required): Required positional or keyword input. - Return annotation: `str` - Calls: isinstance, math.isnan, math.isinf, str, str(value).replace - Return expressions: 'NULL'; '1' if value else '0'; str(value); f"'{escaped}'" ## `vllm_mlx.bench_serve.format_sql` - Kind: function - Signature: `def format_sql(results: list[BenchServeResult]) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/bench_serve.py#L1945-L1964 - Implementation: Function `format_sql` calls `_result_to_dict`, `', '.join`, `_sql_escape`, `lines.append`; returns `'\n'.join(lines)`. Emit a SQL ``CREATE TABLE IF NOT EXISTS`` statement and INSERT rows. The schema follows the exact column order defined in the bench-serve spec. Args: results: List of :class:`BenchServeResult` instances. Returns: SQL string containing the CREATE TABLE statement followed by one INSERT statement per result. - Inputs: - `results` (list[BenchServeResult]; required): List of :class:`BenchServeResult` instances. - Return annotation: `str` - Calls: _result_to_dict, ', '.join, _sql_escape, lines.append, '\n'.join - Return expressions: '\n'.join(lines) ## `vllm_mlx.bench_serve._write_sqlite_rows` - Kind: function - Signature: `def _write_sqlite_rows(output_path: str, *, table: str, schema: str, columns: list[str], rows: list[dict]) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/bench_serve.py#L1967-L1990 - Implementation: Function `_write_sqlite_rows` calls `Path(output_path).expanduser`, `Path`, `_validate_sql_identifier`, `', '.join`. Append benchmark rows to a SQLite database. - Inputs: - `output_path` (str; required): Required positional or keyword input. - `table` (str; required): Required keyword-only input. - `schema` (str; required): Required keyword-only input. - `columns` (list[str]; required): Required keyword-only input. - `rows` (list[dict]; required): Required keyword-only input. - Return annotation: `None` - Calls: Path(output_path).expanduser, Path, _validate_sql_identifier, ', '.join, row.get, sqlite3.connect, conn.execute, conn.executemany, conn.commit ## `vllm_mlx.bench_serve._validate_sql_identifier` - Kind: function - Signature: `def _validate_sql_identifier(identifier: str, *, kind: str) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/bench_serve.py#L1993-L1996 - Implementation: Function `_validate_sql_identifier` calls `_SQL_IDENTIFIER_RE.fullmatch`, `ValueError`; can raise `ValueError`. Reject unsafe SQL identifiers before string interpolation. - Inputs: - `identifier` (str; required): Required positional or keyword input. - `kind` (str; required): Required keyword-only input. - Return annotation: `None` - Calls: _SQL_IDENTIFIER_RE.fullmatch, ValueError - Raises directly: ValueError ## `vllm_mlx.bench_serve.write_sqlite` - Kind: function - Signature: `def write_sqlite(results: list[BenchServeResult], output_path: str) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/bench_serve.py#L1999-L2009 - Implementation: Function `write_sqlite` calls `_result_to_dict`, `_write_sqlite_rows`. Append prompt-sweep benchmark results to a SQLite database. - Inputs: - `results` (list[BenchServeResult]; required): Required positional or keyword input. - `output_path` (str; required): Required positional or keyword input. - Return annotation: `None` - Calls: _result_to_dict, _write_sqlite_rows ## `vllm_mlx.bench_serve._workload_record_to_row` - Kind: function - Signature: `def _workload_record_to_row(record: dict) -> dict` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/bench_serve.py#L2069-L2119 - Implementation: Function `_workload_record_to_row` calls `record.get`, `metrics.get`, `','.join`, `hardware.get`; returns `{'run_id': record.get('run_id', ''), 'timestamp': record.get('timestamp', ''), 'workload': record.get('workload', ''), …`. Function `_workload_record_to_row` calls `record.get`, `metrics.get`, `','.join`, `hardware.get`; returns `{'run_id': record.get('run_id', ''), 'timestamp': record.get('timestamp', ''), 'workload': record.get('workload', ''), …`. - Inputs: - `record` (dict; required): Required positional or keyword input. - Return annotation: `dict` - Calls: record.get, metrics.get, ','.join, hardware.get, runtime.get, request.get, json.dumps, policy.get, metal.get, quality.get - Return expressions: {'run_id': record.get('run_id', ''), 'timestamp': record.get('timestamp', ''), 'workload': record.get('workload', ''), … ## `vllm_mlx.bench_serve.format_workload_table` - Kind: function - Signature: `def format_workload_table(payload: dict) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/bench_serve.py#L2122-L2134 - Implementation: Function `format_workload_table` calls `payload.get`, `_workload_record_to_row`, `rows.append`, `isinstance`; returns `_tabulate(rows, headers=_WORKLOAD_TABLE_COLUMNS, tablefmt='simple')`. Format workload result records as a compact human-readable table. - Inputs: - `payload` (dict; required): Required positional or keyword input. - Return annotation: `str` - Calls: payload.get, _workload_record_to_row, rows.append, isinstance, round, _tabulate - Return expressions: _tabulate(rows, headers=_WORKLOAD_TABLE_COLUMNS, tablefmt='simple') ## `vllm_mlx.bench_serve.format_workload_json` - Kind: function - Signature: `def format_workload_json(payload: dict) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/bench_serve.py#L2137-L2140 - Implementation: Function `format_workload_json` calls `json.dumps`; returns `json.dumps(payload, indent=2)`. Serialize a workload result payload as indented JSON. - Inputs: - `payload` (dict; required): Required positional or keyword input. - Return annotation: `str` - Calls: json.dumps - Return expressions: json.dumps(payload, indent=2) ## `vllm_mlx.bench_serve.format_workload_csv` - Kind: function - Signature: `def format_workload_csv(payload: dict) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/bench_serve.py#L2143-L2151 - Implementation: Function `format_workload_csv` calls `io.StringIO`, `csv_mod.DictWriter`, `writer.writeheader`, `payload.get`; returns `buf.getvalue()`. Serialize workload result records with the stable CSV column contract. - Inputs: - `payload` (dict; required): Required positional or keyword input. - Return annotation: `str` - Calls: io.StringIO, csv_mod.DictWriter, writer.writeheader, payload.get, writer.writerow, _workload_record_to_row, buf.getvalue - Return expressions: buf.getvalue() ## `vllm_mlx.bench_serve.format_workload_sql` - Kind: function - Signature: `def format_workload_sql(payload: dict) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/bench_serve.py#L2170-L2180 - Implementation: Function `format_workload_sql` calls `payload.get`, `_workload_record_to_row`, `', '.join`, `_sql_escape`; returns `'\n'.join(lines)`. Render SQL statements that create and populate the workload table. - Inputs: - `payload` (dict; required): Required positional or keyword input. - Return annotation: `str` - Calls: payload.get, _workload_record_to_row, ', '.join, _sql_escape, lines.append, '\n'.join - Return expressions: '\n'.join(lines) ## `vllm_mlx.bench_serve.write_workload_sqlite` - Kind: function - Signature: `def write_workload_sqlite(payload: dict, output_path: str) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/bench_serve.py#L2183-L2193 - Implementation: Function `write_workload_sqlite` calls `_workload_record_to_row`, `payload.get`, `_write_sqlite_rows`. Append workload result records to a SQLite database. - Inputs: - `payload` (dict; required): Required positional or keyword input. - `output_path` (str; required): Required positional or keyword input. - Return annotation: `None` - Calls: _workload_record_to_row, payload.get, _write_sqlite_rows ## `vllm_mlx.bench_serve.format_workload_payload` - Kind: function - Signature: `def format_workload_payload(payload: dict, fmt: str='json') -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/bench_serve.py#L2196-L2211 - Implementation: Function `format_workload_payload` calls `format_workload_json`, `format_workload_csv`, `format_workload_sql`, `format_workload_table`; can raise `ValueError`; has 4 explicit return paths. Serialize a workload payload in the requested text output format. Raises: ValueError: If ``fmt`` is not ``json``, ``csv``, ``sql``, or ``table``. - Inputs: - `payload` (dict; required): Required positional or keyword input. - `fmt` (str; optional; default `'json'`): Optional positional or keyword input; defaults to `'json'`. - Return annotation: `str` - Calls: format_workload_json, format_workload_csv, format_workload_sql, format_workload_table, ValueError - Raises directly: ValueError - Return expressions: format_workload_json(payload); format_workload_csv(payload); format_workload_sql(payload); format_workload_table(payload) ## `vllm_mlx.bench_serve.run_bench_serve` - Kind: function - Signature: `async def run_bench_serve(url: str='http://127.0.0.1:8080', model: Optional[str]=None, prompt_sets: list[str]=None, prompt_file: Optional[str]=None, concurrencies: list[int]=None, max_tokens: int=256, repetitions: int=3, warmup: int=1, thinking_values: list[Optional[bool]]=None, extra_bodies: list[str]=None, output_path: Optional[str]=None, fmt: str='table', do_validate: bool=True, scrape: bool=True, tag: Optional[str]=None, override_fields: Optional[dict]=None, system_prompt_file: Optional[str]=None, skip_preflight_token_count: bool=False) -> list[BenchServeResult]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/bench_serve.py#L2221-L2638 - Implementation: Function `run_bench_serve` calls `str`, `uuid.uuid4`, `datetime.now(timezone.utc).isoformat`, `datetime.now`; awaits asynchronous work; can raise `ValueError`; has 2 explicit return paths. Run the full bench-serve sweep against a running vllm-mlx server. Args: url: Base URL of the server. model: Model ID to use. If ``None``, auto-detected from the server. prompt_sets: List of prompt set names or paths. Defaults to ``["short", "medium", "long"]``. prompt_file: Optional path to an extra prompt file to include. concurrencies: Concurrency levels to sweep. Defaults to ``[1, 4]``. max_tokens: Maximum tokens to generate per request. repetitions: Number of repetitions per sweep config. warmup: Number of warmup rounds before the first measured repetition. thinking_values: Values for ``enable_thinking``. Defaults to ``[None]``. extra_bodies: JSON strings for extra body parameters. Defaults to ``[""]`` (no extra body). output_path: File path to write results to. If ``None``, prints to stdout. fmt: Output format — one of ``"table"``, ``"json"``, ``"csv"``, ``"sql"``, or ``"sqlite"``. do_validate: Whether to validate each response. scrape: Whether to scrape ``/metrics`` before and after each run. tag: Optional tag string stored in every result row. override_fields: Dict of field names to override on every result. Returns: List of :class:`BenchServeResult` instances. - Inputs: - `url` (str; optional; default `'http://127.0.0.1:8080'`): Base URL of the server. - `model` (Optional[str]; optional; default `None`): Model ID to use. If ``None``, auto-detected from the server. - `prompt_sets` (list[str]; optional; default `None`): List of prompt set names or paths. Defaults to ``["short", "medium", "long"]``. - `prompt_file` (Optional[str]; optional; default `None`): Optional path to an extra prompt file to include. - `concurrencies` (list[int]; optional; default `None`): Concurrency levels to sweep. Defaults to ``[1, 4]``. - `max_tokens` (int; optional; default `256`): Maximum tokens to generate per request. - `repetitions` (int; optional; default `3`): Number of repetitions per sweep config. - `warmup` (int; optional; default `1`): Number of warmup rounds before the first measured repetition. - `thinking_values` (list[Optional[bool]]; optional; default `None`): Values for ``enable_thinking``. Defaults to ``[None]``. - `extra_bodies` (list[str]; optional; default `None`): JSON strings for extra body parameters. Defaults to ``[""]`` (no extra body). - `output_path` (Optional[str]; optional; default `None`): File path to write results to. If ``None``, prints to stdout. - `fmt` (str; optional; default `'table'`): Output format — one of ``"table"``, ``"json"``, ``"csv"``, ``"sql"``, or ``"sqlite"``. - `do_validate` (bool; optional; default `True`): Whether to validate each response. - `scrape` (bool; optional; default `True`): Whether to scrape ``/metrics`` before and after each run. - `tag` (Optional[str]; optional; default `None`): Optional tag string stored in every result row. - `override_fields` (Optional[dict]; optional; default `None`): Dict of field names to override on every result. - `system_prompt_file` (Optional[str]; optional; default `None`): Optional positional or keyword input; defaults to `None`. - `skip_preflight_token_count` (bool; optional; default `False`): Optional positional or keyword input; defaults to `False`. - Return annotation: `list[BenchServeResult]` - Calls: str, uuid.uuid4, datetime.now(timezone.utc).isoformat, datetime.now, httpx.AsyncClient, httpx.Timeout, print, auto_detect_runtime, detect_hardware_fingerprint, runtime.get, hw.get, load_prompt_set, Path(system_prompt_file).expanduser, Path, sys_path.exists, sys_path.read_text, all_prompts.items, msgs[0].get, patched.append, len, count_prompt_tokens, expand_sweep, list, all_prompts.keys, set, json.loads, label_parts.append, ' '.join, warmed_up.add, range, run_concurrent_requests, scrape_metrics, metrics_after.get, metrics_before.get, client.get, resp.raise_for_status, parse_status_response, resp.json, status_data.get, BenchServeResult, prompt_token_counts.get, _mean, sum, r.get, max, all, override_fields.items, hasattr, setattr, results.append, ValueError, write_sqlite, formatters.get, formatter, Path(output_path).write_text - Raises directly: ValueError - Return expressions: []; results ## `vllm_mlx.bench_serve.run_bench_serve._mean` - Kind: nested function - Signature: `def _mean(key: str) -> float` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/bench_serve.py#L2522-L2526 - Implementation: Nested Function `run_bench_serve._mean` calls `statistics.mean`; returns `statistics.mean(vals) if vals else 0.0`. Nested Function `run_bench_serve._mean` calls `statistics.mean`; returns `statistics.mean(vals) if vals else 0.0`. - Inputs: - `key` (str; required): Required positional or keyword input. - Return annotation: `float` - Calls: statistics.mean - Return expressions: statistics.mean(vals) if vals else 0.0 # Module `vllm_mlx.benchmark` Performance Benchmark for vllm-mlx. Measures key performance metrics for LLM and MLLM (Multimodal Language Model) inference: - Time to First Token (TTFT) - Time Per Output Token (TPOT) - Tokens Per Second (TPS) - both input processing and output generation - End-to-End Latency - Throughput - Memory Usage (process and MLX cache) - MLLM: Image resolution performance - MLLM: Video frame count performance Usage: # LLM benchmark python -m vllm_mlx.benchmark --model mlx-community/Llama-3.2-1B-Instruct-4bit python -m vllm_mlx.benchmark --model mlx-community/Llama-3.2-3B-Instruct-4bit --prompts 10 --max-tokens 256 # MLLM image benchmark (auto-detected or use --mllm flag) python -m vllm_mlx.benchmark --model mlx-community/Qwen3-VL-4B-Instruct-3bit python -m vllm_mlx.benchmark --model mlx-community/Qwen3-VL-4B-Instruct-3bit --mllm --quick # MLLM video benchmark python -m vllm_mlx.benchmark --model mlx-community/Qwen3-VL-4B-Instruct-3bit --video python -m vllm_mlx.benchmark --model mlx-community/Qwen3-VL-4B-Instruct-3bit --video --video-url https://example.com/video.mp4 Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/benchmark.py#L1-L1684 ## `vllm_mlx.benchmark.ResourceMetrics` - Kind: class - Signature: `class ResourceMetrics` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/benchmark.py#L72-L80 - Implementation: Class `ResourceMetrics` declares 0 direct member(s). Resource usage metrics during benchmark. - Inputs: - `process_memory_gb` (float; optional; default `0.0`): Optional constructor field; defaults to `0.0`. - `mlx_cache_gb` (float; optional; default `0.0`): Optional constructor field; defaults to `0.0`. - `mlx_peak_memory_gb` (float; optional; default `0.0`): Optional constructor field; defaults to `0.0`. - `system_memory_used_gb` (float; optional; default `0.0`): Optional constructor field; defaults to `0.0`. - `system_memory_total_gb` (float; optional; default `0.0`): Optional constructor field; defaults to `0.0`. - Constructs: `vllm_mlx.benchmark.ResourceMetrics` - Decorators: dataclass ## `vllm_mlx.benchmark.reset_mlx_peak_memory` - Kind: function - Signature: `def reset_mlx_peak_memory()` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/benchmark.py#L83-L95 - Implementation: Function `reset_mlx_peak_memory` calls `hasattr`, `mx.reset_peak_memory`, `mx.metal.reset_peak_memory`; returns `None`. Reset MLX peak memory counter. - Inputs: none - Return annotation: `not annotated` - Calls: hasattr, mx.reset_peak_memory, mx.metal.reset_peak_memory - Return expressions: None ## `vllm_mlx.benchmark.get_mlx_memory_info` - Kind: function - Signature: `def get_mlx_memory_info(reset_peak: bool=True) -> dict` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/benchmark.py#L98-L135 - Implementation: Function `get_mlx_memory_info` calls `hasattr`, `mx.get_cache_memory`, `mx.get_peak_memory`, `mx.get_active_memory`; has 2 explicit return paths. Get MLX memory usage information. Args: reset_peak: If True, reset peak memory counter after reading. - Inputs: - `reset_peak` (bool; optional; default `True`): If True, reset peak memory counter after reading. - Return annotation: `dict` - Calls: hasattr, mx.get_cache_memory, mx.get_peak_memory, mx.get_active_memory, mx.metal.get_cache_memory, mx.metal.get_peak_memory, mx.metal.get_active_memory, reset_mlx_peak_memory - Return expressions: {}; info ## `vllm_mlx.benchmark.get_process_memory` - Kind: function - Signature: `def get_process_memory() -> float` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/benchmark.py#L138-L147 - Implementation: Function `get_process_memory` calls `psutil.Process`, `process.memory_info`; has 2 explicit return paths. Get current process memory usage in GB. - Inputs: none - Return annotation: `float` - Calls: psutil.Process, process.memory_info - Return expressions: 0.0; process.memory_info().rss / 1024 ** 3 ## `vllm_mlx.benchmark.get_system_memory` - Kind: function - Signature: `def get_system_memory() -> tuple[float, float]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/benchmark.py#L150-L159 - Implementation: Function `get_system_memory` calls `psutil.virtual_memory`; has 2 explicit return paths. Get system memory (used, total) in GB. - Inputs: none - Return annotation: `tuple[float, float]` - Calls: psutil.virtual_memory - Return expressions: (0.0, 0.0); (mem.used / 1024 ** 3, mem.total / 1024 ** 3) ## `vllm_mlx.benchmark.ResourceMonitor` - Kind: class - Signature: `class ResourceMonitor` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/benchmark.py#L162-L213 - Implementation: Class `ResourceMonitor` declares 4 direct member(s). Monitor system resources during benchmark runs. - Inputs: none - Constructs: `vllm_mlx.benchmark.ResourceMonitor` ## `vllm_mlx.benchmark.ResourceMonitor.__init__` - Kind: method - Signature: `def __init__(self)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/benchmark.py#L165-L168 - Implementation: Method `ResourceMonitor.__init__` updates `self.samples`, `self._start_time`, `self._start_memory`. Method `ResourceMonitor.__init__` updates `self.samples`, `self._start_time`, `self._start_memory`. - Inputs: none - Return annotation: `not annotated` - State writes: self.samples, self._start_time, self._start_memory ## `vllm_mlx.benchmark.ResourceMonitor.start` - Kind: method - Signature: `def start(self)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/benchmark.py#L170-L176 - Implementation: Method `ResourceMonitor.start` updates `self._start_time`, `self._start_memory`; calls `time.perf_counter`, `get_process_memory`, `reset_mlx_peak_memory`. Start monitoring. - Inputs: none - Return annotation: `not annotated` - Calls: time.perf_counter, get_process_memory, reset_mlx_peak_memory - State writes: self._start_time, self._start_memory ## `vllm_mlx.benchmark.ResourceMonitor.sample` - Kind: method - Signature: `def sample(self) -> ResourceMetrics` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/benchmark.py#L178-L192 - Implementation: Method `ResourceMonitor.sample` calls `get_mlx_memory_info`, `get_system_memory`, `ResourceMetrics`, `get_process_memory`; returns `metrics`. Take a resource sample. - Inputs: none - Return annotation: `ResourceMetrics` - Calls: get_mlx_memory_info, get_system_memory, ResourceMetrics, get_process_memory, mlx_info.get, self.samples.append - State reads: self.samples.append, self.samples - Return expressions: metrics ## `vllm_mlx.benchmark.ResourceMonitor.get_summary` - Kind: method - Signature: `def get_summary(self) -> ResourceMetrics` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/benchmark.py#L194-L213 - Implementation: Method `ResourceMonitor.get_summary` calls `ResourceMetrics`, `max`; has 2 explicit return paths. Get summary of all samples. - Inputs: none - Return annotation: `ResourceMetrics` - Calls: ResourceMetrics, max - State reads: self.samples - Return expressions: ResourceMetrics(); ResourceMetrics(process_memory_gb=peak_process, mlx_cache_gb=peak_mlx_cache, mlx_peak_memory_gb=peak_mlx, system_memory… ## `vllm_mlx.benchmark.BenchmarkResult` - Kind: class - Signature: `class BenchmarkResult` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/benchmark.py#L235-L268 - Implementation: Class `BenchmarkResult` declares 1 direct member(s). Results from a single benchmark run. - Inputs: - `prompt` (str; required): Required constructor field. - `prompt_tokens` (int; required): Required constructor field. - `generated_tokens` (int; required): Required constructor field. - `ttft` (float; required): Required constructor field. - `total_time` (float; required): Required constructor field. - `tpot` (float; optional; default `0.0`): Optional constructor field; defaults to `0.0`. - `generation_tps` (float; optional; default `0.0`): Optional constructor field; defaults to `0.0`. - `processing_tps` (float; optional; default `0.0`): Optional constructor field; defaults to `0.0`. - Constructs: `vllm_mlx.benchmark.BenchmarkResult` - Decorators: dataclass ## `vllm_mlx.benchmark.BenchmarkResult.__post_init__` - Kind: method - Signature: `def __post_init__(self)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/benchmark.py#L251-L268 - Implementation: Method `BenchmarkResult.__post_init__` updates `self.tpot`, `self.generation_tps`, `self.processing_tps`. Method `BenchmarkResult.__post_init__` updates `self.tpot`, `self.generation_tps`, `self.processing_tps`. - Inputs: none - Return annotation: `not annotated` - State reads: self.generated_tokens, self.total_time, self.ttft, self.prompt_tokens - State writes: self.tpot, self.generation_tps, self.processing_tps ## `vllm_mlx.benchmark.BenchmarkSummary` - Kind: class - Signature: `class BenchmarkSummary` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/benchmark.py#L272-L315 - Implementation: Class `BenchmarkSummary` declares 0 direct member(s). Summary statistics across all benchmark runs. - Inputs: - `model_name` (str; required): Required constructor field. - `num_runs` (int; required): Required constructor field. - `total_prompt_tokens` (int; required): Required constructor field. - `total_generated_tokens` (int; required): Required constructor field. - `total_time` (float; required): Required constructor field. - `ttft_mean` (float; required): Required constructor field. - `ttft_min` (float; required): Required constructor field. - `ttft_max` (float; required): Required constructor field. - `ttft_p50` (float; required): Required constructor field. - `ttft_p95` (float; required): Required constructor field. - `tpot_mean` (float; required): Required constructor field. - `tpot_min` (float; required): Required constructor field. - `tpot_max` (float; required): Required constructor field. - `generation_tps_mean` (float; required): Required constructor field. - `generation_tps_max` (float; required): Required constructor field. - `processing_tps_mean` (float; required): Required constructor field. - `latency_mean` (float; required): Required constructor field. - `latency_min` (float; required): Required constructor field. - `latency_max` (float; required): Required constructor field. - `latency_p50` (float; required): Required constructor field. - `latency_p95` (float; required): Required constructor field. - `total_throughput_tps` (float; required): Required constructor field. - `requests_per_second` (float; required): Required constructor field. - `hardware_chip` (str; optional; default `''`): Optional constructor field; defaults to `''`. - `hardware_memory_gb` (float; optional; default `0.0`): Optional constructor field; defaults to `0.0`. - `hardware_bandwidth_gbs` (float; optional; default `0.0`): Optional constructor field; defaults to `0.0`. - `resources` (ResourceMetrics; optional; default `field(default_factory=ResourceMetrics)`): Optional constructor field; defaults to `field(default_factory=ResourceMetrics)`. - Constructs: `vllm_mlx.benchmark.BenchmarkSummary` - Decorators: dataclass ## `vllm_mlx.benchmark.calculate_percentile` - Kind: function - Signature: `def calculate_percentile(data: list, percentile: float) -> float` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/benchmark.py#L318-L325 - Implementation: Function `calculate_percentile` calls `sorted`, `int`, `len`, `min`; has 2 explicit return paths. Calculate percentile from a list. - Inputs: - `data` (list; required): Required positional or keyword input. - `percentile` (float; required): Required positional or keyword input. - Return annotation: `float` - Calls: sorted, int, len, min - Return expressions: 0.0; sorted_data[index] ## `vllm_mlx.benchmark.benchmark_single_prompt` - Kind: function - Signature: `def benchmark_single_prompt(model, tokenizer, prompt: str, max_tokens: int=256, temperature: float=0.7) -> Optional[BenchmarkResult]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/benchmark.py#L328-L394 - Implementation: Function `benchmark_single_prompt` calls `tokenizer.encode`, `len`, `make_sampler`, `time.perf_counter`; has 2 explicit return paths. Benchmark a single prompt with detailed timing. Args: model: The loaded MLX model tokenizer: The tokenizer prompt: The prompt to benchmark max_tokens: Maximum tokens to generate temperature: Sampling temperature Returns: BenchmarkResult with timing metrics - Inputs: - `model` (not annotated; required): The loaded MLX model - `tokenizer` (not annotated; required): The tokenizer - `prompt` (str; required): The prompt to benchmark - `max_tokens` (int; optional; default `256`): Maximum tokens to generate - `temperature` (float; optional; default `0.7`): Sampling temperature - Return annotation: `Optional[BenchmarkResult]` - Calls: tokenizer.encode, len, make_sampler, time.perf_counter, stream_generate, BenchmarkResult, print, traceback.print_exc - Return expressions: BenchmarkResult(prompt=prompt[:50] + '...' if len(prompt) > 50 else prompt, prompt_tokens=prompt_token_count, generated…; None ## `vllm_mlx.benchmark.run_benchmark` - Kind: function - Signature: `def run_benchmark(model_name: str, num_prompts: int=5, max_tokens: int=256, temperature: float=0.7, warmup_runs: int=1) -> Optional[BenchmarkSummary]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/benchmark.py#L397-L610 - Implementation: Function `run_benchmark` calls `detect_hardware`, `len`, `print`, `tabulate`; has 2 explicit return paths. Run the full benchmark suite. Args: model_name: HuggingFace model name or local path num_prompts: Number of prompts to test max_tokens: Maximum tokens per generation temperature: Sampling temperature warmup_runs: Number of warmup runs before measuring Returns: BenchmarkSummary with aggregate statistics - Inputs: - `model_name` (str; required): HuggingFace model name or local path - `num_prompts` (int; optional; default `5`): Number of prompts to test - `max_tokens` (int; optional; default `256`): Maximum tokens per generation - `temperature` (float; optional; default `0.7`): Sampling temperature - `warmup_runs` (int; optional; default `1`): Number of warmup runs before measuring - Return annotation: `Optional[BenchmarkSummary]` - Calls: detect_hardware, len, print, tabulate, ResourceMonitor, monitor.start, time.perf_counter, load_model_with_fallback, tokenizer.encode, sum, range, benchmark_single_prompt, get_mlx_memory_info, mlx_info.get, enumerate, results.append, run_data.append, monitor.sample, BenchmarkSummary, statistics.mean, min, max, calculate_percentile, monitor.get_summary - Return expressions: None; summary ## `vllm_mlx.benchmark.is_mllm_model` - Kind: function - Signature: `def is_mllm_model(model_name: str) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/benchmark.py#L651-L657 - Implementation: Function `is_mllm_model` calls `model_name.lower`, `pattern.lower`; has 2 explicit return paths. Check if model name indicates a multimodal language model. - Inputs: - `model_name` (str; required): Required positional or keyword input. - Return annotation: `bool` - Calls: model_name.lower, pattern.lower - Return expressions: True; False ## `vllm_mlx.benchmark.MLLMBenchmarkResult` - Kind: class - Signature: `class MLLMBenchmarkResult` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/benchmark.py#L661-L674 - Implementation: Class `MLLMBenchmarkResult` declares 0 direct member(s). Result from a single MLLM benchmark run. - Inputs: - `resolution` (str; required): Required constructor field. - `width` (int; required): Required constructor field. - `height` (int; required): Required constructor field. - `pixels` (int; required): Required constructor field. - `time_seconds` (float; required): Required constructor field. - `tokens_generated` (int; required): Required constructor field. - `tokens_per_second` (float; required): Required constructor field. - `response_preview` (str; required): Required constructor field. - `memory_gb` (float; optional; default `0.0`): Optional constructor field; defaults to `0.0`. - `mlx_memory_gb` (float; optional; default `0.0`): Optional constructor field; defaults to `0.0`. - Constructs: `vllm_mlx.benchmark.MLLMBenchmarkResult` - Decorators: dataclass ## `vllm_mlx.benchmark.download_test_image` - Kind: function - Signature: `def download_test_image(url: str, timeout: int=30) -> Image.Image` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/benchmark.py#L677-L684 - Implementation: Function `download_test_image` calls `requests.get`, `response.raise_for_status`, `Image.open`, `io.BytesIO`; returns `Image.open(io.BytesIO(response.content))`. Download image from URL and return PIL Image. - Inputs: - `url` (str; required): Required positional or keyword input. - `timeout` (int; optional; default `30`): Optional positional or keyword input; defaults to `30`. - Return annotation: `Image.Image` - Calls: requests.get, response.raise_for_status, Image.open, io.BytesIO - Return expressions: Image.open(io.BytesIO(response.content)) ## `vllm_mlx.benchmark.resize_image` - Kind: function - Signature: `def resize_image(img: Image.Image, width: int, height: int) -> Image.Image` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/benchmark.py#L687-L689 - Implementation: Function `resize_image` calls `img.resize`; returns `img.resize((width, height), Image.Resampling.LANCZOS)`. Resize image to specified dimensions. - Inputs: - `img` (Image.Image; required): Required positional or keyword input. - `width` (int; required): Required positional or keyword input. - `height` (int; required): Required positional or keyword input. - Return annotation: `Image.Image` - Calls: img.resize - Return expressions: img.resize((width, height), Image.Resampling.LANCZOS) ## `vllm_mlx.benchmark.image_to_base64` - Kind: function - Signature: `def image_to_base64(img: Image.Image, format: str='JPEG') -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/benchmark.py#L692-L705 - Implementation: Function `image_to_base64` calls `Image.new`, `background.paste`, `img.split`, `img.convert`; returns `f'data:{mime};base64,{b64}'`. Convert PIL Image to base64 data URL. - Inputs: - `img` (Image.Image; required): Required positional or keyword input. - `format` (str; optional; default `'JPEG'`): Optional positional or keyword input; defaults to `'JPEG'`. - Return annotation: `str` - Calls: Image.new, background.paste, img.split, img.convert, io.BytesIO, img.save, base64.b64encode(buffer.getvalue()).decode, base64.b64encode, buffer.getvalue - Return expressions: f'data:{mime};base64,{b64}' ## `vllm_mlx.benchmark.benchmark_mllm_resolution` - Kind: function - Signature: `def benchmark_mllm_resolution(model, processor, config, base_image: Image.Image, width: int, height: int, max_tokens: int=256, warmup: bool=False) -> MLLMBenchmarkResult` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/benchmark.py#L708-L800 - Implementation: Function `benchmark_mllm_resolution` calls `reset_mlx_peak_memory`, `resize_image`, `tempfile.NamedTemporaryFile`, `img.save`; returns `MLLMBenchmarkResult(resolution=resolution_name, width=width, height=height, pixels=pixels, time_seconds=elapsed, tokens…`. Run MLLM benchmark for a specific resolution. - Inputs: - `model` (not annotated; required): Required positional or keyword input. - `processor` (not annotated; required): Required positional or keyword input. - `config` (not annotated; required): Required positional or keyword input. - `base_image` (Image.Image; required): Required positional or keyword input. - `width` (int; required): Required positional or keyword input. - `height` (int; required): Required positional or keyword input. - `max_tokens` (int; optional; default `256`): Optional positional or keyword input; defaults to `256`. - `warmup` (bool; optional; default `False`): Optional positional or keyword input; defaults to `False`. - Return annotation: `MLLMBenchmarkResult` - Calls: reset_mlx_peak_memory, resize_image, tempfile.NamedTemporaryFile, img.save, print, apply_chat_template, time.perf_counter, generate, hasattr, getattr, len, text.split, str, get_mlx_memory_info, get_process_memory, mlx_info.get, os.unlink, MLLMBenchmarkResult - Return expressions: MLLMBenchmarkResult(resolution=resolution_name, width=width, height=height, pixels=pixels, time_seconds=elapsed, tokens… ## `vllm_mlx.benchmark.run_mllm_benchmark` - Kind: function - Signature: `def run_mllm_benchmark(model_name: str, quick: bool=False, max_tokens: int=256, warmup_runs: int=1) -> list[MLLMBenchmarkResult]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/benchmark.py#L803-L913 - Implementation: Function `run_mllm_benchmark` calls `detect_hardware`, `print`, `len`, `tabulate`; has 2 explicit return paths. Run MLLM benchmark across multiple image resolutions. Args: model_name: HuggingFace model name quick: If True, test only 4 resolutions max_tokens: Max tokens to generate warmup_runs: Number of warmup runs Returns: List of MLLMBenchmarkResult - Inputs: - `model_name` (str; required): HuggingFace model name - `quick` (bool; optional; default `False`): If True, test only 4 resolutions - `max_tokens` (int; optional; default `256`): Max tokens to generate - `warmup_runs` (int; optional; default `1`): Number of warmup runs - Return annotation: `list[MLLMBenchmarkResult]` - Calls: detect_hardware, print, len, tabulate, time.perf_counter, load, load_config, download_test_image, range, benchmark_mllm_resolution, get_mlx_memory_info, mlx_info.get, results.append - Return expressions: []; results ## `vllm_mlx.benchmark.print_mllm_summary` - Kind: function - Signature: `def print_mllm_summary(results: list[MLLMBenchmarkResult], model_name: str)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/benchmark.py#L916-L975 - Implementation: Function `print_mllm_summary` calls `print`, `table_data.append`, `tabulate`, `sum`; returns `None`. Print MLLM benchmark summary. - Inputs: - `results` (list[MLLMBenchmarkResult]; required): Required positional or keyword input. - `model_name` (str; required): Required positional or keyword input. - Return annotation: `not annotated` - Calls: print, table_data.append, tabulate, sum, max, min - Return expressions: None ## `vllm_mlx.benchmark.VideoBenchmarkResult` - Kind: class - Signature: `class VideoBenchmarkResult` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/benchmark.py#L984-L999 - Implementation: Class `VideoBenchmarkResult` declares 0 direct member(s). Result from a single video benchmark run. - Inputs: - `config_name` (str; required): Required constructor field. - `fps` (float; required): Required constructor field. - `max_frames` (int; required): Required constructor field. - `frames_extracted` (int; required): Required constructor field. - `video_duration` (float; required): Required constructor field. - `time_seconds` (float; required): Required constructor field. - `prompt_tokens` (int; required): Required constructor field. - `completion_tokens` (int; required): Required constructor field. - `tokens_per_second` (float; required): Required constructor field. - `response_preview` (str; required): Required constructor field. - `memory_gb` (float; optional; default `0.0`): Optional constructor field; defaults to `0.0`. - `mlx_memory_gb` (float; optional; default `0.0`): Optional constructor field; defaults to `0.0`. - Constructs: `vllm_mlx.benchmark.VideoBenchmarkResult` - Decorators: dataclass ## `vllm_mlx.benchmark.create_test_video` - Kind: function - Signature: `def create_test_video(duration: float=10.0, fps: float=30.0, width: int=640, height: int=480) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/benchmark.py#L1002-L1056 - Implementation: Function `create_test_video` calls `tempfile.NamedTemporaryFile`, `temp_file.close`, `cv2.VideoWriter_fourcc`, `cv2.VideoWriter`; returns `temp_file.name`. Create a synthetic test video with colored frames and text. - Inputs: - `duration` (float; optional; default `10.0`): Optional positional or keyword input; defaults to `10.0`. - `fps` (float; optional; default `30.0`): Optional positional or keyword input; defaults to `30.0`. - `width` (int; optional; default `640`): Optional positional or keyword input; defaults to `640`. - `height` (int; optional; default `480`): Optional positional or keyword input; defaults to `480`. - Return annotation: `str` - Calls: tempfile.NamedTemporaryFile, temp_file.close, cv2.VideoWriter_fourcc, cv2.VideoWriter, int, len, range, np.zeros, min, cv2.putText, out.write, out.release - Return expressions: temp_file.name ## `vllm_mlx.benchmark.download_video` - Kind: function - Signature: `def download_video(url: str, timeout: int=120) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/benchmark.py#L1059-L1075 - Implementation: Function `download_video` calls `print`, `requests.get`, `response.raise_for_status`, `tempfile.NamedTemporaryFile`; returns `temp_file.name`. Download video from URL and return local path. - Inputs: - `url` (str; required): Required positional or keyword input. - `timeout` (int; optional; default `120`): Optional positional or keyword input; defaults to `120`. - Return annotation: `str` - Calls: print, requests.get, response.raise_for_status, tempfile.NamedTemporaryFile, response.iter_content, temp_file.write, temp_file.close, Path(temp_file.name).stat, Path - Return expressions: temp_file.name ## `vllm_mlx.benchmark.get_video_info` - Kind: function - Signature: `def get_video_info(video_path: str) -> dict` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/benchmark.py#L1078-L1094 - Implementation: Function `get_video_info` calls `cv2.VideoCapture`, `cap.isOpened`, `int`, `cap.get`; has 2 explicit return paths. Get information about a video file. - Inputs: - `video_path` (str; required): Required positional or keyword input. - Return annotation: `dict` - Calls: cv2.VideoCapture, cap.isOpened, int, cap.get, cap.release - Return expressions: {'error': 'Cannot open video'}; info ## `vllm_mlx.benchmark.benchmark_video_config` - Kind: function - Signature: `def benchmark_video_config(model, video_path: str, fps: float, max_frames: int, config_name: str, video_info: dict, max_tokens: int=150, warmup: bool=False) -> VideoBenchmarkResult` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/benchmark.py#L1097-L1162 - Implementation: Function `benchmark_video_config` calls `reset_mlx_peak_memory`, `print`, `time.perf_counter`, `model.generate`; returns `VideoBenchmarkResult(config_name=config_name, fps=fps, max_frames=max_frames, frames_extracted=frames_extracted, video_…`. Run a single video benchmark configuration. - Inputs: - `model` (not annotated; required): Required positional or keyword input. - `video_path` (str; required): Required positional or keyword input. - `fps` (float; required): Required positional or keyword input. - `max_frames` (int; required): Required positional or keyword input. - `config_name` (str; required): Required positional or keyword input. - `video_info` (dict; required): Required positional or keyword input. - `max_tokens` (int; optional; default `150`): Optional positional or keyword input; defaults to `150`. - `warmup` (bool; optional; default `False`): Optional positional or keyword input; defaults to `False`. - Return annotation: `VideoBenchmarkResult` - Calls: reset_mlx_peak_memory, print, time.perf_counter, model.generate, int, min, get_mlx_memory_info, get_process_memory, mlx_info.get, VideoBenchmarkResult, len - Return expressions: VideoBenchmarkResult(config_name=config_name, fps=fps, max_frames=max_frames, frames_extracted=frames_extracted, video_… ## `vllm_mlx.benchmark.run_video_benchmark` - Kind: function - Signature: `def run_video_benchmark(model_name: str, video_url: str=None, video_path: str=None, quick: bool=False, max_tokens: int=150, warmup_runs: int=1) -> list[VideoBenchmarkResult]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/benchmark.py#L1165-L1285 - Implementation: Function `run_video_benchmark` calls `detect_hardware`, `print`, `len`, `tabulate`; returns `results`. Run video benchmark across multiple frame configurations. Args: model_name: HuggingFace MLLM model name video_url: URL to download video from video_path: Local video file path quick: If True, test only 3 configurations max_tokens: Max tokens to generate warmup_runs: Number of warmup runs Returns: List of VideoBenchmarkResult - Inputs: - `model_name` (str; required): HuggingFace MLLM model name - `video_url` (str; optional; default `None`): URL to download video from - `video_path` (str; optional; default `None`): Local video file path - `quick` (bool; optional; default `False`): If True, test only 3 configurations - `max_tokens` (int; optional; default `150`): Max tokens to generate - `warmup_runs` (int; optional; default `1`): Number of warmup runs - Return annotation: `list[VideoBenchmarkResult]` - Calls: detect_hardware, print, len, tabulate, time.perf_counter, MLXMultimodalLM, model.load, Path(video_path).exists, Path, download_video, get_video_info, range, benchmark_video_config, get_mlx_memory_info, mlx_info.get, results.append - Return expressions: results ## `vllm_mlx.benchmark.print_video_summary` - Kind: function - Signature: `def print_video_summary(results: list[VideoBenchmarkResult], model_name: str)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/benchmark.py#L1288-L1341 - Implementation: Function `print_video_summary` calls `print`, `sorted`, `table_data.append`, `tabulate`; returns `None`. Print video benchmark summary. - Inputs: - `results` (list[VideoBenchmarkResult]; required): Required positional or keyword input. - `model_name` (str; required): Required positional or keyword input. - Return annotation: `not annotated` - Calls: print, sorted, table_data.append, tabulate, sum, max, min - Return expressions: None ## `vllm_mlx.benchmark.print_summary` - Kind: function - Signature: `def print_summary(summary: BenchmarkSummary)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/benchmark.py#L1349-L1441 - Implementation: Function `print_summary` calls `print`, `tabulate`, `resource_data.append`. Print a formatted summary of benchmark results using tabulate. - Inputs: - `summary` (BenchmarkSummary; required): Required positional or keyword input. - Return annotation: `not annotated` - Calls: print, tabulate, resource_data.append ## `vllm_mlx.benchmark.main` - Kind: function - Signature: `def main()` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/benchmark.py#L1444-L1680 - Implementation: Function `main` calls `argparse.ArgumentParser`, `parser.add_argument`, `parser.parse_args`, `is_mllm_model`. Run the benchmark. - Inputs: none - Return annotation: `not annotated` - Calls: argparse.ArgumentParser, parser.add_argument, parser.parse_args, is_mllm_model, run_video_benchmark, print_video_summary, open, json.dump, print, run_mllm_benchmark, print_mllm_summary, run_benchmark, print_summary # Module `vllm_mlx.cli` CLI for vllm-mlx. Commands: vllm-mlx serve --port 8000 Start OpenAI-compatible server vllm-mlx bench Run benchmark Usage: vllm-mlx serve mlx-community/Llama-3.2-3B-Instruct-4bit --port 8000 vllm-mlx bench mlx-community/Llama-3.2-1B-Instruct-4bit --num-prompts 10 Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1-L2138 ## `vllm_mlx.cli.serve_command` - Kind: function - Signature: `def serve_command(args)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L22-L393 - Implementation: Function `serve_command` calls `logging.getLogger`, `getattr`, `print`, `sys.exit`. Start the OpenAI-compatible server. - Inputs: - `args` (not annotated; required): Required positional or keyword input. - Return annotation: `not annotated` - Calls: logging.getLogger, getattr, print, sys.exit, server._metrics.configure, RateLimiter, get_parser, parser_cls, logger.info, DownloadConfig, ensure_model_downloaded, is_mllm_model, server.load_embedding_model, server.load_reranker_model, SchedulerConfig, RegistryServeDefaults, load_model_registry, load_model, uvicorn.run ## `vllm_mlx.cli.download_command` - Kind: function - Signature: `def download_command(args)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L396-L410 - Implementation: Function `download_command` calls `DownloadConfig`, `print`, `ensure_model_downloaded`. Download a model to local cache without starting a server. - Inputs: - `args` (not annotated; required): Required positional or keyword input. - Return annotation: `not annotated` - Calls: DownloadConfig, print, ensure_model_downloaded ## `vllm_mlx.cli.model_command` - Kind: function - Signature: `def model_command(args)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L413-L503 - Implementation: Function `model_command` calls `inspect_model`, `acquire_model`, `AcquisitionOptions`, `convert_model`; can raise `ValueError`. Run model lifecycle helper commands. - Inputs: - `args` (not annotated; required): Required positional or keyword input. - Return annotation: `not annotated` - Calls: inspect_model, acquire_model, AcquisitionOptions, convert_model, ConversionOptions, payload.get, print, json.dumps, sys.exit, register_model, RegistrationOptions, qualify_model, QualificationOptions, ValueError - Raises directly: ValueError ## `vllm_mlx.cli.bench_command` - Kind: function - Signature: `def bench_command(args)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L506-L625 - Implementation: Function `bench_command` calls `asyncio.run`, `run_benchmark`. Run benchmark. - Inputs: - `args` (not annotated; required): Required positional or keyword input. - Return annotation: `not annotated` - Calls: asyncio.run, run_benchmark ## `vllm_mlx.cli.bench_command.run_benchmark` - Kind: nested function - Signature: `async def run_benchmark()` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L520-L623 - Implementation: Nested Function `bench_command.run_benchmark` calls `print`, `load`, `SchedulerConfig`, `EngineConfig`; awaits asynchronous work. Nested Function `bench_command.run_benchmark` calls `print`, `load`, `SchedulerConfig`, `EngineConfig`; awaits asynchronous work. - Inputs: none - Return annotation: `not annotated` - Calls: print, load, SchedulerConfig, EngineConfig, SamplingParams, len, AsyncEngineCore, asyncio.sleep, time.perf_counter, engine.add_request, request_ids.append, asyncio.gather, get_output ## `vllm_mlx.cli.bench_command.run_benchmark.get_output` - Kind: nested function - Signature: `async def get_output(rid)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L597-L601 - Implementation: Nested Function `bench_command.run_benchmark.get_output` calls `engine.stream_outputs`; has 2 explicit return paths. Nested Function `bench_command.run_benchmark.get_output` calls `engine.stream_outputs`; has 2 explicit return paths. - Inputs: - `rid` (not annotated; required): Required positional or keyword input. - Return annotation: `not annotated` - Calls: engine.stream_outputs - Return expressions: out; None ## `vllm_mlx.cli.bench_detok_command` - Kind: function - Signature: `def bench_detok_command(args)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L628-L740 - Implementation: Function `bench_detok_command` calls `print`, `load`, `generate`, `tokenizer.encode`. Benchmark streaming detokenizer optimization. - Inputs: - `args` (not annotated; required): Required positional or keyword input. - Return annotation: `not annotated` - Calls: print, load, generate, tokenizer.encode, len, range, time.perf_counter, tokenizer.decode, naive_times.append, statistics.mean, detok_class, detok.reset, detok.add_token, detok.finalize, streaming_times.append, detok.text.strip, batch_result.strip, min, repr ## `vllm_mlx.cli.bench_kv_cache_command` - Kind: function - Signature: `def bench_kv_cache_command(args)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L743-L886 - Implementation: Function `bench_kv_cache_command` calls `print`, `range`, `KVCache`, `mx.random.normal`. Benchmark KV cache quantization memory savings and quality. - Inputs: - `args` (not annotated; required): Required positional or keyword input. - Return annotation: `not annotated` - Calls: print, range, KVCache, mx.random.normal, cache.append, mx.eval, estimate_kv_cache_memory, time.perf_counter, _quantize_cache, hasattr, _dequantize_cache, zip, mx.abs(orig.keys - rest.keys).mean().item, mx.abs(orig.keys - rest.keys).mean, mx.abs, mx.abs(orig.values - rest.values).mean().item, mx.abs(orig.values - rest.values).mean, mx.abs(orig.keys - rest.keys).max().item, mx.abs(orig.keys - rest.keys).max, mx.abs(orig.values - rest.values).max().item, mx.abs(orig.values - rest.values).max, max, results.append ## `vllm_mlx.cli.bench_serve_command` - Kind: function - Signature: `def bench_serve_command(args)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L889-L990 - Implementation: Function `bench_serve_command` calls `sweep_only_warnings.append`, `print`, `', '.join`, `asyncio.run`; returns `None`. Run serving benchmark. - Inputs: - `args` (not annotated; required): Required positional or keyword input. - Return annotation: `not annotated` - Calls: sweep_only_warnings.append, print, ', '.join, asyncio.run, run_bench_serve_workload, args.prompts.split, int, args.concurrency.split, args.enable_thinking.split, v.strip().lower, v.strip, thinking_values.append, s.strip().strip, s.strip, re.split, kv.split, run_bench_serve, bool - Return expressions: None ## `vllm_mlx.cli.create_parser` - Kind: function - Signature: `def create_parser() -> argparse.ArgumentParser` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L993-L2105 - Implementation: Function `create_parser` calls `argparse.ArgumentParser`, `parser.add_subparsers`, `subparsers.add_parser`, `serve_parser.add_argument`; returns `parser`. Build the top-level CLI parser. - Inputs: none - Return annotation: `argparse.ArgumentParser` - Calls: argparse.ArgumentParser, parser.add_subparsers, subparsers.add_parser, serve_parser.add_argument, make_positive_int_arg_parser, list_parsers, ', '.join, make_json_object_arg_parser, bench_parser.add_argument, detok_parser.add_argument, kv_cache_parser.add_argument, download_parser.add_argument, model_parser.add_subparsers, model_subparsers.add_parser, model_inspect_parser.add_argument, model_acquire_parser.add_argument, model_convert_parser.add_argument, model_register_parser.add_argument, model_register_parser.add_mutually_exclusive_group, mllm_group.add_argument, model_qualify_parser.add_argument, bench_serve_parser.add_argument - Return expressions: parser ## `vllm_mlx.cli.main` - Kind: function - Signature: `def main()` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L2112-L2134 - Implementation: Function `main` calls `create_parser`, `parser.parse_args`, `serve_command`, `bench_command`. Parse the command line and dispatch to the selected vllm-mlx command. - Inputs: none - Return annotation: `not annotated` - Calls: create_parser, parser.parse_args, serve_command, bench_command, bench_detok_command, bench_kv_cache_command, download_command, model_command, bench_serve_command, parser.print_help, sys.exit # Module `vllm_mlx.cli_arg_types` Argparse type helpers shared by CLI entrypoints. Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli_arg_types.py#L1-L62 ## `vllm_mlx.cli_arg_types.parse_json_object_arg` - Kind: function - Signature: `def parse_json_object_arg(value: str, option_name: str) -> dict[str, Any]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli_arg_types.py#L10-L22 - Implementation: Function `parse_json_object_arg` calls `json.loads`, `argparse.ArgumentTypeError`, `isinstance`; can raise `argparse.ArgumentTypeError`; returns `parsed`. Parse and validate that an option value is a JSON object. - Inputs: - `value` (str; required): Required positional or keyword input. - `option_name` (str; required): Required positional or keyword input. - Return annotation: `dict[str, Any]` - Calls: json.loads, argparse.ArgumentTypeError, isinstance - Raises directly: argparse.ArgumentTypeError - Return expressions: parsed ## `vllm_mlx.cli_arg_types.make_json_object_arg_parser` - Kind: function - Signature: `def make_json_object_arg_parser(option_name: str) -> Callable[[str], dict[str, Any]]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli_arg_types.py#L25-L31 - Implementation: Function `make_json_object_arg_parser` returns `_parser`. Create an argparse type parser for JSON object options. - Inputs: - `option_name` (str; required): Required positional or keyword input. - Return annotation: `Callable[[str], dict[str, Any]]` - Return expressions: _parser ## `vllm_mlx.cli_arg_types.make_json_object_arg_parser._parser` - Kind: nested function - Signature: `def _parser(value: str) -> dict[str, Any]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli_arg_types.py#L28-L29 - Implementation: Nested Function `make_json_object_arg_parser._parser` calls `parse_json_object_arg`; returns `parse_json_object_arg(value, option_name)`. Nested Function `make_json_object_arg_parser._parser` calls `parse_json_object_arg`; returns `parse_json_object_arg(value, option_name)`. - Inputs: - `value` (str; required): Required positional or keyword input. - Return annotation: `dict[str, Any]` - Calls: parse_json_object_arg - Return expressions: parse_json_object_arg(value, option_name) ## `vllm_mlx.cli_arg_types.positive_int_arg` - Kind: function - Signature: `def positive_int_arg(value: str) -> int` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli_arg_types.py#L34-L42 - Implementation: Function `positive_int_arg` calls `int`, `argparse.ArgumentTypeError`; can raise `argparse.ArgumentTypeError`; returns `parsed`. Parse an argparse integer that must be greater than zero. - Inputs: - `value` (str; required): Required positional or keyword input. - Return annotation: `int` - Calls: int, argparse.ArgumentTypeError - Raises directly: argparse.ArgumentTypeError - Return expressions: parsed ## `vllm_mlx.cli_arg_types.parse_positive_int_arg` - Kind: function - Signature: `def parse_positive_int_arg(value: str, option_name: str) -> int` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli_arg_types.py#L45-L53 - Implementation: Function `parse_positive_int_arg` calls `int`, `argparse.ArgumentTypeError`; can raise `argparse.ArgumentTypeError`; returns `parsed`. Parse and validate that an option value is a positive integer. - Inputs: - `value` (str; required): Required positional or keyword input. - `option_name` (str; required): Required positional or keyword input. - Return annotation: `int` - Calls: int, argparse.ArgumentTypeError - Raises directly: argparse.ArgumentTypeError - Return expressions: parsed ## `vllm_mlx.cli_arg_types.make_positive_int_arg_parser` - Kind: function - Signature: `def make_positive_int_arg_parser(option_name: str) -> Callable[[str], int]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli_arg_types.py#L56-L62 - Implementation: Function `make_positive_int_arg_parser` returns `_parser`. Create an argparse type parser for positive integer options. - Inputs: - `option_name` (str; required): Required positional or keyword input. - Return annotation: `Callable[[str], int]` - Return expressions: _parser ## `vllm_mlx.cli_arg_types.make_positive_int_arg_parser._parser` - Kind: nested function - Signature: `def _parser(value: str) -> int` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli_arg_types.py#L59-L60 - Implementation: Nested Function `make_positive_int_arg_parser._parser` calls `parse_positive_int_arg`; returns `parse_positive_int_arg(value, option_name)`. Nested Function `make_positive_int_arg_parser._parser` calls `parse_positive_int_arg`; returns `parse_positive_int_arg(value, option_name)`. - Inputs: - `value` (str; required): Required positional or keyword input. - Return annotation: `int` - Calls: parse_positive_int_arg - Return expressions: parse_positive_int_arg(value, option_name) # Module `vllm_mlx.constrained` Constrained decoding for grammar-guided generation. Provides logits processors that mask token probabilities during generation so the model can only emit sequences matching a target grammar (e.g. a JSON schema). Used by the ``response_format`` parameter on the chat completion and Anthropic Messages endpoints. Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/constrained/__init__.py#L1-L23 # Module `vllm_mlx.constrained.cache` Cache of ``TokenEnforcerTokenizerData`` objects keyed by tokenizer identity. Building ``TokenEnforcerTokenizerData`` requires iterating over the entire vocabulary (up to 200k tokens on MiniMax/GLM) and decoding each token. The cost is ~1-2 seconds per model and the result is independent of the JSON schema, so we cache it for the lifetime of the process. Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/constrained/cache.py#L1-L186 ## `vllm_mlx.constrained.cache._resolve_inner_tokenizer` - Kind: function - Signature: `def _resolve_inner_tokenizer(tokenizer: Any) -> Any` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/constrained/cache.py#L26-L53 - Implementation: Function `_resolve_inner_tokenizer` calls `getattr`, `hasattr`; returns `tokenizer`. VLM processors wrap the actual tokenizer under ``processor.tokenizer``. ``mlx_lm.tokenizer_utils.TokenizerWrapper`` exposes it via ``_tokenizer``. Return the most-unwrapped tokenizer that still has the HF ``all_special_ids`` / ``eos_token_id`` surface. Note: on HF ``PreTrainedTokenizerFast``, ``_tokenizer`` points at the rust-level object which lacks ``all_special_ids``; unwrapping to that level would cause every special token (````, ````, ``\n``, ``<|think|>`` …) to leak into ``regular_tokens`` and end up in ``TokenizerPrefixTree.root`` as an always-allowed token. We only unwrap when the inner layer still exposes ``all_special_ids``. - Inputs: - `tokenizer` (Any; required): Required positional or keyword input. - Return annotation: `Any` - Calls: getattr, hasattr - Return expressions: tokenizer ## `vllm_mlx.constrained.cache._build_regular_tokens_list` - Kind: function - Signature: `def _build_regular_tokens_list(tokenizer: Any, vocab_size: int) -> list[tuple[int, str, bool]]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/constrained/cache.py#L56-L95 - Implementation: Function `_build_regular_tokens_list` calls `set`, `tokenizer.encode`, `range`, `tokenizer.decode`; returns `regular_tokens`. Enumerate the regular (non-special) tokens in the vocabulary and produce the ``(token_id, decoded_with_leading_space_marker, is_word_start)`` tuples required by ``TokenEnforcerTokenizerData``. Mirrors the reference implementation in ``lmformatenforcer.integrations. transformers`` but works with the HF tokenizer surface only (so we do not need a hard transformers dependency at the right version). - Inputs: - `tokenizer` (Any; required): Required positional or keyword input. - `vocab_size` (int; required): Required positional or keyword input. - Return annotation: `list[tuple[int, str, bool]]` - Calls: set, tokenizer.encode, range, tokenizer.decode, len, regular_tokens.append - Return expressions: regular_tokens ## `vllm_mlx.constrained.cache._get_eos_token_id` - Kind: function - Signature: `def _get_eos_token_id(tokenizer: Any) -> int | list[int]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/constrained/cache.py#L98-L111 - Implementation: Function `_get_eos_token_id` calls `getattr`, `isinstance`, `list`; has 3 explicit return paths. Function `_get_eos_token_id` calls `getattr`, `isinstance`, `list`; has 3 explicit return paths. - Inputs: - `tokenizer` (Any; required): Required positional or keyword input. - Return annotation: `int | list[int]` - Calls: getattr, isinstance, list - Return expressions: list(eos_list); eos; 0 ## `vllm_mlx.constrained.cache._get_vocab_size` - Kind: function - Signature: `def _get_vocab_size(tokenizer: Any) -> int` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/constrained/cache.py#L114-L125 - Implementation: Function `_get_vocab_size` calls `getattr`, `isinstance`, `len`, `callable`; can raise `ValueError`; has 3 explicit return paths. Function `_get_vocab_size` calls `getattr`, `isinstance`, `len`, `callable`; can raise `ValueError`; has 3 explicit return paths. - Inputs: - `tokenizer` (Any; required): Required positional or keyword input. - Return annotation: `int` - Calls: getattr, isinstance, len, callable, get_vocab, ValueError - Raises directly: ValueError - Return expressions: vs; len(tokenizer); len(get_vocab()) ## `vllm_mlx.constrained.cache._decode_function` - Kind: function - Signature: `def _decode_function(tokenizer: Any, tokens: list[int]) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/constrained/cache.py#L128-L133 - Implementation: Function `_decode_function` calls `tokenizer.decode`, `isinstance`, `decoded.rstrip`; has 2 explicit return paths. Function `_decode_function` calls `tokenizer.decode`, `isinstance`, `decoded.rstrip`; has 2 explicit return paths. - Inputs: - `tokenizer` (Any; required): Required positional or keyword input. - `tokens` (list[int]; required): Required positional or keyword input. - Return annotation: `str` - Calls: tokenizer.decode, isinstance, decoded.rstrip - Return expressions: ''; decoded.rstrip('�') if isinstance(decoded, str) else '' ## `vllm_mlx.constrained.cache.get_tokenizer_data` - Kind: function - Signature: `def get_tokenizer_data(tokenizer: Any) -> Any | None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/constrained/cache.py#L136-L180 - Implementation: Function `get_tokenizer_data` calls `_resolve_inner_tokenizer`, `id`, `_CACHE.get`, `_get_vocab_size`; has 3 explicit return paths. Return a cached ``TokenEnforcerTokenizerData`` for ``tokenizer``. Returns ``None`` if ``lm-format-enforcer`` is not installed or the tokenizer cannot be adapted. - Inputs: - `tokenizer` (Any; required): Required positional or keyword input. - Return annotation: `Any | None` - Calls: _resolve_inner_tokenizer, id, _CACHE.get, _get_vocab_size, logger.warning, _build_regular_tokens_list, functools.partial, _get_eos_token_id, TokenEnforcerTokenizerData - Return expressions: None; cached; data ## `vllm_mlx.constrained.cache.clear_cache` - Kind: function - Signature: `def clear_cache() -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/constrained/cache.py#L183-L186 - Implementation: Function `clear_cache` calls `_CACHE.clear`. Drop the cache (mainly for tests). - Inputs: none - Return annotation: `None` - Calls: _CACHE.clear # Module `vllm_mlx.constrained.json_schema_processor` ``JSONSchemaLogitsProcessor`` — a ``mlx_lm``-compatible logits processor that masks the vocabulary so the model can only emit tokens forming a valid JSON value (optionally matching a JSON schema). The processor implements the signature expected by ``mlx_lm.generate.generate_step`` and ``vllm_mlx``'s batched engine alike: processor(tokens: mx.array, logits: mx.array) -> mx.array ``tokens`` contains the full sequence generated for this request so far (prompt + previously emitted tokens), and ``logits`` is the last-step logits row. Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/constrained/json_schema_processor.py#L1-L924 ## `vllm_mlx.constrained.json_schema_processor.LMFormatEnforcerNotAvailableError` - Kind: class - Signature: `class LMFormatEnforcerNotAvailableError(RuntimeError)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/constrained/json_schema_processor.py#L33-L34 - Implementation: Class `LMFormatEnforcerNotAvailableError` derives from `RuntimeError` and declares 0 direct member(s). Raised when ``lm-format-enforcer`` is required but not installed. - Inputs: none - Constructs: `vllm_mlx.constrained.json_schema_processor.LMFormatEnforcerNotAvailableError` ## `vllm_mlx.constrained.json_schema_processor._canonical_schema_key` - Kind: function - Signature: `def _canonical_schema_key(schema: dict | None) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/constrained/json_schema_processor.py#L50-L54 - Implementation: Function `_canonical_schema_key` calls `json.dumps(schema, sort_keys=True, separators=(',', ':')).encode`, `json.dumps`, `hashlib.sha256(blob).hexdigest`, `hashlib.sha256`; has 2 explicit return paths. Function `_canonical_schema_key` calls `json.dumps(schema, sort_keys=True, separators=(',', ':')).encode`, `json.dumps`, `hashlib.sha256(blob).hexdigest`, `hashlib.sha256`; has 2 explicit return paths. - Inputs: - `schema` (dict | None; required): Required positional or keyword input. - Return annotation: `str` - Calls: json.dumps(schema, sort_keys=True, separators=(',', ':')).encode, json.dumps, hashlib.sha256(blob).hexdigest, hashlib.sha256 - Return expressions: '__none__'; hashlib.sha256(blob).hexdigest() ## `vllm_mlx.constrained.json_schema_processor._get_or_build_parser` - Kind: function - Signature: `def _get_or_build_parser(schema: dict | None) -> tuple[dict, Any]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/constrained/json_schema_processor.py#L57-L73 - Implementation: Function `_get_or_build_parser` calls `_canonical_schema_key`, `_parser_cache.get`, `_simplify_schema`, `_force_no_additional_properties`; has 2 explicit return paths. Return (parser_schema, JsonSchemaParser) for ``schema``, memoised. - Inputs: - `schema` (dict | None; required): Required positional or keyword input. - Return annotation: `tuple[dict, Any]` - Calls: _canonical_schema_key, _parser_cache.get, _simplify_schema, _force_no_additional_properties, JsonSchemaParser - Return expressions: cached; (parser_schema, parser) ## `vllm_mlx.constrained.json_schema_processor.is_available` - Kind: function - Signature: `def is_available() -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/constrained/json_schema_processor.py#L76-L82 - Implementation: Function `is_available` has 2 explicit return paths. Return ``True`` iff ``lm-format-enforcer`` is importable. - Inputs: none - Return annotation: `bool` - Return expressions: False; True ## `vllm_mlx.constrained.json_schema_processor._simplify_schema` - Kind: function - Signature: `def _simplify_schema(schema: dict) -> dict` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/constrained/json_schema_processor.py#L97-L207 - Implementation: Function `_simplify_schema` calls `copy.deepcopy`, `definitions.update`, `schema.pop`, `set`; returns `_resolve(schema)`. Pre-process a JSON Schema for ``lm-format-enforcer`` compatibility. ``lm-format-enforcer`` does not support ``$ref``, ``not``, ``type`` as an array, or recursive definitions. This function: 1. Resolves ``$ref`` by inlining referenced definitions (with cycle detection so recursive definitions are truncated to ``{}``). 2. Removes ``not`` sub-schemas (makes the schema more permissive). 3. Strips metadata / serialisation-hint keywords that the enforcer does not understand: ``default``, ``examples``, ``title``, ``description``, ``$schema``, ``$id``. 4. Converts ``type: [t1, t2, ...]`` to ``anyOf: [{type: t1}, ...]``. 5. Cleans up empty ``anyOf`` / ``oneOf`` branches. 6. Flattens nested ``anyOf``/``oneOf`` (e.g. ``anyOf: [{anyOf: [A, B]}, C]`` → ``anyOf: [A, B, C]``). - Inputs: - `schema` (dict; required): Required positional or keyword input. - Return annotation: `dict` - Calls: copy.deepcopy, definitions.update, schema.pop, set, _resolve - Return expressions: _resolve(schema) ## `vllm_mlx.constrained.json_schema_processor._simplify_schema._resolve` - Kind: nested function - Signature: `def _resolve(node: Any, depth: int=0) -> Any` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/constrained/json_schema_processor.py#L121-L205 - Implementation: Nested Function `_simplify_schema._resolve` calls `isinstance`, `ref.split`, `len`, `resolving.add`; has 3 explicit return paths. Nested Function `_simplify_schema._resolve` calls `isinstance`, `ref.split`, `len`, `resolving.add`; has 3 explicit return paths. - Inputs: - `node` (Any; required): Required positional or keyword input. - `depth` (int; optional; default `0`): Optional positional or keyword input; defaults to `0`. - Return annotation: `Any` - Calls: isinstance, ref.split, len, resolving.add, copy.deepcopy, node.items, _resolve, resolving.discard, node.pop, node.get, branches.append, list, flattened.extend, flattened.append - Return expressions: node; result; {} ## `vllm_mlx.constrained.json_schema_processor._force_no_additional_properties` - Kind: function - Signature: `def _force_no_additional_properties(schema: dict) -> dict` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/constrained/json_schema_processor.py#L210-L224 - Implementation: Function `_force_no_additional_properties` calls `copy.deepcopy`, `_inject_no_additional_props`; returns `schema`. Return a deep copy of *schema* with ``additionalProperties: false`` injected into every object-type sub-schema that declares ``properties``. ``lm-format-enforcer`` has a bug where multi-character tokens spanning JSON structural boundaries (e.g., a single token that decodes to ``""``) can produce empty or whitespace-only keys, causing ``KeyError`` crashes in ``jsonschemaparser.py``. Setting ``additionalProperties: false`` tells the enforcer's trie traversal that only the declared property names are valid keys, which significantly narrows the allowed tokens and prevents most of these boundary-spanning issues. - Inputs: - `schema` (dict; required): Required positional or keyword input. - Return annotation: `dict` - Calls: copy.deepcopy, _inject_no_additional_props - Return expressions: schema ## `vllm_mlx.constrained.json_schema_processor._inject_no_additional_props` - Kind: function - Signature: `def _inject_no_additional_props(node: Any) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/constrained/json_schema_processor.py#L227-L238 - Implementation: Function `_inject_no_additional_props` calls `isinstance`, `node.values`, `_inject_no_additional_props`; returns `None`. Recursively inject ``additionalProperties: false`` into *node*. - Inputs: - `node` (Any; required): Required positional or keyword input. - Return annotation: `None` - Calls: isinstance, node.values, _inject_no_additional_props - Return expressions: None ## `vllm_mlx.constrained.json_schema_processor._collect_property_names` - Kind: function - Signature: `def _collect_property_names(schema: dict | None) -> set[str]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/constrained/json_schema_processor.py#L241-L247 - Implementation: Function `_collect_property_names` calls `set`, `_walk_properties`; returns `names`. Collect all property names declared anywhere in *schema*. - Inputs: - `schema` (dict | None; required): Required positional or keyword input. - Return annotation: `set[str]` - Calls: set, _walk_properties - Return expressions: names ## `vllm_mlx.constrained.json_schema_processor._walk_properties` - Kind: function - Signature: `def _walk_properties(node: Any, names: set[str]) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/constrained/json_schema_processor.py#L250-L264 - Implementation: Function `_walk_properties` calls `isinstance`, `node.get`, `names.update`, `props.keys`; returns `None`. Function `_walk_properties` calls `isinstance`, `node.get`, `names.update`, `props.keys`; returns `None`. - Inputs: - `node` (Any; required): Required positional or keyword input. - `names` (set[str]; required): Required positional or keyword input. - Return annotation: `None` - Calls: isinstance, node.get, names.update, props.keys, props.values, _walk_properties - Return expressions: None ## `vllm_mlx.constrained.json_schema_processor._complete_json_eos_logits` - Kind: function - Signature: `def _complete_json_eos_logits(eos_set: set[int], suffix: list[int], logits: mx.array, is_complete_json, build_allow_mask) -> mx.array | None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/constrained/json_schema_processor.py#L267-L276 - Implementation: Function `_complete_json_eos_logits` calls `is_complete_json`, `_eos_logits`; has 2 explicit return paths. Function `_complete_json_eos_logits` calls `is_complete_json`, `_eos_logits`; has 2 explicit return paths. - Inputs: - `eos_set` (set[int]; required): Required positional or keyword input. - `suffix` (list[int]; required): Required positional or keyword input. - `logits` (mx.array; required): Required positional or keyword input. - `is_complete_json` (not annotated; required): Required positional or keyword input. - `build_allow_mask` (not annotated; required): Required positional or keyword input. - Return annotation: `mx.array | None` - Calls: is_complete_json, _eos_logits - Return expressions: None; _eos_logits(eos_set, logits, build_allow_mask) ## `vllm_mlx.constrained.json_schema_processor._eos_logits` - Kind: function - Signature: `def _eos_logits(eos_set: set[int], logits: mx.array, build_allow_mask) -> mx.array | None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/constrained/json_schema_processor.py#L279-L290 - Implementation: Function `_eos_logits` calls `build_allow_mask`, `sorted`; has 2 explicit return paths. Function `_eos_logits` calls `build_allow_mask`, `sorted`; has 2 explicit return paths. - Inputs: - `eos_set` (set[int]; required): Required positional or keyword input. - `logits` (mx.array; required): Required positional or keyword input. - `build_allow_mask` (not annotated; required): Required positional or keyword input. - Return annotation: `mx.array | None` - Calls: build_allow_mask, sorted - Return expressions: None; logits + mask ## `vllm_mlx.constrained.json_schema_processor._eos_logits_or_original` - Kind: function - Signature: `def _eos_logits_or_original(eos_set: set[int], logits: mx.array, build_allow_mask) -> mx.array` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/constrained/json_schema_processor.py#L293-L299 - Implementation: Function `_eos_logits_or_original` calls `_eos_logits`; returns `logits if masked is None else masked`. Function `_eos_logits_or_original` calls `_eos_logits`; returns `logits if masked is None else masked`. - Inputs: - `eos_set` (set[int]; required): Required positional or keyword input. - `logits` (mx.array; required): Required positional or keyword input. - `build_allow_mask` (not annotated; required): Required positional or keyword input. - Return annotation: `mx.array` - Calls: _eos_logits - Return expressions: logits if masked is None else masked ## `vllm_mlx.constrained.json_schema_processor.JSONSchemaLogitsProcessor` - Kind: class - Signature: `class JSONSchemaLogitsProcessor` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/constrained/json_schema_processor.py#L302-L924 - Implementation: Class `JSONSchemaLogitsProcessor` declares 15 direct member(s). Logits processor that constrains generation to valid JSON. Parameters ---------- schema: The JSON Schema the output must match. When ``None``, any valid JSON object/array is accepted (``json_object`` mode). tokenizer: The tokenizer used for generation. Its vocabulary is iterated once (via :mod:`vllm_mlx.constrained.cache`) and cached for subsequent requests. - Inputs: - `schema` (dict | None; required): Required positional or keyword input. - `tokenizer` (Any; required): Required positional or keyword input. - Constructs: `vllm_mlx.constrained.json_schema_processor.JSONSchemaLogitsProcessor` ## `vllm_mlx.constrained.json_schema_processor.JSONSchemaLogitsProcessor.__init__` - Kind: method - Signature: `def __init__(self, schema: dict | None, tokenizer: Any) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/constrained/json_schema_processor.py#L317-L414 - Implementation: Method `JSONSchemaLogitsProcessor.__init__` updates `self._tokenizer`, `self._schema`, `self._tok_data`, `self._disabled`; calls `is_available`, `LMFormatEnforcerNotAvailableError`, `get_tokenizer_data`, `_get_or_build_parser`; can raise `LMFormatEnforcerNotAvailableError`. Method `JSONSchemaLogitsProcessor.__init__` updates `self._tokenizer`, `self._schema`, `self._tok_data`, `self._disabled`; calls `is_available`, `LMFormatEnforcerNotAvailableError`, `get_tokenizer_data`, `_get_or_build_parser`; can raise `LMFormatEnforcerNotAvailableError`. - Inputs: - `schema` (dict | None; required): Required positional or keyword input. - `tokenizer` (Any; required): Required positional or keyword input. - Return annotation: `None` - Calls: is_available, LMFormatEnforcerNotAvailableError, get_tokenizer_data, _get_or_build_parser, TokenEnforcer, logger.warning, self._enforcer.get_allowed_tokens, _get_vocab_size, getattr, isinstance, int, set, _collect_property_names - State reads: self._tok_data, self._parser, self._disabled, self._enforcer.get_allowed_tokens, self._enforcer - State writes: self._tokenizer, self._schema, self._tok_data, self._disabled, self._parser, self._enforcer, self._prompt_len, self._vocab_size, self._eos_set, self._valid_key_first_chars, self._valid_key_names, self._token_decode_cache, self._cached_suffix_text, self._cached_suffix_len, self._json_ctx_in_string, self._json_ctx_last_quote_pos, self._json_ctx_scanned_len, self._brace_depth, self._bracket_depth, self._container_stack - Raises directly: LMFormatEnforcerNotAvailableError ## `vllm_mlx.constrained.json_schema_processor.JSONSchemaLogitsProcessor._suffix` - Kind: method - Signature: `def _suffix(self, tokens_list: list[int]) -> list[int]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/constrained/json_schema_processor.py#L418-L426 - Implementation: Method `JSONSchemaLogitsProcessor._suffix` updates `self._prompt_len`; calls `len`; returns `tokens_list[self._prompt_len:]`. Return the slice of ``tokens`` that corresponds to generated output. - Inputs: - `tokens_list` (list[int]; required): Required positional or keyword input. - Return annotation: `list[int]` - Calls: len - State reads: self._prompt_len - State writes: self._prompt_len - Return expressions: tokens_list[self._prompt_len:] ## `vllm_mlx.constrained.json_schema_processor.JSONSchemaLogitsProcessor._decode_token_cached` - Kind: method - Signature: `def _decode_token_cached(self, tok_id: int) -> str | None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/constrained/json_schema_processor.py#L428-L442 - Implementation: Method `JSONSchemaLogitsProcessor._decode_token_cached` calls `self._token_decode_cache.get`, `self._tokenizer.decode`, `isinstance`; has 3 explicit return paths. Return the decoded text for a single token (cached). - Inputs: - `tok_id` (int; required): Required positional or keyword input. - Return annotation: `str | None` - Calls: self._token_decode_cache.get, self._tokenizer.decode, isinstance - State reads: self._token_decode_cache.get, self._token_decode_cache, self._tokenizer.decode, self._tokenizer - Return expressions: cached; None; result ## `vllm_mlx.constrained.json_schema_processor.JSONSchemaLogitsProcessor._decode_suffix` - Kind: method - Signature: `def _decode_suffix(self, suffix: list[int]) -> str | None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/constrained/json_schema_processor.py#L444-L493 - Implementation: Method `JSONSchemaLogitsProcessor._decode_suffix` updates `self._cached_suffix_text`, `self._cached_suffix_len`, `self._json_ctx_scanned_len`, `self._json_ctx_in_string`; calls `len`, `self._tokenizer.decode`, `list`, `isinstance`; has 4 explicit return paths. Decode suffix tokens to text. Always uses full ``tokenizer.decode(suffix)`` which is correct for all tokenizer families (BPE, SentencePiece, etc.). Per-token concatenation is NOT safe because whitespace may be encoded as a token prefix (e.g. ``decode([1526]) = "world"`` but in context ``decode([22557, 1526]) = "Hello world"``). Results are cached by suffix length to avoid redundant decodes within the same generation step (``_get_json_context`` and ``_suffix_is_complete_json`` both call this method). - Inputs: - `suffix` (list[int]; required): Required positional or keyword input. - Return annotation: `str | None` - Calls: len, self._tokenizer.decode, list, isinstance, result.startswith - State reads: self._cached_suffix_len, self._cached_suffix_text, self._tokenizer.decode, self._tokenizer - State writes: self._cached_suffix_text, self._cached_suffix_len, self._json_ctx_scanned_len, self._json_ctx_in_string, self._json_ctx_last_quote_pos, self._brace_depth, self._bracket_depth, self._container_stack - Return expressions: ''; self._cached_suffix_text; None; result ## `vllm_mlx.constrained.json_schema_processor.JSONSchemaLogitsProcessor._suffix_is_complete_json` - Kind: method - Signature: `def _suffix_is_complete_json(self, suffix: list[int]) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/constrained/json_schema_processor.py#L495-L520 - Implementation: Method `JSONSchemaLogitsProcessor._suffix_is_complete_json` calls `self._decode_suffix`, `text.strip`, `json.loads`; has 2 explicit return paths. Return True if the decoded ``suffix`` parses as a complete JSON value. Uses cached bracket/brace depth from ``_get_json_context`` as a fast pre-check: JSON cannot be complete when brackets are unbalanced or we are inside a string. This avoids the expensive ``json.loads`` call on ~99% of steps. - Inputs: - `suffix` (list[int]; required): Required positional or keyword input. - Return annotation: `bool` - Calls: self._decode_suffix, text.strip, json.loads - State reads: self._brace_depth, self._bracket_depth, self._json_ctx_in_string, self._decode_suffix - Return expressions: False; True ## `vllm_mlx.constrained.json_schema_processor.JSONSchemaLogitsProcessor._get_json_context` - Kind: method - Signature: `def _get_json_context(self, suffix: list[int]) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/constrained/json_schema_processor.py#L522-L643 - Implementation: Method `JSONSchemaLogitsProcessor._get_json_context` updates `self._json_ctx_in_string`, `self._json_ctx_last_quote_pos`, `self._json_ctx_scanned_len`, `self._brace_depth`; calls `self._decode_suffix`, `len`, `container_stack.append`, `container_stack.pop`; has 3 explicit return paths. Determine the JSON structural context of the current suffix. Processes only newly appended characters instead of re-scanning the full decoded text on every call (O(1) amortised per step instead of O(n)). Returns one of: - ``"key_start"``: expecting a new key (after ``{`` or ``,``) - ``"in_key"``: inside an open key string - ``"other"``: any other position - Inputs: - `suffix` (list[int]; required): Required positional or keyword input. - Return annotation: `str` - Calls: self._decode_suffix, len, container_stack.append, container_stack.pop, text[:self._json_ctx_last_quote_pos].rstrip, text.rstrip - State reads: self._decode_suffix, self._json_ctx_scanned_len, self._json_ctx_in_string, self._json_ctx_last_quote_pos, self._brace_depth, self._bracket_depth, self._container_stack - State writes: self._json_ctx_in_string, self._json_ctx_last_quote_pos, self._json_ctx_scanned_len, self._brace_depth, self._bracket_depth, self._container_stack - Return expressions: 'other'; 'in_key'; 'key_start' ## `vllm_mlx.constrained.json_schema_processor.JSONSchemaLogitsProcessor._filter_at_key_context` - Kind: method - Signature: `def _filter_at_key_context(self, context: str, suffix: list[int], allowed: list[int]) -> list[int]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/constrained/json_schema_processor.py#L645-L662 - Implementation: Method `JSONSchemaLogitsProcessor._filter_at_key_context` calls `self._filter_key_start_tokens`, `self._filter_in_key_tokens`; has 3 explicit return paths. Apply schema-aware filtering when in key-related context. At ``key_start``: only allow tokens that begin a valid key, whitespace, ``}``, or just ``"``. At ``in_key``: only allow tokens compatible with continuing a valid property name (no leading whitespace; content must be a valid prefix). - Inputs: - `context` (str; required): Required positional or keyword input. - `suffix` (list[int]; required): Required positional or keyword input. - `allowed` (list[int]; required): Required positional or keyword input. - Return annotation: `list[int]` - Calls: self._filter_key_start_tokens, self._filter_in_key_tokens - State reads: self._valid_key_names, self._filter_key_start_tokens, self._filter_in_key_tokens - Return expressions: allowed; self._filter_key_start_tokens(suffix, allowed); self._filter_in_key_tokens(suffix, allowed) ## `vllm_mlx.constrained.json_schema_processor.JSONSchemaLogitsProcessor._filter_key_start_tokens` - Kind: method - Signature: `def _filter_key_start_tokens(self, suffix: list[int], allowed: list[int]) -> list[int]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/constrained/json_schema_processor.py#L664-L717 - Implementation: Method `JSONSchemaLogitsProcessor._filter_key_start_tokens` calls `self._decode_token_cached`, `result.append`, `tok_text.lstrip`, `rest.find`; returns `result if result else allowed`. Filter tokens at key-start position. Only permit tokens that: - Are whitespace-only (before the key ``"``) - Decode to ``}`` (close object) - Start a valid key: ``"`` followed by a valid first char - Inputs: - `suffix` (list[int]; required): Required positional or keyword input. - `allowed` (list[int]; required): Required positional or keyword input. - Return annotation: `list[int]` - Calls: self._decode_token_cached, result.append, tok_text.lstrip, rest.find, self._is_valid_key_prefix - State reads: self._eos_set, self._decode_token_cached, self._valid_key_first_chars, self._is_valid_key_prefix, self._valid_key_names - Return expressions: result if result else allowed ## `vllm_mlx.constrained.json_schema_processor.JSONSchemaLogitsProcessor._filter_in_key_tokens` - Kind: method - Signature: `def _filter_in_key_tokens(self, suffix: list[int], allowed: list[int]) -> list[int]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/constrained/json_schema_processor.py#L719-L758 - Implementation: Method `JSONSchemaLogitsProcessor._filter_in_key_tokens` calls `self._decode_suffix`, `text.rfind`, `self._decode_token_cached`, `result.append`; has 2 explicit return paths. Filter tokens when we're inside an open key string. Only allow tokens whose content continues a valid property name. Reject whitespace-only/leading-whitespace tokens. - Inputs: - `suffix` (list[int]; required): Required positional or keyword input. - `allowed` (list[int]; required): Required positional or keyword input. - Return annotation: `list[int]` - Calls: self._decode_suffix, text.rfind, self._decode_token_cached, result.append, tok_text.find, self._is_valid_key_prefix - State reads: self._decode_suffix, self._decode_token_cached, self._valid_key_names, self._is_valid_key_prefix - Return expressions: allowed; result if result else allowed ## `vllm_mlx.constrained.json_schema_processor.JSONSchemaLogitsProcessor._is_valid_key_prefix` - Kind: method - Signature: `def _is_valid_key_prefix(self, prefix: str) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/constrained/json_schema_processor.py#L760-L762 - Implementation: Method `JSONSchemaLogitsProcessor._is_valid_key_prefix` calls `any`, `name.startswith`; returns `any((name.startswith(prefix) for name in self._valid_key_names))`. Return True if *prefix* is a prefix of at least one valid key name. - Inputs: - `prefix` (str; required): Required positional or keyword input. - Return annotation: `bool` - Calls: any, name.startswith - State reads: self._valid_key_names - Return expressions: any((name.startswith(prefix) for name in self._valid_key_names)) ## `vllm_mlx.constrained.json_schema_processor.JSONSchemaLogitsProcessor._filter_nonprogress_whitespace_tokens` - Kind: method - Signature: `def _filter_nonprogress_whitespace_tokens(self, suffix: list[int], allowed: list[int]) -> list[int]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/constrained/json_schema_processor.py#L764-L793 - Implementation: Method `JSONSchemaLogitsProcessor._filter_nonprogress_whitespace_tokens` calls `self._decode_suffix`, `len`, `text.rstrip`, `self._decode_token_cached`; has 2 explicit return paths. Stop constrained JSON from spending a long run on pure whitespace. JSON permits arbitrary whitespace around structural tokens. That is valid, but with non-streaming requests a model can keep selecting whitespace-only tokens for minutes without producing useful JSON content. Once the decoded suffix has a long trailing whitespace run outside a string, remove pure-whitespace tokens from the next-step allowed set so generation must make structural/content progress. - Inputs: - `suffix` (list[int]; required): Required positional or keyword input. - `allowed` (list[int]; required): Required positional or keyword input. - Return annotation: `list[int]` - Calls: self._decode_suffix, len, text.rstrip, self._decode_token_cached, filtered.append, all - State reads: self._decode_suffix, self._decode_token_cached - Return expressions: allowed; filtered if filtered else allowed ## `vllm_mlx.constrained.json_schema_processor.JSONSchemaLogitsProcessor._build_allow_mask` - Kind: method - Signature: `def _build_allow_mask(self, allowed: list[int], vocab_size: int) -> mx.array` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/constrained/json_schema_processor.py#L795-L810 - Implementation: Method `JSONSchemaLogitsProcessor._build_allow_mask` calls `mx.full`, `float`, `np.full`, `mx.array`; has 2 explicit return paths. Build a 1-D mask of length ``vocab_size`` where allowed positions are ``0`` and disallowed positions are ``-inf``. Uses numpy for mask construction (C-level speed) instead of a Python loop over ``vocab_size`` elements. - Inputs: - `allowed` (list[int]; required): Required positional or keyword input. - `vocab_size` (int; required): Required positional or keyword input. - Return annotation: `mx.array` - Calls: mx.full, float, np.full, mx.array - Return expressions: mx.full((vocab_size,), -float('inf')); mx.array(buf) ## `vllm_mlx.constrained.json_schema_processor.JSONSchemaLogitsProcessor.__call__` - Kind: method - Signature: `def __call__(self, tokens: mx.array, logits: mx.array) -> mx.array` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/constrained/json_schema_processor.py#L814-L910 - Implementation: Method `JSONSchemaLogitsProcessor.__call__` updates `self._disabled`; calls `_eos_logits_or_original`, `hasattr`, `tokens.tolist`, `list`; has 4 explicit return paths. Apply the allowed-tokens mask to ``logits``. - Inputs: - `tokens` (mx.array; required): Required positional or keyword input. - `logits` (mx.array; required): Required positional or keyword input. - Return annotation: `mx.array` - Calls: _eos_logits_or_original, hasattr, tokens.tolist, list, isinstance, self._suffix, _complete_json_eos_logits, self._enforcer.get_allowed_tokens, getattr, self._get_json_context, self._filter_nonprogress_whitespace_tokens, self._filter_at_key_context, any, self._suffix_is_complete_json, sorted, self._tokenizer.decode, logger.warning, len, self._build_allow_mask, logger.error - State reads: self._disabled, self._eos_set, self._build_allow_mask, self._suffix, self._suffix_is_complete_json, self._prompt_len, self._enforcer.get_allowed_tokens, self._enforcer, self._get_json_context, self._json_ctx_in_string, self._filter_nonprogress_whitespace_tokens, self._filter_at_key_context, self._tokenizer.decode, self._tokenizer - State writes: self._disabled - Return expressions: _eos_logits_or_original(self._eos_set, logits, self._build_allow_mask); eos_logits; logits; logits + mask ## `vllm_mlx.constrained.json_schema_processor.JSONSchemaLogitsProcessor.schema` - Kind: method - Signature: `def schema(self) -> dict | None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/constrained/json_schema_processor.py#L915-L918 - Implementation: Method `JSONSchemaLogitsProcessor.schema` returns `self._schema`. Return the normalized JSON Schema enforced for this request. - Inputs: none - Return annotation: `dict | None` - Decorators: property - State reads: self._schema - Return expressions: self._schema ## `vllm_mlx.constrained.json_schema_processor.JSONSchemaLogitsProcessor.vocab_size` - Kind: method - Signature: `def vocab_size(self) -> int` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/constrained/json_schema_processor.py#L921-L924 - Implementation: Method `JSONSchemaLogitsProcessor.vocab_size` returns `self._vocab_size`. Return the tokenizer vocabulary size used to construct masks. - Inputs: none - Return annotation: `int` - Decorators: property - State reads: self._vocab_size - Return expressions: self._vocab_size # Module `vllm_mlx.constrained.thinking_processor` Thinking-aware logits processor for reasoning models. Manages the full thinking lifecycle: budget enforcement, phase transitions, and content-phase constrained decoding delegation. Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/constrained/thinking_processor.py#L1-L287 ## `vllm_mlx.constrained.thinking_processor.BoundedSuffixMatcher` - Kind: class - Signature: `class BoundedSuffixMatcher` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/constrained/thinking_processor.py#L16-L48 - Implementation: Class `BoundedSuffixMatcher` declares 5 direct member(s). Detect a target token sequence in a stream using a rolling suffix buffer. Unlike a naive sequential matcher that resets to position 0 on mismatch, this uses a bounded buffer that catches overlapping prefixes. - Inputs: - `target_ids` (list[int]; required): Required positional or keyword input. - Constructs: `vllm_mlx.constrained.thinking_processor.BoundedSuffixMatcher` ## `vllm_mlx.constrained.thinking_processor.BoundedSuffixMatcher.__init__` - Kind: method - Signature: `def __init__(self, target_ids: list[int]) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/constrained/thinking_processor.py#L25-L30 - Implementation: Method `BoundedSuffixMatcher.__init__` updates `self.target`, `self._max_len`, `self._buf`; calls `ValueError`, `tuple`, `len`, `deque`; can raise `ValueError`. Method `BoundedSuffixMatcher.__init__` updates `self.target`, `self._max_len`, `self._buf`; calls `ValueError`, `tuple`, `len`, `deque`; can raise `ValueError`. - Inputs: - `target_ids` (list[int]; required): Required positional or keyword input. - Return annotation: `None` - Calls: ValueError, tuple, len, deque - State reads: self._max_len - State writes: self.target, self._max_len, self._buf - Raises directly: ValueError ## `vllm_mlx.constrained.thinking_processor.BoundedSuffixMatcher.feed` - Kind: method - Signature: `def feed(self, token_id: int) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/constrained/thinking_processor.py#L32-L35 - Implementation: Method `BoundedSuffixMatcher.feed` calls `self._buf.append`, `len`, `tuple`; returns `len(self._buf) == self._max_len and tuple(self._buf) == self.target`. Feed one token. Returns True when the buffer suffix equals the target. - Inputs: - `token_id` (int; required): Required positional or keyword input. - Return annotation: `bool` - Calls: self._buf.append, len, tuple - State reads: self._buf.append, self._buf, self._max_len, self.target - Return expressions: len(self._buf) == self._max_len and tuple(self._buf) == self.target ## `vllm_mlx.constrained.thinking_processor.BoundedSuffixMatcher.reset` - Kind: method - Signature: `def reset(self) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/constrained/thinking_processor.py#L37-L39 - Implementation: Method `BoundedSuffixMatcher.reset` calls `self._buf.clear`. Clear the buffer. - Inputs: none - Return annotation: `None` - Calls: self._buf.clear - State reads: self._buf.clear, self._buf ## `vllm_mlx.constrained.thinking_processor.BoundedSuffixMatcher.snapshot` - Kind: method - Signature: `def snapshot(self) -> tuple[int, ...]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/constrained/thinking_processor.py#L41-L43 - Implementation: Method `BoundedSuffixMatcher.snapshot` calls `tuple`; returns `tuple(self._buf)`. Return a serializable copy of the current suffix buffer. - Inputs: none - Return annotation: `tuple[int, ...]` - Calls: tuple - State reads: self._buf - Return expressions: tuple(self._buf) ## `vllm_mlx.constrained.thinking_processor.BoundedSuffixMatcher.restore` - Kind: method - Signature: `def restore(self, state: tuple[int, ...]) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/constrained/thinking_processor.py#L45-L48 - Implementation: Method `BoundedSuffixMatcher.restore` calls `self._buf.clear`, `self._buf.extend`. Restore the suffix buffer from a previous snapshot. - Inputs: - `state` (tuple[int, ...]; required): Required positional or keyword input. - Return annotation: `None` - Calls: self._buf.clear, self._buf.extend - State reads: self._buf.clear, self._buf, self._buf.extend ## `vllm_mlx.constrained.thinking_processor.Phase` - Kind: class - Signature: `class Phase(enum.Enum)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/constrained/thinking_processor.py#L51-L57 - Implementation: Class `Phase` derives from `enum.Enum` and declares 0 direct member(s). Thinking lifecycle phases. - Inputs: none - Constructs: `vllm_mlx.constrained.thinking_processor.Phase` ## `vllm_mlx.constrained.thinking_processor.ThinkingAwareLogitsProcessor` - Kind: class - Signature: `class ThinkingAwareLogitsProcessor` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/constrained/thinking_processor.py#L60-L287 - Implementation: Class `ThinkingAwareLogitsProcessor` declares 12 direct member(s). Unified logits processor for thinking-model lifecycle management. Manages a four-phase state machine: IDLE -> THINKING -> TRANSITIONING -> CONTENT - IDLE: before reasoning start tokens. Pass through. - THINKING: inside reasoning span. Count tokens, pass through. - TRANSITIONING: forcing reasoning end sequence via logits masking. - CONTENT: after reasoning closed. Delegate to inner processor. No re-entry into THINKING after CONTENT is reached. - Inputs: - `start_token_ids` (list[int]; required): Required positional or keyword input. - `end_token_ids` (list[int]; required): Required positional or keyword input. - `thinking_token_budget` (int; required): Required positional or keyword input. - `inner` (Callable[[mx.array, mx.array], mx.array] | None; optional; default `None`): Optional positional or keyword input; defaults to `None`. - `vocab_size` (int; optional; default `152064`): Optional positional or keyword input; defaults to `152064`. - `prompt_has_think_tag` (bool; optional; default `False`): Optional positional or keyword input; defaults to `False`. - `no_final_content_token_limit` (int | None; optional; default `None`): Optional positional or keyword input; defaults to `None`. - Constructs: `vllm_mlx.constrained.thinking_processor.ThinkingAwareLogitsProcessor` ## `vllm_mlx.constrained.thinking_processor.ThinkingAwareLogitsProcessor.__init__` - Kind: method - Signature: `def __init__(self, start_token_ids: list[int], end_token_ids: list[int], thinking_token_budget: int, inner: Callable[[mx.array, mx.array], mx.array] | None=None, vocab_size: int=152064, prompt_has_think_tag: bool=False, no_final_content_token_limit: int | None=None) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/constrained/thinking_processor.py#L92-L129 - Implementation: Method `ThinkingAwareLogitsProcessor.__init__` updates `self._start_matcher`, `self._end_matcher`, `self._end_token_ids`, `self._content_phase_mask_ids`; calls `BoundedSuffixMatcher`, `list`, `tuple`, `dict.fromkeys`. Method `ThinkingAwareLogitsProcessor.__init__` updates `self._start_matcher`, `self._end_matcher`, `self._end_token_ids`, `self._content_phase_mask_ids`; calls `BoundedSuffixMatcher`, `list`, `tuple`, `dict.fromkeys`. - Inputs: - `start_token_ids` (list[int]; required): Required positional or keyword input. - `end_token_ids` (list[int]; required): Required positional or keyword input. - `thinking_token_budget` (int; required): Required positional or keyword input. - `inner` (Callable[[mx.array, mx.array], mx.array] | None; optional; default `None`): Optional positional or keyword input; defaults to `None`. - `vocab_size` (int; optional; default `152064`): Optional positional or keyword input; defaults to `152064`. - `prompt_has_think_tag` (bool; optional; default `False`): Optional positional or keyword input; defaults to `False`. - `no_final_content_token_limit` (int | None; optional; default `None`): Optional positional or keyword input; defaults to `None`. - Return annotation: `None` - Calls: BoundedSuffixMatcher, list, tuple, dict.fromkeys, self._snapshot_state - State reads: self._snapshot_state - State writes: self._start_matcher, self._end_matcher, self._end_token_ids, self._content_phase_mask_ids, self._thinking_token_budget, self._inner, self._vocab_size, self._thinking_tokens, self._transition_index, self.watchdog_was_enforced, self._no_final_content_token_limit, self._state, self._processed_len, self._processed_token_ids, self._snapshots ## `vllm_mlx.constrained.thinking_processor.ThinkingAwareLogitsProcessor.state` - Kind: method - Signature: `def state(self) -> Phase` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/constrained/thinking_processor.py#L132-L135 - Implementation: Method `ThinkingAwareLogitsProcessor.state` returns `self._state`. Return the current reasoning lifecycle phase. - Inputs: none - Return annotation: `Phase` - Decorators: property - State reads: self._state - Return expressions: self._state ## `vllm_mlx.constrained.thinking_processor.ThinkingAwareLogitsProcessor.thinking_tokens` - Kind: method - Signature: `def thinking_tokens(self) -> int` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/constrained/thinking_processor.py#L138-L141 - Implementation: Method `ThinkingAwareLogitsProcessor.thinking_tokens` returns `self._thinking_tokens`. Return the number of generated tokens counted as reasoning. - Inputs: none - Return annotation: `int` - Decorators: property - State reads: self._thinking_tokens - Return expressions: self._thinking_tokens ## `vllm_mlx.constrained.thinking_processor.ThinkingAwareLogitsProcessor.is_retired` - Kind: method - Signature: `def is_retired(self) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/constrained/thinking_processor.py#L144-L150 - Implementation: Method `ThinkingAwareLogitsProcessor.is_retired` returns `self._state == Phase.CONTENT and self._inner is None`. True when the processor is in CONTENT with no inner constraint. The engine can use this signal to drop the processor and re-enable MTP for the remaining content generation (Phase 2 optimization). - Inputs: none - Return annotation: `bool` - Decorators: property - State reads: self._state, self._inner - Return expressions: self._state == Phase.CONTENT and self._inner is None ## `vllm_mlx.constrained.thinking_processor.ThinkingAwareLogitsProcessor.__call__` - Kind: method - Signature: `def __call__(self, tokens: mx.array, logits: mx.array) -> mx.array` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/constrained/thinking_processor.py#L152-L170 - Implementation: Method `ThinkingAwareLogitsProcessor.__call__` calls `self._force_transition`, `self._call_inner`, `self._sync_to_tokens`; has 3 explicit return paths. Method `ThinkingAwareLogitsProcessor.__call__` calls `self._force_transition`, `self._call_inner`, `self._sync_to_tokens`; has 3 explicit return paths. - Inputs: - `tokens` (mx.array; required): Required positional or keyword input. - `logits` (mx.array; required): Required positional or keyword input. - Return annotation: `mx.array` - Calls: self._force_transition, self._call_inner, self._sync_to_tokens - State reads: self._state, self._force_transition, self._call_inner, self._sync_to_tokens - Return expressions: self._force_transition(logits); self._call_inner(tokens, logits); logits ## `vllm_mlx.constrained.thinking_processor.ThinkingAwareLogitsProcessor._force_transition` - Kind: method - Signature: `def _force_transition(self, logits: mx.array) -> mx.array` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/constrained/thinking_processor.py#L172-L182 - Implementation: Method `ThinkingAwareLogitsProcessor._force_transition` calls `mx.full`, `float`; returns `masked`. Force the next token in the reasoning end sequence. - Inputs: - `logits` (mx.array; required): Required positional or keyword input. - Return annotation: `mx.array` - Calls: mx.full, float - State reads: self._end_token_ids, self._transition_index - Return expressions: masked ## `vllm_mlx.constrained.thinking_processor.ThinkingAwareLogitsProcessor._call_inner` - Kind: method - Signature: `def _call_inner(self, tokens: mx.array, logits: mx.array) -> mx.array` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/constrained/thinking_processor.py#L184-L188 - Implementation: Method `ThinkingAwareLogitsProcessor._call_inner` calls `self._inner`, `self._mask_content_phase_control_tokens`; returns `self._mask_content_phase_control_tokens(logits)`. Delegate to inner processor if present. - Inputs: - `tokens` (mx.array; required): Required positional or keyword input. - `logits` (mx.array; required): Required positional or keyword input. - Return annotation: `mx.array` - Calls: self._inner, self._mask_content_phase_control_tokens - State reads: self._inner, self._mask_content_phase_control_tokens - Return expressions: self._mask_content_phase_control_tokens(logits) ## `vllm_mlx.constrained.thinking_processor.ThinkingAwareLogitsProcessor._mask_content_phase_control_tokens` - Kind: method - Signature: `def _mask_content_phase_control_tokens(self, logits: mx.array) -> mx.array` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/constrained/thinking_processor.py#L190-L197 - Implementation: Method `ThinkingAwareLogitsProcessor._mask_content_phase_control_tokens` calls `float`; returns `logits`. Prevent reserved think-tag starts from leaking into final content. - Inputs: - `logits` (mx.array; required): Required positional or keyword input. - Return annotation: `mx.array` - Calls: float - State reads: self._content_phase_mask_ids - Return expressions: logits ## `vllm_mlx.constrained.thinking_processor.ThinkingAwareLogitsProcessor._snapshot_state` - Kind: method - Signature: `def _snapshot_state(self) -> tuple[Phase, int, int, tuple[int, ...], tuple[int, ...], bool]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/constrained/thinking_processor.py#L199-L209 - Implementation: Method `ThinkingAwareLogitsProcessor._snapshot_state` calls `self._start_matcher.snapshot`, `self._end_matcher.snapshot`; returns `(self._state, self._thinking_tokens, self._transition_index, self._start_matcher.snapshot(), self._end_matcher.snapshot…`. Method `ThinkingAwareLogitsProcessor._snapshot_state` calls `self._start_matcher.snapshot`, `self._end_matcher.snapshot`; returns `(self._state, self._thinking_tokens, self._transition_index, self._start_matcher.snapshot(), self._end_matcher.snapshot…`. - Inputs: none - Return annotation: `tuple[Phase, int, int, tuple[int, ...], tuple[int, ...], bool]` - Calls: self._start_matcher.snapshot, self._end_matcher.snapshot - State reads: self._state, self._thinking_tokens, self._transition_index, self._start_matcher.snapshot, self._start_matcher, self._end_matcher.snapshot, self._end_matcher, self.watchdog_was_enforced - Return expressions: (self._state, self._thinking_tokens, self._transition_index, self._start_matcher.snapshot(), self._end_matcher.snapshot… ## `vllm_mlx.constrained.thinking_processor.ThinkingAwareLogitsProcessor._restore_snapshot` - Kind: method - Signature: `def _restore_snapshot(self, processed_len: int) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/constrained/thinking_processor.py#L211-L229 - Implementation: Method `ThinkingAwareLogitsProcessor._restore_snapshot` updates `self._state`, `self._thinking_tokens`, `self._transition_index`, `self.watchdog_was_enforced`; calls `min`, `len`, `self._start_matcher.restore`, `self._end_matcher.restore`. Method `ThinkingAwareLogitsProcessor._restore_snapshot` updates `self._state`, `self._thinking_tokens`, `self._transition_index`, `self.watchdog_was_enforced`; calls `min`, `len`, `self._start_matcher.restore`, `self._end_matcher.restore`. - Inputs: - `processed_len` (int; required): Required positional or keyword input. - Return annotation: `None` - Calls: min, len, self._start_matcher.restore, self._end_matcher.restore - State reads: self._snapshots, self._start_matcher.restore, self._start_matcher, self._end_matcher.restore, self._end_matcher, self._processed_token_ids - State writes: self._state, self._thinking_tokens, self._transition_index, self.watchdog_was_enforced, self._processed_len, self._processed_token_ids, self._snapshots ## `vllm_mlx.constrained.thinking_processor.ThinkingAwareLogitsProcessor._sync_to_tokens` - Kind: method - Signature: `def _sync_to_tokens(self, tokens: mx.array) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/constrained/thinking_processor.py#L231-L252 - Implementation: Method `ThinkingAwareLogitsProcessor._sync_to_tokens` updates `self._processed_len`; calls `int`, `tokens.tolist`, `min`, `self._restore_snapshot`; returns `None`. Method `ThinkingAwareLogitsProcessor._sync_to_tokens` updates `self._processed_len`; calls `int`, `tokens.tolist`, `min`, `self._restore_snapshot`; returns `None`. - Inputs: - `tokens` (mx.array; required): Required positional or keyword input. - Return annotation: `None` - Calls: int, tokens.tolist, min, self._restore_snapshot, self._advance_with_token, self._processed_token_ids.append, self._snapshots.append, self._snapshot_state - State reads: self._processed_len, self._processed_token_ids, self._restore_snapshot, self._advance_with_token, self._processed_token_ids.append, self._state, self._snapshots.append, self._snapshots, self._snapshot_state - State writes: self._processed_len - Return expressions: None ## `vllm_mlx.constrained.thinking_processor.ThinkingAwareLogitsProcessor._advance_with_token` - Kind: method - Signature: `def _advance_with_token(self, token_id: int) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/constrained/thinking_processor.py#L254-L287 - Implementation: Method `ThinkingAwareLogitsProcessor._advance_with_token` updates `self._state`, `self._transition_index`, `self._thinking_tokens`, `self.watchdog_was_enforced`; calls `self._start_matcher.feed`, `self._end_matcher.feed`, `len`, `self._end_matcher.reset`; returns `None`. Method `ThinkingAwareLogitsProcessor._advance_with_token` updates `self._state`, `self._transition_index`, `self._thinking_tokens`, `self.watchdog_was_enforced`; calls `self._start_matcher.feed`, `self._end_matcher.feed`, `len`, `self._end_matcher.reset`; returns `None`. - Inputs: - `token_id` (int; required): Required positional or keyword input. - Return annotation: `None` - Calls: self._start_matcher.feed, self._end_matcher.feed, len, self._end_matcher.reset - State reads: self._state, self._start_matcher.feed, self._start_matcher, self._thinking_token_budget, self._end_matcher.feed, self._end_matcher, self._thinking_tokens, self._no_final_content_token_limit, self._end_token_ids, self._transition_index, self._end_matcher.reset - State writes: self._state, self._transition_index, self._thinking_tokens, self.watchdog_was_enforced - Return expressions: None # Module `vllm_mlx.embedding` Embedding engine using mlx-embeddings. Provides lazy-loaded model management and batch embedding generation for the OpenAI-compatible /v1/embeddings endpoint. Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/embedding.py#L1-L131 ## `vllm_mlx.embedding.EmbeddingEngine` - Kind: class - Signature: `class EmbeddingEngine` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/embedding.py#L19-L131 - Implementation: Class `EmbeddingEngine` declares 7 direct member(s). Wrapper around mlx-embeddings for text embedding generation. Supports lazy model loading and batch embedding with proper tokenization and pooling. - Inputs: - `model_name` (str; required): Required positional or keyword input. - Constructs: `vllm_mlx.embedding.EmbeddingEngine` ## `vllm_mlx.embedding.EmbeddingEngine.__init__` - Kind: method - Signature: `def __init__(self, model_name: str)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/embedding.py#L27-L31 - Implementation: Method `EmbeddingEngine.__init__` updates `self.model_name`, `self._model`, `self._tokenizer`, `self._max_length`. Method `EmbeddingEngine.__init__` updates `self.model_name`, `self._model`, `self._tokenizer`, `self._max_length`. - Inputs: - `model_name` (str; required): Required positional or keyword input. - Return annotation: `not annotated` - State writes: self.model_name, self._model, self._tokenizer, self._max_length ## `vllm_mlx.embedding.EmbeddingEngine.is_loaded` - Kind: method - Signature: `def is_loaded(self) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/embedding.py#L34-L37 - Implementation: Method `EmbeddingEngine.is_loaded` returns `self._model is not None`. Return whether the embedding model has been loaded. - Inputs: none - Return annotation: `bool` - Decorators: property - State reads: self._model - Return expressions: self._model is not None ## `vllm_mlx.embedding.EmbeddingEngine.load` - Kind: method - Signature: `def load(self) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/embedding.py#L39-L47 - Implementation: Method `EmbeddingEngine.load` updates `self._model`, `self._tokenizer`; calls `logger.info`, `time.perf_counter`, `load`. Load the embedding model and tokenizer. - Inputs: none - Return annotation: `None` - Calls: logger.info, time.perf_counter, load - State reads: self.model_name - State writes: self._model, self._tokenizer ## `vllm_mlx.embedding.EmbeddingEngine._ensure_loaded` - Kind: method - Signature: `def _ensure_loaded(self) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/embedding.py#L49-L51 - Implementation: Method `EmbeddingEngine._ensure_loaded` calls `self.load`. Method `EmbeddingEngine._ensure_loaded` calls `self.load`. - Inputs: none - Return annotation: `None` - Calls: self.load - State reads: self.is_loaded, self.load ## `vllm_mlx.embedding.EmbeddingEngine._resolve_max_length` - Kind: method - Signature: `def _resolve_max_length(self) -> int` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/embedding.py#L53-L60 - Implementation: Method `EmbeddingEngine._resolve_max_length` updates `self._max_length`; calls `resolve_max_length`, `getattr`; returns `self._max_length`. Tokenizer truncation length from the model config (cached). - Inputs: none - Return annotation: `int` - Calls: resolve_max_length, getattr - State reads: self._max_length, self._model, self._tokenizer - State writes: self._max_length - Return expressions: self._max_length ## `vllm_mlx.embedding.EmbeddingEngine.embed` - Kind: method - Signature: `def embed(self, texts: str | list[str]) -> list[list[float]]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/embedding.py#L62-L109 - Implementation: Method `EmbeddingEngine.embed` calls `self._ensure_loaded`, `isinstance`, `inner_tokenizer`, `inner_tok`; returns `result`. Generate embeddings for one or more texts. Args: texts: A single string or list of strings. Returns: List of embedding vectors (one per input text). - Inputs: - `texts` (str | list[str]; required): A single string or list of strings. - Return annotation: `list[list[float]]` - Calls: self._ensure_loaded, isinstance, inner_tokenizer, inner_tok, self._resolve_max_length, mx.array, self._model, embeds.tolist, mx.clear_cache - State reads: self._ensure_loaded, self._tokenizer, self._resolve_max_length, self._model - Return expressions: result ## `vllm_mlx.embedding.EmbeddingEngine.count_tokens` - Kind: method - Signature: `def count_tokens(self, texts: str | list[str]) -> int` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/embedding.py#L111-L131 - Implementation: Method `EmbeddingEngine.count_tokens` calls `self._ensure_loaded`, `isinstance`, `self._tokenizer.encode`, `len`; returns `total`. Approximate token count for usage reporting. - Inputs: - `texts` (str | list[str]; required): Required positional or keyword input. - Return annotation: `int` - Calls: self._ensure_loaded, isinstance, self._tokenizer.encode, len, hasattr, max - State reads: self._ensure_loaded, self._tokenizer.encode, self._tokenizer - Return expressions: total # Module `vllm_mlx.endpoint_model_policies` Request-time model resolution policies for optional endpoints. These endpoints intentionally do not expose arbitrary Hugging Face loading from user-controlled request bodies. Unknown model names must be rejected before any engine instantiation or download path is reached. Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/endpoint_model_policies.py#L1-L118 ## `vllm_mlx.endpoint_model_policies._with_identity_aliases` - Kind: function - Signature: `def _with_identity_aliases(model_map: dict[str, str]) -> dict[str, str]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/endpoint_model_policies.py#L42-L46 - Implementation: Function `_with_identity_aliases` calls `dict`, `model_map.values`; returns `expanded`. Function `_with_identity_aliases` calls `dict`, `model_map.values`; returns `expanded`. - Inputs: - `model_map` (dict[str, str]; required): Required positional or keyword input. - Return annotation: `dict[str, str]` - Calls: dict, model_map.values - Return expressions: expanded ## `vllm_mlx.endpoint_model_policies._reject_unknown_embedding_model` - Kind: function - Signature: `def _reject_unknown_embedding_model(requested_model: str) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/endpoint_model_policies.py#L53-L63 - Implementation: Function `_reject_unknown_embedding_model` calls `', '.join`, `sorted`, `HTTPException`; can raise `HTTPException`. Function `_reject_unknown_embedding_model` calls `', '.join`, `sorted`, `HTTPException`; can raise `HTTPException`. - Inputs: - `requested_model` (str; required): Required positional or keyword input. - Return annotation: `None` - Calls: ', '.join, sorted, HTTPException - Raises directly: HTTPException ## `vllm_mlx.endpoint_model_policies._reject_unknown_audio_model` - Kind: function - Signature: `def _reject_unknown_audio_model(endpoint: str, requested_model: str, supported_aliases: dict[str, str]) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/endpoint_model_policies.py#L66-L79 - Implementation: Function `_reject_unknown_audio_model` calls `', '.join`, `sorted`, `HTTPException`; can raise `HTTPException`. Function `_reject_unknown_audio_model` calls `', '.join`, `sorted`, `HTTPException`; can raise `HTTPException`. - Inputs: - `endpoint` (str; required): Required positional or keyword input. - `requested_model` (str; required): Required positional or keyword input. - `supported_aliases` (dict[str, str]; required): Required positional or keyword input. - Return annotation: `None` - Calls: ', '.join, sorted, HTTPException - Raises directly: HTTPException ## `vllm_mlx.endpoint_model_policies.resolve_embedding_model_name` - Kind: function - Signature: `def resolve_embedding_model_name(requested_model: str, *, locked_model: str | None=None) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/endpoint_model_policies.py#L82-L104 - Implementation: Function `resolve_embedding_model_name` calls `HTTPException`, `_reject_unknown_embedding_model`; can raise `HTTPException`; has 2 explicit return paths. Resolve the embedding model for a request or raise HTTP 400. - Inputs: - `requested_model` (str; required): Required positional or keyword input. - `locked_model` (str | None; optional; default `None`): Optional keyword-only input; defaults to `None`. - Return annotation: `str` - Calls: HTTPException, _reject_unknown_embedding_model - Raises directly: HTTPException - Return expressions: locked_model; requested_model ## `vllm_mlx.endpoint_model_policies.resolve_stt_model_name` - Kind: function - Signature: `def resolve_stt_model_name(requested_model: str) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/endpoint_model_policies.py#L107-L111 - Implementation: Function `resolve_stt_model_name` calls `_reject_unknown_audio_model`; returns `_STT_MODEL_MAP[requested_model]`. Resolve an STT request model alias or configured model ID. - Inputs: - `requested_model` (str; required): Required positional or keyword input. - Return annotation: `str` - Calls: _reject_unknown_audio_model - Return expressions: _STT_MODEL_MAP[requested_model] ## `vllm_mlx.endpoint_model_policies.resolve_tts_model_name` - Kind: function - Signature: `def resolve_tts_model_name(requested_model: str) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/endpoint_model_policies.py#L114-L118 - Implementation: Function `resolve_tts_model_name` calls `_reject_unknown_audio_model`; returns `_TTS_MODEL_MAP[requested_model]`. Resolve a TTS request model alias or configured model ID. - Inputs: - `requested_model` (str; required): Required positional or keyword input. - Return annotation: `str` - Calls: _reject_unknown_audio_model - Return expressions: _TTS_MODEL_MAP[requested_model] # Module `vllm_mlx.engine` Engine abstraction for vllm-mlx inference. The package stays intentionally light at import time so server- and contract-level tests can import API modules without eagerly importing MLX, engine_core, or the batched engine stack. Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/__init__.py#L1-L54 ## `vllm_mlx.engine.__getattr__` - Kind: function - Signature: `def __getattr__(name: str)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/__init__.py#L34-L54 - Implementation: Function `__getattr__` calls `AttributeError`; can raise `AttributeError`; has 3 explicit return paths. Function `__getattr__` calls `AttributeError`; can raise `AttributeError`; has 3 explicit return paths. - Inputs: - `name` (str; required): Required positional or keyword input. - Return annotation: `not annotated` - Calls: AttributeError - Raises directly: AttributeError - Return expressions: SimpleEngine; BatchedEngine; {'EngineCore': EngineCore, 'AsyncEngineCore': AsyncEngineCore, 'EngineConfig': EngineConfig}[name] # Module `vllm_mlx.engine.base` Base engine interface for vllm-mlx inference. Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/base.py#L1-L288 ## `vllm_mlx.engine.base.GenerationOutput` - Kind: class - Signature: `class GenerationOutput` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/base.py#L18-L37 - Implementation: Class `GenerationOutput` declares 0 direct member(s). Output from generation. Compatible with both simple and batched engines. - Inputs: - `text` (str; required): Required constructor field. - `tokens` (list[int]; optional; default `field(default_factory=list)`): Optional constructor field; defaults to `field(default_factory=list)`. - `prompt_tokens` (int; optional; default `0`): Optional constructor field; defaults to `0`. - `completion_tokens` (int; optional; default `0`): Optional constructor field; defaults to `0`. - `finish_reason` (str | None; optional; default `'stop'`): Optional constructor field; defaults to `'stop'`. - `mtp_drafts` (int; optional; default `0`): Optional constructor field; defaults to `0`. - `mtp_accepted` (int; optional; default `0`): Optional constructor field; defaults to `0`. - `new_text` (str; optional; default `''`): Optional constructor field; defaults to `''`. - `finished` (bool; optional; default `True`): Optional constructor field; defaults to `True`. - `mtp_drafts` (int; optional; default `0`): Optional constructor field; defaults to `0`. - `mtp_accepted` (int; optional; default `0`): Optional constructor field; defaults to `0`. - Constructs: `vllm_mlx.engine.base.GenerationOutput` - Decorators: dataclass ## `vllm_mlx.engine.base.EngineBusy` - Kind: class - Signature: `class EngineBusy(RuntimeError)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/base.py#L40-L43 - Implementation: Class `EngineBusy` derives from `RuntimeError` and declares 0 direct member(s). Raised when a serialized engine route is already serving a request. - Inputs: none - Constructs: `vllm_mlx.engine.base.EngineBusy` ## `vllm_mlx.engine.base.suspend_cancellation` - Kind: function - Signature: `def suspend_cancellation()` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/base.py#L47-L67 - Implementation: Function `suspend_cancellation` calls `asyncio.current_task`, `getattr`, `cancelling`, `range`; yields values incrementally; returns `None`. Temporarily clear task cancellation so cleanup can finish deterministically. - Inputs: none - Return annotation: `not annotated` - Decorators: contextmanager - Calls: asyncio.current_task, getattr, cancelling, range, uncancel, task.cancel - Return expressions: None ## `vllm_mlx.engine.base.run_blocking_startup_work` - Kind: function - Signature: `async def run_blocking_startup_work(work: Callable[[], Any]) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/base.py#L70-L84 - Implementation: Function `run_blocking_startup_work` calls `asyncio.create_task`, `asyncio.to_thread`, `asyncio.shield`, `suspend_cancellation`; awaits asynchronous work. Run blocking startup work off-loop without leaking cancellation races. - Inputs: - `work` (Callable[[], Any]; required): Required positional or keyword input. - Return annotation: `None` - Calls: asyncio.create_task, asyncio.to_thread, asyncio.shield, suspend_cancellation, task.done ## `vllm_mlx.engine.base.cleanup_startup_cancellation` - Kind: function - Signature: `async def cleanup_startup_cancellation(cleanup: Callable[[], Awaitable[None]]) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/base.py#L87-L98 - Implementation: Function `cleanup_startup_cancellation` calls `suspend_cancellation`, `cleanup`, `isinstance`, `logger.error`; awaits asynchronous work. Run startup cleanup without letting cleanup failures replace cancellation. - Inputs: - `cleanup` (Callable[[], Awaitable[None]]; required): Required positional or keyword input. - Return annotation: `None` - Calls: suspend_cancellation, cleanup, isinstance, logger.error, type ## `vllm_mlx.engine.base.BaseEngine` - Kind: class - Signature: `class BaseEngine(ABC)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/base.py#L101-L288 - Implementation: Class `BaseEngine` derives from `ABC` and declares 16 direct member(s). Abstract base class for inference engines. Both SimpleEngine and BatchedEngine implement this interface, allowing the server to use either without code changes. - Inputs: none - Constructs: `vllm_mlx.engine.base.BaseEngine` ## `vllm_mlx.engine.base.BaseEngine.model_name` - Kind: method - Signature: `def model_name(self) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/base.py#L111-L113 - Implementation: Method `BaseEngine.model_name` contains no state mutation, call, raise, return, await, or yield. Get the model name. - Inputs: none - Return annotation: `str` - Decorators: property, abstractmethod ## `vllm_mlx.engine.base.BaseEngine.is_mllm` - Kind: method - Signature: `def is_mllm(self) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/base.py#L117-L119 - Implementation: Method `BaseEngine.is_mllm` contains no state mutation, call, raise, return, await, or yield. Check if this is a multimodal model. - Inputs: none - Return annotation: `bool` - Decorators: property, abstractmethod ## `vllm_mlx.engine.base.BaseEngine.tokenizer` - Kind: method - Signature: `def tokenizer(self) -> Any` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/base.py#L123-L125 - Implementation: Method `BaseEngine.tokenizer` contains no state mutation, call, raise, return, await, or yield. Get the tokenizer. - Inputs: none - Return annotation: `Any` - Decorators: property, abstractmethod ## `vllm_mlx.engine.base.BaseEngine.preserve_native_tool_format` - Kind: method - Signature: `def preserve_native_tool_format(self) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/base.py#L128-L135 - Implementation: Method `BaseEngine.preserve_native_tool_format` calls `getattr`; returns `getattr(self, '_preserve_native_tool_format', False)`. Whether to preserve native tool message format. When True, role="tool" messages and tool_calls fields are preserved instead of being converted to text. Set by server based on tool parser. - Inputs: none - Return annotation: `bool` - Decorators: property - Calls: getattr - Return expressions: getattr(self, '_preserve_native_tool_format', False) ## `vllm_mlx.engine.base.BaseEngine.preserve_native_tool_format` - Kind: method - Signature: `def preserve_native_tool_format(self, value: bool) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/base.py#L138-L141 - Implementation: Method `BaseEngine.preserve_native_tool_format` updates `self._preserve_native_tool_format`. Enable or disable preservation of model-native tool messages. - Inputs: - `value` (bool; required): Required positional or keyword input. - Return annotation: `None` - Decorators: preserve_native_tool_format.setter - State writes: self._preserve_native_tool_format ## `vllm_mlx.engine.base.BaseEngine.prepare_for_start` - Kind: method - Signature: `def prepare_for_start(self) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/base.py#L143-L150 - Implementation: Method `BaseEngine.prepare_for_start` returns `None`. Run blocking startup work before async engine start. Engines can override this to perform heavyweight synchronous model loads off the serving event loop. The default implementation is a no-op so lightweight engines do not need extra plumbing. - Inputs: none - Return annotation: `None` - Return expressions: None ## `vllm_mlx.engine.base.BaseEngine.start` - Kind: method - Signature: `async def start(self) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/base.py#L153-L155 - Implementation: Method `BaseEngine.start` contains no state mutation, call, raise, return, await, or yield. Start the engine (load model if not loaded). - Inputs: none - Return annotation: `None` - Decorators: abstractmethod ## `vllm_mlx.engine.base.BaseEngine.stop` - Kind: method - Signature: `async def stop(self) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/base.py#L158-L160 - Implementation: Method `BaseEngine.stop` contains no state mutation, call, raise, return, await, or yield. Stop the engine and cleanup resources. - Inputs: none - Return annotation: `None` - Decorators: abstractmethod ## `vllm_mlx.engine.base.BaseEngine.generate` - Kind: method - Signature: `async def generate(self, prompt: str, max_tokens: int=256, temperature: float=0.7, top_p: float=0.9, stop: list[str] | None=None, **kwargs) -> GenerationOutput` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/base.py#L163-L186 - Implementation: Method `BaseEngine.generate` contains no state mutation, call, raise, return, await, or yield. Generate a complete response (non-streaming). Args: prompt: Input text max_tokens: Maximum tokens to generate temperature: Sampling temperature top_p: Top-p sampling stop: Stop sequences **kwargs: Additional model-specific parameters Returns: GenerationOutput with complete text - Inputs: - `prompt` (str; required): Input text - `max_tokens` (int; optional; default `256`): Maximum tokens to generate - `temperature` (float; optional; default `0.7`): Sampling temperature - `top_p` (float; optional; default `0.9`): Top-p sampling - `stop` (list[str] | None; optional; default `None`): Stop sequences - `**kwargs` (not annotated; optional): Additional model-specific parameters - Return annotation: `GenerationOutput` - Decorators: abstractmethod ## `vllm_mlx.engine.base.BaseEngine.stream_generate` - Kind: method - Signature: `async def stream_generate(self, prompt: str, max_tokens: int=256, temperature: float=0.7, top_p: float=0.9, stop: list[str] | None=None, **kwargs) -> AsyncIterator[GenerationOutput]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/base.py#L189-L212 - Implementation: Method `BaseEngine.stream_generate` contains no state mutation, call, raise, return, await, or yield. Stream generation token by token. Args: prompt: Input text max_tokens: Maximum tokens to generate temperature: Sampling temperature top_p: Top-p sampling stop: Stop sequences **kwargs: Additional model-specific parameters Yields: GenerationOutput with incremental text - Inputs: - `prompt` (str; required): Input text - `max_tokens` (int; optional; default `256`): Maximum tokens to generate - `temperature` (float; optional; default `0.7`): Sampling temperature - `top_p` (float; optional; default `0.9`): Top-p sampling - `stop` (list[str] | None; optional; default `None`): Stop sequences - `**kwargs` (not annotated; optional): Additional model-specific parameters - Return annotation: `AsyncIterator[GenerationOutput]` - Decorators: abstractmethod ## `vllm_mlx.engine.base.BaseEngine.chat` - Kind: method - Signature: `async def chat(self, messages: list[dict[str, Any]], max_tokens: int=256, temperature: float=0.7, top_p: float=0.9, tools: list[dict] | None=None, images: list[str] | None=None, videos: list[str] | None=None, **kwargs) -> GenerationOutput` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/base.py#L215-L242 - Implementation: Method `BaseEngine.chat` contains no state mutation, call, raise, return, await, or yield. Chat completion (non-streaming). Args: messages: List of chat messages max_tokens: Maximum tokens to generate temperature: Sampling temperature top_p: Top-p sampling tools: Optional tool definitions images: Optional image URLs/paths videos: Optional video URLs/paths **kwargs: Additional model-specific parameters Returns: GenerationOutput with assistant response - Inputs: - `messages` (list[dict[str, Any]]; required): List of chat messages - `max_tokens` (int; optional; default `256`): Maximum tokens to generate - `temperature` (float; optional; default `0.7`): Sampling temperature - `top_p` (float; optional; default `0.9`): Top-p sampling - `tools` (list[dict] | None; optional; default `None`): Optional tool definitions - `images` (list[str] | None; optional; default `None`): Optional image URLs/paths - `videos` (list[str] | None; optional; default `None`): Optional video URLs/paths - `**kwargs` (not annotated; optional): Additional model-specific parameters - Return annotation: `GenerationOutput` - Decorators: abstractmethod ## `vllm_mlx.engine.base.BaseEngine.stream_chat` - Kind: method - Signature: `async def stream_chat(self, messages: list[dict[str, Any]], max_tokens: int=256, temperature: float=0.7, top_p: float=0.9, tools: list[dict] | None=None, images: list[str] | None=None, videos: list[str] | None=None, **kwargs) -> AsyncIterator[GenerationOutput]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/base.py#L245-L272 - Implementation: Method `BaseEngine.stream_chat` contains no state mutation, call, raise, return, await, or yield. Stream chat completion token by token. Args: messages: List of chat messages max_tokens: Maximum tokens to generate temperature: Sampling temperature top_p: Top-p sampling tools: Optional tool definitions images: Optional image URLs/paths videos: Optional video URLs/paths **kwargs: Additional model-specific parameters Yields: GenerationOutput with incremental text - Inputs: - `messages` (list[dict[str, Any]]; required): List of chat messages - `max_tokens` (int; optional; default `256`): Maximum tokens to generate - `temperature` (float; optional; default `0.7`): Sampling temperature - `top_p` (float; optional; default `0.9`): Top-p sampling - `tools` (list[dict] | None; optional; default `None`): Optional tool definitions - `images` (list[str] | None; optional; default `None`): Optional image URLs/paths - `videos` (list[str] | None; optional; default `None`): Optional video URLs/paths - `**kwargs` (not annotated; optional): Additional model-specific parameters - Return annotation: `AsyncIterator[GenerationOutput]` - Decorators: abstractmethod ## `vllm_mlx.engine.base.BaseEngine.get_stats` - Kind: method - Signature: `def get_stats(self) -> dict[str, Any]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/base.py#L274-L276 - Implementation: Method `BaseEngine.get_stats` returns `{}`. Get engine statistics. Override in subclasses. - Inputs: none - Return annotation: `dict[str, Any]` - Return expressions: {} ## `vllm_mlx.engine.base.BaseEngine.get_cache_stats` - Kind: method - Signature: `def get_cache_stats(self) -> dict[str, Any] | None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/base.py#L278-L280 - Implementation: Method `BaseEngine.get_cache_stats` returns `None`. Get cache statistics. Override in subclasses. - Inputs: none - Return annotation: `dict[str, Any] | None` - Return expressions: None ## `vllm_mlx.engine.base.BaseEngine.clear_runtime_caches` - Kind: method - Signature: `def clear_runtime_caches(self) -> dict[str, Any] | None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/base.py#L282-L284 - Implementation: Method `BaseEngine.clear_runtime_caches` returns `None`. Clear engine-managed runtime caches. Override in subclasses. - Inputs: none - Return annotation: `dict[str, Any] | None` - Return expressions: None ## `vllm_mlx.engine.base.BaseEngine.abort_request` - Kind: method - Signature: `async def abort_request(self, request_id: str) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/base.py#L286-L288 - Implementation: Method `BaseEngine.abort_request` returns `False`. Abort an active or queued request when the engine supports it. - Inputs: - `request_id` (str; required): Required positional or keyword input. - Return annotation: `bool` - Return expressions: False # Module `vllm_mlx.engine.batched` Batched engine for continuous batching with multiple concurrent users. This engine wraps AsyncEngineCore to provide continuous batching for better throughput when serving multiple concurrent requests. For MLLM models, all requests (text-only and multimodal) are routed through the MLLMScheduler, which handles vision encoding and batched generation via MLLMBatchGenerator. MLLM models only initialise the MLLM scheduler (not the LLM engine), so text-only requests must also be routed through it. Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/batched.py#L1-L1231 ## `vllm_mlx.engine.batched._resolve_metal_buffer_cache_limit` - Kind: function - Signature: `def _resolve_metal_buffer_cache_limit(max_recommended: int, gpu_memory_utilization: float) -> tuple[int, str]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/batched.py#L35-L58 - Implementation: Function `_resolve_metal_buffer_cache_limit` calls `os.environ.get`, `int`, `logger.warning`; has 2 explicit return paths. Resolve the MLX retained-buffer cache cap for Metal startup. - Inputs: - `max_recommended` (int; required): Required positional or keyword input. - `gpu_memory_utilization` (float; required): Required positional or keyword input. - Return annotation: `tuple[int, str]` - Calls: os.environ.get, int, logger.warning - Return expressions: (limit, 'MLX_BUFFER_CACHE_LIMIT'); (int(max_recommended * gpu_memory_utilization), 'device-scaled') ## `vllm_mlx.engine.batched._normalize_tool_call_arguments_for_template` - Kind: function - Signature: `def _normalize_tool_call_arguments_for_template(messages: list[dict]) -> list[dict]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/batched.py#L61-L63 - Implementation: Function `_normalize_tool_call_arguments_for_template` calls `normalize_messages_for_chat_template`; returns `normalize_messages_for_chat_template(messages)`. Normalize OpenAI tool-call replay for templates expecting mappings. - Inputs: - `messages` (list[dict]; required): Required positional or keyword input. - Return annotation: `list[dict]` - Calls: normalize_messages_for_chat_template - Return expressions: normalize_messages_for_chat_template(messages) ## `vllm_mlx.engine.batched._extract_media_from_messages` - Kind: function - Signature: `def _extract_media_from_messages(messages: list[dict[str, Any]]) -> tuple` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/batched.py#L66-L137 - Implementation: Function `_extract_media_from_messages` calls `msg.get`, `isinstance`, `hasattr`, `item.model_dump`; returns `(has_media, images, videos, audios)`. Extract images, videos, and audio from OpenAI-format messages. Returns: Tuple of (has_media, images_list, videos_list, audios_list) - Inputs: - `messages` (list[dict[str, Any]]; required): Required positional or keyword input. - Return annotation: `tuple` - Calls: msg.get, isinstance, hasattr, item.model_dump, item.dict().items, item.dict, item.get, images.append, img_url.get, videos.append, vid_url.get, audios.append, audio_url.get, bool - Return expressions: (has_media, images, videos, audios) ## `vllm_mlx.engine.batched.MLLMModelWrapper` - Kind: class - Signature: `class MLLMModelWrapper` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/batched.py#L140-L175 - Implementation: Class `MLLMModelWrapper` declares 3 direct member(s). Wrapper for MLLM models to make them compatible with BatchGenerator. BatchGenerator expects model output to be subscriptable (logits array), but MLLM models return LanguageModelOutput objects. This wrapper extracts the logits from the output. Also handles Gemma 3's required pixel_values argument by injecting None for text-only requests. - Inputs: - `model` (not annotated; required): Required positional or keyword input. - Constructs: `vllm_mlx.engine.batched.MLLMModelWrapper` ## `vllm_mlx.engine.batched.MLLMModelWrapper.__init__` - Kind: method - Signature: `def __init__(self, model)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/batched.py#L152-L158 - Implementation: Method `MLLMModelWrapper.__init__` updates `self._model`, `self._is_gemma3`; calls `hasattr`, `str(getattr(model, 'model_type', '')).lower`, `str`, `getattr`. Method `MLLMModelWrapper.__init__` updates `self._model`, `self._is_gemma3`; calls `hasattr`, `str(getattr(model, 'model_type', '')).lower`, `str`, `getattr`. - Inputs: - `model` (not annotated; required): Required positional or keyword input. - Return annotation: `not annotated` - Calls: hasattr, str(getattr(model, 'model_type', '')).lower, str, getattr - State writes: self._model, self._is_gemma3 ## `vllm_mlx.engine.batched.MLLMModelWrapper.__call__` - Kind: method - Signature: `def __call__(self, *args, **kwargs)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/batched.py#L160-L171 - Implementation: Method `MLLMModelWrapper.__call__` calls `self._model`, `hasattr`; has 2 explicit return paths. Call the model and extract logits from LanguageModelOutput. - Inputs: - `*args` (not annotated; optional): Additional variadic positional inputs accepted by this callable. - `**kwargs` (not annotated; optional): Additional variadic keyword inputs accepted by this callable. - Return annotation: `not annotated` - Calls: self._model, hasattr - State reads: self._is_gemma3, self._model - Return expressions: output.logits; output ## `vllm_mlx.engine.batched.MLLMModelWrapper.__getattr__` - Kind: method - Signature: `def __getattr__(self, name)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/batched.py#L173-L175 - Implementation: Method `MLLMModelWrapper.__getattr__` calls `getattr`; returns `getattr(self._model, name)`. Forward all other attributes to the wrapped model. - Inputs: - `name` (not annotated; required): Required positional or keyword input. - Return annotation: `not annotated` - Calls: getattr - State reads: self._model - Return expressions: getattr(self._model, name) ## `vllm_mlx.engine.batched.BatchedEngine` - Kind: class - Signature: `class BatchedEngine(BaseEngine)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/batched.py#L178-L1231 - Implementation: Class `BatchedEngine` derives from `BaseEngine` and declares 28 direct member(s). Batched engine for continuous batching. This engine provides better throughput when serving multiple concurrent users by batching requests together. For MLLM (multimodal) models, this engine uses MLLMScheduler which handles images and videos alongside text generation. - Inputs: - `model_name` (str; required): HuggingFace model name or local path - `trust_remote_code` (bool; optional; default `False`): Whether to trust remote code - `scheduler_config` (Any | None; optional; default `None`): Optional scheduler configuration - `stream_interval` (int; optional; default `1`): Tokens to batch before streaming (1=every token) - `force_mllm` (bool; optional; default `False`): Force loading as MLLM even if not auto-detected - `gpu_memory_utilization` (float; optional; default `0.9`): Fraction of device memory for Metal allocation limit and emergency threshold (0.0-1.0, default 0.90) - Constructs: `vllm_mlx.engine.batched.BatchedEngine` ## `vllm_mlx.engine.batched.BatchedEngine.__init__` - Kind: method - Signature: `def __init__(self, model_name: str, trust_remote_code: bool=False, scheduler_config: Any | None=None, stream_interval: int=1, force_mllm: bool=False, gpu_memory_utilization: float=0.9)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/batched.py#L189-L224 - Implementation: Method `BatchedEngine.__init__` updates `self._model_name`, `self._created_at`, `self._trust_remote_code`, `self._scheduler_config`; calls `time.time`, `is_mllm_model`. Initialize the batched engine. Args: model_name: HuggingFace model name or local path trust_remote_code: Whether to trust remote code scheduler_config: Optional scheduler configuration stream_interval: Tokens to batch before streaming (1=every token) force_mllm: Force loading as MLLM even if not auto-detected gpu_memory_utilization: Fraction of device memory for Metal allocation limit and emergency threshold (0.0-1.0, default 0.90) - Inputs: - `model_name` (str; required): HuggingFace model name or local path - `trust_remote_code` (bool; optional; default `False`): Whether to trust remote code - `scheduler_config` (Any | None; optional; default `None`): Optional scheduler configuration - `stream_interval` (int; optional; default `1`): Tokens to batch before streaming (1=every token) - `force_mllm` (bool; optional; default `False`): Force loading as MLLM even if not auto-detected - `gpu_memory_utilization` (float; optional; default `0.9`): Fraction of device memory for Metal allocation limit and emergency threshold (0.0-1.0, default 0.90) - Return annotation: `not annotated` - Calls: time.time, is_mllm_model - State writes: self._model_name, self._created_at, self._trust_remote_code, self._scheduler_config, self._stream_interval, self._gpu_memory_utilization, self._is_mllm, self._model, self._processor, self._tokenizer, self._engine, self._mllm_scheduler, self._mllm_instance, self._loaded ## `vllm_mlx.engine.batched.BatchedEngine.model_name` - Kind: method - Signature: `def model_name(self) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/batched.py#L227-L229 - Implementation: Method `BatchedEngine.model_name` returns `self._model_name`. Get the model name. - Inputs: none - Return annotation: `str` - Decorators: property - State reads: self._model_name - Return expressions: self._model_name ## `vllm_mlx.engine.batched.BatchedEngine.is_mllm` - Kind: method - Signature: `def is_mllm(self) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/batched.py#L232-L234 - Implementation: Method `BatchedEngine.is_mllm` returns `self._is_mllm`. Check if this is a multimodal model. - Inputs: none - Return annotation: `bool` - Decorators: property - State reads: self._is_mllm - Return expressions: self._is_mllm ## `vllm_mlx.engine.batched.BatchedEngine.tokenizer` - Kind: method - Signature: `def tokenizer(self) -> Any` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/batched.py#L237-L241 - Implementation: Method `BatchedEngine.tokenizer` calls `getattr`; has 2 explicit return paths. Get the tokenizer. - Inputs: none - Return annotation: `Any` - Decorators: property - Calls: getattr - State reads: self._is_mllm, self._processor, self._tokenizer - Return expressions: getattr(self._processor, 'tokenizer', self._processor); self._tokenizer ## `vllm_mlx.engine.batched.BatchedEngine.prepare_for_start` - Kind: method - Signature: `def prepare_for_start(self) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/batched.py#L243-L251 - Implementation: Method `BatchedEngine.prepare_for_start` calls `self._prepare_mllm_model`, `self._prepare_llm_model`; returns `None`. Load heavyweight model state off the serving event loop. - Inputs: none - Return annotation: `None` - Calls: self._prepare_mllm_model, self._prepare_llm_model - State reads: self._model, self._is_mllm, self._prepare_mllm_model, self._prepare_llm_model - Return expressions: None ## `vllm_mlx.engine.batched.BatchedEngine.start` - Kind: method - Signature: `async def start(self) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/batched.py#L253-L282 - Implementation: Method `BatchedEngine.start` updates `self._loaded`; calls `self._uses_default_prepare_for_start`, `self.prepare_for_start`, `run_blocking_startup_work`, `self._start_mllm`; awaits asynchronous work; returns `None`. Start the engine (load model if not loaded). - Inputs: none - Return annotation: `None` - Calls: self._uses_default_prepare_for_start, self.prepare_for_start, run_blocking_startup_work, self._start_mllm, self._start_llm, logger.info, cleanup_startup_cancellation - State reads: self._loaded, self._model, self._uses_default_prepare_for_start, self.prepare_for_start, self._is_mllm, self._start_mllm, self._start_llm, self._model_name, self.stop - State writes: self._loaded - Return expressions: None ## `vllm_mlx.engine.batched.BatchedEngine._uses_default_prepare_for_start` - Kind: method - Signature: `def _uses_default_prepare_for_start(self) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/batched.py#L284-L287 - Implementation: Method `BatchedEngine._uses_default_prepare_for_start` calls `getattr`; returns `method is BatchedEngine.prepare_for_start`. Return True when prepare_for_start is the class implementation. - Inputs: none - Return annotation: `bool` - Calls: getattr - State reads: self.prepare_for_start - Return expressions: method is BatchedEngine.prepare_for_start ## `vllm_mlx.engine.batched.BatchedEngine._prepare_mllm_model` - Kind: method - Signature: `def _prepare_mllm_model(self) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/batched.py#L289-L334 - Implementation: Method `BatchedEngine._prepare_mllm_model` updates `self._mllm_instance`, `self._model`, `self._processor`; calls `getattr`, `MLXMultimodalLM`, `self._mllm_instance.load`, `mx.metal.is_available`. Load the MLLM model before scheduler startup. - Inputs: none - Return annotation: `None` - Calls: getattr, MLXMultimodalLM, self._mllm_instance.load, mx.metal.is_available, mx.device_info, device_info.get, int, _resolve_metal_buffer_cache_limit, mx.set_memory_limit, mx.set_cache_limit, logger.info, logger.warning, self._inject_mtp_mllm - State reads: self._scheduler_config, self._model_name, self._trust_remote_code, self._mllm_instance.load, self._mllm_instance, self._mllm_instance.model, self._mllm_instance.processor, self._gpu_memory_utilization, self._scheduler_config.enable_mtp, self._inject_mtp_mllm - State writes: self._mllm_instance, self._model, self._processor ## `vllm_mlx.engine.batched.BatchedEngine._start_mllm` - Kind: method - Signature: `async def _start_mllm(self) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/batched.py#L336-L429 - Implementation: Method `BatchedEngine._start_mllm` updates `self._mllm_scheduler`; calls `self._prepare_mllm_model`, `hasattr`, `getattr`, `MLLMSchedulerConfig`; awaits asynchronous work. Start the MLLM engine with MLLMScheduler (continuous batching). - Inputs: none - Return annotation: `None` - Calls: self._prepare_mllm_model, hasattr, getattr, MLLMSchedulerConfig, MLLMScheduler, self._mllm_scheduler.start, logger.info - State reads: self._model, self._processor, self._prepare_mllm_model, self._scheduler_config, self._scheduler_config.max_num_seqs, self._scheduler_config.enable_mtp, self._mllm_scheduler.start, self._mllm_scheduler - State writes: self._mllm_scheduler ## `vllm_mlx.engine.batched.BatchedEngine._inject_mtp_mllm` - Kind: method - Signature: `def _inject_mtp_mllm(self) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/batched.py#L431-L477 - Implementation: Method `BatchedEngine._inject_mtp_mllm` calls `Path`, `_download`, `config_path.exists`, `logger.warning`; returns `None`. Inject MTP weights into the MLLM model's language_model. - Inputs: none - Return annotation: `None` - Calls: Path, _download, config_path.exists, logger.warning, open, json.load, config.get, text_config.get, logger.info, hasattr, getattr, inject_mtp_support - State reads: self._model, self._model_name - Return expressions: None ## `vllm_mlx.engine.batched.BatchedEngine._prepare_llm_model` - Kind: method - Signature: `def _prepare_llm_model(self) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/batched.py#L479-L511 - Implementation: Method `BatchedEngine._prepare_llm_model` updates `self._model`, `self._tokenizer`; calls `self._model_name.lower`, `load_model_with_fallback`, `validate_mtp_support`, `validate_35`; returns `None`. Load the LLM model/tokenizer before engine loop startup. - Inputs: none - Return annotation: `None` - Calls: self._model_name.lower, load_model_with_fallback, validate_mtp_support, validate_35, logger.info, logger.warning, self._configure_metal_memory_limits - State reads: self._model, self._tokenizer, self._trust_remote_code, self._model_name.lower, self._model_name, self._scheduler_config, self._scheduler_config.enable_mtp, self._configure_metal_memory_limits - State writes: self._model, self._tokenizer - Return expressions: None ## `vllm_mlx.engine.batched.BatchedEngine._configure_metal_memory_limits` - Kind: method - Signature: `def _configure_metal_memory_limits(self) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/batched.py#L513-L541 - Implementation: Method `BatchedEngine._configure_metal_memory_limits` calls `mx.metal.is_available`, `mx.device_info`, `device_info.get`, `int`. Make MLX allocation failures graceful during startup. - Inputs: none - Return annotation: `None` - Calls: mx.metal.is_available, mx.device_info, device_info.get, int, _resolve_metal_buffer_cache_limit, mx.set_memory_limit, mx.set_cache_limit, logger.info, logger.warning - State reads: self._gpu_memory_utilization ## `vllm_mlx.engine.batched.BatchedEngine._start_llm` - Kind: method - Signature: `async def _start_llm(self) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/batched.py#L543-L579 - Implementation: Method `BatchedEngine._start_llm` updates `self._engine`; calls `self._prepare_llm_model`, `validate_mtp_support`, `logger.info`, `logger.warning`; awaits asynchronous work. Start the LLM engine with AsyncEngineCore. - Inputs: none - Return annotation: `None` - Calls: self._prepare_llm_model, validate_mtp_support, logger.info, logger.warning, SchedulerConfig, EngineConfig, AsyncEngineCore, self._engine.engine.start - State reads: self._model, self._tokenizer, self._prepare_llm_model, self._scheduler_config, self._scheduler_config.enable_mtp, self._model_name, self._stream_interval, self._gpu_memory_utilization, self._engine.engine.start, self._engine.engine, self._engine - State writes: self._engine ## `vllm_mlx.engine.batched.BatchedEngine.stop` - Kind: method - Signature: `async def stop(self) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/batched.py#L581-L597 - Implementation: Method `BatchedEngine.stop` updates `self._mllm_scheduler`, `self._engine`, `self._model`, `self._tokenizer`; calls `self._mllm_scheduler.stop`, `self._engine.stop`, `self._engine.engine.close`, `logger.info`; awaits asynchronous work. Stop the engine and cleanup resources. - Inputs: none - Return annotation: `None` - Calls: self._mllm_scheduler.stop, self._engine.stop, self._engine.engine.close, logger.info - State reads: self._mllm_scheduler, self._mllm_scheduler.stop, self._engine, self._engine.stop, self._engine.engine.close, self._engine.engine - State writes: self._mllm_scheduler, self._engine, self._model, self._tokenizer, self._processor, self._mllm_instance, self._loaded ## `vllm_mlx.engine.batched.BatchedEngine._apply_chat_template` - Kind: method - Signature: `def _apply_chat_template(self, messages: list[dict[str, Any]], tools: list[dict] | None=None, num_images: int=0, num_audios: int=0, chat_template_kwargs: dict[str, Any] | None=None, enable_thinking: bool | None=None) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/batched.py#L599-L687 - Implementation: Method `BatchedEngine._apply_chat_template` calls `_normalize_tool_call_arguments_for_template`, `hasattr`, `self._prepare_mllm_messages`, `self._model_name.lower`; has 3 explicit return paths. Apply chat template to messages. Uses the processor's (or tokenizer's) apply_chat_template with the full message list so that system prompts and conversation history are preserved. The previous implementation extracted only the last user message text via mlx_vlm.prompt_utils.apply_chat_template, which dropped system prompts and all prior turns. - Inputs: - `messages` (list[dict[str, Any]]; required): Required positional or keyword input. - `tools` (list[dict] | None; optional; default `None`): Optional positional or keyword input; defaults to `None`. - `num_images` (int; optional; default `0`): Optional positional or keyword input; defaults to `0`. - `num_audios` (int; optional; default `0`): Optional positional or keyword input; defaults to `0`. - `chat_template_kwargs` (dict[str, Any] | None; optional; default `None`): Optional positional or keyword input; defaults to `None`. - `enable_thinking` (bool | None; optional; default `None`): Optional positional or keyword input; defaults to `None`. - Return annotation: `str` - Calls: _normalize_tool_call_arguments_for_template, hasattr, self._prepare_mllm_messages, self._model_name.lower, template_kwargs.update, template_applicator.apply_chat_template, str, tokenizer_applicator.apply_chat_template, logger.debug, (chat_template_kwargs or {}).keys, template_kwargs.pop, '\n'.join - State reads: self._is_mllm, self._processor, self.tokenizer, self._prepare_mllm_messages, self._model_name.lower, self._model_name - Return expressions: template_applicator.apply_chat_template(messages, **template_kwargs); tokenizer_applicator.apply_chat_template(messages, **template_kwargs); prompt + '\nassistant:' ## `vllm_mlx.engine.batched.BatchedEngine._prepare_mllm_messages` - Kind: method - Signature: `def _prepare_mllm_messages(messages: list[dict[str, Any]]) -> list[dict[str, Any]]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/batched.py#L690-L726 - Implementation: Method `BatchedEngine._prepare_mllm_messages` calls `isinstance`, `msg.get`, `part.get`, `new_content.append`; returns `prepared`. Convert OpenAI-style multimodal content to HuggingFace format. The OpenAI API uses ``{"type": "image_url", "image_url": {"url": ...}}`` and ``{"type": "audio_url", "audio_url": {"url": ...}}`` while HuggingFace processors expect ``{"type": "image"}`` / ``{"type": "audio"}``. Args: messages: List of chat messages in OpenAI format. Each message is a dict with at least ``role`` and ``content`` keys. Returns: A new list of messages with ``image_url`` / ``audio_url`` parts replaced by ``{"type": "image"}`` / ``{"type": "audio"}`` entries for the HuggingFace processor. - Inputs: - `messages` (list[dict[str, Any]]; required): List of chat messages in OpenAI format. Each message is a dict with at least ``role`` and ``content`` keys. - Return annotation: `list[dict[str, Any]]` - Decorators: staticmethod - Calls: isinstance, msg.get, part.get, new_content.append, prepared.append - Return expressions: prepared ## `vllm_mlx.engine.batched.BatchedEngine.generate` - Kind: method - Signature: `async def generate(self, prompt: str, max_tokens: int=256, temperature: float=0.7, top_p: float=0.9, stop: list[str] | None=None, images: list[str] | None=None, videos: list[str] | None=None, audio: list[str] | None=None, **kwargs) -> GenerationOutput` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/batched.py#L728-L817 - Implementation: Method `BatchedEngine.generate` calls `self.start`, `self._mllm_scheduler.generate`, `kwargs.pop`, `GenerationOutput`; awaits asynchronous work; has 2 explicit return paths. Generate a complete response (non-streaming). Args: prompt: Input text max_tokens: Maximum tokens to generate temperature: Sampling temperature top_p: Top-p sampling stop: Stop sequences images: Optional image URLs/paths (for MLLM) videos: Optional video URLs/paths (for MLLM) audio: Optional audio URLs/paths (for MLLM) **kwargs: Additional model-specific parameters Returns: GenerationOutput with complete text - Inputs: - `prompt` (str; required): Input text - `max_tokens` (int; optional; default `256`): Maximum tokens to generate - `temperature` (float; optional; default `0.7`): Sampling temperature - `top_p` (float; optional; default `0.9`): Top-p sampling - `stop` (list[str] | None; optional; default `None`): Stop sequences - `images` (list[str] | None; optional; default `None`): Optional image URLs/paths (for MLLM) - `videos` (list[str] | None; optional; default `None`): Optional video URLs/paths (for MLLM) - `audio` (list[str] | None; optional; default `None`): Optional audio URLs/paths (for MLLM) - `**kwargs` (not annotated; optional): Additional model-specific parameters - Return annotation: `GenerationOutput` - Calls: self.start, self._mllm_scheduler.generate, kwargs.pop, GenerationOutput, clean_output_text, SamplingParams, self._engine.generate - State reads: self._loaded, self.start, self._is_mllm, self._mllm_scheduler, self._mllm_scheduler.generate, self._engine.generate, self._engine - Return expressions: GenerationOutput(text=clean_output_text(output.output_text), tokens=output.output_token_ids, prompt_tokens=output.promp…; GenerationOutput(text=text, tokens=output.output_token_ids, prompt_tokens=output.prompt_tokens, completion_tokens=outpu… ## `vllm_mlx.engine.batched.BatchedEngine.stream_generate` - Kind: method - Signature: `async def stream_generate(self, prompt: str, max_tokens: int=256, temperature: float=0.7, top_p: float=0.9, stop: list[str] | None=None, images: list[str] | None=None, videos: list[str] | None=None, audio: list[str] | None=None, **kwargs) -> AsyncIterator[GenerationOutput]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/batched.py#L819-L913 - Implementation: Method `BatchedEngine.stream_generate` calls `self.start`, `self._mllm_scheduler.add_request_async`, `kwargs.pop`, `self._mllm_scheduler.stream_outputs`; awaits asynchronous work; yields values incrementally; returns `None`. Stream generation token by token. Args: prompt: Input text max_tokens: Maximum tokens to generate temperature: Sampling temperature top_p: Top-p sampling stop: Stop sequences images: Optional image URLs/paths (for MLLM) videos: Optional video URLs/paths (for MLLM) audio: Optional audio URLs/paths (for MLLM) **kwargs: Additional model-specific parameters Yields: GenerationOutput with incremental text - Inputs: - `prompt` (str; required): Input text - `max_tokens` (int; optional; default `256`): Maximum tokens to generate - `temperature` (float; optional; default `0.7`): Sampling temperature - `top_p` (float; optional; default `0.9`): Top-p sampling - `stop` (list[str] | None; optional; default `None`): Stop sequences - `images` (list[str] | None; optional; default `None`): Optional image URLs/paths (for MLLM) - `videos` (list[str] | None; optional; default `None`): Optional video URLs/paths (for MLLM) - `audio` (list[str] | None; optional; default `None`): Optional audio URLs/paths (for MLLM) - `**kwargs` (not annotated; optional): Additional model-specific parameters - Return annotation: `AsyncIterator[GenerationOutput]` - Calls: self.start, self._mllm_scheduler.add_request_async, kwargs.pop, self._mllm_scheduler.stream_outputs, GenerationOutput, clean_output_text, SamplingParams, self._engine.add_request, self._engine.stream_outputs - State reads: self._loaded, self.start, self._is_mllm, self._mllm_scheduler, self._mllm_scheduler.add_request_async, self._mllm_scheduler.stream_outputs, self._engine.add_request, self._engine, self._engine.stream_outputs - Return expressions: None ## `vllm_mlx.engine.batched.BatchedEngine.chat` - Kind: method - Signature: `async def chat(self, messages: list[dict[str, Any]], max_tokens: int=256, temperature: float=0.7, top_p: float=0.9, tools: list[dict] | None=None, images: list[str] | None=None, videos: list[str] | None=None, **kwargs) -> GenerationOutput` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/batched.py#L915-L984 - Implementation: Method `BatchedEngine.chat` calls `self.start`, `extract_multimodal_content`, `convert_tools_for_template`, `dict`; awaits asynchronous work; returns `await self.generate(prompt=prompt, max_tokens=max_tokens, temperature=temperature, top_p=top_p, images=all_images if al…`. Chat completion (non-streaming). For MLLM models, all requests (including text-only) are routed through the MLLMScheduler for vision-aware batched generation. For non-MLLM models, uses the LLM engine with BatchGenerator. Args: messages: List of chat messages (OpenAI format) max_tokens: Maximum tokens to generate temperature: Sampling temperature top_p: Top-p sampling tools: Optional tool definitions images: Optional image URLs/paths videos: Optional video URLs/paths **kwargs: Additional model-specific parameters Returns: GenerationOutput with assistant response - Inputs: - `messages` (list[dict[str, Any]]; required): List of chat messages (OpenAI format) - `max_tokens` (int; optional; default `256`): Maximum tokens to generate - `temperature` (float; optional; default `0.7`): Sampling temperature - `top_p` (float; optional; default `0.9`): Top-p sampling - `tools` (list[dict] | None; optional; default `None`): Optional tool definitions - `images` (list[str] | None; optional; default `None`): Optional image URLs/paths - `videos` (list[str] | None; optional; default `None`): Optional video URLs/paths - `**kwargs` (not annotated; optional): Additional model-specific parameters - Return annotation: `GenerationOutput` - Calls: self.start, extract_multimodal_content, convert_tools_for_template, dict, kwargs.pop, self._apply_chat_template, len, self.generate - State reads: self._loaded, self.start, self._apply_chat_template, self.generate - Return expressions: await self.generate(prompt=prompt, max_tokens=max_tokens, temperature=temperature, top_p=top_p, images=all_images if al… ## `vllm_mlx.engine.batched.BatchedEngine._compute_prefix_boundary` - Kind: method - Signature: `def _compute_prefix_boundary(self, messages: list[dict[str, Any]], tools: list[dict] | None=None, chat_template_kwargs: dict[str, Any] | None=None) -> int` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/batched.py#L986-L1046 - Implementation: Method `BatchedEngine._compute_prefix_boundary` calls `range`, `len`, `messages[i].get`, `convert_tools_for_template`; has 2 explicit return paths. Compute token count for the shared prefix across message variations. Uses a two-tokenization approach: tokenize the full prompt twice (once as-is, once with the last user message replaced by a dummy) and find the longest common prefix (LCP). This gives the exact boundary where different user suffixes diverge, avoiding template discrepancies (e.g. Qwen3 markers on last assistant). - Inputs: - `messages` (list[dict[str, Any]]; required): Required positional or keyword input. - `tools` (list[dict] | None; optional; default `None`): Optional positional or keyword input; defaults to `None`. - `chat_template_kwargs` (dict[str, Any] | None; optional; default `None`): Optional positional or keyword input; defaults to `None`. - Return annotation: `int` - Calls: range, len, messages[i].get, convert_tools_for_template, self._apply_chat_template, list, hasattr, tokenizer.encode, min - State reads: self._apply_chat_template, self.tokenizer - Return expressions: 0; lcp ## `vllm_mlx.engine.batched.BatchedEngine.stream_chat` - Kind: method - Signature: `async def stream_chat(self, messages: list[dict[str, Any]], max_tokens: int=256, temperature: float=0.7, top_p: float=0.9, tools: list[dict] | None=None, images: list[str] | None=None, videos: list[str] | None=None, **kwargs) -> AsyncIterator[GenerationOutput]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/batched.py#L1048-L1127 - Implementation: Method `BatchedEngine.stream_chat` calls `self.start`, `extract_multimodal_content`, `convert_tools_for_template`, `dict`; awaits asynchronous work; yields values incrementally. Stream chat completion token by token. For MLLM models, all requests (including text-only) are streamed through the MLLMScheduler for vision-aware batched generation. For non-MLLM models, uses the LLM engine with BatchGenerator. Args: messages: List of chat messages (OpenAI format) max_tokens: Maximum tokens to generate temperature: Sampling temperature top_p: Top-p sampling tools: Optional tool definitions images: Optional image URLs/paths videos: Optional video URLs/paths **kwargs: Additional model-specific parameters Yields: GenerationOutput with incremental text - Inputs: - `messages` (list[dict[str, Any]]; required): List of chat messages (OpenAI format) - `max_tokens` (int; optional; default `256`): Maximum tokens to generate - `temperature` (float; optional; default `0.7`): Sampling temperature - `top_p` (float; optional; default `0.9`): Top-p sampling - `tools` (list[dict] | None; optional; default `None`): Optional tool definitions - `images` (list[str] | None; optional; default `None`): Optional image URLs/paths - `videos` (list[str] | None; optional; default `None`): Optional video URLs/paths - `**kwargs` (not annotated; optional): Additional model-specific parameters - Return annotation: `AsyncIterator[GenerationOutput]` - Calls: self.start, extract_multimodal_content, convert_tools_for_template, dict, kwargs.pop, self._apply_chat_template, len, self._compute_prefix_boundary, self.stream_generate - State reads: self._loaded, self.start, self._apply_chat_template, self._compute_prefix_boundary, self.stream_generate ## `vllm_mlx.engine.batched.BatchedEngine.get_stats` - Kind: method - Signature: `def get_stats(self) -> dict[str, Any]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/batched.py#L1129-L1169 - Implementation: Method `BatchedEngine.get_stats` calls `time.time`, `self._mllm_scheduler.get_stats`, `stats.update`, `self._engine.get_stats`; returns `stats`. Get engine statistics. - Inputs: none - Return annotation: `dict[str, Any]` - Calls: time.time, self._mllm_scheduler.get_stats, stats.update, self._engine.get_stats - State reads: self._model_name, self._created_at, self._is_mllm, self._loaded, self._stream_interval, self._mllm_scheduler, self._mllm_scheduler.get_stats, self._engine, self._engine.get_stats - Return expressions: stats ## `vllm_mlx.engine.batched.BatchedEngine.get_cache_stats` - Kind: method - Signature: `def get_cache_stats(self) -> dict[str, Any] | None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/batched.py#L1171-L1180 - Implementation: Method `BatchedEngine.get_cache_stats` calls `self._mllm_scheduler.batch_generator.get_prefix_cache_stats`, `self._mllm_scheduler.batch_generator.get_vision_cache_stats`, `self._engine.get_cache_stats`; has 3 explicit return paths. Get cache statistics. - Inputs: none - Return annotation: `dict[str, Any] | None` - Calls: self._mllm_scheduler.batch_generator.get_prefix_cache_stats, self._mllm_scheduler.batch_generator.get_vision_cache_stats, self._engine.get_cache_stats - State reads: self._mllm_scheduler, self._mllm_scheduler.batch_generator, self._mllm_scheduler.batch_generator.get_prefix_cache_stats, self._mllm_scheduler.batch_generator.get_vision_cache_stats, self._engine, self._engine.get_cache_stats - Return expressions: {'prefix_cache': self._mllm_scheduler.batch_generator.get_prefix_cache_stats(), 'vision_embedding_cache': self._mllm_sc…; self._engine.get_cache_stats(); None ## `vllm_mlx.engine.batched.BatchedEngine.clear_runtime_caches` - Kind: method - Signature: `def clear_runtime_caches(self) -> dict[str, Any] | None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/batched.py#L1182-L1188 - Implementation: Method `BatchedEngine.clear_runtime_caches` calls `self._mllm_scheduler.clear_runtime_caches`, `self._engine.clear_runtime_caches`; has 3 explicit return paths. Clear engine-managed runtime caches. - Inputs: none - Return annotation: `dict[str, Any] | None` - Calls: self._mllm_scheduler.clear_runtime_caches, self._engine.clear_runtime_caches - State reads: self._mllm_scheduler, self._mllm_scheduler.clear_runtime_caches, self._engine, self._engine.clear_runtime_caches - Return expressions: self._mllm_scheduler.clear_runtime_caches(); self._engine.clear_runtime_caches(); None ## `vllm_mlx.engine.batched.BatchedEngine.abort_request` - Kind: method - Signature: `async def abort_request(self, request_id: str) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/batched.py#L1190-L1199 - Implementation: Method `BatchedEngine.abort_request` calls `self._mllm_scheduler.abort_request`, `hasattr`, `self._engine.abort_request`, `inspect.isawaitable`; awaits asynchronous work; has 4 explicit return paths. Abort an active or queued batched request by request ID. - Inputs: - `request_id` (str; required): Required positional or keyword input. - Return annotation: `bool` - Calls: self._mllm_scheduler.abort_request, hasattr, self._engine.abort_request, inspect.isawaitable - State reads: self._mllm_scheduler, self._mllm_scheduler.abort_request, self._engine, self._engine.abort_request - Return expressions: self._mllm_scheduler.abort_request(request_id); await result; result; False ## `vllm_mlx.engine.batched.BatchedEngine.save_cache_to_disk` - Kind: method - Signature: `def save_cache_to_disk(self, cache_dir: str) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/batched.py#L1201-L1209 - Implementation: Method `BatchedEngine.save_cache_to_disk` calls `pc.save_to_disk`, `self._engine.save_cache_to_disk`; has 3 explicit return paths. Save prefix cache to disk for persistence across restarts. - Inputs: - `cache_dir` (str; required): Required positional or keyword input. - Return annotation: `bool` - Calls: pc.save_to_disk, self._engine.save_cache_to_disk - State reads: self._mllm_scheduler, self._mllm_scheduler.batch_generator, self._mllm_scheduler.batch_generator.prefix_cache, self._engine, self._engine.save_cache_to_disk - Return expressions: pc.save_to_disk(cache_dir); self._engine.save_cache_to_disk(cache_dir); False ## `vllm_mlx.engine.batched.BatchedEngine.load_cache_from_disk` - Kind: method - Signature: `def load_cache_from_disk(self, cache_dir: str) -> int` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/batched.py#L1211-L1220 - Implementation: Method `BatchedEngine.load_cache_from_disk` calls `self._mllm_scheduler._ensure_batch_generator`, `pc.load_from_disk`, `self._engine.load_cache_from_disk`; has 3 explicit return paths. Load prefix cache from disk. Returns number of entries loaded. - Inputs: - `cache_dir` (str; required): Required positional or keyword input. - Return annotation: `int` - Calls: self._mllm_scheduler._ensure_batch_generator, pc.load_from_disk, self._engine.load_cache_from_disk - State reads: self._mllm_scheduler, self._mllm_scheduler._ensure_batch_generator, self._mllm_scheduler.batch_generator.prefix_cache, self._mllm_scheduler.batch_generator, self._engine, self._engine.load_cache_from_disk - Return expressions: pc.load_from_disk(cache_dir); self._engine.load_cache_from_disk(cache_dir); 0 ## `vllm_mlx.engine.batched.BatchedEngine.clear_prefix_cache` - Kind: method - Signature: `def clear_prefix_cache(self) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/batched.py#L1222-L1231 - Implementation: Method `BatchedEngine.clear_prefix_cache` calls `hasattr`, `pc.clear`, `self._engine.clear_prefix_cache`; returns `None`. Clear the in-memory prefix cache. Used by bench-serve for clean cold-start measurements between configurations. - Inputs: none - Return annotation: `None` - Calls: hasattr, pc.clear, self._engine.clear_prefix_cache - State reads: self._mllm_scheduler, self._mllm_scheduler.batch_generator, self._mllm_scheduler.batch_generator.prefix_cache, self._engine, self._engine.clear_prefix_cache - Return expressions: None # Module `vllm_mlx.engine.chat_template_safety` Safety normalization for messages before Jinja chat-template rendering. Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/chat_template_safety.py#L1-L90 ## `vllm_mlx.engine.chat_template_safety._close_dangling_think_before_tool_call` - Kind: function - Signature: `def _close_dangling_think_before_tool_call(content: str) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/chat_template_safety.py#L8-L29 - Implementation: Function `_close_dangling_think_before_tool_call` calls `content.rfind`, `content.find`; has 3 explicit return paths. Keep raw tool XML out of an unterminated ```` section. Qwen 3.6 can produce assistant history where ```` is opened and a raw ```` follows before ````. Rendering that history as-is conditions the next turn as though the tool call is still reasoning. Close the dangling thinking span immediately before the first tool call. This mirrors the template-side repair described by Cheuk-Yiu Chan: https://allanchan339.github.io/bug-fixes/2026/05/02/Qwen36-27B-updated-jinja.html - Inputs: - `content` (str; required): Required positional or keyword input. - Return annotation: `str` - Calls: content.rfind, content.find - Return expressions: content; content[:tool_pos] + '' + content[tool_pos:]; content + '' ## `vllm_mlx.engine.chat_template_safety._message_to_dict` - Kind: function - Signature: `def _message_to_dict(message: Any) -> dict[str, Any] | Any` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/chat_template_safety.py#L32-L46 - Implementation: Function `_message_to_dict` calls `isinstance`, `dict`, `getattr`, `callable`; has 4 explicit return paths. Convert OpenAI message model objects without stringifying them. - Inputs: - `message` (Any; required): Required positional or keyword input. - Return annotation: `dict[str, Any] | Any` - Calls: isinstance, dict, getattr, callable, model_dump(exclude_none=True).items, model_dump, legacy_dict().items, legacy_dict - Return expressions: dict(message); {key: value for key, value in model_dump(exclude_none=True).items() if value is not None}; {k: v for k, v in legacy_dict().items() if v is not None}; message ## `vllm_mlx.engine.chat_template_safety.normalize_messages_for_chat_template` - Kind: function - Signature: `def normalize_messages_for_chat_template(messages: list[Any]) -> list[dict]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/chat_template_safety.py#L49-L90 - Implementation: Function `normalize_messages_for_chat_template` calls `json.loads`, `json.dumps`, `_message_to_dict`, `isinstance`; returns `normalized`. Return a JSON-safe copy of messages for chat-template rendering. Normalizations: - close dangling ```` spans before raw ```` XML in assistant content - convert OpenAI tool-call argument JSON strings to mappings for templates that iterate argument keys - Inputs: - `messages` (list[Any]; required): Required positional or keyword input. - Return annotation: `list[dict]` - Calls: json.loads, json.dumps, _message_to_dict, isinstance, message.get, _close_dangling_think_before_tool_call, tool_call.get, function.get - Return expressions: normalized # Module `vllm_mlx.engine.simple` Simple engine for maximum single-user throughput. This engine wraps mlx-lm directly with zero overhead for optimal performance when serving a single user at a time. Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/simple.py#L1-L2912 ## `vllm_mlx.engine.simple._bind_worker_generation_streams` - Kind: function - Signature: `def _bind_worker_generation_streams() -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/simple.py#L48-L50 - Implementation: Function `_bind_worker_generation_streams` calls `bind_generation_streams`. Rebind mlx generation streams inside the current worker thread. - Inputs: none - Return annotation: `None` - Calls: bind_generation_streams ## `vllm_mlx.engine.simple._seed_logits_processors` - Kind: function - Signature: `def _seed_logits_processors(seed_tokens: mx.array | None, processors: list[Any] | None) -> list[Any] | None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/simple.py#L53-L77 - Implementation: Function `_seed_logits_processors` calls `list`, `_wrap`; has 3 explicit return paths. Wrap logits processors so continuation decode sees the full prompt. - Inputs: - `seed_tokens` (mx.array | None; required): Required positional or keyword input. - `processors` (list[Any] | None; required): Required positional or keyword input. - Return annotation: `list[Any] | None` - Calls: list, _wrap - Return expressions: None; list(processors); [_wrap(processor) for processor in processors] ## `vllm_mlx.engine.simple._seed_logits_processors._wrap` - Kind: nested function - Signature: `def _wrap(processor)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/simple.py#L63-L75 - Implementation: Nested Function `_seed_logits_processors._wrap` returns `_seeded`. Nested Function `_seed_logits_processors._wrap` returns `_seeded`. - Inputs: - `processor` (not annotated; required): Required positional or keyword input. - Return annotation: `not annotated` - Return expressions: _seeded ## `vllm_mlx.engine.simple._seed_logits_processors._wrap._seeded` - Kind: nested function - Signature: `def _seeded(tokens, logits)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/simple.py#L64-L73 - Implementation: Nested Function `_seed_logits_processors._wrap._seeded` calls `isinstance`, `mx.array`, `mx.concatenate`, `processor`; returns `processor(merged, logits)`. Nested Function `_seed_logits_processors._wrap._seeded` calls `isinstance`, `mx.array`, `mx.concatenate`, `processor`; returns `processor(merged, logits)`. - Inputs: - `tokens` (not annotated; required): Required positional or keyword input. - `logits` (not annotated; required): Required positional or keyword input. - Return annotation: `not annotated` - Calls: isinstance, mx.array, mx.concatenate, processor - Return expressions: processor(merged, logits) ## `vllm_mlx.engine.simple._sample_with_processors` - Kind: function - Signature: `def _sample_with_processors(tokens: mx.array | None, logits: mx.array, sampler: Any, logits_processors: list[Any] | None) -> tuple[mx.array, mx.array]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/simple.py#L80-L97 - Implementation: Function `_sample_with_processors` calls `processor`, `logits.squeeze`, `mx.logsumexp`, `sampler`; returns `(tok, logprobs)`. Sample a token while honoring any active logits processors. - Inputs: - `tokens` (mx.array | None; required): Required positional or keyword input. - `logits` (mx.array; required): Required positional or keyword input. - `sampler` (Any; required): Required positional or keyword input. - `logits_processors` (list[Any] | None; required): Required positional or keyword input. - Return annotation: `tuple[mx.array, mx.array]` - Calls: processor, logits.squeeze, mx.logsumexp, sampler - Return expressions: (tok, logprobs) ## `vllm_mlx.engine.simple._processors_can_retire` - Kind: function - Signature: `def _processors_can_retire(processors: list[Any] | None) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/simple.py#L100-L106 - Implementation: Function `_processors_can_retire` calls `os.getenv`, `bool`, `any`, `isinstance`; has 2 explicit return paths. True when any processor advertises a retire-to-content transition. - Inputs: - `processors` (list[Any] | None; required): Required positional or keyword input. - Return annotation: `bool` - Calls: os.getenv, bool, any, isinstance, getattr - Return expressions: False; bool(processors) and any((isinstance(getattr(p, 'is_retired', None), bool) for p in processors)) ## `vllm_mlx.engine.simple._processors_retired` - Kind: function - Signature: `def _processors_retired(processors: list[Any] | None) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/simple.py#L109-L115 - Implementation: Function `_processors_retired` calls `os.getenv`, `bool`, `any`, `getattr`; has 2 explicit return paths. True when any retire-capable processor has entered its retired state. - Inputs: - `processors` (list[Any] | None; required): Required positional or keyword input. - Return annotation: `bool` - Calls: os.getenv, bool, any, getattr - Return expressions: False; bool(processors) and any((getattr(p, 'is_retired', False) is True for p in processors)) ## `vllm_mlx.engine.simple._SpecPrefillCancelled` - Kind: class - Signature: `class _SpecPrefillCancelled(Exception)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/simple.py#L118-L119 - Implementation: Class `_SpecPrefillCancelled` derives from `Exception` and declares 0 direct member(s). Cooperative cancellation sentinel for blocking SpecPrefill workers. - Inputs: none - Constructs: `vllm_mlx.engine.simple._SpecPrefillCancelled` ## `vllm_mlx.engine.simple.SimpleEngine` - Kind: class - Signature: `class SimpleEngine(BaseEngine)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/simple.py#L122-L2912 - Implementation: Class `SimpleEngine` derives from `BaseEngine` and declares 31 direct member(s). Simple engine for direct model calls. This engine provides maximum throughput for single-user scenarios by calling mlx-lm/mlx-vlm directly without batching overhead. - Inputs: - `model_name` (str; required): HuggingFace model name or local path - `trust_remote_code` (bool; optional; default `False`): Whether to trust remote code - `enable_cache` (bool; optional; default `True`): Enable VLM cache for multimodal models - `force_mllm` (bool; optional; default `False`): Force loading as MLLM even if not auto-detected - `mtp` (bool; optional; default `False`): Enable native MTP speculative decoding (model must have MTP head) - `mtp_num_draft_tokens` (int; optional; default `1`): Draft tokens per speculative MTP step - `prefill_step_size` (int; optional; default `2048`): Chunk size for prompt prefill processing (default: 2048) - `specprefill_enabled` (bool; optional; default `False`): Enable SpecPrefill (attention-based sparse prefill) - `specprefill_threshold` (int; optional; default `8192`): Minimum suffix tokens to trigger SpecPrefill - `specprefill_keep_pct` (float; optional; default `0.3`): Fraction of tokens to keep (default: 0.3) - `specprefill_backbone_pct` (float; optional; default `0.0`): Fraction of chunks to reserve for evenly spaced coverage (default: 0.0) - `specprefill_draft_model` (str | None; optional; default `None`): Path to small draft model for importance scoring - `max_kv_size` (int; optional; default `0`): Maximum KV cache size per sequence (0 = unbounded) - `mllm_draft_model` (str | None; optional; default `None`): Optional MLLM speculative draft/assistant model path - `mllm_draft_kind` (str | None; optional; default `None`): Optional mlx-vlm draft kind, for example "mtp" - `mllm_draft_block_size` (int | None; optional; default `None`): Optional speculative block size for mlx-vlm - Constructs: `vllm_mlx.engine.simple.SimpleEngine` ## `vllm_mlx.engine.simple.SimpleEngine.__init__` - Kind: method - Signature: `def __init__(self, model_name: str, trust_remote_code: bool=False, enable_cache: bool=True, force_mllm: bool=False, mtp: bool=False, mtp_num_draft_tokens: int=1, prefill_step_size: int=2048, specprefill_enabled: bool=False, specprefill_threshold: int=8192, specprefill_keep_pct: float=0.3, specprefill_backbone_pct: float=0.0, specprefill_draft_model: str | None=None, max_kv_size: int=0, mllm_draft_model: str | None=None, mllm_draft_kind: str | None=None, mllm_draft_block_size: int | None=None)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/simple.py#L130-L257 - Implementation: Method `SimpleEngine.__init__` updates `self._model_name`, `self._created_at`, `self._trust_remote_code`, `self._enable_cache`; calls `time.time`, `is_mllm_model`, `deque`, `asyncio.Lock`. Initialize the simple engine. Args: model_name: HuggingFace model name or local path trust_remote_code: Whether to trust remote code enable_cache: Enable VLM cache for multimodal models force_mllm: Force loading as MLLM even if not auto-detected mtp: Enable native MTP speculative decoding (model must have MTP head) mtp_num_draft_tokens: Draft tokens per speculative MTP step prefill_step_size: Chunk size for prompt prefill processing (default: 2048) specprefill_enabled: Enable SpecPrefill (attention-based sparse prefill) specprefill_threshold: Minimum suffix tokens to trigger SpecPrefill specprefill_keep_pct: Fraction of tokens to keep (default: 0.3) specprefill_backbone_pct: Fraction of chunks to reserve for evenly spaced coverage (default: 0.0) specprefill_draft_model: Path to small draft model for importance scoring max_kv_size: Maximum KV cache size per sequence (0 = unbounded) mllm_draft_model: Optional MLLM speculative draft/assistant model path mllm_draft_kind: Optional mlx-vlm draft kind, for example "mtp" mllm_draft_block_size: Optional speculative block size for mlx-vlm - Inputs: - `model_name` (str; required): HuggingFace model name or local path - `trust_remote_code` (bool; optional; default `False`): Whether to trust remote code - `enable_cache` (bool; optional; default `True`): Enable VLM cache for multimodal models - `force_mllm` (bool; optional; default `False`): Force loading as MLLM even if not auto-detected - `mtp` (bool; optional; default `False`): Enable native MTP speculative decoding (model must have MTP head) - `mtp_num_draft_tokens` (int; optional; default `1`): Draft tokens per speculative MTP step - `prefill_step_size` (int; optional; default `2048`): Chunk size for prompt prefill processing (default: 2048) - `specprefill_enabled` (bool; optional; default `False`): Enable SpecPrefill (attention-based sparse prefill) - `specprefill_threshold` (int; optional; default `8192`): Minimum suffix tokens to trigger SpecPrefill - `specprefill_keep_pct` (float; optional; default `0.3`): Fraction of tokens to keep (default: 0.3) - `specprefill_backbone_pct` (float; optional; default `0.0`): Fraction of chunks to reserve for evenly spaced coverage (default: 0.0) - `specprefill_draft_model` (str | None; optional; default `None`): Path to small draft model for importance scoring - `max_kv_size` (int; optional; default `0`): Maximum KV cache size per sequence (0 = unbounded) - `mllm_draft_model` (str | None; optional; default `None`): Optional MLLM speculative draft/assistant model path - `mllm_draft_kind` (str | None; optional; default `None`): Optional mlx-vlm draft kind, for example "mtp" - `mllm_draft_block_size` (int | None; optional; default `None`): Optional speculative block size for mlx-vlm - Return annotation: `not annotated` - Calls: time.time, is_mllm_model, deque, asyncio.Lock, os.environ.get('VLLM_MLX_SIMPLE_ENGINE_LOCK_ADMISSION', 'fail_fast').strip().lower, os.environ.get('VLLM_MLX_SIMPLE_ENGINE_LOCK_ADMISSION', 'fail_fast').strip, os.environ.get, logger.warning, max, int, OrderedDict - State reads: self._generation_lock_admission - State writes: self._model_name, self._created_at, self._trust_remote_code, self._enable_cache, self._is_mllm, self._mtp, self._mtp_num_draft_tokens, self._prefill_step_size, self._total_requests_processed, self._total_prompt_tokens, self._total_completion_tokens, self._num_running, self._recent_completions, self._active_requests, self._specprefill_enabled, self._specprefill_threshold, self._specprefill_keep_pct, self._specprefill_backbone_pct, self._specprefill_draft_model_path, self._mllm_draft_model_path, self._mllm_draft_kind, self._mllm_draft_block_size, self._max_kv_size, self._model, self._loaded, self._text_model, self._text_tokenizer, self._draft_model, self._generation_lock, self._generation_lock_admission, self._generation_waiters, self._generation_busy_rejections, self._system_kv_capacity, self._system_kv_cache, self._system_kv_cache_stats, self._supports_system_kv_cache ## `vllm_mlx.engine.simple.SimpleEngine._clone_cache_state` - Kind: method - Signature: `def _clone_cache_state(value: Any) -> Any` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/simple.py#L260-L266 - Implementation: Method `SimpleEngine._clone_cache_state` calls `isinstance`, `tuple`, `SimpleEngine._clone_cache_state`; has 3 explicit return paths. Copy cache state containers without duplicating immutable MLX arrays. - Inputs: - `value` (Any; required): Required positional or keyword input. - Return annotation: `Any` - Decorators: staticmethod - Calls: isinstance, tuple, SimpleEngine._clone_cache_state - Return expressions: tuple((SimpleEngine._clone_cache_state(v) for v in value)); [SimpleEngine._clone_cache_state(v) for v in value]; value ## `vllm_mlx.engine.simple.SimpleEngine._snapshot_prompt_cache` - Kind: method - Signature: `def _snapshot_prompt_cache(cls, prompt_cache: list[Any]) -> list[Any]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/simple.py#L269-L271 - Implementation: Method `SimpleEngine._snapshot_prompt_cache` calls `cls._clone_cache_state`; returns `[cls._clone_cache_state(c.state) for c in prompt_cache]`. Capture cache states without aliasing mutable state containers. - Inputs: - `prompt_cache` (list[Any]; required): Required positional or keyword input. - Return annotation: `list[Any]` - Decorators: classmethod - Calls: cls._clone_cache_state - State reads: cls._clone_cache_state - Return expressions: [cls._clone_cache_state(c.state) for c in prompt_cache] ## `vllm_mlx.engine.simple.SimpleEngine._restore_prompt_cache` - Kind: method - Signature: `def _restore_prompt_cache(cls, prompt_cache: list[Any], snapshot: list[Any]) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/simple.py#L274-L279 - Implementation: Method `SimpleEngine._restore_prompt_cache` calls `enumerate`, `cls._clone_cache_state`. Restore cache states without letting decode mutate the saved snapshot. - Inputs: - `prompt_cache` (list[Any]; required): Required positional or keyword input. - `snapshot` (list[Any]; required): Required positional or keyword input. - Return annotation: `None` - Decorators: classmethod - Calls: enumerate, cls._clone_cache_state - State reads: cls._clone_cache_state ## `vllm_mlx.engine.simple.SimpleEngine._iter_cache_state_arrays` - Kind: method - Signature: `def _iter_cache_state_arrays(value: Any)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/simple.py#L282-L287 - Implementation: Method `SimpleEngine._iter_cache_state_arrays` calls `isinstance`, `SimpleEngine._iter_cache_state_arrays`, `hasattr`; yields values incrementally. Method `SimpleEngine._iter_cache_state_arrays` calls `isinstance`, `SimpleEngine._iter_cache_state_arrays`, `hasattr`; yields values incrementally. - Inputs: - `value` (Any; required): Required positional or keyword input. - Return annotation: `not annotated` - Decorators: staticmethod - Calls: isinstance, SimpleEngine._iter_cache_state_arrays, hasattr ## `vllm_mlx.engine.simple.SimpleEngine._eval_cache_snapshot` - Kind: method - Signature: `def _eval_cache_snapshot(cls, snapshot: list[Any]) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/simple.py#L290-L293 - Implementation: Method `SimpleEngine._eval_cache_snapshot` calls `list`, `cls._iter_cache_state_arrays`, `mx.eval`. Method `SimpleEngine._eval_cache_snapshot` calls `list`, `cls._iter_cache_state_arrays`, `mx.eval`. - Inputs: - `snapshot` (list[Any]; required): Required positional or keyword input. - Return annotation: `None` - Decorators: classmethod - Calls: list, cls._iter_cache_state_arrays, mx.eval - State reads: cls._iter_cache_state_arrays ## `vllm_mlx.engine.simple.SimpleEngine._cache_class_is_system_snapshot_safe` - Kind: method - Signature: `def _cache_class_is_system_snapshot_safe(cache_entry: Any) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/simple.py#L296-L303 - Implementation: Method `SimpleEngine._cache_class_is_system_snapshot_safe` calls `isinstance`, `type`; has 2 explicit return paths. Method `SimpleEngine._cache_class_is_system_snapshot_safe` calls `isinstance`, `type`; has 2 explicit return paths. - Inputs: - `cache_entry` (Any; required): Required positional or keyword input. - Return annotation: `bool` - Decorators: staticmethod - Calls: isinstance, type - Return expressions: isinstance(cache_entry, (KVCache, ArraysCache)); cache_type in {'KVCache', 'ArraysCache'} ## `vllm_mlx.engine.simple.SimpleEngine._probe_system_kv_cache_support` - Kind: method - Signature: `def _probe_system_kv_cache_support(cls, model: Any, route: str) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/simple.py#L306-L331 - Implementation: Method `SimpleEngine._probe_system_kv_cache_support` calls `make_prompt_cache`, `bool`, `all`, `cls._cache_class_is_system_snapshot_safe`; has 2 explicit return paths. Method `SimpleEngine._probe_system_kv_cache_support` calls `make_prompt_cache`, `bool`, `all`, `cls._cache_class_is_system_snapshot_safe`; has 2 explicit return paths. - Inputs: - `model` (Any; required): Required positional or keyword input. - `route` (str; required): Required positional or keyword input. - Return annotation: `bool` - Decorators: classmethod - Calls: make_prompt_cache, bool, all, cls._cache_class_is_system_snapshot_safe, sorted, type, logger.info, logger.debug - State reads: cls._cache_class_is_system_snapshot_safe - Return expressions: supported; False ## `vllm_mlx.engine.simple.SimpleEngine.model_name` - Kind: method - Signature: `def model_name(self) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/simple.py#L334-L336 - Implementation: Method `SimpleEngine.model_name` returns `self._model_name`. Get the model name. - Inputs: none - Return annotation: `str` - Decorators: property - State reads: self._model_name - Return expressions: self._model_name ## `vllm_mlx.engine.simple.SimpleEngine.is_mllm` - Kind: method - Signature: `def is_mllm(self) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/simple.py#L339-L341 - Implementation: Method `SimpleEngine.is_mllm` returns `self._is_mllm`. Check if this is a multimodal model. - Inputs: none - Return annotation: `bool` - Decorators: property - State reads: self._is_mllm - Return expressions: self._is_mllm ## `vllm_mlx.engine.simple.SimpleEngine.tokenizer` - Kind: method - Signature: `def tokenizer(self) -> Any` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/simple.py#L344-L350 - Implementation: Method `SimpleEngine.tokenizer` calls `getattr`; has 3 explicit return paths. Get the tokenizer. - Inputs: none - Return annotation: `Any` - Decorators: property - Calls: getattr - State reads: self._loaded, self._model, self._is_mllm, self._model.tokenizer - Return expressions: None; getattr(self._model, 'processor', None); self._model.tokenizer ## `vllm_mlx.engine.simple.SimpleEngine._generation_lock_holder_summary` - Kind: method - Signature: `def _generation_lock_holder_summary(self) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/simple.py#L352-L371 - Implementation: Method `SimpleEngine._generation_lock_holder_summary` calls `time.time`, `self._active_requests.items`, `info.get`, `round`; has 2 explicit return paths. Method `SimpleEngine._generation_lock_holder_summary` calls `time.time`, `self._active_requests.items`, `info.get`, `round`; has 2 explicit return paths. - Inputs: none - Return annotation: `str` - Calls: time.time, self._active_requests.items, info.get, round, holders.append, ','.join - State reads: self._active_requests, self._active_requests.items - Return expressions: 'none'; ','.join(holders) ## `vllm_mlx.engine.simple.SimpleEngine._acquire_generation_slot` - Kind: method - Signature: `async def _acquire_generation_slot(self, request_id: str)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/simple.py#L374-L398 - Implementation: Method `SimpleEngine._acquire_generation_slot` updates `self._generation_busy_rejections`, `self._generation_waiters`; calls `self._generation_lock.locked`, `EngineBusy`, `self._generation_lock_holder_summary`; yields values incrementally; can raise `EngineBusy`. Admission control for SimpleEngine's serialized MLX route. - Inputs: - `request_id` (str; required): Required positional or keyword input. - Return annotation: `not annotated` - Decorators: asynccontextmanager - Calls: self._generation_lock.locked, EngineBusy, self._generation_lock_holder_summary - State reads: self._generation_lock_admission, self._generation_lock.locked, self._generation_lock, self._generation_lock_holder_summary, self._generation_waiters - State writes: self._generation_busy_rejections, self._generation_waiters - Raises directly: EngineBusy ## `vllm_mlx.engine.simple.SimpleEngine.prepare_for_start` - Kind: method - Signature: `def prepare_for_start(self) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/simple.py#L400-L427 - Implementation: Method `SimpleEngine.prepare_for_start` updates `self._model`; calls `MLXMultimodalLM`, `MLXLanguageModel`, `self._model.load`; returns `None`. Load the backing model off the serving event loop. - Inputs: none - Return annotation: `None` - Calls: MLXMultimodalLM, MLXLanguageModel, self._model.load - State reads: self._model, self._is_mllm, self._model_name, self._trust_remote_code, self._enable_cache, self._max_kv_size, self._mllm_draft_model_path, self._mllm_draft_kind, self._mllm_draft_block_size, self._mtp, self._mtp_num_draft_tokens, self._model.load - State writes: self._model - Return expressions: None ## `vllm_mlx.engine.simple.SimpleEngine._uses_default_prepare_for_start` - Kind: method - Signature: `def _uses_default_prepare_for_start(self) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/simple.py#L429-L432 - Implementation: Method `SimpleEngine._uses_default_prepare_for_start` calls `getattr`; returns `method is SimpleEngine.prepare_for_start`. Return True when prepare_for_start is the class implementation. - Inputs: none - Return annotation: `bool` - Calls: getattr - State reads: self.prepare_for_start - Return expressions: method is SimpleEngine.prepare_for_start ## `vllm_mlx.engine.simple.SimpleEngine.start` - Kind: method - Signature: `async def start(self) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/simple.py#L434-L595 - Implementation: Method `SimpleEngine.start` updates `self._loaded`, `self._supports_system_kv_cache`, `self._text_model`, `self._text_tokenizer`; calls `self._uses_default_prepare_for_start`, `self.prepare_for_start`, `run_blocking_startup_work`, `logger.warning`; awaits asynchronous work; returns `None`. Start the engine (load model if not loaded). - Inputs: none - Return annotation: `None` - Calls: self._uses_default_prepare_for_start, self.prepare_for_start, run_blocking_startup_work, logger.warning, getattr, self._probe_system_kv_cache_support, self._should_route_text_through_text_model, build_text_model, self._model.get_tokenizer, self._model_name.lower, self._text_tokenizer.convert_tokens_to_ids, make_prompt_cache, bool, all, isinstance, sorted, type, logger.info, logger.debug, hasattr, logger.error, mlx_lm_load, cleanup_startup_cancellation - State reads: self._loaded, self._model, self._uses_default_prepare_for_start, self.prepare_for_start, self._mtp, self._mtp_num_draft_tokens, self._is_mllm, self._probe_system_kv_cache_support, self._should_route_text_through_text_model, self._model.model, self._model_name, self._text_model, self._model.get_tokenizer, self._model_name.lower, self._text_tokenizer, self._text_tokenizer.convert_tokens_to_ids, self._max_kv_size, self._supports_system_kv_cache, self._text_model.mtp, self._specprefill_enabled, self._specprefill_draft_model_path, self._specprefill_threshold, self._specprefill_keep_pct, self._draft_model, self.stop - State writes: self._loaded, self._supports_system_kv_cache, self._text_model, self._text_tokenizer, self._text_tokenizer.eos_token, self._text_tokenizer.eos_token_id, self._draft_model - Return expressions: None ## `vllm_mlx.engine.simple.SimpleEngine.stop` - Kind: method - Signature: `async def stop(self) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/simple.py#L597-L608 - Implementation: Method `SimpleEngine.stop` updates `self._model`, `self._text_model`, `self._text_tokenizer`, `self._draft_model`; calls `self._system_kv_cache.clear`, `logger.info`. Stop the engine and cleanup resources. - Inputs: none - Return annotation: `None` - Calls: self._system_kv_cache.clear, logger.info - State reads: self._system_kv_cache.clear, self._system_kv_cache, self._system_kv_cache_stats - State writes: self._model, self._text_model, self._text_tokenizer, self._draft_model, self._loaded, self._supports_system_kv_cache ## `vllm_mlx.engine.simple.SimpleEngine._should_route_text_through_text_model` - Kind: method - Signature: `def _should_route_text_through_text_model(self, *, mllm_draft_requested: bool=False) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/simple.py#L610-L614 - Implementation: Method `SimpleEngine._should_route_text_through_text_model` returns `not (mllm_draft_requested and self._mllm_draft_model_path is not None)`. Return whether text-only MLLM requests may use mlx_lm TextModel. - Inputs: - `mllm_draft_requested` (bool; optional; default `False`): Optional keyword-only input; defaults to `False`. - Return annotation: `bool` - State reads: self._mllm_draft_model_path - Return expressions: not (mllm_draft_requested and self._mllm_draft_model_path is not None) ## `vllm_mlx.engine.simple.SimpleEngine._run_blocking_serialized` - Kind: method - Signature: `async def _run_blocking_serialized(self, func, /, *args, request_id: str | None=None, on_cancel=None, **kwargs)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/simple.py#L616-L666 - Implementation: Method `SimpleEngine._run_blocking_serialized` calls `id`, `self._acquire_generation_slot`, `time.time`, `asyncio.create_task`; awaits asynchronous work; returns `await asyncio.shield(task)`. Run a blocking MLX operation under the generation lock. Cancellation must not release the async lock before the worker thread finishes, or a follow-up request can enter MLX/Metal concurrently and corrupt the command-buffer state. - Inputs: - `func` (not annotated; required): Required positional-only input. - `*args` (not annotated; optional): Additional variadic positional inputs accepted by this callable. - `request_id` (str | None; optional; default `None`): Optional keyword-only input; defaults to `None`. - `on_cancel` (not annotated; optional; default `None`): Optional keyword-only input; defaults to `None`. - `**kwargs` (not annotated; optional): Additional variadic keyword inputs accepted by this callable. - Return annotation: `not annotated` - Calls: id, self._acquire_generation_slot, time.time, asyncio.create_task, asyncio.to_thread, asyncio.shield, on_cancel, logger.debug, self._active_requests.pop - State reads: self._acquire_generation_slot, self._active_requests, self._active_requests.pop - Return expressions: await asyncio.shield(task) ## `vllm_mlx.engine.simple.SimpleEngine._run_blocking_serialized.run_bound` - Kind: nested function - Signature: `def run_bound()` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/simple.py#L644-L646 - Implementation: Nested Function `SimpleEngine._run_blocking_serialized.run_bound` calls `_bind_worker_generation_streams`, `func`; returns `func(*args, **kwargs)`. Nested Function `SimpleEngine._run_blocking_serialized.run_bound` calls `_bind_worker_generation_streams`, `func`; returns `func(*args, **kwargs)`. - Inputs: none - Return annotation: `not annotated` - Calls: _bind_worker_generation_streams, func - Return expressions: func(*args, **kwargs) ## `vllm_mlx.engine.simple.SimpleEngine.generate` - Kind: method - Signature: `async def generate(self, prompt: str, max_tokens: int=256, temperature: float=0.7, top_p: float=0.9, stop: list[str] | None=None, **kwargs) -> GenerationOutput` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/simple.py#L668-L730 - Implementation: Method `SimpleEngine.generate` calls `self.start`, `self.stream_generate`, `GenerationOutput`, `clean_output_text`; awaits asynchronous work; has 2 explicit return paths. Generate a complete response (non-streaming). Thin accumulator over stream_generate(). stream_generate() is the only code path that consumes per-request SpecPrefill overrides (`specprefill`, `specprefill_keep_pct`) and routes through _stream_generate_specprefill() when engaged. The prior direct self._model.generate() path silently dropped those overrides for non-streaming /v1/completions callers, so extra_body.specprefill was advertised by the server but had no effect on this route. By iterating stream_generate() and returning the last GenerationOutput, non-streaming clients get the same SpecPrefill engagement, accurate prompt_tokens reporting, and per-request override support as streaming clients. Args: prompt: Input text max_tokens: Maximum tokens to generate temperature: Sampling temperature top_p: Top-p sampling stop: Stop sequences **kwargs: Additional parameters forwarded to stream_generate, including per-request `specprefill` / `specprefill_keep_pct` Returns: GenerationOutput with complete text - Inputs: - `prompt` (str; required): Input text - `max_tokens` (int; optional; default `256`): Maximum tokens to generate - `temperature` (float; optional; default `0.7`): Sampling temperature - `top_p` (float; optional; default `0.9`): Top-p sampling - `stop` (list[str] | None; optional; default `None`): Stop sequences - `**kwargs` (not annotated; optional): Additional parameters forwarded to stream_generate, including per-request `specprefill` / `specprefill_keep_pct` - Return annotation: `GenerationOutput` - Calls: self.start, self.stream_generate, GenerationOutput, clean_output_text, list - State reads: self._loaded, self.start, self.stream_generate - Return expressions: GenerationOutput(text='', finish_reason='stop'); GenerationOutput(text=text, tokens=list(last_output.tokens), prompt_tokens=last_output.prompt_tokens, completion_tokens… ## `vllm_mlx.engine.simple.SimpleEngine._track_request_stream` - Kind: method - Signature: `async def _track_request_stream(self, source_gen: AsyncIterator[GenerationOutput], *, max_tokens: int=0) -> AsyncIterator[GenerationOutput]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/simple.py#L732-L817 - Implementation: Method `SimpleEngine._track_request_stream` updates `self._num_running`, `self._total_requests_processed`, `self._total_prompt_tokens`, `self._total_completion_tokens`; calls `_in_tracker.get`, `_in_tracker.set`, `str`, `uuid.uuid4`; yields values incrementally; returns `None`. Yield-through wrapper that records per-request live state and final ``prompt_tokens``/``completion_tokens`` counters. Mirrors the fields BatchedEngine emits per running request (``request_id``, ``phase``, ``elapsed_s``, ``ttft_s``, ``tokens_per_second``, ``progress``, ...) so dashboards built against ``/v1/status`` show individual in-flight requests for SimpleEngine-backed services as well (Gemma 4 31B + MTP, etc.). Re-entrant calls (e.g. the cache-fallback path inside ``_stream_chat_impl`` that delegates to ``self.stream_generate``) are detected via the ``_in_tracker`` context variable and pass through without a second tracking entry, so each external request is counted exactly once. Note: we deliberately use ``set(True)``/``set(False)`` rather than ``set(token)``/``reset(token)``. FastAPI/uvicorn finalize streaming generators from a different async context than the one that created them; ``ContextVar.reset(token)`` raises ``ValueError`` in that case ("Token was created in a different Context"), which surfaces as a terminal-frame streaming error. ``set(False)`` works in any context and the contextvar is only consumed inside this method, so there is no value to preserve. - Inputs: - `source_gen` (AsyncIterator[GenerationOutput]; required): Required positional or keyword input. - `max_tokens` (int; optional; default `0`): Optional keyword-only input; defaults to `0`. - Return annotation: `AsyncIterator[GenerationOutput]` - Calls: _in_tracker.get, _in_tracker.set, str, uuid.uuid4, time.time, hasattr, round, min, max, self._active_requests.pop, self._recent_completions.append - State reads: self._active_requests, self._active_requests.pop, self._num_running, self._recent_completions.append, self._recent_completions - State writes: self._num_running, self._total_requests_processed, self._total_prompt_tokens, self._total_completion_tokens - Return expressions: None ## `vllm_mlx.engine.simple.SimpleEngine.stream_generate` - Kind: method - Signature: `async def stream_generate(self, prompt: str, max_tokens: int=256, temperature: float=0.7, top_p: float=0.9, stop: list[str] | None=None, **kwargs) -> AsyncIterator[GenerationOutput]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/simple.py#L819-L840 - Implementation: Method `SimpleEngine.stream_generate` calls `self._track_request_stream`, `self._stream_generate_impl`; yields values incrementally. Public stream-generate wrapper with request stats tracking. - Inputs: - `prompt` (str; required): Required positional or keyword input. - `max_tokens` (int; optional; default `256`): Optional positional or keyword input; defaults to `256`. - `temperature` (float; optional; default `0.7`): Optional positional or keyword input; defaults to `0.7`. - `top_p` (float; optional; default `0.9`): Optional positional or keyword input; defaults to `0.9`. - `stop` (list[str] | None; optional; default `None`): Optional positional or keyword input; defaults to `None`. - `**kwargs` (not annotated; optional): Additional variadic keyword inputs accepted by this callable. - Return annotation: `AsyncIterator[GenerationOutput]` - Calls: self._track_request_stream, self._stream_generate_impl - State reads: self._track_request_stream, self._stream_generate_impl ## `vllm_mlx.engine.simple.SimpleEngine._stream_generate_impl` - Kind: method - Signature: `async def _stream_generate_impl(self, prompt: str, max_tokens: int=256, temperature: float=0.7, top_p: float=0.9, stop: list[str] | None=None, **kwargs) -> AsyncIterator[GenerationOutput]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/simple.py#L842-L1012 - Implementation: Method `SimpleEngine._stream_generate_impl` calls `self.start`, `kwargs.pop`, `str`, `id`; awaits asynchronous work; yields values incrementally; returns `None`. Stream generation token by token. Args: prompt: Input text max_tokens: Maximum tokens to generate temperature: Sampling temperature top_p: Top-p sampling stop: Stop sequences **kwargs: Additional model-specific parameters Yields: GenerationOutput with incremental text - Inputs: - `prompt` (str; required): Input text - `max_tokens` (int; optional; default `256`): Maximum tokens to generate - `temperature` (float; optional; default `0.7`): Sampling temperature - `top_p` (float; optional; default `0.9`): Top-p sampling - `stop` (list[str] | None; optional; default `None`): Stop sequences - `**kwargs` (not annotated; optional): Additional model-specific parameters - Return annotation: `AsyncIterator[GenerationOutput]` - Calls: self.start, kwargs.pop, str, id, prompt.startswith, tokenizer.encode, len, logger.warning, self._stream_generate_specprefill, self._acquire_generation_slot, time.time, _bind_worker_generation_streams, self._model.stream_generate, hasattr, self._active_requests[request_id].update, round, getattr, GenerationOutput, self._model.tokenizer.encode, self._active_requests.pop - State reads: self._loaded, self.start, self._is_mllm, self._draft_model, self._model.tokenizer, self._model, self._specprefill_threshold, self._stream_generate_specprefill, self._acquire_generation_slot, self._active_requests, self._model.stream_generate, self._model.tokenizer.encode, self._active_requests.pop - Return expressions: None ## `vllm_mlx.engine.simple.SimpleEngine.chat` - Kind: method - Signature: `async def chat(self, messages: list[dict[str, Any]], max_tokens: int=256, temperature: float=0.7, top_p: float=0.9, tools: list[dict] | None=None, images: list[str] | None=None, videos: list[str] | None=None, **kwargs) -> GenerationOutput` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/simple.py#L1014-L1144 - Implementation: Method `SimpleEngine.chat` calls `self.start`, `dict`, `kwargs.pop`, `aggregate_stream_chat`; awaits asynchronous work; has 3 explicit return paths. Chat completion (non-streaming). Args: messages: List of chat messages max_tokens: Maximum tokens to generate temperature: Sampling temperature top_p: Top-p sampling tools: Optional tool definitions images: Optional image URLs/paths videos: Optional video URLs/paths **kwargs: Additional model-specific parameters Returns: GenerationOutput with assistant response - Inputs: - `messages` (list[dict[str, Any]]; required): List of chat messages - `max_tokens` (int; optional; default `256`): Maximum tokens to generate - `temperature` (float; optional; default `0.7`): Sampling temperature - `top_p` (float; optional; default `0.9`): Top-p sampling - `tools` (list[dict] | None; optional; default `None`): Optional tool definitions - `images` (list[str] | None; optional; default `None`): Optional image URLs/paths - `videos` (list[str] | None; optional; default `None`): Optional video URLs/paths - `**kwargs` (not annotated; optional): Additional model-specific parameters - Return annotation: `GenerationOutput` - Calls: self.start, dict, kwargs.pop, aggregate_stream_chat, kwargs.get, has_media_content, convert_tools_for_template, self._run_blocking_serialized, clean_output_text, GenerationOutput, getattr, tokenizer.apply_chat_template, len - State reads: self._loaded, self.start, self._is_mllm, self._run_blocking_serialized, self._model.chat, self._model, self._model.tokenizer - Return expressions: await aggregate_stream_chat(); GenerationOutput(text=text, prompt_tokens=output.prompt_tokens, completion_tokens=output.completion_tokens, finish_reas…; GenerationOutput(text=text, tokens=output.tokens, prompt_tokens=prompt_token_count, completion_tokens=len(output.tokens… ## `vllm_mlx.engine.simple.SimpleEngine.chat.aggregate_stream_chat` - Kind: nested function - Signature: `async def aggregate_stream_chat() -> GenerationOutput` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/simple.py#L1046-L1069 - Implementation: Nested Function `SimpleEngine.chat.aggregate_stream_chat` calls `GenerationOutput`, `self.stream_chat`, `clean_output_text`, `list`; returns `GenerationOutput(text=text, tokens=list(final_output.tokens), prompt_tokens=final_output.prompt_tokens, completion_toke…`. Nested Function `SimpleEngine.chat.aggregate_stream_chat` calls `GenerationOutput`, `self.stream_chat`, `clean_output_text`, `list`; returns `GenerationOutput(text=text, tokens=list(final_output.tokens), prompt_tokens=final_output.prompt_tokens, completion_toke…`. - Inputs: none - Return annotation: `GenerationOutput` - Calls: GenerationOutput, self.stream_chat, clean_output_text, list - State reads: self.stream_chat - Return expressions: GenerationOutput(text=text, tokens=list(final_output.tokens), prompt_tokens=final_output.prompt_tokens, completion_toke… ## `vllm_mlx.engine.simple.SimpleEngine.stream_chat` - Kind: method - Signature: `async def stream_chat(self, messages: list[dict[str, Any]], max_tokens: int=256, temperature: float=0.7, top_p: float=0.9, tools: list[dict] | None=None, images: list[str] | None=None, videos: list[str] | None=None, **kwargs) -> AsyncIterator[GenerationOutput]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/simple.py#L1146-L1171 - Implementation: Method `SimpleEngine.stream_chat` calls `self._track_request_stream`, `self._stream_chat_impl`; yields values incrementally. Public stream-chat wrapper with request stats tracking. - Inputs: - `messages` (list[dict[str, Any]]; required): Required positional or keyword input. - `max_tokens` (int; optional; default `256`): Optional positional or keyword input; defaults to `256`. - `temperature` (float; optional; default `0.7`): Optional positional or keyword input; defaults to `0.7`. - `top_p` (float; optional; default `0.9`): Optional positional or keyword input; defaults to `0.9`. - `tools` (list[dict] | None; optional; default `None`): Optional positional or keyword input; defaults to `None`. - `images` (list[str] | None; optional; default `None`): Optional positional or keyword input; defaults to `None`. - `videos` (list[str] | None; optional; default `None`): Optional positional or keyword input; defaults to `None`. - `**kwargs` (not annotated; optional): Additional variadic keyword inputs accepted by this callable. - Return annotation: `AsyncIterator[GenerationOutput]` - Calls: self._track_request_stream, self._stream_chat_impl - State reads: self._track_request_stream, self._stream_chat_impl ## `vllm_mlx.engine.simple.SimpleEngine._stream_chat_impl` - Kind: method - Signature: `async def _stream_chat_impl(self, messages: list[dict[str, Any]], max_tokens: int=256, temperature: float=0.7, top_p: float=0.9, tools: list[dict] | None=None, images: list[str] | None=None, videos: list[str] | None=None, **kwargs) -> AsyncIterator[GenerationOutput]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/simple.py#L1173-L1794 - Implementation: Method `SimpleEngine._stream_chat_impl` calls `self.start`, `dict`, `kwargs.pop`, `bool`; awaits asynchronous work; yields values incrementally; can raise `payload`; returns `None`. Stream chat completion token by token. Args: messages: List of chat messages max_tokens: Maximum tokens to generate temperature: Sampling temperature top_p: Top-p sampling tools: Optional tool definitions images: Optional image URLs/paths videos: Optional video URLs/paths **kwargs: Additional model-specific parameters Yields: GenerationOutput with incremental text - Inputs: - `messages` (list[dict[str, Any]]; required): List of chat messages - `max_tokens` (int; optional; default `256`): Maximum tokens to generate - `temperature` (float; optional; default `0.7`): Sampling temperature - `top_p` (float; optional; default `0.9`): Top-p sampling - `tools` (list[dict] | None; optional; default `None`): Optional tool definitions - `images` (list[str] | None; optional; default `None`): Optional image URLs/paths - `videos` (list[str] | None; optional; default `None`): Optional video URLs/paths - `**kwargs` (not annotated; optional): Additional model-specific parameters - Return annotation: `AsyncIterator[GenerationOutput]` - Calls: self.start, dict, kwargs.pop, bool, has_media_content, convert_tools_for_template, self._should_route_text_through_text_model, hasattr, logger.info, self._stream_generate_text, str, id, getattr, self._model._collect_video_inputs, mllm_call_kwargs, self._acquire_generation_slot, _bind_worker_generation_streams, self._model.stream_chat, GenerationOutput, self._run_blocking_serialized, self._model_name.lower, template_kwargs.update, normalize_messages_for_chat_template, chat_template_kwargs.get, _harmony_render_messages, tokenizer.apply_chat_template, chat_template_kwargs.keys, '\n'.join, kwargs.get, cache_blocking_controls.append, _to_msg_dict, any, m.get, _with_user, isinstance, range, min, len, hashlib.sha256(system_prefix_text.encode()).hexdigest, hashlib.sha256, system_prefix_text.encode, prompt.startswith, tokenizer.encode, self._system_kv_cache.get, asyncio.get_running_loop, asyncio.Queue, threading.Event, asyncio.create_task, _produce_responses, response_queue.get, logger.warning, producer_task.done, abort_event.set, self.stream_generate - State reads: self._loaded, self.start, self._is_mllm, self._text_model, self._should_route_text_through_text_model, self._text_model.mtp, self._mtp, self._stream_generate_text, self._model, self._model._collect_video_inputs, self._acquire_generation_slot, self._model.stream_chat, self._run_blocking_serialized, self._model.tokenizer, self._model_name.lower, self._model_name, self._draft_model, self._max_kv_size, self._supports_system_kv_cache, self._system_kv_cache.get, self._system_kv_cache, self.stream_generate - Raises directly: payload - Return expressions: None ## `vllm_mlx.engine.simple.SimpleEngine._stream_chat_impl.mllm_call_kwargs` - Kind: nested function - Signature: `def mllm_call_kwargs() -> dict` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/simple.py#L1236-L1242 - Implementation: Nested Function `SimpleEngine._stream_chat_impl.mllm_call_kwargs` calls `dict`; returns `local_kwargs`. Nested Function `SimpleEngine._stream_chat_impl.mllm_call_kwargs` calls `dict`; returns `local_kwargs`. - Inputs: none - Return annotation: `dict` - Calls: dict - Return expressions: local_kwargs ## `vllm_mlx.engine.simple.SimpleEngine._stream_chat_impl.run_native_video` - Kind: nested function - Signature: `def run_native_video()` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/simple.py#L1299-L1309 - Implementation: Nested Function `SimpleEngine._stream_chat_impl.run_native_video` calls `mllm_call_kwargs`, `list`, `self._model.stream_chat`; returns `list(self._model.stream_chat(messages=messages, max_tokens=max_tokens, temperature=temperature, tools=template_tools, *…`. Nested Function `SimpleEngine._stream_chat_impl.run_native_video` calls `mllm_call_kwargs`, `list`, `self._model.stream_chat`; returns `list(self._model.stream_chat(messages=messages, max_tokens=max_tokens, temperature=temperature, tools=template_tools, *…`. - Inputs: none - Return annotation: `not annotated` - Calls: mllm_call_kwargs, list, self._model.stream_chat - State reads: self._model.stream_chat, self._model - Return expressions: list(self._model.stream_chat(messages=messages, max_tokens=max_tokens, temperature=temperature, tools=template_tools, *… ## `vllm_mlx.engine.simple.SimpleEngine._stream_chat_impl._to_msg_dict` - Kind: nested function - Signature: `def _to_msg_dict(m: Any) -> dict[str, Any]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/simple.py#L1499-L1509 - Implementation: Nested Function `SimpleEngine._stream_chat_impl._to_msg_dict` calls `isinstance`, `hasattr`, `m.model_dump`, `m.dict`; has 4 explicit return paths. Nested Function `SimpleEngine._stream_chat_impl._to_msg_dict` calls `isinstance`, `hasattr`, `m.model_dump`, `m.dict`; has 4 explicit return paths. - Inputs: - `m` (Any; required): Required positional or keyword input. - Return annotation: `dict[str, Any]` - Calls: isinstance, hasattr, m.model_dump, m.dict, getattr - Return expressions: m; m.model_dump(); m.dict(); {'role': getattr(m, 'role', None), 'content': getattr(m, 'content', '')} ## `vllm_mlx.engine.simple.SimpleEngine._stream_chat_impl._with_user` - Kind: nested function - Signature: `def _with_user(user_content: str) -> list[dict[str, Any]]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/simple.py#L1519-L1525 - Implementation: Nested Function `SimpleEngine._stream_chat_impl._with_user` calls `dict`, `msgs[-1].get`; returns `msgs`. Nested Function `SimpleEngine._stream_chat_impl._with_user` calls `dict`, `msgs[-1].get`; returns `msgs`. - Inputs: - `user_content` (str; required): Required positional or keyword input. - Return annotation: `list[dict[str, Any]]` - Calls: dict, msgs[-1].get - Return expressions: msgs ## `vllm_mlx.engine.simple.SimpleEngine._stream_chat_impl._emit_response` - Kind: nested function - Signature: `def _emit_response(resp: Any) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/simple.py#L1609-L1612 - Implementation: Nested Function `SimpleEngine._stream_chat_impl._emit_response` calls `abort_event.is_set`, `loop.call_soon_threadsafe`; returns `None`. Nested Function `SimpleEngine._stream_chat_impl._emit_response` calls `abort_event.is_set`, `loop.call_soon_threadsafe`; returns `None`. - Inputs: - `resp` (Any; required): Required positional or keyword input. - Return annotation: `None` - Calls: abort_event.is_set, loop.call_soon_threadsafe - Return expressions: None ## `vllm_mlx.engine.simple.SimpleEngine._stream_chat_impl._emit_done` - Kind: nested function - Signature: `def _emit_done() -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/simple.py#L1614-L1615 - Implementation: Nested Function `SimpleEngine._stream_chat_impl._emit_done` calls `loop.call_soon_threadsafe`. Nested Function `SimpleEngine._stream_chat_impl._emit_done` calls `loop.call_soon_threadsafe`. - Inputs: none - Return annotation: `None` - Calls: loop.call_soon_threadsafe ## `vllm_mlx.engine.simple.SimpleEngine._stream_chat_impl._emit_error` - Kind: nested function - Signature: `def _emit_error(exc: BaseException) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/simple.py#L1617-L1618 - Implementation: Nested Function `SimpleEngine._stream_chat_impl._emit_error` calls `loop.call_soon_threadsafe`. Nested Function `SimpleEngine._stream_chat_impl._emit_error` calls `loop.call_soon_threadsafe`. - Inputs: - `exc` (BaseException; required): Required positional or keyword input. - Return annotation: `None` - Calls: loop.call_soon_threadsafe ## `vllm_mlx.engine.simple.SimpleEngine._stream_chat_impl._run_with_cache` - Kind: nested function - Signature: `def _run_with_cache() -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/simple.py#L1620-L1705 - Implementation: Nested Function `SimpleEngine._stream_chat_impl._run_with_cache` calls `make_sampler`, `make_prompt_cache`, `self._restore_prompt_cache`, `self._system_kv_cache.move_to_end`. Nested Function `SimpleEngine._stream_chat_impl._run_with_cache` calls `make_sampler`, `make_prompt_cache`, `self._restore_prompt_cache`, `self._system_kv_cache.move_to_end`. - Inputs: none - Return annotation: `None` - Calls: make_sampler, make_prompt_cache, self._restore_prompt_cache, self._system_kv_cache.move_to_end, mx.array, model, self._eval_cache_snapshot, mx.clear_cache, self._snapshot_prompt_cache, len, self._system_kv_cache.popitem, logger.info, sum, mlx_stream_generate, abort_event.is_set, _emit_response - State reads: self._model.model, self._model, self._restore_prompt_cache, self._system_kv_cache, self._system_kv_cache.move_to_end, self._system_kv_cache_stats, self._prefill_step_size, self._eval_cache_snapshot, self._snapshot_prompt_cache, self._system_kv_capacity, self._system_kv_cache.popitem ## `vllm_mlx.engine.simple.SimpleEngine._stream_chat_impl._produce_responses` - Kind: nested function - Signature: `async def _produce_responses() -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/simple.py#L1707-L1718 - Implementation: Nested Function `SimpleEngine._stream_chat_impl._produce_responses` calls `self._run_blocking_serialized`, `_emit_error`, `_emit_done`; awaits asynchronous work. Nested Function `SimpleEngine._stream_chat_impl._produce_responses` calls `self._run_blocking_serialized`, `_emit_error`, `_emit_done`; awaits asynchronous work. - Inputs: none - Return annotation: `None` - Calls: self._run_blocking_serialized, _emit_error, _emit_done - State reads: self._run_blocking_serialized ## `vllm_mlx.engine.simple.SimpleEngine._stream_generate_specprefill` - Kind: method - Signature: `async def _stream_generate_specprefill(self, prompt: str, tokens: list[int], max_tokens: int, temperature: float, top_p: float, stop: list[str] | None=None, specprefill_keep_pct: float | None=None, specprefill_backbone_pct: float | None=None, **kwargs) -> AsyncIterator[GenerationOutput]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/simple.py#L1796-L2000 - Implementation: Method `SimpleEngine._stream_generate_specprefill` calls `len`, `Event`, `self._run_blocking_serialized`, `enumerate`; awaits asynchronous work; yields values incrementally. SpecPrefill path for non-MTP models (Nemotron, GPT-OSS, etc). Scores token importance with the draft model, sparse-prefills the target model, then generates autoregressively. Falls back to normal generation on any error. - Inputs: - `prompt` (str; required): Required positional or keyword input. - `tokens` (list[int]; required): Required positional or keyword input. - `max_tokens` (int; required): Required positional or keyword input. - `temperature` (float; required): Required positional or keyword input. - `top_p` (float; required): Required positional or keyword input. - `stop` (list[str] | None; optional; default `None`): Optional positional or keyword input; defaults to `None`. - `specprefill_keep_pct` (float | None; optional; default `None`): Optional positional or keyword input; defaults to `None`. - `specprefill_backbone_pct` (float | None; optional; default `None`): Optional positional or keyword input; defaults to `None`. - `**kwargs` (not annotated; optional): Additional variadic keyword inputs accepted by this callable. - Return annotation: `AsyncIterator[GenerationOutput]` - Calls: len, Event, self._run_blocking_serialized, enumerate, GenerationOutput - State reads: self._model.model, self._model, self._model.tokenizer, self._run_blocking_serialized ## `vllm_mlx.engine.simple.SimpleEngine._stream_generate_specprefill._request_cancel` - Kind: nested function - Signature: `def _request_cancel() -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/simple.py#L1821-L1822 - Implementation: Nested Function `SimpleEngine._stream_generate_specprefill._request_cancel` calls `cancel_requested.set`. Nested Function `SimpleEngine._stream_generate_specprefill._request_cancel` calls `cancel_requested.set`. - Inputs: none - Return annotation: `None` - Calls: cancel_requested.set ## `vllm_mlx.engine.simple.SimpleEngine._stream_generate_specprefill._cancel_check` - Kind: nested function - Signature: `def _cancel_check() -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/simple.py#L1824-L1826 - Implementation: Nested Function `SimpleEngine._stream_generate_specprefill._cancel_check` calls `cancel_requested.is_set`, `_SpecPrefillCancelled`; can raise `_SpecPrefillCancelled`. Nested Function `SimpleEngine._stream_generate_specprefill._cancel_check` calls `cancel_requested.is_set`, `_SpecPrefillCancelled`; can raise `_SpecPrefillCancelled`. - Inputs: none - Return annotation: `None` - Calls: cancel_requested.is_set, _SpecPrefillCancelled - Raises directly: _SpecPrefillCancelled ## `vllm_mlx.engine.simple.SimpleEngine._stream_generate_specprefill._run_all` - Kind: nested function - Signature: `def _run_all()` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/simple.py#L1828-L1835 - Implementation: Nested Function `SimpleEngine._stream_generate_specprefill._run_all` calls `_run_specprefill`, `logger.error`, `_run_normal`; has 2 explicit return paths. Nested Function `SimpleEngine._stream_generate_specprefill._run_all` calls `_run_specprefill`, `logger.error`, `_run_normal`; has 2 explicit return paths. - Inputs: none - Return annotation: `not annotated` - Calls: _run_specprefill, logger.error, _run_normal - Return expressions: _run_specprefill(); _run_normal() ## `vllm_mlx.engine.simple.SimpleEngine._stream_generate_specprefill._run_specprefill` - Kind: nested function - Signature: `def _run_specprefill()` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/simple.py#L1837-L1939 - Implementation: Nested Function `SimpleEngine._stream_generate_specprefill._run_specprefill` calls `make_prompt_cache`, `time.monotonic`, `score_tokens`, `_cancel_check`; returns `results`. Score tokens, sparse prefill, generate autoregressively. - Inputs: none - Return annotation: `not annotated` - Calls: make_prompt_cache, time.monotonic, score_tokens, _cancel_check, select_chunks, sparse_prefill, logger.info, make_sampler, sampler(logits[:, -1, :]).item, sampler, tokenizer.decode, SimpleNamespace, self._model.stream_generate, mx.array, hasattr, str, results.append, getattr, cleanup_rope - State reads: self._max_kv_size, self._draft_model, self._prefill_step_size, self._specprefill_keep_pct, self._specprefill_backbone_pct, self._model.stream_generate, self._model - Return expressions: results ## `vllm_mlx.engine.simple.SimpleEngine._stream_generate_specprefill._run_normal` - Kind: nested function - Signature: `def _run_normal()` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/simple.py#L1941-L1962 - Implementation: Nested Function `SimpleEngine._stream_generate_specprefill._run_normal` calls `self._model.stream_generate`, `_cancel_check`, `hasattr`, `str`; returns `results`. Fallback: normal generation without specprefill. - Inputs: none - Return annotation: `not annotated` - Calls: self._model.stream_generate, _cancel_check, hasattr, str, results.append, SimpleNamespace, getattr - State reads: self._model.stream_generate, self._model - Return expressions: results ## `vllm_mlx.engine.simple.SimpleEngine._stream_generate_text` - Kind: method - Signature: `async def _stream_generate_text(self, messages: list[dict[str, Any]], max_tokens: int, temperature: float, top_p: float, tools: list | None=None, **kwargs) -> AsyncIterator[GenerationOutput]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/simple.py#L2002-L2734 - Implementation: Method `SimpleEngine._stream_generate_text` calls `kwargs.pop`, `dict`, `threading.Event`, `os.environ.get`; awaits asynchronous work; yields values incrementally; can raise `payload`. Text-only generation via mlx_lm TextModel. Used when text-only MLLM routing is active and the request has no media. Runs the full generation in a single thread to maintain Metal safety. System prompt KV caching: on the first request, prefills system tokens and snapshots backbone KV state. Subsequent requests with the same system prompt restore the snapshot and only prefill the suffix tokens. - Inputs: - `messages` (list[dict[str, Any]]; required): Required positional or keyword input. - `max_tokens` (int; required): Required positional or keyword input. - `temperature` (float; required): Required positional or keyword input. - `top_p` (float; required): Required positional or keyword input. - `tools` (list | None; optional; default `None`): Optional positional or keyword input; defaults to `None`. - `**kwargs` (not annotated; optional): Additional variadic keyword inputs accepted by this callable. - Return annotation: `AsyncIterator[GenerationOutput]` - Calls: kwargs.pop, dict, threading.Event, os.environ.get, enable_thinking_env.lower, template_kwargs.update, normalize_messages_for_chat_template, self._text_tokenizer.apply_chat_template, template_kwargs.pop, make_sampler, make_logits_processors, bool, cache_blocking_controls.append, logger.info, any, m.get, full_prompt.find, hashlib.sha256(system_prefix_text.encode()).hexdigest, hashlib.sha256, system_prefix_text.encode, full_prompt.startswith, tokenizer.encode, len, self._system_kv_cache.get, self._run_blocking_serialized, self._system_kv_cache.move_to_end, logger.debug, logger.warning, asyncio.get_running_loop, asyncio.Queue, asyncio.create_task, _produce_responses, response_queue.get, hasattr, str, getattr, GenerationOutput, producer_task.done, abort_event.set - State reads: self._text_tokenizer.apply_chat_template, self._text_tokenizer, self._supports_system_kv_cache, self._text_model, self._system_kv_cache.get, self._system_kv_cache, self._run_blocking_serialized, self._system_kv_cache.move_to_end, self._system_kv_cache_stats, self._draft_model, self._specprefill_threshold - Raises directly: payload ## `vllm_mlx.engine.simple.SimpleEngine._stream_generate_text.make_cache_with_snapshot` - Kind: nested function - Signature: `def make_cache_with_snapshot(text_model, system_kv_snapshot, _max_kv_size=self._max_kv_size)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/simple.py#L2159-L2176 - Implementation: Nested Function `SimpleEngine._stream_generate_text.make_cache_with_snapshot` calls `make_prompt_cache`, `SimpleEngine._restore_prompt_cache`, `mx.array`; returns `(backbone_cache, prompt_to_send)`. Nested Function `SimpleEngine._stream_generate_text.make_cache_with_snapshot` calls `make_prompt_cache`, `SimpleEngine._restore_prompt_cache`, `mx.array`; returns `(backbone_cache, prompt_to_send)`. - Inputs: - `text_model` (not annotated; required): Required positional or keyword input. - `system_kv_snapshot` (not annotated; required): Required positional or keyword input. - `_max_kv_size` (not annotated; optional; default `self._max_kv_size`): Optional positional or keyword input; defaults to `self._max_kv_size`. - Return annotation: `not annotated` - Calls: make_prompt_cache, SimpleEngine._restore_prompt_cache, mx.array - Return expressions: (backbone_cache, prompt_to_send) ## `vllm_mlx.engine.simple.SimpleEngine._stream_generate_text._emit_response` - Kind: nested function - Signature: `def _emit_response(resp: Any) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/simple.py#L2272-L2275 - Implementation: Nested Function `SimpleEngine._stream_generate_text._emit_response` calls `abort_event.is_set`, `loop.call_soon_threadsafe`; returns `None`. Nested Function `SimpleEngine._stream_generate_text._emit_response` calls `abort_event.is_set`, `loop.call_soon_threadsafe`; returns `None`. - Inputs: - `resp` (Any; required): Required positional or keyword input. - Return annotation: `None` - Calls: abort_event.is_set, loop.call_soon_threadsafe - Return expressions: None ## `vllm_mlx.engine.simple.SimpleEngine._stream_generate_text._emit_done` - Kind: nested function - Signature: `def _emit_done() -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/simple.py#L2277-L2278 - Implementation: Nested Function `SimpleEngine._stream_generate_text._emit_done` calls `loop.call_soon_threadsafe`. Nested Function `SimpleEngine._stream_generate_text._emit_done` calls `loop.call_soon_threadsafe`. - Inputs: none - Return annotation: `None` - Calls: loop.call_soon_threadsafe ## `vllm_mlx.engine.simple.SimpleEngine._stream_generate_text._emit_error` - Kind: nested function - Signature: `def _emit_error(exc: BaseException) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/simple.py#L2280-L2281 - Implementation: Nested Function `SimpleEngine._stream_generate_text._emit_error` calls `loop.call_soon_threadsafe`. Nested Function `SimpleEngine._stream_generate_text._emit_error` calls `loop.call_soon_threadsafe`. - Inputs: - `exc` (BaseException; required): Required positional or keyword input. - Return annotation: `None` - Calls: loop.call_soon_threadsafe ## `vllm_mlx.engine.simple.SimpleEngine._stream_generate_text._seed_from_last_response` - Kind: nested function - Signature: `def _seed_from_last_response(prompt_cache, last_resp)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/simple.py#L2283-L2291 - Implementation: Nested Function `SimpleEngine._stream_generate_text._seed_from_last_response` calls `getattr`, `cache_module.trim_prompt_cache`, `mx.array`, `self._text_tokenizer.encode`; has 2 explicit return paths. Nested Function `SimpleEngine._stream_generate_text._seed_from_last_response` calls `getattr`, `cache_module.trim_prompt_cache`, `mx.array`, `self._text_tokenizer.encode`; has 2 explicit return paths. - Inputs: - `prompt_cache` (not annotated; required): Required positional or keyword input. - `last_resp` (not annotated; required): Required positional or keyword input. - Return annotation: `not annotated` - Calls: getattr, cache_module.trim_prompt_cache, mx.array, self._text_tokenizer.encode - State reads: self._text_tokenizer.encode, self._text_tokenizer - Return expressions: mx.array([last_tok], dtype=mx.uint32); mx.array(self._text_tokenizer.encode(getattr(last_resp, 'text', '')), dtype=mx.uint32) ## `vllm_mlx.engine.simple.SimpleEngine._stream_generate_text._resume_after_processor_retirement` - Kind: nested function - Signature: `def _resume_after_processor_retirement(model, prompt_cache, prompt, remaining_tokens: int) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/simple.py#L2293-L2320 - Implementation: Nested Function `SimpleEngine._stream_generate_text._resume_after_processor_retirement` calls `dict`, `hasattr`, `model.make_mtp_cache`, `mlx_stream_generate`. Nested Function `SimpleEngine._stream_generate_text._resume_after_processor_retirement` calls `dict`, `hasattr`, `model.make_mtp_cache`, `mlx_stream_generate`. - Inputs: - `model` (not annotated; required): Required positional or keyword input. - `prompt_cache` (not annotated; required): Required positional or keyword input. - `prompt` (not annotated; required): Required positional or keyword input. - `remaining_tokens` (int; required): Required positional or keyword input. - Return annotation: `None` - Calls: dict, hasattr, model.make_mtp_cache, mlx_stream_generate, abort_event.is_set, logger.info, _emit_response - State reads: self._prefill_step_size, self._mtp_num_draft_tokens, self._text_tokenizer ## `vllm_mlx.engine.simple.SimpleEngine._stream_generate_text._run_all` - Kind: nested function - Signature: `def _run_all()` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/simple.py#L2323-L2485 - Implementation: Nested Function `SimpleEngine._stream_generate_text._run_all` calls `_processors_can_retire`, `hasattr`, `logger.info`, `make_prompt_cache`; returns `None`. Nested Function `SimpleEngine._stream_generate_text._run_all` calls `_processors_can_retire`, `hasattr`, `logger.info`, `make_prompt_cache`; returns `None`. - Inputs: none - Return annotation: `not annotated` - Calls: _processors_can_retire, hasattr, logger.info, make_prompt_cache, mx.array, model, self._eval_cache_snapshot, mx.clear_cache, self._snapshot_prompt_cache, self._system_kv_cache.move_to_end, len, self._system_kv_cache.popitem, sum, _run_specprefill, logger.error, model.make_mtp_cache, dict, mlx_stream_generate, abort_event.is_set, _emit_response, _processors_retired, _seed_from_last_response, _resume_after_processor_retirement - State reads: self._text_model, self._mtp, self._max_kv_size, self._prefill_step_size, self._eval_cache_snapshot, self._snapshot_prompt_cache, self._system_kv_cache, self._system_kv_cache.move_to_end, self._system_kv_capacity, self._system_kv_cache.popitem, self._system_kv_cache_stats, self._mtp_num_draft_tokens, self._text_tokenizer - Return expressions: None ## `vllm_mlx.engine.simple.SimpleEngine._stream_generate_text._run_specprefill` - Kind: nested function - Signature: `def _run_specprefill(model, bc, use_mtp)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/simple.py#L2487-L2664 - Implementation: Nested Function `SimpleEngine._stream_generate_text._run_specprefill` calls `make_prompt_cache`, `time.monotonic`, `score_tokens`, `select_chunks`; returns `None`. Score tokens, sparse prefill, then continue on the standard decode path. - Inputs: - `model` (not annotated; required): Required positional or keyword input. - `bc` (not annotated; required): Required positional or keyword input. - `use_mtp` (not annotated; required): Required positional or keyword input. - Return annotation: `not annotated` - Calls: make_prompt_cache, time.monotonic, score_tokens, select_chunks, len, sparse_prefill, logger.info, mx.array, _seed_logits_processors, _sample_with_processors, logits[:, -1, :].squeeze, mx.eval, y.item, generated_ids.append, self._text_tokenizer.decode, _emit_response, SimpleNamespace, abort_event.is_set, hasattr, model.make_mtp_cache, _processors_retired, _resume_after_processor_retirement, mlx_stream_generate, _seed_from_last_response, cleanup_rope - State reads: self._max_kv_size, self._draft_model, self._prefill_step_size, self._specprefill_keep_pct, self._specprefill_backbone_pct, self._text_tokenizer.eos_token_id, self._text_tokenizer, self._text_tokenizer.decode - Return expressions: None ## `vllm_mlx.engine.simple.SimpleEngine._stream_generate_text._produce_responses` - Kind: nested function - Signature: `async def _produce_responses() -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/simple.py#L2666-L2677 - Implementation: Nested Function `SimpleEngine._stream_generate_text._produce_responses` calls `self._run_blocking_serialized`, `_emit_error`, `_emit_done`; awaits asynchronous work. Nested Function `SimpleEngine._stream_generate_text._produce_responses` calls `self._run_blocking_serialized`, `_emit_error`, `_emit_done`; awaits asynchronous work. - Inputs: none - Return annotation: `None` - Calls: self._run_blocking_serialized, _emit_error, _emit_done - State reads: self._run_blocking_serialized ## `vllm_mlx.engine.simple.SimpleEngine.get_stats` - Kind: method - Signature: `def get_stats(self) -> dict[str, Any]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/simple.py#L2736-L2858 - Implementation: Method `SimpleEngine.get_stats` calls `sum`, `time.time`, `self._active_requests.values`, `dict`; returns `stats`. Get engine statistics. - Inputs: none - Return annotation: `dict[str, Any]` - Calls: sum, time.time, self._active_requests.values, dict, requests_snapshot.append, self._generation_lock.locked, self._model.get_cache_stats, raw_cache.get, float, round, self._system_kv_cache.items, isinstance, len, slots.append, mx.metal.is_available, mx.get_active_memory, mx.get_peak_memory, mx.get_cache_memory - State reads: self._recent_completions, self._active_requests.values, self._active_requests, self._model_name, self._created_at, self._is_mllm, self._loaded, self._num_running, self._generation_waiters, self._total_requests_processed, self._total_prompt_tokens, self._total_completion_tokens, self._generation_lock.locked, self._generation_lock, self._generation_lock_admission, self._generation_busy_rejections, self._model, self._model.get_cache_stats, self._draft_model, self._specprefill_draft_model_path, self._specprefill_threshold, self._specprefill_keep_pct, self._specprefill_backbone_pct, self._system_kv_cache, self._system_kv_cache.items, self._system_kv_cache_stats, self._system_kv_capacity - Return expressions: stats ## `vllm_mlx.engine.simple.SimpleEngine.get_cache_stats` - Kind: method - Signature: `def get_cache_stats(self) -> dict[str, Any] | None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/simple.py#L2860-L2878 - Implementation: Method `SimpleEngine.get_cache_stats` calls `dict`, `round`, `len`, `self._model.get_cache_stats`; returns `result or None`. Get cache statistics for the system-prompt KV LRU plus, when the model is multimodal, the MLLM's own cache stats. - Inputs: none - Return annotation: `dict[str, Any] | None` - Calls: dict, round, len, self._model.get_cache_stats - State reads: self._supports_system_kv_cache, self._system_kv_cache_stats, self._system_kv_capacity, self._system_kv_cache, self._is_mllm, self._model, self._model.get_cache_stats - Return expressions: result or None ## `vllm_mlx.engine.simple.SimpleEngine.clear_runtime_caches` - Kind: method - Signature: `def clear_runtime_caches(self) -> dict[str, Any] | None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/simple.py#L2880-L2912 - Implementation: Method `SimpleEngine.clear_runtime_caches` calls `len`, `any`, `self._system_kv_cache_stats.values`, `self._system_kv_cache.clear`; returns `result or None`. Clear engine-managed runtime caches. Includes the multi-slot system-prompt KV LRU — each retained snapshot is multi-GB on the Metal heap, so DELETE /v1/cache must drop them or the operator's reset is silently incomplete. Counters reset alongside so /v1/cache/stats reflects the cleared state immediately. OrderedDict ops are atomic under the GIL: a concurrent worker that has already captured a tuple reference from .get() finishes safely against its own copy; any new request after this call hits MISS and repopulates from scratch. No need to acquire _generation_lock for the clear itself. - Inputs: none - Return annotation: `dict[str, Any] | None` - Calls: len, any, self._system_kv_cache_stats.values, self._system_kv_cache.clear, mx.clear_cache, self._model.clear_cache - State reads: self._system_kv_cache, self._system_kv_cache_stats.values, self._system_kv_cache_stats, self._system_kv_cache.clear, self._is_mllm, self._model, self._model.clear_cache - Return expressions: result or None # Module `vllm_mlx.engine_core` Engine Core for vllm-mlx continuous batching. This module provides the EngineCore class that coordinates: - Model loading and management - Request scheduling via Scheduler - Async request processing - Output streaming The design follows vLLM's engine architecture adapted for MLX. Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine_core.py#L1-L794 ## `vllm_mlx.engine_core._is_stream_thread_error` - Kind: function - Signature: `def _is_stream_thread_error(error: Exception) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine_core.py#L33-L36 - Implementation: Function `_is_stream_thread_error` calls `str`; returns `'no Stream(' in message or 'no Stream(gpu' in message`. True when MLX reports stream ownership mismatch across threads. - Inputs: - `error` (Exception; required): Required positional or keyword input. - Return annotation: `bool` - Calls: str - Return expressions: 'no Stream(' in message or 'no Stream(gpu' in message ## `vllm_mlx.engine_core.EngineConfig` - Kind: class - Signature: `class EngineConfig` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine_core.py#L40-L47 - Implementation: Class `EngineConfig` declares 0 direct member(s). Configuration for the engine. - Inputs: - `model_name` (str; optional; default `''`): Optional constructor field; defaults to `''`. - `scheduler_config` (Optional[SchedulerConfig]; optional; default `None`): Optional constructor field; defaults to `None`. - `step_interval` (float; optional; default `0.001`): Optional constructor field; defaults to `0.001`. - `stream_interval` (int; optional; default `1`): Optional constructor field; defaults to `1`. - `gpu_memory_utilization` (float; optional; default `0.9`): Optional constructor field; defaults to `0.9`. - Constructs: `vllm_mlx.engine_core.EngineConfig` - Decorators: dataclass ## `vllm_mlx.engine_core.EngineCore` - Kind: class - Signature: `class EngineCore` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine_core.py#L50-L698 - Implementation: Class `EngineCore` declares 21 direct member(s). Core engine for vllm-mlx inference with continuous batching. This engine runs the generation loop and manages request lifecycle. It provides both sync and async interfaces for request handling. - Inputs: - `model` (Any; required): The MLX model - `tokenizer` (Any; required): The tokenizer - `config` (Optional[EngineConfig]; optional; default `None`): Engine configuration - `engine_id` (Optional[str]; optional; default `None`): Optional unique ID for this engine (auto-generated if None) - `force_model_ownership` (bool; optional; default `True`): If True (default), forcibly take model ownership from any existing engine. If False, raises ModelOwnershipError if model is in use. - Constructs: `vllm_mlx.engine_core.EngineCore` ## `vllm_mlx.engine_core.EngineCore.__init__` - Kind: method - Signature: `def __init__(self, model: Any, tokenizer: Any, config: Optional[EngineConfig]=None, engine_id: Optional[str]=None, force_model_ownership: bool=True)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine_core.py#L58-L114 - Implementation: Method `EngineCore.__init__` updates `self.model`, `self.tokenizer`, `self.config`, `self._engine_id`; calls `EngineConfig`, `str`, `uuid.uuid4`, `get_registry`. Initialize the engine. Args: model: The MLX model tokenizer: The tokenizer config: Engine configuration engine_id: Optional unique ID for this engine (auto-generated if None) force_model_ownership: If True (default), forcibly take model ownership from any existing engine. If False, raises ModelOwnershipError if model is in use. - Inputs: - `model` (Any; required): The MLX model - `tokenizer` (Any; required): The tokenizer - `config` (Optional[EngineConfig]; optional; default `None`): Engine configuration - `engine_id` (Optional[str]; optional; default `None`): Optional unique ID for this engine (auto-generated if None) - `force_model_ownership` (bool; optional; default `True`): If True (default), forcibly take model ownership from any existing engine. If False, raises ModelOwnershipError if model is in use. - Return annotation: `not annotated` - Calls: EngineConfig, str, uuid.uuid4, get_registry, registry.acquire, SchedulerConfig, Scheduler, logger.debug - State reads: self._engine_id, self.config.scheduler_config, self.config - State writes: self.model, self.tokenizer, self.config, self._engine_id, self._owns_model, self._closed, self.scheduler, self._output_collectors, self._stream_states, self._finished_events, self._running, self._task, self._start_time, self._steps_executed ## `vllm_mlx.engine_core.EngineCore.start` - Kind: method - Signature: `async def start(self) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine_core.py#L116-L124 - Implementation: Method `EngineCore.start` updates `self._running`, `self._start_time`, `self._task`; calls `time.time`, `asyncio.create_task`, `self._engine_loop`, `logger.info`; returns `None`. Start the engine loop. - Inputs: none - Return annotation: `None` - Calls: time.time, asyncio.create_task, self._engine_loop, logger.info - State reads: self._running, self._engine_loop - State writes: self._running, self._start_time, self._task - Return expressions: None ## `vllm_mlx.engine_core.EngineCore.stop` - Kind: method - Signature: `async def stop(self) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine_core.py#L126-L140 - Implementation: Method `EngineCore.stop` updates `self._running`, `self._task`; calls `self._task.cancel`, `self.scheduler._close_batch_generator`, `logger.info`; awaits asynchronous work. Stop the engine loop. - Inputs: none - Return annotation: `None` - Calls: self._task.cancel, self.scheduler._close_batch_generator, logger.info - State reads: self._task, self._task.cancel, self.scheduler._close_batch_generator, self.scheduler - State writes: self._running, self._task ## `vllm_mlx.engine_core.EngineCore.is_running` - Kind: method - Signature: `def is_running(self) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine_core.py#L142-L144 - Implementation: Method `EngineCore.is_running` returns `self._running`. Check if engine is running. - Inputs: none - Return annotation: `bool` - State reads: self._running - Return expressions: self._running ## `vllm_mlx.engine_core.EngineCore._engine_loop` - Kind: method - Signature: `async def _engine_loop(self) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine_core.py#L146-L334 - Implementation: Method `EngineCore._engine_loop` calls `asyncio.get_running_loop`, `ThreadPoolExecutor`, `mx.device_info().get`, `mx.device_info`; awaits asynchronous work. Main engine loop. scheduler.step runs on one dedicated worker thread. MLX streams are thread-local, so we rebind generation streams inside that worker. - Inputs: none - Return annotation: `None` - Calls: asyncio.get_running_loop, ThreadPoolExecutor, mx.device_info().get, mx.device_info, int, min, self.scheduler.has_requests, loop.run_in_executor, _is_stream_thread_error, _bind_model_streams_once, logger.warning, _step_on_model_thread, asyncio.sleep, collectors.get, collector.put, states.get, state.should_send, state.mark_sent, events.get, event.set, mx.clear_cache, logger.error, traceback.format_exc, self.scheduler._close_batch_generator, worker.shutdown - State reads: self.config.step_interval, self.config, self.config.stream_interval, self.config.gpu_memory_utilization, self._running, self.scheduler.has_requests, self.scheduler, self._output_collectors, self._stream_states, self._finished_events, self.scheduler._close_batch_generator ## `vllm_mlx.engine_core.EngineCore._engine_loop._bind_worker_streams_once` - Kind: nested function - Signature: `def _bind_worker_streams_once() -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine_core.py#L160-L164 - Implementation: Nested Function `EngineCore._engine_loop._bind_worker_streams_once` calls `bind_generation_streams`. Nested Function `EngineCore._engine_loop._bind_worker_streams_once` calls `bind_generation_streams`. - Inputs: none - Return annotation: `None` - Calls: bind_generation_streams ## `vllm_mlx.engine_core.EngineCore._engine_loop._bind_model_streams_once` - Kind: nested function - Signature: `def _bind_model_streams_once() -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine_core.py#L166-L170 - Implementation: Nested Function `EngineCore._engine_loop._bind_model_streams_once` calls `bind_generation_streams`. Nested Function `EngineCore._engine_loop._bind_model_streams_once` calls `bind_generation_streams`. - Inputs: none - Return annotation: `None` - Calls: bind_generation_streams ## `vllm_mlx.engine_core.EngineCore._engine_loop._step_on_worker` - Kind: nested function - Signature: `def _step_on_worker()` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine_core.py#L172-L190 - Implementation: Nested Function `EngineCore._engine_loop._step_on_worker` updates `self._steps_executed`; calls `_bind_worker_streams_once`, `self.scheduler.step`, `mx.get_active_memory`, `mx.clear_cache`; returns `output`. Nested Function `EngineCore._engine_loop._step_on_worker` updates `self._steps_executed`; calls `_bind_worker_streams_once`, `self.scheduler.step`, `mx.get_active_memory`, `mx.clear_cache`; returns `output`. - Inputs: none - Return annotation: `not annotated` - Calls: _bind_worker_streams_once, self.scheduler.step, mx.get_active_memory, mx.clear_cache, logger.warning - State reads: self.scheduler.step, self.scheduler, self._steps_executed - State writes: self._steps_executed - Return expressions: output ## `vllm_mlx.engine_core.EngineCore._engine_loop._step_on_model_thread` - Kind: nested function - Signature: `def _step_on_model_thread()` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine_core.py#L192-L210 - Implementation: Nested Function `EngineCore._engine_loop._step_on_model_thread` updates `self._steps_executed`; calls `_bind_model_streams_once`, `self.scheduler.step`, `mx.get_active_memory`, `mx.clear_cache`; returns `output`. Nested Function `EngineCore._engine_loop._step_on_model_thread` updates `self._steps_executed`; calls `_bind_model_streams_once`, `self.scheduler.step`, `mx.get_active_memory`, `mx.clear_cache`; returns `output`. - Inputs: none - Return annotation: `not annotated` - Calls: _bind_model_streams_once, self.scheduler.step, mx.get_active_memory, mx.clear_cache, logger.warning - State reads: self.scheduler.step, self.scheduler, self._steps_executed - State writes: self._steps_executed - Return expressions: output ## `vllm_mlx.engine_core.EngineCore._engine_loop._recover_stream_thread_error_on_worker` - Kind: nested function - Signature: `def _recover_stream_thread_error_on_worker() -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine_core.py#L212-L215 - Implementation: Nested Function `EngineCore._engine_loop._recover_stream_thread_error_on_worker` calls `_bind_worker_streams_once`, `self.scheduler._recover_from_cache_error`, `self.scheduler._reschedule_running_requests`. Nested Function `EngineCore._engine_loop._recover_stream_thread_error_on_worker` calls `_bind_worker_streams_once`, `self.scheduler._recover_from_cache_error`, `self.scheduler._reschedule_running_requests`. - Inputs: none - Return annotation: `None` - Calls: _bind_worker_streams_once, self.scheduler._recover_from_cache_error, self.scheduler._reschedule_running_requests - State reads: self.scheduler._recover_from_cache_error, self.scheduler, self.scheduler._reschedule_running_requests ## `vllm_mlx.engine_core.EngineCore._engine_loop._clear_cache_on_worker` - Kind: nested function - Signature: `def _clear_cache_on_worker() -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine_core.py#L217-L219 - Implementation: Nested Function `EngineCore._engine_loop._clear_cache_on_worker` calls `_bind_worker_streams_once`, `mx.clear_cache`. Nested Function `EngineCore._engine_loop._clear_cache_on_worker` calls `_bind_worker_streams_once`, `mx.clear_cache`. - Inputs: none - Return annotation: `None` - Calls: _bind_worker_streams_once, mx.clear_cache ## `vllm_mlx.engine_core.EngineCore._engine_loop._close_batch_generator_on_worker` - Kind: nested function - Signature: `def _close_batch_generator_on_worker() -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine_core.py#L221-L223 - Implementation: Nested Function `EngineCore._engine_loop._close_batch_generator_on_worker` calls `_bind_worker_streams_once`, `self.scheduler._close_batch_generator`. Nested Function `EngineCore._engine_loop._close_batch_generator_on_worker` calls `_bind_worker_streams_once`, `self.scheduler._close_batch_generator`. - Inputs: none - Return annotation: `None` - Calls: _bind_worker_streams_once, self.scheduler._close_batch_generator - State reads: self.scheduler._close_batch_generator, self.scheduler ## `vllm_mlx.engine_core.EngineCore.add_request` - Kind: method - Signature: `async def add_request(self, prompt: Union[str, List[int]], sampling_params: Optional[SamplingParams]=None, request_id: Optional[str]=None, images: Optional[List[Any]]=None, videos: Optional[List[Any]]=None, prefix_boundary: int=0) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine_core.py#L336-L384 - Implementation: Method `EngineCore.add_request` calls `str`, `uuid.uuid4`, `SamplingParams`, `Request`; returns `request_id`. Add a request for processing. Args: prompt: Input prompt (string or token IDs) sampling_params: Generation parameters request_id: Optional custom request ID images: Optional images for multimodal videos: Optional videos for multimodal prefix_boundary: Token count for shared prefix (for cache) Returns: The request ID - Inputs: - `prompt` (Union[str, List[int]]; required): Input prompt (string or token IDs) - `sampling_params` (Optional[SamplingParams]; optional; default `None`): Generation parameters - `request_id` (Optional[str]; optional; default `None`): Optional custom request ID - `images` (Optional[List[Any]]; optional; default `None`): Optional images for multimodal - `videos` (Optional[List[Any]]; optional; default `None`): Optional videos for multimodal - `prefix_boundary` (int; optional; default `0`): Token count for shared prefix (for cache) - Return annotation: `str` - Calls: str, uuid.uuid4, SamplingParams, Request, RequestOutputCollector, RequestStreamState, asyncio.Event, self.scheduler.add_request - State reads: self._output_collectors, self._stream_states, self.config.stream_interval, self.config, self._finished_events, self.scheduler.add_request, self.scheduler - Return expressions: request_id ## `vllm_mlx.engine_core.EngineCore.abort_request` - Kind: method - Signature: `async def abort_request(self, request_id: str) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine_core.py#L386-L390 - Implementation: Method `EngineCore.abort_request` calls `self.scheduler.abort_request`, `self._cleanup_request`; returns `result`. Abort a request. - Inputs: - `request_id` (str; required): Required positional or keyword input. - Return annotation: `bool` - Calls: self.scheduler.abort_request, self._cleanup_request - State reads: self.scheduler.abort_request, self.scheduler, self._cleanup_request - Return expressions: result ## `vllm_mlx.engine_core.EngineCore._cleanup_request` - Kind: method - Signature: `def _cleanup_request(self, request_id: str) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine_core.py#L392-L399 - Implementation: Method `EngineCore._cleanup_request` calls `self._output_collectors.pop`, `collector.clear`, `self._stream_states.pop`, `self._finished_events.pop`. Clean up request tracking. - Inputs: - `request_id` (str; required): Required positional or keyword input. - Return annotation: `None` - Calls: self._output_collectors.pop, collector.clear, self._stream_states.pop, self._finished_events.pop, self.scheduler.remove_finished_request - State reads: self._output_collectors.pop, self._output_collectors, self._stream_states.pop, self._stream_states, self._finished_events.pop, self._finished_events, self.scheduler.remove_finished_request, self.scheduler ## `vllm_mlx.engine_core.EngineCore.stream_outputs` - Kind: method - Signature: `async def stream_outputs(self, request_id: str, timeout: Optional[float]=None) -> AsyncIterator[RequestOutput]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine_core.py#L401-L488 - Implementation: Method `EngineCore.stream_outputs` calls `_time.monotonic`, `self._output_collectors.get`, `logger.warning`, `logger.info`; awaits asynchronous work; yields values incrementally; returns `None`. Stream outputs for a request with low-latency non-blocking pattern. Uses the vLLM pattern: get_nowait() or await get() This avoids unnecessary task switches when output is available. Args: request_id: The request ID timeout: Optional timeout in seconds Yields: RequestOutput objects as tokens are generated - Inputs: - `request_id` (str; required): The request ID - `timeout` (Optional[float]; optional; default `None`): Optional timeout in seconds - Return annotation: `AsyncIterator[RequestOutput]` - Calls: _time.monotonic, self._output_collectors.get, logger.warning, logger.info, collector.get_nowait, asyncio.wait_for, collector.get, type, self.scheduler.abort_request, self._cleanup_request - State reads: self._output_collectors.get, self._output_collectors, self.scheduler.abort_request, self.scheduler, self._cleanup_request - Return expressions: None ## `vllm_mlx.engine_core.EngineCore.generate` - Kind: method - Signature: `async def generate(self, prompt: Union[str, List[int]], sampling_params: Optional[SamplingParams]=None, request_id: Optional[str]=None, **kwargs) -> RequestOutput` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine_core.py#L490-L552 - Implementation: Method `EngineCore.generate` calls `self.add_request`, `self._finished_events.get`, `RuntimeError`, `event.wait`; awaits asynchronous work; can raise `RuntimeError`; returns `final_output`. Generate a complete response (non-streaming). This method is optimized to avoid streaming overhead when you only need the final result. Args: prompt: Input prompt sampling_params: Generation parameters request_id: Optional request ID Returns: Final RequestOutput with complete text - Inputs: - `prompt` (Union[str, List[int]]; required): Input prompt - `sampling_params` (Optional[SamplingParams]; optional; default `None`): Generation parameters - `request_id` (Optional[str]; optional; default `None`): Optional request ID - `**kwargs` (not annotated; optional): Additional variadic keyword inputs accepted by this callable. - Return annotation: `RequestOutput` - Calls: self.add_request, self._finished_events.get, RuntimeError, event.wait, self._output_collectors.get, collector.get_nowait, logger.info, self.scheduler.abort_request, self._cleanup_request - State reads: self.add_request, self._finished_events.get, self._finished_events, self._output_collectors.get, self._output_collectors, self.scheduler.abort_request, self.scheduler, self._cleanup_request - Raises directly: RuntimeError - Return expressions: final_output ## `vllm_mlx.engine_core.EngineCore.generate_batch_sync` - Kind: method - Signature: `def generate_batch_sync(self, prompts: List[Union[str, List[int]]], sampling_params: Optional[SamplingParams]=None) -> List[RequestOutput]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine_core.py#L554-L609 - Implementation: Method `EngineCore.generate_batch_sync` calls `SamplingParams`, `str`, `uuid_module.uuid4`, `Request`; returns `[results[rid] for rid in request_ids]`. Generate responses synchronously for maximum throughput. This bypasses the async engine loop entirely, running the scheduler directly for optimal batching performance. Use this when you don't need streaming and want maximum throughput. Args: prompts: List of input prompts sampling_params: Generation parameters (same for all) Returns: List of RequestOutput in same order as prompts - Inputs: - `prompts` (List[Union[str, List[int]]]; required): List of input prompts - `sampling_params` (Optional[SamplingParams]; optional; default `None`): Generation parameters (same for all) - Return annotation: `List[RequestOutput]` - Calls: SamplingParams, str, uuid_module.uuid4, Request, self.scheduler.add_request, request_ids.append, bind_generation_streams, self.scheduler.has_requests, self.scheduler.step, self.scheduler.remove_finished_request - State reads: self.scheduler.add_request, self.scheduler, self.scheduler.has_requests, self.scheduler.step, self.scheduler.remove_finished_request - Return expressions: [results[rid] for rid in request_ids] ## `vllm_mlx.engine_core.EngineCore.get_stats` - Kind: method - Signature: `def get_stats(self) -> Dict[str, Any]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine_core.py#L611-L624 - Implementation: Method `EngineCore.get_stats` calls `self.scheduler.get_stats`, `time.time`, `len`, `self.scheduler.get_running_requests_info`; returns `{'running': self._running, 'uptime_seconds': uptime, 'steps_executed': self._steps_executed, 'active_requests': len(sel…`. Get engine statistics. - Inputs: none - Return annotation: `Dict[str, Any]` - Calls: self.scheduler.get_stats, time.time, len, self.scheduler.get_running_requests_info - State reads: self.scheduler.get_stats, self.scheduler, self._start_time, self._running, self._steps_executed, self._output_collectors, self.config.stream_interval, self.config, self.scheduler.get_running_requests_info - Return expressions: {'running': self._running, 'uptime_seconds': uptime, 'steps_executed': self._steps_executed, 'active_requests': len(sel… ## `vllm_mlx.engine_core.EngineCore.get_cache_stats` - Kind: method - Signature: `def get_cache_stats(self) -> Optional[Dict[str, Any]]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine_core.py#L626-L628 - Implementation: Method `EngineCore.get_cache_stats` calls `self.scheduler.get_cache_stats`; returns `self.scheduler.get_cache_stats()`. Get prefix cache statistics. - Inputs: none - Return annotation: `Optional[Dict[str, Any]]` - Calls: self.scheduler.get_cache_stats - State reads: self.scheduler.get_cache_stats, self.scheduler - Return expressions: self.scheduler.get_cache_stats() ## `vllm_mlx.engine_core.EngineCore.save_cache_to_disk` - Kind: method - Signature: `def save_cache_to_disk(self, cache_dir: str) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine_core.py#L630-L632 - Implementation: Method `EngineCore.save_cache_to_disk` calls `self.scheduler.save_cache_to_disk`; returns `self.scheduler.save_cache_to_disk(cache_dir)`. Save prefix cache to disk. - Inputs: - `cache_dir` (str; required): Required positional or keyword input. - Return annotation: `bool` - Calls: self.scheduler.save_cache_to_disk - State reads: self.scheduler.save_cache_to_disk, self.scheduler - Return expressions: self.scheduler.save_cache_to_disk(cache_dir) ## `vllm_mlx.engine_core.EngineCore.load_cache_from_disk` - Kind: method - Signature: `def load_cache_from_disk(self, cache_dir: str) -> int` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine_core.py#L634-L636 - Implementation: Method `EngineCore.load_cache_from_disk` calls `self.scheduler.load_cache_from_disk`; returns `self.scheduler.load_cache_from_disk(cache_dir)`. Load prefix cache from disk. - Inputs: - `cache_dir` (str; required): Required positional or keyword input. - Return annotation: `int` - Calls: self.scheduler.load_cache_from_disk - State reads: self.scheduler.load_cache_from_disk, self.scheduler - Return expressions: self.scheduler.load_cache_from_disk(cache_dir) ## `vllm_mlx.engine_core.EngineCore.clear_runtime_caches` - Kind: method - Signature: `def clear_runtime_caches(self) -> Dict[str, Any] | None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine_core.py#L638-L640 - Implementation: Method `EngineCore.clear_runtime_caches` calls `self.scheduler.clear_runtime_caches`; returns `self.scheduler.clear_runtime_caches()`. Clear scheduler-managed runtime caches. - Inputs: none - Return annotation: `Dict[str, Any] | None` - Calls: self.scheduler.clear_runtime_caches - State reads: self.scheduler.clear_runtime_caches, self.scheduler - Return expressions: self.scheduler.clear_runtime_caches() ## `vllm_mlx.engine_core.EngineCore.clear_prefix_cache` - Kind: method - Signature: `def clear_prefix_cache(self) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine_core.py#L642-L645 - Implementation: Method `EngineCore.clear_prefix_cache` calls `hasattr`, `self.scheduler.clear_prefix_cache`. Clear the prefix cache (delegates to scheduler). - Inputs: none - Return annotation: `None` - Calls: hasattr, self.scheduler.clear_prefix_cache - State reads: self.scheduler, self.scheduler.clear_prefix_cache ## `vllm_mlx.engine_core.EngineCore._release_model` - Kind: method - Signature: `def _release_model(self) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine_core.py#L647-L653 - Implementation: Method `EngineCore._release_model` updates `self._owns_model`; calls `get_registry`, `registry.release`, `logger.debug`. Release model ownership. - Inputs: none - Return annotation: `None` - Calls: get_registry, registry.release, logger.debug - State reads: self._owns_model, self._closed, self.model, self._engine_id - State writes: self._owns_model ## `vllm_mlx.engine_core.EngineCore.close` - Kind: method - Signature: `def close(self) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine_core.py#L655-L685 - Implementation: Method `EngineCore.close` updates `self._owns_model`, `self._closed`; calls `get_registry`, `registry.release`, `logger.debug`, `self.scheduler.deep_reset`; returns `None`. Explicitly close the engine and release resources. This should be called when done using the engine, especially if you plan to create another engine with the same model. - Inputs: none - Return annotation: `None` - Calls: get_registry, registry.release, logger.debug, self.scheduler.deep_reset, self._output_collectors.values, collector.clear, self._output_collectors.clear, self._stream_states.clear, self._finished_events.clear - State reads: self._closed, self._owns_model, self.model, self._engine_id, self.scheduler.deep_reset, self.scheduler, self._output_collectors.values, self._output_collectors, self._output_collectors.clear, self._stream_states.clear, self._stream_states, self._finished_events.clear, self._finished_events - State writes: self._owns_model, self._closed - Return expressions: None ## `vllm_mlx.engine_core.EngineCore.__del__` - Kind: method - Signature: `def __del__(self)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine_core.py#L687-L693 - Implementation: Method `EngineCore.__del__` calls `self._release_model`. Cleanup on destruction. - Inputs: none - Return annotation: `not annotated` - Calls: self._release_model - State reads: self._release_model ## `vllm_mlx.engine_core.EngineCore.engine_id` - Kind: method - Signature: `def engine_id(self) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine_core.py#L696-L698 - Implementation: Method `EngineCore.engine_id` returns `self._engine_id`. Get the engine ID. - Inputs: none - Return annotation: `str` - Decorators: property - State reads: self._engine_id - Return expressions: self._engine_id ## `vllm_mlx.engine_core.AsyncEngineCore` - Kind: class - Signature: `class AsyncEngineCore` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine_core.py#L701-L794 - Implementation: Class `AsyncEngineCore` declares 14 direct member(s). Async context manager wrapper for EngineCore. Usage: async with AsyncEngineCore(model, tokenizer) as engine: request_id = await engine.add_request("Hello") async for output in engine.stream_outputs(request_id): print(output.new_text) - Inputs: - `model` (Any; required): Required positional or keyword input. - `tokenizer` (Any; required): Required positional or keyword input. - `config` (Optional[EngineConfig]; optional; default `None`): Optional positional or keyword input; defaults to `None`. - Constructs: `vllm_mlx.engine_core.AsyncEngineCore` ## `vllm_mlx.engine_core.AsyncEngineCore.__init__` - Kind: method - Signature: `def __init__(self, model: Any, tokenizer: Any, config: Optional[EngineConfig]=None)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine_core.py#L712-L718 - Implementation: Method `AsyncEngineCore.__init__` updates `self.engine`; calls `EngineCore`. Method `AsyncEngineCore.__init__` updates `self.engine`; calls `EngineCore`. - Inputs: - `model` (Any; required): Required positional or keyword input. - `tokenizer` (Any; required): Required positional or keyword input. - `config` (Optional[EngineConfig]; optional; default `None`): Optional positional or keyword input; defaults to `None`. - Return annotation: `not annotated` - Calls: EngineCore - State writes: self.engine ## `vllm_mlx.engine_core.AsyncEngineCore.__aenter__` - Kind: method - Signature: `async def __aenter__(self) -> 'AsyncEngineCore'` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine_core.py#L720-L722 - Implementation: Method `AsyncEngineCore.__aenter__` calls `self.engine.start`; awaits asynchronous work; returns `self`. Method `AsyncEngineCore.__aenter__` calls `self.engine.start`; awaits asynchronous work; returns `self`. - Inputs: none - Return annotation: `'AsyncEngineCore'` - Calls: self.engine.start - State reads: self.engine.start, self.engine - Return expressions: self ## `vllm_mlx.engine_core.AsyncEngineCore.__aexit__` - Kind: method - Signature: `async def __aexit__(self, *args) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine_core.py#L724-L725 - Implementation: Method `AsyncEngineCore.__aexit__` calls `self.engine.stop`; awaits asynchronous work. Method `AsyncEngineCore.__aexit__` calls `self.engine.stop`; awaits asynchronous work. - Inputs: - `*args` (not annotated; optional): Additional variadic positional inputs accepted by this callable. - Return annotation: `None` - Calls: self.engine.stop - State reads: self.engine.stop, self.engine ## `vllm_mlx.engine_core.AsyncEngineCore.start` - Kind: method - Signature: `def start(self) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine_core.py#L727-L729 - Implementation: Method `AsyncEngineCore.start` updates `self._start_task`; calls `asyncio.create_task`, `self.engine.start`. Start engine (creates task in current loop). - Inputs: none - Return annotation: `None` - Calls: asyncio.create_task, self.engine.start - State reads: self.engine.start, self.engine - State writes: self._start_task ## `vllm_mlx.engine_core.AsyncEngineCore.stop` - Kind: method - Signature: `async def stop(self) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine_core.py#L731-L733 - Implementation: Method `AsyncEngineCore.stop` calls `self.engine.stop`; awaits asynchronous work. Stop the engine. - Inputs: none - Return annotation: `None` - Calls: self.engine.stop - State reads: self.engine.stop, self.engine ## `vllm_mlx.engine_core.AsyncEngineCore.add_request` - Kind: method - Signature: `async def add_request(self, prompt: Union[str, List[int]], sampling_params: Optional[SamplingParams]=None, request_id: Optional[str]=None, **kwargs) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine_core.py#L735-L748 - Implementation: Method `AsyncEngineCore.add_request` calls `self.engine.add_request`; awaits asynchronous work; returns `await self.engine.add_request(prompt=prompt, sampling_params=sampling_params, request_id=request_id, **kwargs)`. Add a request. - Inputs: - `prompt` (Union[str, List[int]]; required): Required positional or keyword input. - `sampling_params` (Optional[SamplingParams]; optional; default `None`): Optional positional or keyword input; defaults to `None`. - `request_id` (Optional[str]; optional; default `None`): Optional positional or keyword input; defaults to `None`. - `**kwargs` (not annotated; optional): Additional variadic keyword inputs accepted by this callable. - Return annotation: `str` - Calls: self.engine.add_request - State reads: self.engine.add_request, self.engine - Return expressions: await self.engine.add_request(prompt=prompt, sampling_params=sampling_params, request_id=request_id, **kwargs) ## `vllm_mlx.engine_core.AsyncEngineCore.abort_request` - Kind: method - Signature: `async def abort_request(self, request_id: str) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine_core.py#L750-L752 - Implementation: Method `AsyncEngineCore.abort_request` calls `self.engine.abort_request`; awaits asynchronous work; returns `await self.engine.abort_request(request_id)`. Abort a request. - Inputs: - `request_id` (str; required): Required positional or keyword input. - Return annotation: `bool` - Calls: self.engine.abort_request - State reads: self.engine.abort_request, self.engine - Return expressions: await self.engine.abort_request(request_id) ## `vllm_mlx.engine_core.AsyncEngineCore.stream_outputs` - Kind: method - Signature: `async def stream_outputs(self, request_id: str, timeout: Optional[float]=None) -> AsyncIterator[RequestOutput]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine_core.py#L754-L761 - Implementation: Method `AsyncEngineCore.stream_outputs` calls `self.engine.stream_outputs`; yields values incrementally. Stream outputs. - Inputs: - `request_id` (str; required): Required positional or keyword input. - `timeout` (Optional[float]; optional; default `None`): Optional positional or keyword input; defaults to `None`. - Return annotation: `AsyncIterator[RequestOutput]` - Calls: self.engine.stream_outputs - State reads: self.engine.stream_outputs, self.engine ## `vllm_mlx.engine_core.AsyncEngineCore.generate` - Kind: method - Signature: `async def generate(self, prompt: Union[str, List[int]], sampling_params: Optional[SamplingParams]=None, **kwargs) -> RequestOutput` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine_core.py#L763-L774 - Implementation: Method `AsyncEngineCore.generate` calls `self.engine.generate`; awaits asynchronous work; returns `await self.engine.generate(prompt=prompt, sampling_params=sampling_params, **kwargs)`. Generate complete response. - Inputs: - `prompt` (Union[str, List[int]]; required): Required positional or keyword input. - `sampling_params` (Optional[SamplingParams]; optional; default `None`): Optional positional or keyword input; defaults to `None`. - `**kwargs` (not annotated; optional): Additional variadic keyword inputs accepted by this callable. - Return annotation: `RequestOutput` - Calls: self.engine.generate - State reads: self.engine.generate, self.engine - Return expressions: await self.engine.generate(prompt=prompt, sampling_params=sampling_params, **kwargs) ## `vllm_mlx.engine_core.AsyncEngineCore.get_stats` - Kind: method - Signature: `def get_stats(self) -> Dict[str, Any]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine_core.py#L776-L778 - Implementation: Method `AsyncEngineCore.get_stats` calls `self.engine.get_stats`; returns `self.engine.get_stats()`. Get engine stats. - Inputs: none - Return annotation: `Dict[str, Any]` - Calls: self.engine.get_stats - State reads: self.engine.get_stats, self.engine - Return expressions: self.engine.get_stats() ## `vllm_mlx.engine_core.AsyncEngineCore.get_cache_stats` - Kind: method - Signature: `def get_cache_stats(self) -> Optional[Dict[str, Any]]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine_core.py#L780-L782 - Implementation: Method `AsyncEngineCore.get_cache_stats` calls `self.engine.get_cache_stats`; returns `self.engine.get_cache_stats()`. Get prefix cache statistics. - Inputs: none - Return annotation: `Optional[Dict[str, Any]]` - Calls: self.engine.get_cache_stats - State reads: self.engine.get_cache_stats, self.engine - Return expressions: self.engine.get_cache_stats() ## `vllm_mlx.engine_core.AsyncEngineCore.save_cache_to_disk` - Kind: method - Signature: `def save_cache_to_disk(self, cache_dir: str) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine_core.py#L784-L786 - Implementation: Method `AsyncEngineCore.save_cache_to_disk` calls `self.engine.save_cache_to_disk`; returns `self.engine.save_cache_to_disk(cache_dir)`. Save prefix cache to disk. - Inputs: - `cache_dir` (str; required): Required positional or keyword input. - Return annotation: `bool` - Calls: self.engine.save_cache_to_disk - State reads: self.engine.save_cache_to_disk, self.engine - Return expressions: self.engine.save_cache_to_disk(cache_dir) ## `vllm_mlx.engine_core.AsyncEngineCore.load_cache_from_disk` - Kind: method - Signature: `def load_cache_from_disk(self, cache_dir: str) -> int` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine_core.py#L788-L790 - Implementation: Method `AsyncEngineCore.load_cache_from_disk` calls `self.engine.load_cache_from_disk`; returns `self.engine.load_cache_from_disk(cache_dir)`. Load prefix cache from disk. - Inputs: - `cache_dir` (str; required): Required positional or keyword input. - Return annotation: `int` - Calls: self.engine.load_cache_from_disk - State reads: self.engine.load_cache_from_disk, self.engine - Return expressions: self.engine.load_cache_from_disk(cache_dir) ## `vllm_mlx.engine_core.AsyncEngineCore.clear_runtime_caches` - Kind: method - Signature: `def clear_runtime_caches(self) -> Dict[str, Any] | None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine_core.py#L792-L794 - Implementation: Method `AsyncEngineCore.clear_runtime_caches` calls `self.engine.clear_runtime_caches`; returns `self.engine.clear_runtime_caches()`. Clear scheduler-managed runtime caches. - Inputs: none - Return annotation: `Dict[str, Any] | None` - Calls: self.engine.clear_runtime_caches - State reads: self.engine.clear_runtime_caches, self.engine - Return expressions: self.engine.clear_runtime_caches() # Module `vllm_mlx.gradio_app` Gradio Chatbot Interface for vllm-mlx. A multimodal chat interface that connects to the vllm-mlx server and supports text, images, and video files. Usage: # First start the server with a multimodal model: vllm-mlx serve --served-model-name default mlx-community/Qwen3-VL-4B-Instruct-3bit --port 8000 # Then run the app: vllm-mlx-chat # Or with a different served-model name served on localhost:8000: vllm-mlx-chat --served-model-name --server-url http://localhost:8000 --port 7860 Note: Query the /v1/models endpoint on localhost with `curl` and `jq` to see available models and their names: ```bash curl http://localhost:8000/v1/models | jq ".data[0].id" ``` Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/gradio_app.py#L1-L411 ## `vllm_mlx.gradio_app.encode_file_to_base64` - Kind: function - Signature: `def encode_file_to_base64(file_path: str) -> tuple[str, str]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/gradio_app.py#L33-L76 - Implementation: Function `encode_file_to_base64` calls `Path`, `path.suffix.lower`, `open`, `base64.b64encode(f.read()).decode`; returns `(f'data:{mime_type};base64,{data}', media_type)`. Encode a file to base64 data URL. Returns: Tuple of (data_url, media_type) where media_type is 'image' or 'video' - Inputs: - `file_path` (str; required): Required positional or keyword input. - Return annotation: `tuple[str, str]` - Calls: Path, path.suffix.lower, open, base64.b64encode(f.read()).decode, base64.b64encode, f.read - Return expressions: (f'data:{mime_type};base64,{data}', media_type) ## `vllm_mlx.gradio_app.build_message_content` - Kind: function - Signature: `def build_message_content(text: str, files: list[str] | None=None) -> list | str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/gradio_app.py#L79-L108 - Implementation: Function `build_message_content` calls `content.append`, `encode_file_to_base64`; has 2 explicit return paths. Build OpenAI-compatible message content with text and optional files. Args: text: The text message files: Optional list of file paths (images or videos) Returns: Content in OpenAI multimodal format - Inputs: - `text` (str; required): The text message - `files` (list[str] | None; optional; default `None`): Optional list of file paths (images or videos) - Return annotation: `list | str` - Calls: content.append, encode_file_to_base64 - Return expressions: text; content if content else text ## `vllm_mlx.gradio_app.create_chat_function` - Kind: function - Signature: `def create_chat_function(server_url: str, max_tokens: int, temperature: float, served_model_name: str='default')` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/gradio_app.py#L111-L257 - Implementation: Function `create_chat_function` returns `chat`. Create the chat function for Gradio ChatInterface. Args: server_url: URL of the vllm-mlx server max_tokens: Maximum tokens to generate temperature: Sampling temperature served_model_name: Model name to send in OpenAI-compatible requests Returns: Chat function compatible with gr.ChatInterface - Inputs: - `server_url` (str; required): URL of the vllm-mlx server - `max_tokens` (int; required): Maximum tokens to generate - `temperature` (float; required): Sampling temperature - `served_model_name` (str; optional; default `'default'`): Model name to send in OpenAI-compatible requests - Return annotation: `not annotated` - Return expressions: chat ## `vllm_mlx.gradio_app.create_chat_function.chat` - Kind: nested function - Signature: `def chat(message: dict, history: list) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/gradio_app.py#L132-L255 - Implementation: Nested Function `create_chat_function.chat` calls `isinstance`, `message.get`, `print`, `len`; has 4 explicit return paths. Process a multimodal message and return response. Args: message: Dict with 'text' and optional 'files' keys history: List of previous messages Returns: Assistant response text - Inputs: - `message` (dict; required): Dict with 'text' and optional 'files' keys - `history` (list; required): List of previous messages - Return annotation: `str` - Calls: isinstance, message.get, print, len, sys.stdout.flush, enumerate, msg.get, p.get, ' '.join, str, rebuilt_content.append, messages.append, content.get, build_message_content, encode_file_to_base64, media_items.append, c.get, requests.post, response.raise_for_status, response.json - Return expressions: result['choices'][0]['message']['content']; 'Error: Cannot connect to server. Make sure vllm-mlx is running.'; 'Error: Timeout - server took too long to respond.'; f'Error: {str(e)}' ## `vllm_mlx.gradio_app.main` - Kind: function - Signature: `def main()` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/gradio_app.py#L260-L407 - Implementation: Function `main` calls `argparse.ArgumentParser`, `parser.add_argument`, `parser.parse_args`, `print`. Run the Gradio app. - Inputs: none - Return annotation: `not annotated` - Calls: argparse.ArgumentParser, parser.add_argument, parser.parse_args, print, gr.ChatInterface, create_chat_function, gr.MultimodalTextbox, demo.launch ## `vllm_mlx.gradio_app.main.text_chat` - Kind: nested function - Signature: `def text_chat(message: str, history: list) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/gradio_app.py#L331-L368 - Implementation: Nested Function `main.text_chat` calls `isinstance`, `msg.get`, `p.get`, `' '.join`; has 4 explicit return paths. Process a text-only message. - Inputs: - `message` (str; required): Required positional or keyword input. - `history` (list; required): Required positional or keyword input. - Return annotation: `str` - Calls: isinstance, msg.get, p.get, ' '.join, messages.append, requests.post, response.raise_for_status, response.json, str - Return expressions: result['choices'][0]['message']['content']; 'Error: Cannot connect to server. Make sure vllm-mlx is running.'; 'Error: Timeout - server took too long to respond.'; f'Error: {str(e)}' # Module `vllm_mlx.gradio_text_app` Gradio Text-Only Chatbot Interface for vllm-mlx. A fast, text-only chat interface for LLM models. Use this for text conversations without image/video overhead. Usage: # First start the server with a model: # Without a custom API model name (model path is the name used in the OpenAI API): vllm-mlx serve mlx-community/Llama-3.2-3B-Instruct-4bit --port 8000 # With a custom API model name ("default" is the name used in the OpenAI API): vllm-mlx serve --served-model-name default mlx-community/Llama-3.2-3B-Instruct-4bit --port 8000 # Then run this app: vllm-mlx-text-chat --served-model-name mlx-community/Llama-3.2-3B-Instruct-4bit # Or with vllm-mlx started with served model name is 'default', there is no need to use --served-model-name: vllm-mlx-text-chat --server-url http://localhost:8000 --port 7861 Note: Query the /v1/models endpoint with `curl` and `jq` to see available models and their names: ```bash curl http://localhost:8000/v1/models | jq ".data[0].id" ``` Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/gradio_text_app.py#L1-L205 ## `vllm_mlx.gradio_text_app.create_chat_function` - Kind: function - Signature: `def create_chat_function(server_url: str, max_tokens: int, temperature: float, served_model_name: str='default')` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/gradio_text_app.py#L34-L108 - Implementation: Function `create_chat_function` returns `chat`. Create the chat function for Gradio ChatInterface. Args: server_url: URL of the vllm-mlx server max_tokens: Maximum tokens to generate temperature: Sampling temperature served_model_name: Model name to send in OpenAI-compatible requests Returns: Chat function compatible with gr.ChatInterface - Inputs: - `server_url` (str; required): URL of the vllm-mlx server - `max_tokens` (int; required): Maximum tokens to generate - `temperature` (float; required): Sampling temperature - `served_model_name` (str; optional; default `'default'`): Model name to send in OpenAI-compatible requests - Return annotation: `not annotated` - Return expressions: chat ## `vllm_mlx.gradio_text_app.create_chat_function.chat` - Kind: nested function - Signature: `def chat(message: str, history: list) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/gradio_text_app.py#L53-L106 - Implementation: Nested Function `create_chat_function.chat` calls `isinstance`, `msg.get`, `p.get`, `' '.join`; has 4 explicit return paths. Process a text message and return response. Args: message: User's text message history: List of previous messages Returns: Assistant response text - Inputs: - `message` (str; required): User's text message - `history` (list; required): List of previous messages - Return annotation: `str` - Calls: isinstance, msg.get, p.get, ' '.join, messages.append, requests.post, response.raise_for_status, response.json, str - Return expressions: result['choices'][0]['message']['content']; 'Error: Cannot connect to server. Make sure vllm-mlx is running.'; 'Error: Timeout - server took too long to respond.'; f'Error: {str(e)}' ## `vllm_mlx.gradio_text_app.main` - Kind: function - Signature: `def main()` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/gradio_text_app.py#L111-L201 - Implementation: Function `main` calls `argparse.ArgumentParser`, `parser.add_argument`, `parser.parse_args`, `print`. Run the Gradio app. - Inputs: none - Return annotation: `not annotated` - Calls: argparse.ArgumentParser, parser.add_argument, parser.parse_args, print, create_chat_function, gr.ChatInterface, demo.launch # Module `vllm_mlx.lifecycle` Model lifecycle / residency management for vllm-mlx. Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/lifecycle.py#L1-L493 ## `vllm_mlx.lifecycle.ResidentState` - Kind: class - Signature: `class ResidentState(str, Enum)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/lifecycle.py#L17-L24 - Implementation: Class `ResidentState` derives from `str`, `Enum` and declares 0 direct member(s). Runtime residency state for a configured model. - Inputs: none - Constructs: `vllm_mlx.lifecycle.ResidentState` ## `vllm_mlx.lifecycle.ModelSpec` - Kind: class - Signature: `class ModelSpec` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/lifecycle.py#L28-L44 - Implementation: Class `ModelSpec` declares 0 direct member(s). Immutable engine construction inputs for a resident model. - Inputs: - `model_key` (str; required): Required constructor field. - `model_name` (str; required): Required constructor field. - `use_batching` (bool; optional; default `False`): Optional constructor field; defaults to `False`. - `scheduler_config` (Any | None; optional; default `None`): Optional constructor field; defaults to `None`. - `stream_interval` (int; optional; default `1`): Optional constructor field; defaults to `1`. - `max_tokens` (int; optional; default `32768`): Optional constructor field; defaults to `32768`. - `force_mllm` (bool; optional; default `False`): Optional constructor field; defaults to `False`. - `mtp` (bool; optional; default `False`): Optional constructor field; defaults to `False`. - `prefill_step_size` (int; optional; default `2048`): Optional constructor field; defaults to `2048`. - `specprefill_enabled` (bool; optional; default `False`): Optional constructor field; defaults to `False`. - `specprefill_threshold` (int; optional; default `8192`): Optional constructor field; defaults to `8192`. - `specprefill_keep_pct` (float; optional; default `0.3`): Optional constructor field; defaults to `0.3`. - `specprefill_backbone_pct` (float; optional; default `0.0`): Optional constructor field; defaults to `0.0`. - `specprefill_draft_model` (str | None; optional; default `None`): Optional constructor field; defaults to `None`. - Constructs: `vllm_mlx.lifecycle.ModelSpec` - Decorators: dataclass(frozen=True) ## `vllm_mlx.lifecycle.ResidentModel` - Kind: class - Signature: `class ResidentModel` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/lifecycle.py#L48-L66 - Implementation: Class `ResidentModel` declares 0 direct member(s). Runtime state for a single resident model. - Inputs: - `spec` (ModelSpec; required): Required constructor field. - `state` (ResidentState; optional; default `ResidentState.UNLOADED`): Optional constructor field; defaults to `ResidentState.UNLOADED`. - `engine` (BaseEngine | None; optional; default `None`): Optional constructor field; defaults to `None`. - `active_requests` (int; optional; default `0`): Optional constructor field; defaults to `0`. - `last_used_at` (float | None; optional; default `None`): Optional constructor field; defaults to `None`. - `loaded_at` (float | None; optional; default `None`): Optional constructor field; defaults to `None`. - `last_error` (str | None; optional; default `None`): Optional constructor field; defaults to `None`. - `estimated_memory_bytes` (int | None; optional; default `None`): Optional constructor field; defaults to `None`. - `_load_waiters` (int; optional; default `field(default=0, repr=False)`): Optional constructor field; defaults to `field(default=0, repr=False)`. - `_load_waiter_task` (asyncio.Task[BaseEngine] | None; optional; default `field(default=None, repr=False)`): Optional constructor field; defaults to `field(default=None, repr=False)`. - `_prepare_task` (asyncio.Task[None] | None; optional; default `field(default=None, repr=False)`): Optional constructor field; defaults to `field(default=None, repr=False)`. - `_abandoned_loading_task` (asyncio.Task[BaseEngine] | None; optional; default `field(default=None, repr=False)`): Optional constructor field; defaults to `field(default=None, repr=False)`. - `_loading_task` (asyncio.Task[BaseEngine] | None; optional; default `field(default=None, repr=False)`): Optional constructor field; defaults to `field(default=None, repr=False)`. - `_unloading_task` (asyncio.Task[bool] | None; optional; default `field(default=None, repr=False)`): Optional constructor field; defaults to `field(default=None, repr=False)`. - Constructs: `vllm_mlx.lifecycle.ResidentModel` - Decorators: dataclass ## `vllm_mlx.lifecycle.ResidencyManager` - Kind: class - Signature: `class ResidencyManager` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/lifecycle.py#L69-L493 - Implementation: Class `ResidencyManager` declares 16 direct member(s). Single-flight lifecycle manager for resident models. - Inputs: - `engine_factory` (Callable[[ModelSpec], Awaitable[BaseEngine]]; required): Required positional or keyword input. - `on_engine_loaded` (Callable[[ModelSpec, BaseEngine], Awaitable[None] | None] | None; optional; default `None`): Optional keyword-only input; defaults to `None`. - `on_engine_unloading` (Callable[[ModelSpec, BaseEngine], Awaitable[None] | None] | None; optional; default `None`): Optional keyword-only input; defaults to `None`. - `time_fn` (Callable[[], float] | None; optional; default `None`): Optional keyword-only input; defaults to `None`. - `auto_unload_idle_seconds` (float; optional; default `0`): Optional keyword-only input; defaults to `0`. - Constructs: `vllm_mlx.lifecycle.ResidencyManager` ## `vllm_mlx.lifecycle.ResidencyManager.__init__` - Kind: method - Signature: `def __init__(self, engine_factory: Callable[[ModelSpec], Awaitable[BaseEngine]], *, on_engine_loaded: Callable[[ModelSpec, BaseEngine], Awaitable[None] | None] | None=None, on_engine_unloading: Callable[[ModelSpec, BaseEngine], Awaitable[None] | None] | None=None, time_fn: Callable[[], float] | None=None, auto_unload_idle_seconds: float=0) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/lifecycle.py#L72-L91 - Implementation: Method `ResidencyManager.__init__` updates `self._engine_factory`, `self._on_engine_loaded`, `self._on_engine_unloading`, `self._time_fn`; calls `__import__`, `asyncio.Lock`. Method `ResidencyManager.__init__` updates `self._engine_factory`, `self._on_engine_loaded`, `self._on_engine_unloading`, `self._time_fn`; calls `__import__`, `asyncio.Lock`. - Inputs: - `engine_factory` (Callable[[ModelSpec], Awaitable[BaseEngine]]; required): Required positional or keyword input. - `on_engine_loaded` (Callable[[ModelSpec, BaseEngine], Awaitable[None] | None] | None; optional; default `None`): Optional keyword-only input; defaults to `None`. - `on_engine_unloading` (Callable[[ModelSpec, BaseEngine], Awaitable[None] | None] | None; optional; default `None`): Optional keyword-only input; defaults to `None`. - `time_fn` (Callable[[], float] | None; optional; default `None`): Optional keyword-only input; defaults to `None`. - `auto_unload_idle_seconds` (float; optional; default `0`): Optional keyword-only input; defaults to `0`. - Return annotation: `None` - Calls: __import__, asyncio.Lock - State writes: self._engine_factory, self._on_engine_loaded, self._on_engine_unloading, self._time_fn, self.auto_unload_idle_seconds, self._residents, self._lock ## `vllm_mlx.lifecycle.ResidencyManager.register_model` - Kind: method - Signature: `def register_model(self, spec: ModelSpec) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/lifecycle.py#L93-L111 - Implementation: Method `ResidencyManager.register_model` calls `self._residents.get`, `RuntimeError`, `ResidentModel`; can raise `RuntimeError`; returns `spec.model_key`. Register a model spec, or replace a dormant resident entry. - Inputs: - `spec` (ModelSpec; required): Required positional or keyword input. - Return annotation: `str` - Calls: self._residents.get, RuntimeError, ResidentModel - State reads: self._residents.get, self._residents - Raises directly: RuntimeError - Return expressions: spec.model_key ## `vllm_mlx.lifecycle.ResidencyManager.get_engine` - Kind: method - Signature: `def get_engine(self, model_key: str) -> BaseEngine | None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/lifecycle.py#L113-L115 - Implementation: Method `ResidencyManager.get_engine` calls `self._resident`; returns `self._resident(model_key).engine`. Get the currently loaded engine, if any. - Inputs: - `model_key` (str; required): Required positional or keyword input. - Return annotation: `BaseEngine | None` - Calls: self._resident - State reads: self._resident - Return expressions: self._resident(model_key).engine ## `vllm_mlx.lifecycle.ResidencyManager.get_status` - Kind: method - Signature: `def get_status(self, model_key: str) -> dict[str, Any]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/lifecycle.py#L117-L130 - Implementation: Method `ResidencyManager.get_status` calls `self._resident`; returns `{'model_key': resident.spec.model_key, 'model_name': resident.spec.model_name, 'state': resident.state.value, 'active_r…`. Return a serializable snapshot of resident state. - Inputs: - `model_key` (str; required): Required positional or keyword input. - Return annotation: `dict[str, Any]` - Calls: self._resident - State reads: self._resident, self.auto_unload_idle_seconds - Return expressions: {'model_key': resident.spec.model_key, 'model_name': resident.spec.model_name, 'state': resident.state.value, 'active_r… ## `vllm_mlx.lifecycle.ResidencyManager.ensure_loaded` - Kind: method - Signature: `async def ensure_loaded(self, model_key: str) -> BaseEngine` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/lifecycle.py#L132-L184 - Implementation: Method `ResidencyManager.ensure_loaded` calls `self._resident`, `asyncio.create_task`, `self._load_engine`, `asyncio.shield`; awaits asynchronous work; can raise `RuntimeError`; has 2 explicit return paths. Load and start a resident engine if needed. - Inputs: - `model_key` (str; required): Required positional or keyword input. - Return annotation: `BaseEngine` - Calls: self._resident, asyncio.create_task, self._load_engine, asyncio.shield, RuntimeError, asyncio.current_task, getattr, task.done, task.cancelled, cancelling, self._release_load_waiter - State reads: self._lock, self._resident, self._load_engine, self._release_load_waiter - Raises directly: RuntimeError - Return expressions: resident.engine; await asyncio.shield(task) ## `vllm_mlx.lifecycle.ResidencyManager.acquire` - Kind: method - Signature: `async def acquire(self, model_key: str, *, count_activity: bool=True) -> BaseEngine` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/lifecycle.py#L186-L206 - Implementation: Method `ResidencyManager.acquire` calls `self.ensure_loaded`, `self._resident`, `self._time_fn`; awaits asynchronous work; returns `engine`. Acquire a resident engine for request processing. - Inputs: - `model_key` (str; required): Required positional or keyword input. - `count_activity` (bool; optional; default `True`): Optional keyword-only input; defaults to `True`. - Return annotation: `BaseEngine` - Calls: self.ensure_loaded, self._resident, self._time_fn - State reads: self.ensure_loaded, self._lock, self._resident, self._time_fn - Return expressions: engine ## `vllm_mlx.lifecycle.ResidencyManager.release` - Kind: method - Signature: `async def release(self, model_key: str, *, count_activity: bool=True) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/lifecycle.py#L208-L215 - Implementation: Method `ResidencyManager.release` calls `self._resident`, `self._time_fn`. Release a previously acquired resident engine. - Inputs: - `model_key` (str; required): Required positional or keyword input. - `count_activity` (bool; optional; default `True`): Optional keyword-only input; defaults to `True`. - Return annotation: `None` - Calls: self._resident, self._time_fn - State reads: self._lock, self._resident, self._time_fn ## `vllm_mlx.lifecycle.ResidencyManager.unload_if_idle` - Kind: method - Signature: `async def unload_if_idle(self, model_key: str) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/lifecycle.py#L217-L253 - Implementation: Method `ResidencyManager.unload_if_idle` calls `self._resident`, `self._time_fn`, `asyncio.create_task`, `self._unload_engine`; awaits asynchronous work; has 2 explicit return paths. Unload a resident engine if it has been idle past the threshold. - Inputs: - `model_key` (str; required): Required positional or keyword input. - Return annotation: `bool` - Calls: self._resident, self._time_fn, asyncio.create_task, self._unload_engine, asyncio.shield - State reads: self.auto_unload_idle_seconds, self._lock, self._resident, self._time_fn, self._unload_engine - Return expressions: False; await asyncio.shield(unloading_task) ## `vllm_mlx.lifecycle.ResidencyManager.shutdown` - Kind: method - Signature: `async def shutdown(self) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/lifecycle.py#L255-L312 - Implementation: Method `ResidencyManager.shutdown` calls `list`, `self._residents.keys`, `self._resident`, `resident._loading_task.cancel`; awaits asynchronous work; can raise `RuntimeError`. Stop all loaded residents. - Inputs: none - Return annotation: `None` - Calls: list, self._residents.keys, self._resident, resident._loading_task.cancel, asyncio.create_task, self._unload_engine, suppress, asyncio.shield, suspend_cancellation, failures.append, len, RuntimeError, '; '.join - State reads: self._residents.keys, self._residents, self._lock, self._resident, self._unload_engine - Raises directly: RuntimeError ## `vllm_mlx.lifecycle.ResidencyManager._load_engine` - Kind: method - Signature: `async def _load_engine(self, resident: ResidentModel) -> BaseEngine` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/lifecycle.py#L314-L354 - Implementation: Method `ResidencyManager._load_engine` calls `self._engine_factory`, `self._prepare_engine_start`, `engine.start`, `self._run_hook`; awaits asynchronous work; can raise `asyncio.CancelledError`; returns `engine`. Create and start a resident engine. - Inputs: - `resident` (ResidentModel; required): Required positional or keyword input. - Return annotation: `BaseEngine` - Calls: self._engine_factory, self._prepare_engine_start, engine.start, self._run_hook, self._cleanup_cancelled_load, asyncio.current_task, asyncio.CancelledError, suppress, engine.stop, str, self._time_fn - State reads: self._engine_factory, self._prepare_engine_start, self._run_hook, self._on_engine_loaded, self._cleanup_cancelled_load, self._lock, self._time_fn - Raises directly: asyncio.CancelledError - Return expressions: engine ## `vllm_mlx.lifecycle.ResidencyManager._unload_engine` - Kind: method - Signature: `async def _unload_engine(self, resident: ResidentModel) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/lifecycle.py#L356-L388 - Implementation: Method `ResidencyManager._unload_engine` calls `self._run_hook`, `engine.stop`, `str`; awaits asynchronous work; has 2 explicit return paths. Stop and drop a resident engine. - Inputs: - `resident` (ResidentModel; required): Required positional or keyword input. - Return annotation: `bool` - Calls: self._run_hook, engine.stop, str - State reads: self._lock, self._run_hook, self._on_engine_unloading - Return expressions: False; True ## `vllm_mlx.lifecycle.ResidencyManager._resident` - Kind: method - Signature: `def _resident(self, model_key: str) -> ResidentModel` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/lifecycle.py#L390-L394 - Implementation: Method `ResidencyManager._resident` calls `KeyError`; can raise `KeyError`; returns `self._residents[model_key]`. Method `ResidencyManager._resident` calls `KeyError`; can raise `KeyError`; returns `self._residents[model_key]`. - Inputs: - `model_key` (str; required): Required positional or keyword input. - Return annotation: `ResidentModel` - Calls: KeyError - State reads: self._residents - Raises directly: KeyError - Return expressions: self._residents[model_key] ## `vllm_mlx.lifecycle.ResidencyManager._run_hook` - Kind: method - Signature: `async def _run_hook(self, hook: Callable[[ModelSpec, BaseEngine], Awaitable[None] | None] | None, spec: ModelSpec, engine: BaseEngine) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/lifecycle.py#L396-L407 - Implementation: Method `ResidencyManager._run_hook` calls `hook`, `inspect.isawaitable`; awaits asynchronous work; returns `None`. Method `ResidencyManager._run_hook` calls `hook`, `inspect.isawaitable`; awaits asynchronous work; returns `None`. - Inputs: - `hook` (Callable[[ModelSpec, BaseEngine], Awaitable[None] | None] | None; required): Required positional or keyword input. - `spec` (ModelSpec; required): Required positional or keyword input. - `engine` (BaseEngine; required): Required positional or keyword input. - Return annotation: `None` - Calls: hook, inspect.isawaitable - Return expressions: None ## `vllm_mlx.lifecycle.ResidencyManager._prepare_engine_start` - Kind: method - Signature: `async def _prepare_engine_start(self, resident: ResidentModel, engine: BaseEngine) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/lifecycle.py#L409-L445 - Implementation: Method `ResidencyManager._prepare_engine_start` calls `getattr`, `callable`, `uses_default_prepare`, `prepare_for_start`; awaits asynchronous work; returns `None`. Run blocking startup work away from the serving event loop. - Inputs: - `resident` (ResidentModel; required): Required positional or keyword input. - `engine` (BaseEngine; required): Required positional or keyword input. - Return annotation: `None` - Calls: getattr, callable, uses_default_prepare, prepare_for_start, asyncio.create_task, asyncio.to_thread, asyncio.shield, suspend_cancellation, prepare_task.done - State reads: self._lock - Return expressions: None ## `vllm_mlx.lifecycle.ResidencyManager._cleanup_cancelled_load` - Kind: method - Signature: `async def _cleanup_cancelled_load(self, resident: ResidentModel, engine: BaseEngine | None) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/lifecycle.py#L447-L465 - Implementation: Method `ResidencyManager._cleanup_cancelled_load` calls `suspend_cancellation`, `suppress`, `engine.stop`; awaits asynchronous work. Stop a partially loaded engine and unwind resident state. - Inputs: - `resident` (ResidentModel; required): Required positional or keyword input. - `engine` (BaseEngine | None; required): Required positional or keyword input. - Return annotation: `None` - Calls: suspend_cancellation, suppress, engine.stop - State reads: self._lock ## `vllm_mlx.lifecycle.ResidencyManager._release_load_waiter` - Kind: method - Signature: `async def _release_load_waiter(self, model_key: str, task: asyncio.Task[BaseEngine]) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/lifecycle.py#L467-L493 - Implementation: Method `ResidencyManager._release_load_waiter` calls `self._resident`, `task.done`, `suspend_cancellation`, `task_to_cancel.cancel`; awaits asynchronous work; returns `None`. Drop one waiter from a shared load, canceling abandoned solo loads. - Inputs: - `model_key` (str; required): Required positional or keyword input. - `task` (asyncio.Task[BaseEngine]; required): Required positional or keyword input. - Return annotation: `None` - Calls: self._resident, task.done, suspend_cancellation, task_to_cancel.cancel, suppress - State reads: self._lock, self._resident - Return expressions: None # Module `vllm_mlx.mcp` MCP (Model Context Protocol) client support for vllm-mlx. This module provides integration with MCP servers, allowing the vllm-mlx server to discover and execute tools from external MCP servers. Example usage: from vllm_mlx.mcp import MCPClientManager, load_mcp_config config = load_mcp_config("./mcp.json") manager = MCPClientManager(config) await manager.start() # Get all available tools in OpenAI format tools = manager.get_all_tools() # Execute a tool call result = await manager.execute_tool("filesystem__read_file", {"path": "/tmp/test.txt"}) Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mcp/__init__.py#L1-L85 # Module `vllm_mlx.mcp.client` MCP client for connecting to individual MCP servers. Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mcp/client.py#L1-L328 ## `vllm_mlx.mcp.client.MCPClient` - Kind: class - Signature: `class MCPClient` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mcp/client.py#L23-L328 - Implementation: Class `MCPClient` declares 15 direct member(s). Client for connecting to a single MCP server. Supports both stdio and SSE transports. - Inputs: - `config` (MCPServerConfig; required): Server configuration - Constructs: `vllm_mlx.mcp.client.MCPClient` ## `vllm_mlx.mcp.client.MCPClient.__init__` - Kind: method - Signature: `def __init__(self, config: MCPServerConfig)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mcp/client.py#L30-L45 - Implementation: Method `MCPClient.__init__` updates `self.config`, `self._session`, `self._read`, `self._write`; calls `asyncio.Lock`. Initialize MCP client. Args: config: Server configuration - Inputs: - `config` (MCPServerConfig; required): Server configuration - Return annotation: `not annotated` - Calls: asyncio.Lock - State writes: self.config, self._session, self._read, self._write, self._tools, self._state, self._error, self._last_connected, self._lock ## `vllm_mlx.mcp.client.MCPClient.name` - Kind: method - Signature: `def name(self) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mcp/client.py#L48-L50 - Implementation: Method `MCPClient.name` returns `self.config.name`. Get server name. - Inputs: none - Return annotation: `str` - Decorators: property - State reads: self.config.name, self.config - Return expressions: self.config.name ## `vllm_mlx.mcp.client.MCPClient.state` - Kind: method - Signature: `def state(self) -> MCPServerState` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mcp/client.py#L53-L55 - Implementation: Method `MCPClient.state` returns `self._state`. Get current connection state. - Inputs: none - Return annotation: `MCPServerState` - Decorators: property - State reads: self._state - Return expressions: self._state ## `vllm_mlx.mcp.client.MCPClient.is_connected` - Kind: method - Signature: `def is_connected(self) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mcp/client.py#L58-L60 - Implementation: Method `MCPClient.is_connected` returns `self._state == MCPServerState.CONNECTED`. Check if connected to server. - Inputs: none - Return annotation: `bool` - Decorators: property - State reads: self._state - Return expressions: self._state == MCPServerState.CONNECTED ## `vllm_mlx.mcp.client.MCPClient.tools` - Kind: method - Signature: `def tools(self) -> List[MCPTool]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mcp/client.py#L63-L65 - Implementation: Method `MCPClient.tools` returns `self._tools`. Get discovered tools. - Inputs: none - Return annotation: `List[MCPTool]` - Decorators: property - State reads: self._tools - Return expressions: self._tools ## `vllm_mlx.mcp.client.MCPClient.get_status` - Kind: method - Signature: `def get_status(self) -> MCPServerStatus` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mcp/client.py#L67-L76 - Implementation: Method `MCPClient.get_status` calls `MCPServerStatus`, `len`; returns `MCPServerStatus(name=self.name, state=self._state, transport=self.config.transport, tools_count=len(self._tools), error…`. Get server status. - Inputs: none - Return annotation: `MCPServerStatus` - Calls: MCPServerStatus, len - State reads: self.name, self._state, self.config.transport, self.config, self._tools, self._error, self._last_connected - Return expressions: MCPServerStatus(name=self.name, state=self._state, transport=self.config.transport, tools_count=len(self._tools), error… ## `vllm_mlx.mcp.client.MCPClient.connect` - Kind: method - Signature: `async def connect(self) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mcp/client.py#L78-L122 - Implementation: Method `MCPClient.connect` updates `self._state`, `self._error`, `self._last_connected`; calls `logger.info`, `self._connect_stdio`, `self._connect_sse`, `ValueError`; awaits asynchronous work; can raise `ValueError`; has 2 explicit return paths. Connect to the MCP server. Returns: True if connection successful, False otherwise - Inputs: none - Return annotation: `bool` - Calls: logger.info, self._connect_stdio, self._connect_sse, ValueError, self._initialize_session, self._discover_tools, time.time, len, str, logger.error - State reads: self._lock, self._state, self.config.enabled, self.config, self.name, self.config.transport, self._connect_stdio, self._connect_sse, self._initialize_session, self._discover_tools, self._tools - State writes: self._state, self._error, self._last_connected - Raises directly: ValueError - Return expressions: True; False ## `vllm_mlx.mcp.client.MCPClient._connect_stdio` - Kind: method - Signature: `async def _connect_stdio(self)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mcp/client.py#L124-L152 - Implementation: Method `MCPClient._connect_stdio` updates `self._stdio_client`, `self._read`, `self._write`, `self._session`; calls `ImportError`, `logger.info`, `' '.join`, `StdioServerParameters`; awaits asynchronous work; can raise `ImportError`. Connect via stdio transport. - Inputs: none - Return annotation: `not annotated` - Calls: ImportError, logger.info, ' '.join, StdioServerParameters, stdio_client, self._stdio_client.__aenter__, ClientSession, self._session.__aenter__ - State reads: self.name, self.config.command, self.config, self.config.args, self.config.env, self._stdio_client.__aenter__, self._stdio_client, self._read, self._write, self._session.__aenter__, self._session - State writes: self._stdio_client, self._read, self._write, self._session - Raises directly: ImportError ## `vllm_mlx.mcp.client.MCPClient._connect_sse` - Kind: method - Signature: `async def _connect_sse(self)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mcp/client.py#L154-L170 - Implementation: Method `MCPClient._connect_sse` updates `self._sse_client`, `self._read`, `self._write`, `self._session`; calls `ImportError`, `sse_client`, `self._sse_client.__aenter__`, `ClientSession`; awaits asynchronous work; can raise `ImportError`. Connect via SSE transport. - Inputs: none - Return annotation: `not annotated` - Calls: ImportError, sse_client, self._sse_client.__aenter__, ClientSession, self._session.__aenter__ - State reads: self.config.url, self.config, self._sse_client.__aenter__, self._sse_client, self._read, self._write, self._session.__aenter__, self._session - State writes: self._sse_client, self._read, self._write, self._session - Raises directly: ImportError ## `vllm_mlx.mcp.client.MCPClient._initialize_session` - Kind: method - Signature: `async def _initialize_session(self)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mcp/client.py#L172-L183 - Implementation: Method `MCPClient._initialize_session` calls `RuntimeError`, `self._session.initialize`, `logger.debug`; awaits asynchronous work; can raise `RuntimeError`. Initialize the MCP session. - Inputs: none - Return annotation: `not annotated` - Calls: RuntimeError, self._session.initialize, logger.debug - State reads: self._session, self._session.initialize, self.name - Raises directly: RuntimeError ## `vllm_mlx.mcp.client.MCPClient._discover_tools` - Kind: method - Signature: `async def _discover_tools(self)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mcp/client.py#L185-L208 - Implementation: Method `MCPClient._discover_tools` updates `self._tools`; calls `RuntimeError`, `self._session.list_tools`, `MCPTool`, `hasattr`; awaits asynchronous work; can raise `RuntimeError`. Discover available tools from the server. - Inputs: none - Return annotation: `not annotated` - Calls: RuntimeError, self._session.list_tools, MCPTool, hasattr, self._tools.append, logger.debug, logger.warning - State reads: self._session, self._session.list_tools, self.name, self._tools.append, self._tools - State writes: self._tools - Raises directly: RuntimeError ## `vllm_mlx.mcp.client.MCPClient.disconnect` - Kind: method - Signature: `async def disconnect(self)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mcp/client.py#L210-L235 - Implementation: Method `MCPClient.disconnect` updates `self._session`, `self._stdio_client`, `self._sse_client`, `self._state`; calls `self._session.__aexit__`, `hasattr`, `self._stdio_client.__aexit__`, `self._sse_client.__aexit__`; awaits asynchronous work; returns `None`. Disconnect from the MCP server. - Inputs: none - Return annotation: `not annotated` - Calls: self._session.__aexit__, hasattr, self._stdio_client.__aexit__, self._sse_client.__aexit__, logger.warning, logger.info - State reads: self._lock, self._state, self._session, self._session.__aexit__, self._stdio_client, self._stdio_client.__aexit__, self._sse_client, self._sse_client.__aexit__, self.name - State writes: self._session, self._stdio_client, self._sse_client, self._state, self._tools - Return expressions: None ## `vllm_mlx.mcp.client.MCPClient.call_tool` - Kind: method - Signature: `async def call_tool(self, tool_name: str, arguments: Dict[str, Any], timeout: Optional[float]=None) -> MCPToolResult` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mcp/client.py#L237-L301 - Implementation: Method `MCPClient.call_tool` calls `MCPToolResult`, `asyncio.wait_for`, `self._session.call_tool`, `self._extract_content`; awaits asynchronous work; has 5 explicit return paths. Call a tool on the MCP server. Args: tool_name: Name of the tool (without server prefix) arguments: Tool arguments timeout: Optional timeout in seconds Returns: MCPToolResult with the result or error - Inputs: - `tool_name` (str; required): Name of the tool (without server prefix) - `arguments` (Dict[str, Any]; required): Tool arguments - `timeout` (Optional[float]; optional; default `None`): Optional timeout in seconds - Return annotation: `MCPToolResult` - Calls: MCPToolResult, asyncio.wait_for, self._session.call_tool, self._extract_content, hasattr, str - State reads: self.is_connected, self.name, self._session, self.config.timeout, self.config, self._session.call_tool, self._extract_content - Return expressions: MCPToolResult(tool_name=tool_name, content=None, is_error=True, error_message=f"Not connected to server '{self.name}'"); MCPToolResult(tool_name=tool_name, content=None, is_error=True, error_message='Session not initialized'); MCPToolResult(tool_name=tool_name, content=content, is_error=result.isError if hasattr(result, 'isError') else False); MCPToolResult(tool_name=tool_name, content=None, is_error=True, error_message=f'Tool call timed out after {timeout}s'); MCPToolResult(tool_name=tool_name, content=None, is_error=True, error_message=str(e)) ## `vllm_mlx.mcp.client.MCPClient._extract_content` - Kind: method - Signature: `def _extract_content(self, result) -> Any` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mcp/client.py#L303-L321 - Implementation: Method `MCPClient._extract_content` calls `hasattr`, `contents.append`, `str`, `len`; has 3 explicit return paths. Extract content from MCP tool result. - Inputs: - `result` (not annotated; required): Required positional or keyword input. - Return annotation: `Any` - Calls: hasattr, contents.append, str, len - Return expressions: None; contents[0]; contents ## `vllm_mlx.mcp.client.MCPClient.refresh_tools` - Kind: method - Signature: `async def refresh_tools(self)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mcp/client.py#L323-L328 - Implementation: Method `MCPClient.refresh_tools` calls `self._discover_tools`; awaits asynchronous work; returns `None`. Refresh the list of available tools. - Inputs: none - Return annotation: `not annotated` - Calls: self._discover_tools - State reads: self.is_connected, self._discover_tools - Return expressions: None # Module `vllm_mlx.mcp.config` MCP configuration loading and validation. Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mcp/config.py#L1-L199 ## `vllm_mlx.mcp.config.load_mcp_config` - Kind: function - Signature: `def load_mcp_config(path: Optional[Union[str, Path]]=None) -> MCPConfig` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mcp/config.py#L26-L70 - Implementation: Function `load_mcp_config` calls `_find_config_file`, `logger.info`, `MCPConfig`, `Path(config_path).expanduser`; can raise `ImportError`; has 2 explicit return paths. Load MCP configuration from file. Search order: 1. Explicit path argument 2. VLLM_MLX_MCP_CONFIG environment variable 3. ~/.config/vllm-mlx/mcp.json or mcp.yaml Args: path: Optional explicit path to config file Returns: MCPConfig object Raises: FileNotFoundError: If no config file found ValueError: If config is invalid - Inputs: - `path` (Optional[Union[str, Path]]; optional; default `None`): Optional explicit path to config file - Return annotation: `MCPConfig` - Calls: _find_config_file, logger.info, MCPConfig, Path(config_path).expanduser, Path, config_path.read_text, yaml.safe_load, ImportError, json.loads, validate_config - Raises directly: ImportError - Return expressions: MCPConfig(); validate_config(data) ## `vllm_mlx.mcp.config._find_config_file` - Kind: function - Signature: `def _find_config_file(explicit_path: Optional[Union[str, Path]]=None) -> Optional[Path]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mcp/config.py#L73-L98 - Implementation: Function `_find_config_file` calls `Path(explicit_path).expanduser`, `Path`, `path.exists`, `FileNotFoundError`; can raise `FileNotFoundError`; has 2 explicit return paths. Find the config file to use. - Inputs: - `explicit_path` (Optional[Union[str, Path]]; optional; default `None`): Optional positional or keyword input; defaults to `None`. - Return annotation: `Optional[Path]` - Calls: Path(explicit_path).expanduser, Path, path.exists, FileNotFoundError, os.environ.get, Path(env_path).expanduser, logger.warning, Path(search_path).expanduser - Raises directly: FileNotFoundError - Return expressions: path; None ## `vllm_mlx.mcp.config.validate_config` - Kind: function - Signature: `def validate_config(data: Dict[str, Any]) -> MCPConfig` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mcp/config.py#L101-L163 - Implementation: Function `validate_config` calls `isinstance`, `ValueError`, `data.get`, `servers_data.items`; can raise `ValueError`; returns `MCPConfig(servers=servers, max_tool_calls=max_tool_calls, default_timeout=default_timeout, allowed_high_risk_tools=set(…`. Validate and parse configuration dictionary. Args: data: Raw configuration dictionary Returns: Validated MCPConfig object Raises: ValueError: If configuration is invalid - Inputs: - `data` (Dict[str, Any]; required): Raw configuration dictionary - Return annotation: `MCPConfig` - Calls: isinstance, ValueError, data.get, servers_data.items, server_data.copy, MCPServerConfig, any, tool.strip, MCPConfig, set - Raises directly: ValueError - Return expressions: MCPConfig(servers=servers, max_tool_calls=max_tool_calls, default_timeout=default_timeout, allowed_high_risk_tools=set(… ## `vllm_mlx.mcp.config.create_example_config` - Kind: function - Signature: `def create_example_config() -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mcp/config.py#L166-L199 - Implementation: Function `create_example_config` calls `json.dumps`; returns `json.dumps(example, indent=2)`. Create an example MCP configuration. Returns: JSON string with example configuration - Inputs: none - Return annotation: `str` - Calls: json.dumps - Return expressions: json.dumps(example, indent=2) # Module `vllm_mlx.mcp.executor` Tool executor for handling tool calls from model responses. Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mcp/executor.py#L1-L500 ## `vllm_mlx.mcp.executor.ToolArgumentValidationError` - Kind: class - Signature: `class ToolArgumentValidationError(Exception)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mcp/executor.py#L22-L25 - Implementation: Class `ToolArgumentValidationError` derives from `Exception` and declares 0 direct member(s). Raised when tool arguments fail validation against schema. - Inputs: none - Constructs: `vllm_mlx.mcp.executor.ToolArgumentValidationError` ## `vllm_mlx.mcp.executor.validate_tool_arguments` - Kind: function - Signature: `def validate_tool_arguments(tool: MCPTool, arguments: Dict[str, Any], strict: bool=True) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mcp/executor.py#L28-L61 - Implementation: Function `validate_tool_arguments` calls `logger.debug`, `jsonschema.validate`, `'.'.join`, `str`; can raise `ToolArgumentValidationError`; returns `None`. Validate tool arguments against the tool's input schema. Args: tool: The MCP tool with input_schema arguments: Arguments to validate strict: If True, raise exception on validation failure Raises: ToolArgumentValidationError: If validation fails and strict=True - Inputs: - `tool` (MCPTool; required): The MCP tool with input_schema - `arguments` (Dict[str, Any]; required): Arguments to validate - `strict` (bool; optional; default `True`): If True, raise exception on validation failure - Return annotation: `None` - Calls: logger.debug, jsonschema.validate, '.'.join, str, logger.warning, ToolArgumentValidationError - Raises directly: ToolArgumentValidationError - Return expressions: None ## `vllm_mlx.mcp.executor.ToolExecutor` - Kind: class - Signature: `class ToolExecutor` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mcp/executor.py#L64-L479 - Implementation: Class `ToolExecutor` declares 11 direct member(s). Handles execution of tool calls from model responses. Provides utilities for: - Extracting tool calls from responses - Executing multiple tool calls (parallel or sequential) - Formatting results for conversation - Validating tool arguments against schemas - Inputs: - `manager` (MCPClientManager; required): MCP client manager - `max_parallel` (int; optional; default `5`): Maximum parallel tool executions - `default_timeout` (Optional[float]; optional; default `None`): Default timeout for tool calls - `validate_arguments` (bool; optional; default `True`): If True, validate arguments against tool schemas - `sandbox` (Optional[ToolSandbox]; optional; default `None`): Optional tool sandbox for security controls. Uses global if None. - Constructs: `vllm_mlx.mcp.executor.ToolExecutor` ## `vllm_mlx.mcp.executor.ToolExecutor.__init__` - Kind: method - Signature: `def __init__(self, manager: MCPClientManager, max_parallel: int=5, default_timeout: Optional[float]=None, validate_arguments: bool=True, sandbox: Optional[ToolSandbox]=None)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mcp/executor.py#L75-L97 - Implementation: Method `ToolExecutor.__init__` updates `self.manager`, `self.max_parallel`, `self.default_timeout`, `self.validate_arguments`; calls `get_sandbox`. Initialize tool executor. Args: manager: MCP client manager max_parallel: Maximum parallel tool executions default_timeout: Default timeout for tool calls validate_arguments: If True, validate arguments against tool schemas sandbox: Optional tool sandbox for security controls. Uses global if None. - Inputs: - `manager` (MCPClientManager; required): MCP client manager - `max_parallel` (int; optional; default `5`): Maximum parallel tool executions - `default_timeout` (Optional[float]; optional; default `None`): Default timeout for tool calls - `validate_arguments` (bool; optional; default `True`): If True, validate arguments against tool schemas - `sandbox` (Optional[ToolSandbox]; optional; default `None`): Optional tool sandbox for security controls. Uses global if None. - Return annotation: `not annotated` - Calls: get_sandbox - State writes: self.manager, self.max_parallel, self.default_timeout, self.validate_arguments, self.sandbox ## `vllm_mlx.mcp.executor.ToolExecutor.execute_tool_calls` - Kind: method - Signature: `async def execute_tool_calls(self, tool_calls: List[Dict[str, Any]], parallel: bool=True) -> List[Tuple[MCPToolResult, str]]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mcp/executor.py#L99-L120 - Implementation: Method `ToolExecutor.execute_tool_calls` calls `self._execute_parallel`, `self._execute_sequential`; awaits asynchronous work; has 3 explicit return paths. Execute multiple tool calls. Args: tool_calls: List of OpenAI tool call objects parallel: Execute in parallel (True) or sequential (False) Returns: List of (MCPToolResult, tool_call_id) tuples - Inputs: - `tool_calls` (List[Dict[str, Any]]; required): List of OpenAI tool call objects - `parallel` (bool; optional; default `True`): Execute in parallel (True) or sequential (False) - Return annotation: `List[Tuple[MCPToolResult, str]]` - Calls: self._execute_parallel, self._execute_sequential - State reads: self._execute_parallel, self._execute_sequential - Return expressions: []; await self._execute_parallel(tool_calls); await self._execute_sequential(tool_calls) ## `vllm_mlx.mcp.executor.ToolExecutor._get_tool_by_name` - Kind: method - Signature: `def _get_tool_by_name(self, full_name: str) -> Optional[MCPTool]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mcp/executor.py#L122-L132 - Implementation: Method `ToolExecutor._get_tool_by_name` calls `self.manager.get_all_tools`; has 2 explicit return paths. Get a tool by its full name (server__tool or just tool). - Inputs: - `full_name` (str; required): Required positional or keyword input. - Return annotation: `Optional[MCPTool]` - Calls: self.manager.get_all_tools - State reads: self.manager.get_all_tools, self.manager - Return expressions: tool; None ## `vllm_mlx.mcp.executor.ToolExecutor._validate_tool_call` - Kind: method - Signature: `def _validate_tool_call(self, tool_call: Dict[str, Any]) -> Optional[str]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mcp/executor.py#L134-L165 - Implementation: Method `ToolExecutor._validate_tool_call` calls `tool_call.get`, `func.get`, `isinstance`, `json.loads`; has 3 explicit return paths. Validate a tool call's arguments against the tool's schema. Returns: Error message if validation fails, None if valid - Inputs: - `tool_call` (Dict[str, Any]; required): Required positional or keyword input. - Return annotation: `Optional[str]` - Calls: tool_call.get, func.get, isinstance, json.loads, self._get_tool_by_name, validate_tool_arguments, str - State reads: self.validate_arguments, self._get_tool_by_name - Return expressions: None; f"Invalid JSON in arguments for tool '{name}'"; str(e) ## `vllm_mlx.mcp.executor.ToolExecutor._validate_sandbox` - Kind: method - Signature: `def _validate_sandbox(self, tool_name: str, server_name: str, arguments: Dict[str, Any]) -> Optional[str]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mcp/executor.py#L167-L183 - Implementation: Method `ToolExecutor._validate_sandbox` calls `self.sandbox.validate_tool_execution`, `str`; has 2 explicit return paths. Validate tool execution against sandbox policy. Returns: Error message if blocked, None if allowed - Inputs: - `tool_name` (str; required): Required positional or keyword input. - `server_name` (str; required): Required positional or keyword input. - `arguments` (Dict[str, Any]; required): Required positional or keyword input. - Return annotation: `Optional[str]` - Calls: self.sandbox.validate_tool_execution, str - State reads: self.sandbox.validate_tool_execution, self.sandbox - Return expressions: None; str(e) ## `vllm_mlx.mcp.executor.ToolExecutor._get_server_for_tool` - Kind: method - Signature: `def _get_server_for_tool(self, full_name: str) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mcp/executor.py#L185-L193 - Implementation: Method `ToolExecutor._get_server_for_tool` calls `full_name.split`, `self.manager.get_all_tools`; has 3 explicit return paths. Extract server name from full tool name or find it. - Inputs: - `full_name` (str; required): Required positional or keyword input. - Return annotation: `str` - Calls: full_name.split, self.manager.get_all_tools - State reads: self.manager.get_all_tools, self.manager - Return expressions: full_name.split('__')[0]; tool.server_name; 'unknown' ## `vllm_mlx.mcp.executor.ToolExecutor._execute_parallel` - Kind: method - Signature: `async def _execute_parallel(self, tool_calls: List[Dict[str, Any]]) -> List[Tuple[MCPToolResult, str]]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mcp/executor.py#L195-L305 - Implementation: Method `ToolExecutor._execute_parallel` calls `asyncio.Semaphore`, `execute_with_semaphore`, `asyncio.gather`, `enumerate`; awaits asynchronous work; returns `processed`. Execute tool calls in parallel with concurrency limit. - Inputs: - `tool_calls` (List[Dict[str, Any]]; required): Required positional or keyword input. - Return annotation: `List[Tuple[MCPToolResult, str]]` - Calls: asyncio.Semaphore, execute_with_semaphore, asyncio.gather, enumerate, tool_calls[i].get, isinstance, processed.append, MCPToolResult, tool_calls[i].get('function', {}).get, str - State reads: self.max_parallel - Return expressions: processed ## `vllm_mlx.mcp.executor.ToolExecutor._execute_parallel.execute_with_semaphore` - Kind: nested function - Signature: `async def execute_with_semaphore(tool_call: Dict[str, Any])` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mcp/executor.py#L202-L281 - Implementation: Nested Function `ToolExecutor._execute_parallel.execute_with_semaphore` calls `tool_call.get`, `func.get`, `isinstance`, `json.loads`; awaits asynchronous work; has 3 explicit return paths. Nested Function `ToolExecutor._execute_parallel.execute_with_semaphore` calls `tool_call.get`, `func.get`, `isinstance`, `json.loads`; awaits asynchronous work; has 3 explicit return paths. - Inputs: - `tool_call` (Dict[str, Any]; required): Required positional or keyword input. - Return annotation: `not annotated` - Calls: tool_call.get, func.get, isinstance, json.loads, self._get_server_for_tool, name.split, self._validate_tool_call, self.sandbox.record_execution, MCPToolResult, self._validate_sandbox, time.time, self.manager.execute_tool_call - State reads: self._get_server_for_tool, self._validate_tool_call, self.sandbox.record_execution, self.sandbox, self._validate_sandbox, self.manager.execute_tool_call, self.manager, self.default_timeout - Return expressions: (MCPToolResult(tool_name=name, content=None, is_error=True, error_message=validation_error), call_id); (MCPToolResult(tool_name=name, content=None, is_error=True, error_message=sandbox_error), call_id); (result, call_id) ## `vllm_mlx.mcp.executor.ToolExecutor._execute_sequential` - Kind: method - Signature: `async def _execute_sequential(self, tool_calls: List[Dict[str, Any]]) -> List[Tuple[MCPToolResult, str]]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mcp/executor.py#L307-L415 - Implementation: Method `ToolExecutor._execute_sequential` calls `tool_call.get`, `func.get`, `isinstance`, `json.loads`; awaits asynchronous work; returns `results`. Execute tool calls sequentially. - Inputs: - `tool_calls` (List[Dict[str, Any]]; required): Required positional or keyword input. - Return annotation: `List[Tuple[MCPToolResult, str]]` - Calls: tool_call.get, func.get, isinstance, json.loads, self._get_server_for_tool, name.split, self._validate_tool_call, self.sandbox.record_execution, results.append, MCPToolResult, self._validate_sandbox, time.time, self.manager.execute_tool_call, str - State reads: self._get_server_for_tool, self._validate_tool_call, self.sandbox.record_execution, self.sandbox, self._validate_sandbox, self.manager.execute_tool_call, self.manager, self.default_timeout - Return expressions: results ## `vllm_mlx.mcp.executor.ToolExecutor.execute_and_format` - Kind: method - Signature: `async def execute_and_format(self, tool_calls: List[Dict[str, Any]], parallel: bool=True) -> List[Dict[str, Any]]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mcp/executor.py#L417-L433 - Implementation: Method `ToolExecutor.execute_and_format` calls `self.execute_tool_calls`, `format_tool_result`; awaits asynchronous work; returns `[format_tool_result(result, call_id) for result, call_id in results]`. Execute tool calls and format results as messages. Args: tool_calls: List of OpenAI tool call objects parallel: Execute in parallel Returns: List of tool result messages ready for conversation - Inputs: - `tool_calls` (List[Dict[str, Any]]; required): List of OpenAI tool call objects - `parallel` (bool; optional; default `True`): Execute in parallel - Return annotation: `List[Dict[str, Any]]` - Calls: self.execute_tool_calls, format_tool_result - State reads: self.execute_tool_calls - Return expressions: [format_tool_result(result, call_id) for result, call_id in results] ## `vllm_mlx.mcp.executor.ToolExecutor.extract_and_validate` - Kind: method - Signature: `def extract_and_validate(self, response: Dict[str, Any]) -> Tuple[List[Dict[str, Any]], bool]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mcp/executor.py#L435-L464 - Implementation: Method `ToolExecutor.extract_and_validate` calls `extract_tool_calls`, `tc.get`, `func.get`, `self._tool_exists`; has 2 explicit return paths. Extract tool calls from response and validate them. Args: response: Model response in OpenAI format Returns: Tuple of (tool_calls, all_valid) - Inputs: - `response` (Dict[str, Any]; required): Model response in OpenAI format - Return annotation: `Tuple[List[Dict[str, Any]], bool]` - Calls: extract_tool_calls, tc.get, func.get, self._tool_exists, logger.warning - State reads: self._tool_exists - Return expressions: ([], True); (tool_calls, all_valid) ## `vllm_mlx.mcp.executor.ToolExecutor._tool_exists` - Kind: method - Signature: `def _tool_exists(self, full_name: str) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mcp/executor.py#L466-L479 - Implementation: Method `ToolExecutor._tool_exists` calls `self.manager.get_all_tools`; has 2 explicit return paths. Check if a tool exists in any connected server. - Inputs: - `full_name` (str; required): Required positional or keyword input. - Return annotation: `bool` - Calls: self.manager.get_all_tools - State reads: self.manager.get_all_tools, self.manager - Return expressions: True; False ## `vllm_mlx.mcp.executor.execute_single_tool` - Kind: function - Signature: `async def execute_single_tool(manager: MCPClientManager, tool_name: str, arguments: Dict[str, Any], timeout: Optional[float]=None) -> MCPToolResult` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mcp/executor.py#L482-L500 - Implementation: Function `execute_single_tool` calls `manager.execute_tool`; awaits asynchronous work; returns `await manager.execute_tool(tool_name, arguments, timeout)`. Convenience function to execute a single tool. Args: manager: MCP client manager tool_name: Full tool name (server__tool) arguments: Tool arguments timeout: Optional timeout Returns: MCPToolResult - Inputs: - `manager` (MCPClientManager; required): MCP client manager - `tool_name` (str; required): Full tool name (server__tool) - `arguments` (Dict[str, Any]; required): Tool arguments - `timeout` (Optional[float]; optional; default `None`): Optional timeout - Return annotation: `MCPToolResult` - Calls: manager.execute_tool - Return expressions: await manager.execute_tool(tool_name, arguments, timeout) # Module `vllm_mlx.mcp.manager` MCP Client Manager for handling multiple MCP server connections. Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mcp/manager.py#L1-L301 ## `vllm_mlx.mcp.manager.MCPClientManager` - Kind: class - Signature: `class MCPClientManager` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mcp/manager.py#L22-L301 - Implementation: Class `MCPClientManager` declares 14 direct member(s). Manages multiple MCP server connections. Provides a unified interface for: - Connecting to multiple MCP servers - Discovering and aggregating tools - Executing tool calls - Managing connection lifecycle - Inputs: - `config` (MCPConfig; required): MCP configuration with server definitions - Constructs: `vllm_mlx.mcp.manager.MCPClientManager` ## `vllm_mlx.mcp.manager.MCPClientManager.__init__` - Kind: method - Signature: `def __init__(self, config: MCPConfig)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mcp/manager.py#L33-L47 - Implementation: Method `MCPClientManager.__init__` updates `self.config`, `self._clients`, `self._started`, `self._lock`; calls `asyncio.Lock`, `config.servers.items`, `MCPClient`. Initialize MCP Client Manager. Args: config: MCP configuration with server definitions - Inputs: - `config` (MCPConfig; required): MCP configuration with server definitions - Return annotation: `not annotated` - Calls: asyncio.Lock, config.servers.items, MCPClient - State reads: self._clients - State writes: self.config, self._clients, self._started, self._lock ## `vllm_mlx.mcp.manager.MCPClientManager.is_started` - Kind: method - Signature: `def is_started(self) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mcp/manager.py#L50-L52 - Implementation: Method `MCPClientManager.is_started` returns `self._started`. Check if manager has been started. - Inputs: none - Return annotation: `bool` - Decorators: property - State reads: self._started - Return expressions: self._started ## `vllm_mlx.mcp.manager.MCPClientManager.start` - Kind: method - Signature: `async def start(self)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mcp/manager.py#L54-L96 - Implementation: Method `MCPClientManager.start` updates `self._started`; calls `logger.info`, `len`, `client.connect`, `self._clients.values`; awaits asynchronous work; returns `None`. Start the manager and connect to all enabled servers. Connections are made in parallel for faster startup. - Inputs: none - Return annotation: `not annotated` - Calls: logger.info, len, client.connect, self._clients.values, asyncio.gather, zip, isinstance, logger.error, sum - State reads: self._lock, self._started, self._clients, self._clients.values - State writes: self._started - Return expressions: None ## `vllm_mlx.mcp.manager.MCPClientManager.stop` - Kind: method - Signature: `async def stop(self)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mcp/manager.py#L98-L112 - Implementation: Method `MCPClientManager.stop` updates `self._started`; calls `logger.info`, `client.disconnect`, `self._clients.values`, `asyncio.gather`; awaits asynchronous work; returns `None`. Stop the manager and disconnect from all servers. - Inputs: none - Return annotation: `not annotated` - Calls: logger.info, client.disconnect, self._clients.values, asyncio.gather - State reads: self._lock, self._started, self._clients.values, self._clients - State writes: self._started - Return expressions: None ## `vllm_mlx.mcp.manager.MCPClientManager.get_all_tools` - Kind: method - Signature: `def get_all_tools(self) -> List[MCPTool]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mcp/manager.py#L114-L125 - Implementation: Method `MCPClientManager.get_all_tools` calls `self._clients.values`, `tools.extend`; returns `tools`. Get all tools from all connected servers. Returns: List of MCPTool instances - Inputs: none - Return annotation: `List[MCPTool]` - Calls: self._clients.values, tools.extend - State reads: self._clients.values, self._clients - Return expressions: tools ## `vllm_mlx.mcp.manager.MCPClientManager.get_all_tools_openai` - Kind: method - Signature: `def get_all_tools_openai(self) -> List[Dict[str, Any]]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mcp/manager.py#L127-L134 - Implementation: Method `MCPClientManager.get_all_tools_openai` calls `mcp_tools_to_openai`, `self.get_all_tools`; returns `mcp_tools_to_openai(self.get_all_tools())`. Get all tools in OpenAI function calling format. Returns: List of OpenAI-compatible tool definitions - Inputs: none - Return annotation: `List[Dict[str, Any]]` - Calls: mcp_tools_to_openai, self.get_all_tools - State reads: self.get_all_tools - Return expressions: mcp_tools_to_openai(self.get_all_tools()) ## `vllm_mlx.mcp.manager.MCPClientManager.get_merged_tools` - Kind: method - Signature: `def get_merged_tools(self, user_tools: Optional[List[Dict[str, Any]]]=None) -> List[Dict[str, Any]]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mcp/manager.py#L136-L151 - Implementation: Method `MCPClientManager.get_merged_tools` calls `merge_tools`, `self.get_all_tools`; returns `merge_tools(self.get_all_tools(), user_tools)`. Get MCP tools merged with user-provided tools. User tools take precedence on name conflicts. Args: user_tools: Optional user-provided tools in OpenAI format Returns: Combined list of tools in OpenAI format - Inputs: - `user_tools` (Optional[List[Dict[str, Any]]]; optional; default `None`): Optional user-provided tools in OpenAI format - Return annotation: `List[Dict[str, Any]]` - Calls: merge_tools, self.get_all_tools - State reads: self.get_all_tools - Return expressions: merge_tools(self.get_all_tools(), user_tools) ## `vllm_mlx.mcp.manager.MCPClientManager.get_server_status` - Kind: method - Signature: `def get_server_status(self) -> List[MCPServerStatus]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mcp/manager.py#L153-L160 - Implementation: Method `MCPClientManager.get_server_status` calls `client.get_status`, `self._clients.values`; returns `[client.get_status() for client in self._clients.values()]`. Get status of all servers. Returns: List of MCPServerStatus for each server - Inputs: none - Return annotation: `List[MCPServerStatus]` - Calls: client.get_status, self._clients.values - State reads: self._clients.values, self._clients - Return expressions: [client.get_status() for client in self._clients.values()] ## `vllm_mlx.mcp.manager.MCPClientManager.get_client` - Kind: method - Signature: `def get_client(self, server_name: str) -> Optional[MCPClient]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mcp/manager.py#L162-L172 - Implementation: Method `MCPClientManager.get_client` calls `self._clients.get`; returns `self._clients.get(server_name)`. Get client for a specific server. Args: server_name: Name of the server Returns: MCPClient instance or None if not found - Inputs: - `server_name` (str; required): Name of the server - Return annotation: `Optional[MCPClient]` - Calls: self._clients.get - State reads: self._clients.get, self._clients - Return expressions: self._clients.get(server_name) ## `vllm_mlx.mcp.manager.MCPClientManager.execute_tool` - Kind: method - Signature: `async def execute_tool(self, full_name: str, arguments: Dict[str, Any], timeout: Optional[float]=None) -> MCPToolResult` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mcp/manager.py#L174-L232 - Implementation: Method `MCPClientManager.execute_tool` calls `openai_call_to_mcp`, `self._find_tool_server`, `MCPToolResult`, `self._clients.get`; awaits asynchronous work; has 4 explicit return paths. Execute a tool by its full name (server__tool). Args: full_name: Full tool name with server prefix arguments: Tool arguments timeout: Optional timeout in seconds Returns: MCPToolResult with the result or error - Inputs: - `full_name` (str; required): Full tool name with server prefix - `arguments` (Dict[str, Any]; required): Tool arguments - `timeout` (Optional[float]; optional; default `None`): Optional timeout in seconds - Return annotation: `MCPToolResult` - Calls: openai_call_to_mcp, self._find_tool_server, MCPToolResult, self._clients.get, client.call_tool - State reads: self._find_tool_server, self._clients.get, self._clients, self.config.default_timeout, self.config - Return expressions: MCPToolResult(tool_name=full_name, content=None, is_error=True, error_message=f"Tool '{full_name}' not found in any con…; MCPToolResult(tool_name=full_name, content=None, is_error=True, error_message=f"Server '{server_name}' not found"); MCPToolResult(tool_name=full_name, content=None, is_error=True, error_message=f"Server '{server_name}' is not connected…; await client.call_tool(tool_name, arguments, timeout=timeout or self.config.default_timeout) ## `vllm_mlx.mcp.manager.MCPClientManager.execute_tool_call` - Kind: method - Signature: `async def execute_tool_call(self, tool_call: Dict[str, Any], timeout: Optional[float]=None) -> MCPToolResult` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mcp/manager.py#L234-L256 - Implementation: Method `MCPClientManager.execute_tool_call` calls `openai_call_to_mcp`, `self.execute_tool`; awaits asynchronous work; returns `await self.execute_tool(full_name, arguments, timeout)`. Execute a tool call from OpenAI format. Args: tool_call: OpenAI tool call object timeout: Optional timeout in seconds Returns: MCPToolResult with the result or error - Inputs: - `tool_call` (Dict[str, Any]; required): OpenAI tool call object - `timeout` (Optional[float]; optional; default `None`): Optional timeout in seconds - Return annotation: `MCPToolResult` - Calls: openai_call_to_mcp, self.execute_tool - State reads: self.execute_tool - Return expressions: await self.execute_tool(full_name, arguments, timeout) ## `vllm_mlx.mcp.manager.MCPClientManager._find_tool_server` - Kind: method - Signature: `def _find_tool_server(self, tool_name: str) -> Optional[str]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mcp/manager.py#L258-L273 - Implementation: Method `MCPClientManager._find_tool_server` calls `self._clients.values`; has 2 explicit return paths. Find which server has a tool by name. Args: tool_name: Tool name (without server prefix) Returns: Server name or None if not found - Inputs: - `tool_name` (str; required): Tool name (without server prefix) - Return annotation: `Optional[str]` - Calls: self._clients.values - State reads: self._clients.values, self._clients - Return expressions: client.name; None ## `vllm_mlx.mcp.manager.MCPClientManager.refresh_tools` - Kind: method - Signature: `async def refresh_tools(self)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mcp/manager.py#L275-L283 - Implementation: Method `MCPClientManager.refresh_tools` calls `client.refresh_tools`, `self._clients.values`, `asyncio.gather`; awaits asynchronous work. Refresh tools from all connected servers. - Inputs: none - Return annotation: `not annotated` - Calls: client.refresh_tools, self._clients.values, asyncio.gather - State reads: self._clients.values, self._clients ## `vllm_mlx.mcp.manager.MCPClientManager.reconnect` - Kind: method - Signature: `async def reconnect(self, server_name: Optional[str]=None)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mcp/manager.py#L285-L301 - Implementation: Method `MCPClientManager.reconnect` calls `self._clients.get`, `client.disconnect`, `client.connect`, `self._clients.values`; awaits asynchronous work. Reconnect to server(s). Args: server_name: Specific server to reconnect, or None for all - Inputs: - `server_name` (Optional[str]; optional; default `None`): Specific server to reconnect, or None for all - Return annotation: `not annotated` - Calls: self._clients.get, client.disconnect, client.connect, self._clients.values - State reads: self._clients.get, self._clients, self._clients.values # Module `vllm_mlx.mcp.security` MCP security module for command validation and sandboxing. This module provides security controls to prevent command injection and other attacks via MCP server configurations. Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mcp/security.py#L1-L852 ## `vllm_mlx.mcp.security.MCPSecurityError` - Kind: class - Signature: `class MCPSecurityError(Exception)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mcp/security.py#L106-L109 - Implementation: Class `MCPSecurityError` derives from `Exception` and declares 0 direct member(s). Raised when MCP security validation fails. - Inputs: none - Constructs: `vllm_mlx.mcp.security.MCPSecurityError` ## `vllm_mlx.mcp.security.MCPCommandValidator` - Kind: class - Signature: `class MCPCommandValidator` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mcp/security.py#L112-L427 - Implementation: Class `MCPCommandValidator` declares 8 direct member(s). Validates MCP server commands for security. This class provides methods to validate commands and arguments before they are executed, preventing command injection attacks. - Inputs: - `allowed_commands` (Optional[Set[str]]; optional; default `None`): Set of allowed command names. If None, uses default whitelist. - `allow_unsafe` (bool; optional; default `False`): If True, allows any command (for development only). - `custom_whitelist` (Optional[Set[str]]; optional; default `None`): Additional commands to allow beyond the default whitelist. - `check_path_exists` (bool; optional; default `True`): If True, verify command exists in PATH. Set to False for testing. - Constructs: `vllm_mlx.mcp.security.MCPCommandValidator` ## `vllm_mlx.mcp.security.MCPCommandValidator.__init__` - Kind: method - Signature: `def __init__(self, allowed_commands: Optional[Set[str]]=None, allow_unsafe: bool=False, custom_whitelist: Optional[Set[str]]=None, check_path_exists: bool=True)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mcp/security.py#L120-L149 - Implementation: Method `MCPCommandValidator.__init__` updates `self.allow_unsafe`, `self.allowed_commands`, `self.check_path_exists`; calls `ALLOWED_COMMANDS.copy`, `self.allowed_commands.update`, `logger.warning`. Initialize the command validator. Args: allowed_commands: Set of allowed command names. If None, uses default whitelist. allow_unsafe: If True, allows any command (for development only). WARNING: This disables security checks! custom_whitelist: Additional commands to allow beyond the default whitelist. check_path_exists: If True, verify command exists in PATH. Set to False for testing. - Inputs: - `allowed_commands` (Optional[Set[str]]; optional; default `None`): Set of allowed command names. If None, uses default whitelist. - `allow_unsafe` (bool; optional; default `False`): If True, allows any command (for development only). - `custom_whitelist` (Optional[Set[str]]; optional; default `None`): Additional commands to allow beyond the default whitelist. - `check_path_exists` (bool; optional; default `True`): If True, verify command exists in PATH. Set to False for testing. - Return annotation: `not annotated` - Calls: ALLOWED_COMMANDS.copy, self.allowed_commands.update, logger.warning - State reads: self.allowed_commands.update, self.allowed_commands - State writes: self.allow_unsafe, self.allowed_commands, self.check_path_exists ## `vllm_mlx.mcp.security.MCPCommandValidator._check_control_chars` - Kind: method - Signature: `def _check_control_chars(self, value: str, context: str, server_name: str) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mcp/security.py#L151-L157 - Implementation: Method `MCPCommandValidator._check_control_chars` calls `any`, `MCPSecurityError`; can raise `MCPSecurityError`. Block command separators carried via literal newlines. - Inputs: - `value` (str; required): Required positional or keyword input. - `context` (str; required): Required positional or keyword input. - `server_name` (str; required): Required positional or keyword input. - Return annotation: `None` - Calls: any, MCPSecurityError - Raises directly: MCPSecurityError ## `vllm_mlx.mcp.security.MCPCommandValidator._check_path_traversal` - Kind: method - Signature: `def _check_path_traversal(self, value: str, context: str, server_name: str) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mcp/security.py#L159-L194 - Implementation: Method `MCPCommandValidator._check_path_traversal` calls `unquote`, `candidates.append`, `value.lower`, `posixpath.normpath`; can raise `MCPSecurityError`. Block parent-directory traversal, including URL-encoded forms. This normalizes likely path-like inputs rather than relying only on the simple ``../`` regex, which can be bypassed by percent-encoding. - Inputs: - `value` (str; required): Required positional or keyword input. - `context` (str; required): Required positional or keyword input. - `server_name` (str; required): Required positional or keyword input. - Return annotation: `None` - Calls: unquote, candidates.append, value.lower, posixpath.normpath, candidate.replace, normalized.startswith, MCPSecurityError, candidate.replace('\\', '/').split, any - Raises directly: MCPSecurityError ## `vllm_mlx.mcp.security.MCPCommandValidator.validate_command` - Kind: method - Signature: `def validate_command(self, command: str, server_name: str) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mcp/security.py#L196-L258 - Implementation: Method `MCPCommandValidator.validate_command` calls `logger.warning`, `self._check_control_chars`, `self._check_path_traversal`, `pattern.search`; can raise `MCPSecurityError`; returns `None`. Validate that a command is safe to execute. Args: command: The command to validate server_name: Name of the MCP server (for logging) Raises: MCPSecurityError: If the command is not allowed - Inputs: - `command` (str; required): The command to validate - `server_name` (str; required): Name of the MCP server (for logging) - Return annotation: `None` - Calls: logger.warning, self._check_control_chars, self._check_path_traversal, pattern.search, MCPSecurityError, Path, os.path.isabs, os.path.isfile, os.access, logger.info, sorted, shutil.which, logger.debug - State reads: self.allow_unsafe, self._check_control_chars, self._check_path_traversal, self.allowed_commands, self.check_path_exists - Raises directly: MCPSecurityError - Return expressions: None ## `vllm_mlx.mcp.security.MCPCommandValidator.validate_args` - Kind: method - Signature: `def validate_args(self, args: List[str], server_name: str) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mcp/security.py#L260-L286 - Implementation: Method `MCPCommandValidator.validate_args` calls `enumerate`, `self._check_control_chars`, `self._check_path_traversal`, `pattern.search`; can raise `MCPSecurityError`; returns `None`. Validate command arguments for dangerous patterns. Args: args: List of command arguments server_name: Name of the MCP server (for logging) Raises: MCPSecurityError: If any argument contains dangerous patterns - Inputs: - `args` (List[str]; required): List of command arguments - `server_name` (str; required): Name of the MCP server (for logging) - Return annotation: `None` - Calls: enumerate, self._check_control_chars, self._check_path_traversal, pattern.search, MCPSecurityError, logger.debug, len - State reads: self.allow_unsafe, self._check_control_chars, self._check_path_traversal - Raises directly: MCPSecurityError - Return expressions: None ## `vllm_mlx.mcp.security.MCPCommandValidator.validate_command_args` - Kind: method - Signature: `def validate_command_args(self, command: str, args: List[str], server_name: str) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mcp/security.py#L288-L330 - Implementation: Method `MCPCommandValidator.validate_command_args` calls `Path`, `BLOCKED_COMMAND_ARG_RULES.get`, `enumerate`, `MCPSecurityError`; can raise `MCPSecurityError`; returns `None`. Validate command-specific argument combinations. Some whitelisted runtimes (python, node, npx) remain acceptable for launching packaged MCP servers, but inline evaluator flags such as ``python -c`` and ``node -e`` must be rejected. - Inputs: - `command` (str; required): Required positional or keyword input. - `args` (List[str]; required): Required positional or keyword input. - `server_name` (str; required): Required positional or keyword input. - Return annotation: `None` - Calls: Path, BLOCKED_COMMAND_ARG_RULES.get, enumerate, MCPSecurityError, arg.startswith, logger.debug - State reads: self.allow_unsafe - Raises directly: MCPSecurityError - Return expressions: None ## `vllm_mlx.mcp.security.MCPCommandValidator.validate_env` - Kind: method - Signature: `def validate_env(self, env: Optional[Dict[str, str]], server_name: str) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mcp/security.py#L332-L383 - Implementation: Method `MCPCommandValidator.validate_env` calls `env.items`, `self._check_control_chars`, `self._check_path_traversal`, `key.upper`; can raise `MCPSecurityError`; returns `None`. Validate environment variables for dangerous values. Args: env: Dictionary of environment variables server_name: Name of the MCP server (for logging) Raises: MCPSecurityError: If any env var contains dangerous patterns - Inputs: - `env` (Optional[Dict[str, str]]; required): Dictionary of environment variables - `server_name` (str; required): Name of the MCP server (for logging) - Return annotation: `None` - Calls: env.items, self._check_control_chars, self._check_path_traversal, key.upper, MCPSecurityError, pattern.search, logger.debug, len - State reads: self.allow_unsafe, self._check_control_chars, self._check_path_traversal - Raises directly: MCPSecurityError - Return expressions: None ## `vllm_mlx.mcp.security.MCPCommandValidator.validate_url` - Kind: method - Signature: `def validate_url(self, url: str, server_name: str) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mcp/security.py#L385-L427 - Implementation: Method `MCPCommandValidator.validate_url` calls `self._check_control_chars`, `url.startswith`, `MCPSecurityError`, `logger.warning`; can raise `MCPSecurityError`; returns `None`. Validate SSE URL for security. Args: url: The SSE URL to validate server_name: Name of the MCP server (for logging) Raises: MCPSecurityError: If the URL is not safe - Inputs: - `url` (str; required): The SSE URL to validate - `server_name` (str; required): Name of the MCP server (for logging) - Return annotation: `None` - Calls: self._check_control_chars, url.startswith, MCPSecurityError, logger.warning, urlparse, self._check_path_traversal, pattern.search, logger.debug - State reads: self.allow_unsafe, self._check_control_chars, self._check_path_traversal - Raises directly: MCPSecurityError - Return expressions: None ## `vllm_mlx.mcp.security.get_validator` - Kind: function - Signature: `def get_validator() -> MCPCommandValidator` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mcp/security.py#L434-L441 - Implementation: Function `get_validator` calls `MCPCommandValidator`, `os.environ.get`; returns `_validator`. Get the global command validator instance. - Inputs: none - Return annotation: `MCPCommandValidator` - Calls: MCPCommandValidator, os.environ.get - Return expressions: _validator ## `vllm_mlx.mcp.security.set_validator` - Kind: function - Signature: `def set_validator(validator: MCPCommandValidator) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mcp/security.py#L444-L447 - Implementation: Function `set_validator` contains no state mutation, call, raise, return, await, or yield. Set a custom global validator. - Inputs: - `validator` (MCPCommandValidator; required): Required positional or keyword input. - Return annotation: `None` ## `vllm_mlx.mcp.security.validate_mcp_server_config` - Kind: function - Signature: `def validate_mcp_server_config(server_name: str, command: Optional[str]=None, args: Optional[List[str]]=None, env: Optional[Dict[str, str]]=None, url: Optional[str]=None) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mcp/security.py#L450-L486 - Implementation: Function `validate_mcp_server_config` calls `get_validator`, `validator.validate_command`, `validator.validate_args`, `validator.validate_command_args`. Validate MCP server configuration for security. This is a convenience function that uses the global validator. Args: server_name: Name of the MCP server command: Command to execute (for stdio transport) args: Command arguments env: Environment variables url: SSE URL (for sse transport) Raises: MCPSecurityError: If validation fails - Inputs: - `server_name` (str; required): Name of the MCP server - `command` (Optional[str]; optional; default `None`): Command to execute (for stdio transport) - `args` (Optional[List[str]]; optional; default `None`): Command arguments - `env` (Optional[Dict[str, str]]; optional; default `None`): Environment variables - `url` (Optional[str]; optional; default `None`): SSE URL (for sse transport) - Return annotation: `None` - Calls: get_validator, validator.validate_command, validator.validate_args, validator.validate_command_args, validator.validate_env, validator.validate_url ## `vllm_mlx.mcp.security.ToolExecutionAudit` - Kind: class - Signature: `class ToolExecutionAudit` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mcp/security.py#L516-L525 - Implementation: Class `ToolExecutionAudit` declares 0 direct member(s). Record of a tool execution for audit purposes. - Inputs: - `timestamp` (float; required): Required constructor field. - `tool_name` (str; required): Required constructor field. - `server_name` (str; required): Required constructor field. - `arguments` (Dict[str, Any]; required): Required constructor field. - `success` (bool; required): Required constructor field. - `error_message` (Optional[str]; optional; default `None`): Optional constructor field; defaults to `None`. - `execution_time_ms` (Optional[float]; optional; default `None`): Optional constructor field; defaults to `None`. - Constructs: `vllm_mlx.mcp.security.ToolExecutionAudit` - Decorators: dataclass ## `vllm_mlx.mcp.security.ToolSandbox` - Kind: class - Signature: `class ToolSandbox` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mcp/security.py#L528-L834 - Implementation: Class `ToolSandbox` declares 10 direct member(s). Sandboxing controls for MCP tool execution. Provides: - Tool allowlisting/blocklisting - Argument sanitization - Audit logging - Rate limiting - Inputs: - `allowed_tools` (Optional[Set[str]]; optional; default `None`): If set, only these tools can be executed (whitelist mode). - `blocked_tools` (Optional[Set[str]]; optional; default `None`): Tools that are always blocked (blacklist mode). - `allowed_high_risk_tools` (Optional[Set[str]]; optional; default `None`): High-risk tools that are explicitly allowed. - `blocked_arg_patterns` (Optional[List[re.Pattern]]; optional; default `None`): Patterns to block in tool arguments. - `max_calls_per_minute` (int; optional; default `60`): Rate limit for tool calls (0 = unlimited). - `audit_callback` (Optional[Callable[[ToolExecutionAudit], None]]; optional; default `None`): Optional callback for audit events. - `enabled` (bool; optional; default `True`): If False, sandbox checks are bypassed (dev mode only). - Constructs: `vllm_mlx.mcp.security.ToolSandbox` ## `vllm_mlx.mcp.security.ToolSandbox.__init__` - Kind: method - Signature: `def __init__(self, allowed_tools: Optional[Set[str]]=None, blocked_tools: Optional[Set[str]]=None, allowed_high_risk_tools: Optional[Set[str]]=None, blocked_arg_patterns: Optional[List[re.Pattern]]=None, max_calls_per_minute: int=60, audit_callback: Optional[Callable[[ToolExecutionAudit], None]]=None, enabled: bool=True)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mcp/security.py#L539-L586 - Implementation: Method `ToolSandbox.__init__` updates `self.allowed_tools`, `self.blocked_tools`, `self.allowed_high_risk_tools`, `self.blocked_arg_patterns`; calls `set`, `tool.lower`, `DANGEROUS_TOOL_ARG_PATTERNS.copy`, `defaultdict`. Initialize tool sandbox. Args: allowed_tools: If set, only these tools can be executed (whitelist mode). blocked_tools: Tools that are always blocked (blacklist mode). allowed_high_risk_tools: High-risk tools that are explicitly allowed. blocked_arg_patterns: Patterns to block in tool arguments. max_calls_per_minute: Rate limit for tool calls (0 = unlimited). audit_callback: Optional callback for audit events. enabled: If False, sandbox checks are bypassed (dev mode only). - Inputs: - `allowed_tools` (Optional[Set[str]]; optional; default `None`): If set, only these tools can be executed (whitelist mode). - `blocked_tools` (Optional[Set[str]]; optional; default `None`): Tools that are always blocked (blacklist mode). - `allowed_high_risk_tools` (Optional[Set[str]]; optional; default `None`): High-risk tools that are explicitly allowed. - `blocked_arg_patterns` (Optional[List[re.Pattern]]; optional; default `None`): Patterns to block in tool arguments. - `max_calls_per_minute` (int; optional; default `60`): Rate limit for tool calls (0 = unlimited). - `audit_callback` (Optional[Callable[[ToolExecutionAudit], None]]; optional; default `None`): Optional callback for audit events. - `enabled` (bool; optional; default `True`): If False, sandbox checks are bypassed (dev mode only). - Return annotation: `not annotated` - Calls: set, tool.lower, DANGEROUS_TOOL_ARG_PATTERNS.copy, defaultdict, Lock, logger.warning - State writes: self.allowed_tools, self.blocked_tools, self.allowed_high_risk_tools, self.blocked_arg_patterns, self.max_calls_per_minute, self.audit_callback, self.enabled, self._call_times, self._rate_limit_lock, self._audit_log, self._audit_log_max_size, self._audit_lock ## `vllm_mlx.mcp.security.ToolSandbox.validate_tool_execution` - Kind: method - Signature: `def validate_tool_execution(self, tool_name: str, server_name: str, arguments: Dict[str, Any]) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mcp/security.py#L588-L634 - Implementation: Method `ToolSandbox.validate_tool_execution` calls `logger.debug`, `self._is_blocked`, `MCPSecurityError`, `self._check_high_risk_tool`; can raise `MCPSecurityError`; returns `None`. Validate that a tool execution is allowed. Args: tool_name: Name of the tool to execute server_name: MCP server providing the tool arguments: Tool arguments Raises: MCPSecurityError: If execution is not allowed - Inputs: - `tool_name` (str; required): Name of the tool to execute - `server_name` (str; required): MCP server providing the tool - `arguments` (Dict[str, Any]; required): Tool arguments - Return annotation: `None` - Calls: logger.debug, self._is_blocked, MCPSecurityError, self._check_high_risk_tool, self._validate_arguments, self._check_rate_limit - State reads: self.enabled, self._is_blocked, self.allowed_tools, self._check_high_risk_tool, self._validate_arguments, self._check_rate_limit - Raises directly: MCPSecurityError - Return expressions: None ## `vllm_mlx.mcp.security.ToolSandbox._is_blocked` - Kind: method - Signature: `def _is_blocked(self, tool_name: str, full_name: str) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mcp/security.py#L636-L642 - Implementation: Method `ToolSandbox._is_blocked` calls `tool_name.lower`; returns `tool_name in self.blocked_tools or full_name in self.blocked_tools or tool_name.lower() in self.blocked_tools`. Check if tool is in blocklist. - Inputs: - `tool_name` (str; required): Required positional or keyword input. - `full_name` (str; required): Required positional or keyword input. - Return annotation: `bool` - Calls: tool_name.lower - State reads: self.blocked_tools - Return expressions: tool_name in self.blocked_tools or full_name in self.blocked_tools or tool_name.lower() in self.blocked_tools ## `vllm_mlx.mcp.security.ToolSandbox._check_high_risk_tool` - Kind: method - Signature: `def _check_high_risk_tool(self, tool_name: str, full_name: str) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mcp/security.py#L644-L663 - Implementation: Method `ToolSandbox._check_high_risk_tool` calls `tool_name.lower`, `full_name.lower`, `logger.warning`, `MCPSecurityError`; can raise `MCPSecurityError`; returns `None`. Check if tool matches high-risk patterns. - Inputs: - `tool_name` (str; required): Required positional or keyword input. - `full_name` (str; required): Required positional or keyword input. - Return annotation: `None` - Calls: tool_name.lower, full_name.lower, logger.warning, MCPSecurityError - State reads: self.allowed_high_risk_tools - Raises directly: MCPSecurityError - Return expressions: None ## `vllm_mlx.mcp.security.ToolSandbox._validate_arguments` - Kind: method - Signature: `def _validate_arguments(self, tool_name: str, arguments: Dict[str, Any]) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mcp/security.py#L665-L686 - Implementation: Method `ToolSandbox._validate_arguments` calls `arguments.items`, `check_value`. Validate tool arguments for dangerous patterns. - Inputs: - `tool_name` (str; required): Required positional or keyword input. - `arguments` (Dict[str, Any]; required): Required positional or keyword input. - Return annotation: `None` - Calls: arguments.items, check_value ## `vllm_mlx.mcp.security.ToolSandbox._validate_arguments.check_value` - Kind: nested function - Signature: `def check_value(key: str, value: Any, path: str='') -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mcp/security.py#L668-L683 - Implementation: Nested Function `ToolSandbox._validate_arguments.check_value` calls `isinstance`, `pattern.search`, `MCPSecurityError`, `value.items`; can raise `MCPSecurityError`. Nested Function `ToolSandbox._validate_arguments.check_value` calls `isinstance`, `pattern.search`, `MCPSecurityError`, `value.items`; can raise `MCPSecurityError`. - Inputs: - `key` (str; required): Required positional or keyword input. - `value` (Any; required): Required positional or keyword input. - `path` (str; optional; default `''`): Optional positional or keyword input; defaults to `''`. - Return annotation: `None` - Calls: isinstance, pattern.search, MCPSecurityError, value.items, check_value, enumerate - State reads: self.blocked_arg_patterns - Raises directly: MCPSecurityError ## `vllm_mlx.mcp.security.ToolSandbox._check_rate_limit` - Kind: method - Signature: `def _check_rate_limit(self, full_name: str) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mcp/security.py#L688-L710 - Implementation: Method `ToolSandbox._check_rate_limit` calls `time.time`, `len`, `MCPSecurityError`, `self._call_times[full_name].append`; can raise `MCPSecurityError`; returns `None`. Check and enforce rate limit for tool calls. - Inputs: - `full_name` (str; required): Required positional or keyword input. - Return annotation: `None` - Calls: time.time, len, MCPSecurityError, self._call_times[full_name].append - State reads: self.max_calls_per_minute, self._rate_limit_lock, self._call_times - Raises directly: MCPSecurityError - Return expressions: None ## `vllm_mlx.mcp.security.ToolSandbox.record_execution` - Kind: method - Signature: `def record_execution(self, tool_name: str, server_name: str, arguments: Dict[str, Any], success: bool, error_message: Optional[str]=None, execution_time_ms: Optional[float]=None) -> ToolExecutionAudit` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mcp/security.py#L712-L772 - Implementation: Method `ToolSandbox.record_execution` updates `self._audit_log`; calls `ToolExecutionAudit`, `time.time`, `self._sanitize_arguments_for_log`, `self._audit_log.append`; returns `audit`. Record a tool execution for audit purposes. Args: tool_name: Name of the executed tool server_name: MCP server that executed the tool arguments: Arguments passed to the tool success: Whether execution succeeded error_message: Error message if failed execution_time_ms: Execution time in milliseconds Returns: The audit record - Inputs: - `tool_name` (str; required): Name of the executed tool - `server_name` (str; required): MCP server that executed the tool - `arguments` (Dict[str, Any]; required): Arguments passed to the tool - `success` (bool; required): Whether execution succeeded - `error_message` (Optional[str]; optional; default `None`): Error message if failed - `execution_time_ms` (Optional[float]; optional; default `None`): Execution time in milliseconds - Return annotation: `ToolExecutionAudit` - Calls: ToolExecutionAudit, time.time, self._sanitize_arguments_for_log, self._audit_log.append, len, logger.info, logger.warning, self.audit_callback, logger.error - State reads: self._sanitize_arguments_for_log, self._audit_lock, self._audit_log.append, self._audit_log, self._audit_log_max_size, self.audit_callback - State writes: self._audit_log - Return expressions: audit ## `vllm_mlx.mcp.security.ToolSandbox._sanitize_arguments_for_log` - Kind: method - Signature: `def _sanitize_arguments_for_log(self, arguments: Dict[str, Any]) -> Dict[str, Any]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mcp/security.py#L774-L794 - Implementation: Method `ToolSandbox._sanitize_arguments_for_log` calls `sanitize`; returns `sanitize(arguments)`. Sanitize arguments for logging (redact sensitive data). - Inputs: - `arguments` (Dict[str, Any]; required): Required positional or keyword input. - Return annotation: `Dict[str, Any]` - Calls: sanitize - Return expressions: sanitize(arguments) ## `vllm_mlx.mcp.security.ToolSandbox._sanitize_arguments_for_log.sanitize` - Kind: nested function - Signature: `def sanitize(obj: Any) -> Any` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mcp/security.py#L778-L792 - Implementation: Nested Function `ToolSandbox._sanitize_arguments_for_log.sanitize` calls `isinstance`, `any`, `k.lower`, `sanitize`; has 4 explicit return paths. Nested Function `ToolSandbox._sanitize_arguments_for_log.sanitize` calls `isinstance`, `any`, `k.lower`, `sanitize`; has 4 explicit return paths. - Inputs: - `obj` (Any; required): Required positional or keyword input. - Return annotation: `Any` - Calls: isinstance, any, k.lower, sanitize, obj.items, len - Return expressions: {k: '[REDACTED]' if any((s in k.lower() for s in sensitive_keys)) else sanitize(v) for k, v in obj.items()}; [sanitize(item) for item in obj]; obj[:100] + f'... [truncated, {len(obj)} chars total]'; obj ## `vllm_mlx.mcp.security.ToolSandbox.get_audit_log` - Kind: method - Signature: `def get_audit_log(self, limit: int=100, tool_filter: Optional[str]=None, server_filter: Optional[str]=None, errors_only: bool=False) -> List[ToolExecutionAudit]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mcp/security.py#L796-L827 - Implementation: Method `ToolSandbox.get_audit_log` calls `self._audit_log.copy`; returns `entries[-limit:]`. Get audit log entries. Args: limit: Maximum entries to return tool_filter: Filter by tool name (substring match) server_filter: Filter by server name errors_only: Only return failed executions Returns: List of audit entries - Inputs: - `limit` (int; optional; default `100`): Maximum entries to return - `tool_filter` (Optional[str]; optional; default `None`): Filter by tool name (substring match) - `server_filter` (Optional[str]; optional; default `None`): Filter by server name - `errors_only` (bool; optional; default `False`): Only return failed executions - Return annotation: `List[ToolExecutionAudit]` - Calls: self._audit_log.copy - State reads: self._audit_lock, self._audit_log.copy, self._audit_log - Return expressions: entries[-limit:] ## `vllm_mlx.mcp.security.ToolSandbox.clear_audit_log` - Kind: method - Signature: `def clear_audit_log(self) -> int` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mcp/security.py#L829-L834 - Implementation: Method `ToolSandbox.clear_audit_log` calls `len`, `self._audit_log.clear`; returns `count`. Clear audit log and return number of entries cleared. - Inputs: none - Return annotation: `int` - Calls: len, self._audit_log.clear - State reads: self._audit_lock, self._audit_log, self._audit_log.clear - Return expressions: count ## `vllm_mlx.mcp.security.get_sandbox` - Kind: function - Signature: `def get_sandbox() -> ToolSandbox` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mcp/security.py#L841-L846 - Implementation: Function `get_sandbox` calls `ToolSandbox`; returns `_sandbox`. Get the global tool sandbox instance. - Inputs: none - Return annotation: `ToolSandbox` - Calls: ToolSandbox - Return expressions: _sandbox ## `vllm_mlx.mcp.security.set_sandbox` - Kind: function - Signature: `def set_sandbox(sandbox: ToolSandbox) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mcp/security.py#L849-L852 - Implementation: Function `set_sandbox` contains no state mutation, call, raise, return, await, or yield. Set a custom global sandbox. - Inputs: - `sandbox` (ToolSandbox; required): Required positional or keyword input. - Return annotation: `None` # Module `vllm_mlx.mcp.tools` Tool schema conversion utilities for MCP <-> OpenAI formats. Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mcp/tools.py#L1-L174 ## `vllm_mlx.mcp.tools.mcp_tool_to_openai` - Kind: function - Signature: `def mcp_tool_to_openai(tool: MCPTool) -> Dict[str, Any]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mcp/tools.py#L12-L33 - Implementation: Function `mcp_tool_to_openai` returns `{'type': 'function', 'function': {'name': tool.full_name, 'description': tool.description, 'parameters': tool.input_sch…`. Convert MCP tool schema to OpenAI function calling format. Args: tool: MCPTool instance Returns: OpenAI-compatible tool definition - Inputs: - `tool` (MCPTool; required): MCPTool instance - Return annotation: `Dict[str, Any]` - Return expressions: {'type': 'function', 'function': {'name': tool.full_name, 'description': tool.description, 'parameters': tool.input_sch… ## `vllm_mlx.mcp.tools.mcp_tools_to_openai` - Kind: function - Signature: `def mcp_tools_to_openai(tools: List[MCPTool]) -> List[Dict[str, Any]]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mcp/tools.py#L36-L46 - Implementation: Function `mcp_tools_to_openai` calls `mcp_tool_to_openai`; returns `[mcp_tool_to_openai(tool) for tool in tools]`. Convert list of MCP tools to OpenAI format. Args: tools: List of MCPTool instances Returns: List of OpenAI-compatible tool definitions - Inputs: - `tools` (List[MCPTool]; required): List of MCPTool instances - Return annotation: `List[Dict[str, Any]]` - Calls: mcp_tool_to_openai - Return expressions: [mcp_tool_to_openai(tool) for tool in tools] ## `vllm_mlx.mcp.tools.openai_call_to_mcp` - Kind: function - Signature: `def openai_call_to_mcp(tool_call: Dict[str, Any]) -> Tuple[str, str, Dict[str, Any]]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mcp/tools.py#L49-L84 - Implementation: Function `openai_call_to_mcp` calls `tool_call.get`, `function.get`, `isinstance`, `json.loads`; returns `(server_name, tool_name, arguments)`. Parse OpenAI tool call back to MCP format. Args: tool_call: OpenAI tool call from model response Returns: Tuple of (server_name, tool_name, arguments) Raises: ValueError: If tool call format is invalid - Inputs: - `tool_call` (Dict[str, Any]; required): OpenAI tool call from model response - Return annotation: `Tuple[str, str, Dict[str, Any]]` - Calls: tool_call.get, function.get, isinstance, json.loads, full_name.split - Return expressions: (server_name, tool_name, arguments) ## `vllm_mlx.mcp.tools.format_tool_result` - Kind: function - Signature: `def format_tool_result(result: MCPToolResult, tool_call_id: str) -> Dict[str, Any]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mcp/tools.py#L87-L98 - Implementation: Function `format_tool_result` calls `result.to_message`; returns `result.to_message(tool_call_id)`. Format tool result for inclusion in conversation messages. Args: result: MCPToolResult from tool execution tool_call_id: ID of the tool call this is responding to Returns: OpenAI-compatible tool result message - Inputs: - `result` (MCPToolResult; required): MCPToolResult from tool execution - `tool_call_id` (str; required): ID of the tool call this is responding to - Return annotation: `Dict[str, Any]` - Calls: result.to_message - Return expressions: result.to_message(tool_call_id) ## `vllm_mlx.mcp.tools.format_tool_results` - Kind: function - Signature: `def format_tool_results(results: List[Tuple[MCPToolResult, str]]) -> List[Dict[str, Any]]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mcp/tools.py#L101-L113 - Implementation: Function `format_tool_results` calls `format_tool_result`; returns `[format_tool_result(result, call_id) for result, call_id in results]`. Format multiple tool results as messages. Args: results: List of (MCPToolResult, tool_call_id) tuples Returns: List of OpenAI-compatible tool result messages - Inputs: - `results` (List[Tuple[MCPToolResult, str]]; required): List of (MCPToolResult, tool_call_id) tuples - Return annotation: `List[Dict[str, Any]]` - Calls: format_tool_result - Return expressions: [format_tool_result(result, call_id) for result, call_id in results] ## `vllm_mlx.mcp.tools.merge_tools` - Kind: function - Signature: `def merge_tools(mcp_tools: List[MCPTool], user_tools: Optional[List[Dict[str, Any]]]=None) -> List[Dict[str, Any]]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mcp/tools.py#L116-L143 - Implementation: Function `merge_tools` calls `mcp_tool_to_openai`, `tool.get`, `func.get`, `list`; returns `list(all_tools.values())`. Merge MCP tools with user-provided tools. User tools take precedence if there are name conflicts. Args: mcp_tools: Tools discovered from MCP servers user_tools: User-provided tools in OpenAI format Returns: Combined list of tools in OpenAI format - Inputs: - `mcp_tools` (List[MCPTool]; required): Tools discovered from MCP servers - `user_tools` (Optional[List[Dict[str, Any]]]; optional; default `None`): User-provided tools in OpenAI format - Return annotation: `List[Dict[str, Any]]` - Calls: mcp_tool_to_openai, tool.get, func.get, list, all_tools.values - Return expressions: list(all_tools.values()) ## `vllm_mlx.mcp.tools.extract_tool_calls` - Kind: function - Signature: `def extract_tool_calls(response: Dict[str, Any]) -> List[Dict[str, Any]]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mcp/tools.py#L146-L161 - Implementation: Function `extract_tool_calls` calls `response.get`, `choices[0].get`, `message.get`; has 2 explicit return paths. Extract tool calls from model response. Args: response: OpenAI-format model response Returns: List of tool calls - Inputs: - `response` (Dict[str, Any]; required): OpenAI-format model response - Return annotation: `List[Dict[str, Any]]` - Calls: response.get, choices[0].get, message.get - Return expressions: []; message.get('tool_calls', []) ## `vllm_mlx.mcp.tools.has_tool_calls` - Kind: function - Signature: `def has_tool_calls(response: Dict[str, Any]) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mcp/tools.py#L164-L174 - Implementation: Function `has_tool_calls` calls `len`, `extract_tool_calls`; returns `len(extract_tool_calls(response)) > 0`. Check if response contains tool calls. Args: response: OpenAI-format model response Returns: True if response contains tool calls - Inputs: - `response` (Dict[str, Any]; required): OpenAI-format model response - Return annotation: `bool` - Calls: len, extract_tool_calls - Return expressions: len(extract_tool_calls(response)) > 0 # Module `vllm_mlx.mcp.types` Type definitions for MCP client support. Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mcp/types.py#L1-L179 ## `vllm_mlx.mcp.types.MCPTransport` - Kind: class - Signature: `class MCPTransport(str, Enum)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mcp/types.py#L11-L15 - Implementation: Class `MCPTransport` derives from `str`, `Enum` and declares 0 direct member(s). Supported MCP transport types. - Inputs: none - Constructs: `vllm_mlx.mcp.types.MCPTransport` ## `vllm_mlx.mcp.types.MCPServerState` - Kind: class - Signature: `class MCPServerState(str, Enum)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mcp/types.py#L18-L24 - Implementation: Class `MCPServerState` derives from `str`, `Enum` and declares 0 direct member(s). MCP server connection states. - Inputs: none - Constructs: `vllm_mlx.mcp.types.MCPServerState` ## `vllm_mlx.mcp.types.MCPServerConfig` - Kind: class - Signature: `class MCPServerConfig` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mcp/types.py#L28-L78 - Implementation: Class `MCPServerConfig` declares 2 direct member(s). Configuration for a single MCP server. - Inputs: - `name` (str; required): Required constructor field. - `transport` (MCPTransport; optional; default `MCPTransport.STDIO`): Optional constructor field; defaults to `MCPTransport.STDIO`. - `command` (Optional[str]; optional; default `None`): Optional constructor field; defaults to `None`. - `args` (Optional[List[str]]; optional; default `None`): Optional constructor field; defaults to `None`. - `env` (Optional[Dict[str, str]]; optional; default `None`): Optional constructor field; defaults to `None`. - `url` (Optional[str]; optional; default `None`): Optional constructor field; defaults to `None`. - `enabled` (bool; optional; default `True`): Optional constructor field; defaults to `True`. - `timeout` (float; optional; default `30.0`): Optional constructor field; defaults to `30.0`. - Constructs: `vllm_mlx.mcp.types.MCPServerConfig` - Decorators: dataclass ## `vllm_mlx.mcp.types.MCPServerConfig.__post_init__` - Kind: method - Signature: `def __post_init__(self)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mcp/types.py#L46-L63 - Implementation: Method `MCPServerConfig.__post_init__` updates `self.transport`; calls `isinstance`, `MCPTransport`, `ValueError`, `self._validate_security`; can raise `ValueError`. Validate configuration. - Inputs: none - Return annotation: `not annotated` - Calls: isinstance, MCPTransport, ValueError, self._validate_security - State reads: self.transport, self.command, self.name, self.url, self._validate_security - State writes: self.transport - Raises directly: ValueError ## `vllm_mlx.mcp.types.MCPServerConfig._validate_security` - Kind: method - Signature: `def _validate_security(self) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mcp/types.py#L65-L78 - Implementation: Method `MCPServerConfig._validate_security` calls `validate_mcp_server_config`, `ValueError`, `str`; can raise `ValueError`. Validate security of the configuration. - Inputs: none - Return annotation: `None` - Calls: validate_mcp_server_config, ValueError, str - State reads: self.name, self.command, self.args, self.env, self.url - Raises directly: ValueError ## `vllm_mlx.mcp.types.MCPConfig` - Kind: class - Signature: `class MCPConfig` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mcp/types.py#L82-L103 - Implementation: Class `MCPConfig` declares 1 direct member(s). Root configuration for MCP client. - Inputs: - `servers` (Dict[str, MCPServerConfig]; optional; default `field(default_factory=dict)`): Optional constructor field; defaults to `field(default_factory=dict)`. - `max_tool_calls` (int; optional; default `10`): Optional constructor field; defaults to `10`. - `default_timeout` (float; optional; default `30.0`): Optional constructor field; defaults to `30.0`. - `allowed_high_risk_tools` (Set[str]; optional; default `field(default_factory=set)`): Optional constructor field; defaults to `field(default_factory=set)`. - Constructs: `vllm_mlx.mcp.types.MCPConfig` - Decorators: dataclass ## `vllm_mlx.mcp.types.MCPConfig.from_dict` - Kind: method - Signature: `def from_dict(cls, data: Dict[str, Any]) -> 'MCPConfig'` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mcp/types.py#L91-L103 - Implementation: Method `MCPConfig.from_dict` calls `data.get('servers', {}).items`, `data.get`, `MCPServerConfig`, `cls`; returns `cls(servers=servers, max_tool_calls=data.get('max_tool_calls', 10), default_timeout=data.get('default_timeout', 30.0), …`. Create config from dictionary. - Inputs: - `data` (Dict[str, Any]; required): Required positional or keyword input. - Return annotation: `'MCPConfig'` - Decorators: classmethod - Calls: data.get('servers', {}).items, data.get, MCPServerConfig, cls, set - Return expressions: cls(servers=servers, max_tool_calls=data.get('max_tool_calls', 10), default_timeout=data.get('default_timeout', 30.0), … ## `vllm_mlx.mcp.types.MCPTool` - Kind: class - Signature: `class MCPTool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mcp/types.py#L107-L129 - Implementation: Class `MCPTool` declares 2 direct member(s). Normalized tool representation from MCP server. - Inputs: - `server_name` (str; required): Required constructor field. - `name` (str; required): Required constructor field. - `description` (str; required): Required constructor field. - `input_schema` (Dict[str, Any]; optional; default `field(default_factory=dict)`): Optional constructor field; defaults to `field(default_factory=dict)`. - Constructs: `vllm_mlx.mcp.types.MCPTool` - Decorators: dataclass ## `vllm_mlx.mcp.types.MCPTool.full_name` - Kind: method - Signature: `def full_name(self) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mcp/types.py#L116-L118 - Implementation: Method `MCPTool.full_name` returns `f'{self.server_name}__{self.name}'`. Get namespaced tool name (server__tool). - Inputs: none - Return annotation: `str` - Decorators: property - State reads: self.server_name, self.name - Return expressions: f'{self.server_name}__{self.name}' ## `vllm_mlx.mcp.types.MCPTool.to_openai_format` - Kind: method - Signature: `def to_openai_format(self) -> Dict[str, Any]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mcp/types.py#L120-L129 - Implementation: Method `MCPTool.to_openai_format` returns `{'type': 'function', 'function': {'name': self.full_name, 'description': self.description, 'parameters': self.input_sch…`. Convert to OpenAI function calling format. - Inputs: none - Return annotation: `Dict[str, Any]` - State reads: self.full_name, self.description, self.input_schema - Return expressions: {'type': 'function', 'function': {'name': self.full_name, 'description': self.description, 'parameters': self.input_sch… ## `vllm_mlx.mcp.types.MCPToolResult` - Kind: class - Signature: `class MCPToolResult` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mcp/types.py#L133-L156 - Implementation: Class `MCPToolResult` declares 1 direct member(s). Result from a tool execution. - Inputs: - `tool_name` (str; required): Required constructor field. - `content` (Any; required): Required constructor field. - `is_error` (bool; optional; default `False`): Optional constructor field; defaults to `False`. - `error_message` (Optional[str]; optional; default `None`): Optional constructor field; defaults to `None`. - Constructs: `vllm_mlx.mcp.types.MCPToolResult` - Decorators: dataclass ## `vllm_mlx.mcp.types.MCPToolResult.to_message` - Kind: method - Signature: `def to_message(self, tool_call_id: str) -> Dict[str, Any]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mcp/types.py#L141-L156 - Implementation: Method `MCPToolResult.to_message` calls `isinstance`, `json.dumps`; returns `{'role': 'tool', 'tool_call_id': tool_call_id, 'content': content}`. Convert to OpenAI tool result message format. - Inputs: - `tool_call_id` (str; required): Required positional or keyword input. - Return annotation: `Dict[str, Any]` - Calls: isinstance, json.dumps - State reads: self.is_error, self.error_message, self.content - Return expressions: {'role': 'tool', 'tool_call_id': tool_call_id, 'content': content} ## `vllm_mlx.mcp.types.MCPServerStatus` - Kind: class - Signature: `class MCPServerStatus` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mcp/types.py#L160-L179 - Implementation: Class `MCPServerStatus` declares 1 direct member(s). Status of an MCP server connection. - Inputs: - `name` (str; required): Required constructor field. - `state` (MCPServerState; required): Required constructor field. - `transport` (MCPTransport; required): Required constructor field. - `tools_count` (int; optional; default `0`): Optional constructor field; defaults to `0`. - `error` (Optional[str]; optional; default `None`): Optional constructor field; defaults to `None`. - `last_connected` (Optional[float]; optional; default `None`): Optional constructor field; defaults to `None`. - Constructs: `vllm_mlx.mcp.types.MCPServerStatus` - Decorators: dataclass ## `vllm_mlx.mcp.types.MCPServerStatus.to_dict` - Kind: method - Signature: `def to_dict(self) -> Dict[str, Any]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mcp/types.py#L170-L179 - Implementation: Method `MCPServerStatus.to_dict` returns `{'name': self.name, 'state': self.state.value, 'transport': self.transport.value, 'tools_count': self.tools_count, 'err…`. Convert to dictionary for API response. - Inputs: none - Return annotation: `Dict[str, Any]` - State reads: self.name, self.state.value, self.state, self.transport.value, self.transport, self.tools_count, self.error, self.last_connected - Return expressions: {'name': self.name, 'state': self.state.value, 'transport': self.transport.value, 'tools_count': self.tools_count, 'err… # Module `vllm_mlx.memory_cache` Memory-aware prefix cache for vllm-mlx. This module provides a prefix cache implementation that tracks memory usage and evicts entries based on memory pressure rather than entry count. Key features: - Automatic memory limit detection based on available system RAM - Accurate memory tracking for MLX array caches - LRU eviction triggered by memory thresholds - No unnecessary deep copies (MLX arrays are immutable) Example: config = MemoryCacheConfig(max_memory_percent=0.25) cache = MemoryAwarePrefixCache(model, config) # Fetch returns reference (no copy) - safe because MLX arrays are immutable kv_cache, remaining = cache.fetch(tokens) # Store tracks memory automatically cache.store(tokens, kv_cache) Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/memory_cache.py#L1-L1463 ## `vllm_mlx.memory_cache._get_available_memory` - Kind: function - Signature: `def _get_available_memory() -> int` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/memory_cache.py#L47-L63 - Implementation: Function `_get_available_memory` calls `psutil.virtual_memory`, `logger.warning`; has 2 explicit return paths. Get available system memory in bytes. Returns: Available memory in bytes, or 0 if detection fails. - Inputs: none - Return annotation: `int` - Calls: psutil.virtual_memory, logger.warning - Return expressions: psutil.virtual_memory().available; 0 ## `vllm_mlx.memory_cache._array_memory` - Kind: function - Signature: `def _array_memory(arr) -> int` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/memory_cache.py#L66-L88 - Implementation: Function `_array_memory` calls `hasattr`, `math.prod`; has 3 explicit return paths. Estimate array memory from shape+dtype without triggering lazy eval. Accessing .nbytes on a lazy MLX array forces evaluation of the entire computation graph, causing a VRAM spike. This function uses shape and dtype metadata (which are always available without eval) to compute the same value. Args: arr: An MLX array or similar object. Returns: Estimated memory in bytes. - Inputs: - `arr` (not annotated; required): An MLX array or similar object. - Return annotation: `int` - Calls: hasattr, math.prod - Return expressions: math.prod(arr.shape) * dtype.size; arr.nbytes; 0 ## `vllm_mlx.memory_cache._nested_array_memory` - Kind: function - Signature: `def _nested_array_memory(value: Any) -> int` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/memory_cache.py#L91-L105 - Implementation: Function `_nested_array_memory` calls `isinstance`, `sum`, `_nested_array_memory`, `_array_memory`; has 3 explicit return paths. Sum ``_array_memory`` over an arbitrarily nested state structure. Cache ``state`` payloads are not always a flat ``(keys, values)`` pair: CacheList yields a list of sub-cache states and PoolingCache yields ``(buf_kv, buf_gate, pooled)`` with possible ``None`` members. Unpacking those as two values raised, was swallowed, and the entry was accounted as zero bytes — so the dashboard showed 0% cache memory and, far worse, the byte-based LRU eviction never fired for such models. - Inputs: - `value` (Any; required): Required positional or keyword input. - Return annotation: `int` - Calls: isinstance, sum, _nested_array_memory, _array_memory - Return expressions: 0; sum((_nested_array_memory(v) for v in value)); _array_memory(value) ## `vllm_mlx.memory_cache.estimate_kv_cache_memory` - Kind: function - Signature: `def estimate_kv_cache_memory(cache: list[Any]) -> int` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/memory_cache.py#L108-L162 - Implementation: Function `estimate_kv_cache_memory` calls `isinstance`, `_array_memory`, `hasattr`, `getattr`; has 2 explicit return paths. Estimate memory usage of a KV cache in bytes. This function inspects MLX arrays in the cache and calculates their total memory footprint using shape+dtype metadata to avoid triggering lazy evaluation (which would cause a VRAM spike). Args: cache: List of layer cache objects, each containing keys/values tensors. Returns: Estimated memory usage in bytes. - Inputs: - `cache` (list[Any]; required): List of layer cache objects, each containing keys/values tensors. - Return annotation: `int` - Calls: isinstance, _array_memory, hasattr, getattr, _nested_array_memory, callable - Return expressions: 0; total_bytes ## `vllm_mlx.memory_cache.MemoryCacheConfig` - Kind: class - Signature: `class MemoryCacheConfig` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/memory_cache.py#L166-L225 - Implementation: Class `MemoryCacheConfig` declares 2 direct member(s). Configuration for memory-aware prefix cache. Attributes: max_memory_mb: Maximum memory in MB. If None, auto-detects. max_memory_percent: Fraction of available RAM to use (0.0-1.0). max_entries: Hard limit on number of entries (safety net). enable_memory_tracking: Whether to track per-entry memory. kv_quantize: Whether to quantize KV cache layers for reduced memory. kv_bits: Number of bits for KV cache quantization. kv_group_size: Group size for KV cache quantization. kv_min_quantize_tokens: Minimum sequence length for quantization to apply. min_prefix_tokens: Minimum cached prefix length eligible for reuse. - Inputs: - `max_memory_mb` (int | None; optional; default `None`): Optional constructor field; defaults to `None`. - `max_memory_percent` (float; optional; default `_DEFAULT_MEMORY_PERCENT`): Optional constructor field; defaults to `_DEFAULT_MEMORY_PERCENT`. - `max_entries` (int; optional; default `1000`): Optional constructor field; defaults to `1000`. - `enable_memory_tracking` (bool; optional; default `True`): Optional constructor field; defaults to `True`. - `kv_quantize` (bool; optional; default `False`): Optional constructor field; defaults to `False`. - `kv_bits` (int; optional; default `8`): Optional constructor field; defaults to `8`. - `kv_group_size` (int; optional; default `64`): Optional constructor field; defaults to `64`. - `kv_min_quantize_tokens` (int; optional; default `256`): Optional constructor field; defaults to `256`. - `min_prefix_tokens` (int; optional; default `128`): Optional constructor field; defaults to `128`. - Constructs: `vllm_mlx.memory_cache.MemoryCacheConfig` - Decorators: dataclass(frozen=True) ## `vllm_mlx.memory_cache.MemoryCacheConfig.__post_init__` - Kind: method - Signature: `def __post_init__(self) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/memory_cache.py#L192-L206 - Implementation: Method `MemoryCacheConfig.__post_init__` calls `ValueError`; can raise `ValueError`. Method `MemoryCacheConfig.__post_init__` calls `ValueError`; can raise `ValueError`. - Inputs: none - Return annotation: `None` - Calls: ValueError - State reads: self.max_memory_percent, self.max_entries, self.kv_min_quantize_tokens, self.min_prefix_tokens - Raises directly: ValueError ## `vllm_mlx.memory_cache.MemoryCacheConfig.compute_memory_limit` - Kind: method - Signature: `def compute_memory_limit(self) -> int` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/memory_cache.py#L208-L225 - Implementation: Method `MemoryCacheConfig.compute_memory_limit` calls `_get_available_memory`, `int`, `max`; has 3 explicit return paths. Compute the memory limit in bytes. Returns: Memory limit in bytes. - Inputs: none - Return annotation: `int` - Calls: _get_available_memory, int, max - State reads: self.max_memory_mb, self.max_memory_percent - Return expressions: self.max_memory_mb * _BYTES_PER_MB; max(limit, _MIN_MEMORY_BYTES); int(fallback_total * self.max_memory_percent) ## `vllm_mlx.memory_cache.CacheStats` - Kind: class - Signature: `class CacheStats` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/memory_cache.py#L229-L268 - Implementation: Class `CacheStats` declares 3 direct member(s). Statistics for cache performance monitoring. - Inputs: - `hits` (int; optional; default `0`): Optional constructor field; defaults to `0`. - `misses` (int; optional; default `0`): Optional constructor field; defaults to `0`. - `evictions` (int; optional; default `0`): Optional constructor field; defaults to `0`. - `tokens_saved` (int; optional; default `0`): Optional constructor field; defaults to `0`. - `current_memory_bytes` (int; optional; default `0`): Optional constructor field; defaults to `0`. - `max_memory_bytes` (int; optional; default `0`): Optional constructor field; defaults to `0`. - `entry_count` (int; optional; default `0`): Optional constructor field; defaults to `0`. - Constructs: `vllm_mlx.memory_cache.CacheStats` - Decorators: dataclass ## `vllm_mlx.memory_cache.CacheStats.hit_rate` - Kind: method - Signature: `def hit_rate(self) -> float` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/memory_cache.py#L241-L245 - Implementation: Method `CacheStats.hit_rate` returns `self.hits / total if total > 0 else 0.0`. Return successful lookups divided by all completed lookups. - Inputs: none - Return annotation: `float` - Decorators: property - State reads: self.hits, self.misses - Return expressions: self.hits / total if total > 0 else 0.0 ## `vllm_mlx.memory_cache.CacheStats.memory_utilization` - Kind: method - Signature: `def memory_utilization(self) -> float` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/memory_cache.py#L248-L253 - Implementation: Method `CacheStats.memory_utilization` has 2 explicit return paths. Return the fraction of the configured memory budget in use. - Inputs: none - Return annotation: `float` - Decorators: property - State reads: self.max_memory_bytes, self.current_memory_bytes - Return expressions: 0.0; self.current_memory_bytes / self.max_memory_bytes ## `vllm_mlx.memory_cache.CacheStats.to_dict` - Kind: method - Signature: `def to_dict(self) -> dict[str, Any]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/memory_cache.py#L255-L268 - Implementation: Method `CacheStats.to_dict` calls `round`; returns `{'hits': self.hits, 'misses': self.misses, 'hit_rate': round(self.hit_rate, 4), 'evictions': self.evictions, 'tokens_sa…`. Return rounded cache counters and memory values for APIs and logs. - Inputs: none - Return annotation: `dict[str, Any]` - Calls: round - State reads: self.hits, self.misses, self.hit_rate, self.evictions, self.tokens_saved, self.current_memory_bytes, self.max_memory_bytes, self.memory_utilization, self.entry_count - Return expressions: {'hits': self.hits, 'misses': self.misses, 'hit_rate': round(self.hit_rate, 4), 'evictions': self.evictions, 'tokens_sa… ## `vllm_mlx.memory_cache._CacheEntry` - Kind: class - Signature: `class _CacheEntry` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/memory_cache.py#L272-L287 - Implementation: Class `_CacheEntry` declares 1 direct member(s). Internal cache entry with memory tracking. - Inputs: - `tokens` (tuple[int, ...]; required): Required constructor field. - `cache` (list[Any]; required): Required constructor field. - `memory_bytes` (int; required): Required constructor field. - Constructs: `vllm_mlx.memory_cache._CacheEntry` - Decorators: dataclass ## `vllm_mlx.memory_cache._CacheEntry.create` - Kind: method - Signature: `def create(cls, tokens: list[int], cache: list[Any]) -> _CacheEntry` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/memory_cache.py#L280-L287 - Implementation: Method `_CacheEntry.create` calls `estimate_kv_cache_memory`, `cls`, `tuple`; returns `cls(tokens=tuple(tokens), cache=cache, memory_bytes=memory)`. Create a cache entry with memory estimation. - Inputs: - `tokens` (list[int]; required): Required positional or keyword input. - `cache` (list[Any]; required): Required positional or keyword input. - Return annotation: `_CacheEntry` - Decorators: classmethod - Calls: estimate_kv_cache_memory, cls, tuple - Return expressions: cls(tokens=tuple(tokens), cache=cache, memory_bytes=memory) ## `vllm_mlx.memory_cache._is_cache_layer_trimmable` - Kind: function - Signature: `def _is_cache_layer_trimmable(layer_cache: Any) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/memory_cache.py#L290-L314 - Implementation: Function `_is_cache_layer_trimmable` calls `isinstance`, `hasattr`, `getattr`, `callable`; has 3 explicit return paths. Return whether a cache layer can safely be rewound for partial reuse. - Inputs: - `layer_cache` (Any; required): Required positional or keyword input. - Return annotation: `bool` - Calls: isinstance, hasattr, getattr, callable, bool, is_trimmable, logger.debug, type - Return expressions: False; hasattr(layer_cache, 'offset') and hasattr(layer_cache, 'keys'); bool(is_trimmable()) ## `vllm_mlx.memory_cache._trim_cache_offset` - Kind: function - Signature: `def _trim_cache_offset(cache: list[Any], trim_by: int) -> list[Any]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/memory_cache.py#L317-L481 - Implementation: Function `_trim_cache_offset` calls `isinstance`, `_QuantizedCacheWrapper.__new__`, `max`, `trimmed.append`; returns `trimmed`. Create copies of cache layers with the last ``trim_by`` positions removed. This is used when returning a cached KV state to the scheduler so that the last N positions are "freed" and the model will recompute them on the next forward pass (preventing duplicate KV entries). For plain KVCache: reduces offset (surplus data beyond offset is harmless since merge slices to ``keys[:, :, :offset, :]``). For RotatingKVCache: actually trims the circular buffer — reducing offset alone breaks ``size()`` / ``_temporal_order`` invariants. Supports KVCache, RotatingKVCache, and _QuantizedCacheWrapper. - Inputs: - `cache` (list[Any]; required): Required positional or keyword input. - `trim_by` (int; required): Required positional or keyword input. - Return annotation: `list[Any]` - Calls: isinstance, _QuantizedCacheWrapper.__new__, max, trimmed.append, min, type, orig_cls.__new__, getattr, layer_cache._temporal_order, mx.zeros, mx.concatenate, eval_targets.extend, hasattr, len, setattr, mx.eval - Return expressions: trimmed ## `vllm_mlx.memory_cache._needs_kv_trim` - Kind: function - Signature: `def _needs_kv_trim(layer: Any) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/memory_cache.py#L484-L495 - Implementation: Function `_needs_kv_trim` calls `getattr`, `isinstance`, `len`; has 2 explicit return paths. Check if a cache layer has oversized KV arrays (duck-typed, no MLX import). - Inputs: - `layer` (Any; required): Required positional or keyword input. - Return annotation: `bool` - Calls: getattr, isinstance, len - Return expressions: False; 0 < offset < shape[2] ## `vllm_mlx.memory_cache._trim_to_offset` - Kind: function - Signature: `def _trim_to_offset(cache: list[Any]) -> list[Any]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/memory_cache.py#L498-L538 - Implementation: Function `_trim_to_offset` calls `any`, `_needs_kv_trim`, `isinstance`, `trimmed.append`; has 2 explicit return paths. Trim KV arrays to their actual used size (offset) before storage. KV arrays are often pre-allocated larger than needed (e.g. 4096 slots when only 100 are used). This slices them down to ``offset`` and evaluates the result so the original large buffer can be freed. Args: cache: List of cache layer objects (KVCache or other types). Returns: New list with KVCache layers trimmed to their offset. Non-KVCache layers are passed through unchanged. - Inputs: - `cache` (list[Any]; required): List of cache layer objects (KVCache or other types). - Return annotation: `list[Any]` - Calls: any, _needs_kv_trim, isinstance, trimmed.append, KVCache, eval_targets.extend, mx.eval - Return expressions: cache; trimmed ## `vllm_mlx.memory_cache._QuantizedCacheWrapper` - Kind: class - Signature: `class _QuantizedCacheWrapper` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/memory_cache.py#L541-L571 - Implementation: Class `_QuantizedCacheWrapper` declares 1 direct member(s). Lightweight wrapper storing quantized KV arrays + original cache metadata. Unlike ``QuantizedKVCache``, this preserves enough info to reconstruct the *original* cache type (KVCache, RotatingKVCache, etc.) on dequantize. - Inputs: - `layer` (Any; required): Required positional or keyword input. - `bits` (int; required): Required positional or keyword input. - `group_size` (int; required): Required positional or keyword input. - Constructs: `vllm_mlx.memory_cache._QuantizedCacheWrapper` ## `vllm_mlx.memory_cache._QuantizedCacheWrapper.__init__` - Kind: method - Signature: `def __init__(self, layer: Any, bits: int, group_size: int)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/memory_cache.py#L558-L571 - Implementation: Method `_QuantizedCacheWrapper.__init__` updates `self.keys`, `self.values`, `self.offset`, `self.bits`; calls `mx.quantize`, `type`, `hasattr`, `getattr`. Method `_QuantizedCacheWrapper.__init__` updates `self.keys`, `self.values`, `self.offset`, `self.bits`; calls `mx.quantize`, `type`, `hasattr`, `getattr`. - Inputs: - `layer` (Any; required): Required positional or keyword input. - `bits` (int; required): Required positional or keyword input. - `group_size` (int; required): Required positional or keyword input. - Return annotation: `not annotated` - Calls: mx.quantize, type, hasattr, getattr - State reads: self.orig_attrs - State writes: self.keys, self.values, self.offset, self.bits, self.group_size, self.orig_type, self.orig_attrs ## `vllm_mlx.memory_cache._quantize_cache` - Kind: function - Signature: `def _quantize_cache(cache: list[Any], bits: int=8, group_size: int=64) -> list[Any]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/memory_cache.py#L574-L590 - Implementation: Function `_quantize_cache` calls `type`, `getattr`, `quantized.append`, `_QuantizedCacheWrapper`; returns `quantized`. Quantize KV cache layers to reduce memory. Only plain KVCache layers are quantized. RotatingKVCache (sliding window) is left as-is because its internal _idx/rotation state is tightly coupled with update_and_fetch logic and cannot survive quantize/dequantize roundtrip. RotatingKVCache is typically small (max_size=1024) so skipping it is fine. - Inputs: - `cache` (list[Any]; required): Required positional or keyword input. - `bits` (int; optional; default `8`): Optional positional or keyword input; defaults to `8`. - `group_size` (int; optional; default `64`): Optional positional or keyword input; defaults to `64`. - Return annotation: `list[Any]` - Calls: type, getattr, quantized.append, _QuantizedCacheWrapper - Return expressions: quantized ## `vllm_mlx.memory_cache._dequantize_cache` - Kind: function - Signature: `def _dequantize_cache(cache: list[Any]) -> list[Any]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/memory_cache.py#L593-L645 - Implementation: Function `_dequantize_cache` calls `isinstance`, `orig_cls.__new__`, `mx.dequantize`, `hasattr`; returns `result`. Dequantize _QuantizedCacheWrapper layers and copy non-quantized layers. All layers are copied (never returned by reference) so that the model's ``update_and_fetch`` mutations don't corrupt the stored cache entry. - Inputs: - `cache` (list[Any]; required): Required positional or keyword input. - Return annotation: `list[Any]` - Calls: isinstance, orig_cls.__new__, mx.dequantize, hasattr, len, layer.orig_attrs.items, setattr, result.append, type, mx.array, getattr - Return expressions: result ## `vllm_mlx.memory_cache._compute_model_fingerprint` - Kind: function - Signature: `def _compute_model_fingerprint(model: Any) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/memory_cache.py#L648-L682 - Implementation: Function `_compute_model_fingerprint` calls `getattr`, `parts.append`, `hashlib.sha256('|'.join(parts).encode()).hexdigest`, `hashlib.sha256`; returns `fingerprint`. Compute a fingerprint from model architecture for cache compatibility. Used to reject disk-persisted caches created by a different model or a different quantisation of the same model. The fingerprint is a short hex digest of (num_layers, hidden_size, vocab_size, num_kv_heads, head_dim) — lightweight and deterministic. - Inputs: - `model` (Any; required): Required positional or keyword input. - Return annotation: `str` - Calls: getattr, parts.append, hashlib.sha256('|'.join(parts).encode()).hexdigest, hashlib.sha256, '|'.join(parts).encode, '|'.join, logger.debug, ', '.join - Return expressions: fingerprint ## `vllm_mlx.memory_cache.MemoryAwarePrefixCache` - Kind: class - Signature: `class MemoryAwarePrefixCache` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/memory_cache.py#L685-L1463 - Implementation: Class `MemoryAwarePrefixCache` declares 19 direct member(s). Prefix cache with memory-based eviction. This cache tracks memory usage per entry and evicts based on memory pressure rather than entry count. It uses LRU (Least Recently Used) ordering for eviction decisions. Key design decisions: - No deep copies on fetch: MLX arrays are immutable, so sharing is safe - Memory tracking per entry: Accurate accounting for eviction - Auto-detection of available RAM: Adapts to different systems - OrderedDict for O(1) LRU operations Thread Safety: This class is NOT thread-safe. Use external locking if needed. - Inputs: - `model` (Any; required): The MLX model (used for identification). - `config` (MemoryCacheConfig | None; optional; default `None`): Cache configuration. Uses defaults if None. - Constructs: `vllm_mlx.memory_cache.MemoryAwarePrefixCache` ## `vllm_mlx.memory_cache.MemoryAwarePrefixCache.__init__` - Kind: method - Signature: `def __init__(self, model: Any, config: MemoryCacheConfig | None=None) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/memory_cache.py#L703-L746 - Implementation: Method `MemoryAwarePrefixCache.__init__` updates `self._model_id`, `self._config`, `self._model_fingerprint`, `self._entries`; calls `id`, `MemoryCacheConfig`, `_compute_model_fingerprint`, `OrderedDict`. Initialize the memory-aware prefix cache. Args: model: The MLX model (used for identification). config: Cache configuration. Uses defaults if None. - Inputs: - `model` (Any; required): The MLX model (used for identification). - `config` (MemoryCacheConfig | None; optional; default `None`): Cache configuration. Uses defaults if None. - Return annotation: `None` - Calls: id, MemoryCacheConfig, _compute_model_fingerprint, OrderedDict, self._config.compute_memory_limit, threading.RLock, CacheStats, logger.info - State reads: self._config.compute_memory_limit, self._config, self._max_memory, self._config.max_entries - State writes: self._model_id, self._config, self._model_fingerprint, self._entries, self._sorted_keys, self._max_memory, self._current_memory, self._memory_lock, self._stats, self._last_match_type, self._ssd_tier ## `vllm_mlx.memory_cache.MemoryAwarePrefixCache.fetch` - Kind: method - Signature: `def fetch(self, tokens: list[int]) -> tuple[list[Any] | None, list[int]]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/memory_cache.py#L748-L977 - Implementation: Method `MemoryAwarePrefixCache.fetch` updates `self._stats.misses`, `self._last_match_type`, `self._stats.hits`, `self._stats.tokens_saved`; calls `len`, `tuple`, `self._entries.move_to_end`, `_dequantize_cache`; has 5 explicit return paths. Find cached KV state for the given tokens. This method searches for exact matches, prefix matches, supersequence matches, and longest-common-prefix (LCP) matches. Uses a sorted key index for O(log N) lookup instead of scanning all entries. Returns the cached KV state directly (no copy) since MLX arrays are immutable and safe to share. Args: tokens: Input token sequence. Returns: Tuple of (cache, remaining_tokens): - cache: Cached KV state if found, None otherwise - remaining_tokens: Tokens that still need processing - Inputs: - `tokens` (list[int]; required): Input token sequence. - Return annotation: `tuple[list[Any] | None, list[int]]` - Calls: len, tuple, self._entries.move_to_end, _dequantize_cache, bisect.bisect_left, range, any, _is_cache_layer_trimmable, logger.debug, _trim_cache_offset, min, type - State reads: self._stats, self._config.min_prefix_tokens, self._config, self._entries, self._entries.move_to_end, self._config.kv_quantize, self._sorted_keys - State writes: self._stats.misses, self._last_match_type, self._stats.hits, self._stats.tokens_saved - Return expressions: (None, tokens); (cache_out, []); (trimmed_cache, []); (cache_out, remaining); (trimmed_cache, remaining) ## `vllm_mlx.memory_cache.MemoryAwarePrefixCache.store` - Kind: method - Signature: `def store(self, tokens: list[int], cache: list[Any], evict_prefixes: bool=True) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/memory_cache.py#L979-L1092 - Implementation: Method `MemoryAwarePrefixCache.store` updates `self._current_memory`, `self._stats.evictions`, `self._stats.entry_count`, `self._stats.current_memory_bytes`; calls `len`, `logger.debug`, `tuple`, `self._entries.move_to_end`; has 2 explicit return paths. Store KV cache for future reuse. This method stores the cache reference directly (no copy) and tracks memory usage. If memory limit is exceeded, LRU entries are evicted until there's room. Args: tokens: Token sequence that was processed. cache: The computed KV cache to store. evict_prefixes: If True, evict existing entries whose token sequence is a strict prefix of ``tokens``. Set to False when storing prompt+output entries to preserve prompt-only entries created by prompt_cache_save (those are the entries that future requests will actually match). Returns: True if stored successfully, False if rejected. - Inputs: - `tokens` (list[int]; required): Token sequence that was processed. - `cache` (list[Any]; required): The computed KV cache to store. - `evict_prefixes` (bool; optional; default `True`): If True, evict existing entries whose token sequence is a strict prefix of ``tokens``. Set to False when storing prompt+output entries to preserve prompt-only entries created by prompt_cache_save (those are the entries that future requests will actually match). - Return annotation: `bool` - Calls: len, logger.debug, tuple, self._entries.move_to_end, _trim_to_offset, _quantize_cache, _CacheEntry.create, logger.warning, bisect.bisect_left, range, to_remove.append, self._entries.pop, self._remove_from_sorted, self._evict_lru, bisect.insort - State reads: self._config.min_prefix_tokens, self._config, self._memory_lock, self._entries, self._entries.move_to_end, self._config.kv_quantize, self._config.kv_min_quantize_tokens, self._config.kv_bits, self._config.kv_group_size, self._max_memory, self._sorted_keys, self._entries.pop, self._stats, self._remove_from_sorted, self._current_memory, self._config.max_entries, self._evict_lru - State writes: self._current_memory, self._stats.evictions, self._stats.entry_count, self._stats.current_memory_bytes - Return expressions: False; True ## `vllm_mlx.memory_cache.MemoryAwarePrefixCache._remove_from_sorted` - Kind: method - Signature: `def _remove_from_sorted(self, key: tuple[int, ...]) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/memory_cache.py#L1094-L1098 - Implementation: Method `MemoryAwarePrefixCache._remove_from_sorted` calls `bisect.bisect_left`, `len`, `self._sorted_keys.pop`. Remove a key from the sorted index using bisect for O(log N). - Inputs: - `key` (tuple[int, ...]; required): Required positional or keyword input. - Return annotation: `None` - Calls: bisect.bisect_left, len, self._sorted_keys.pop - State reads: self._sorted_keys, self._sorted_keys.pop ## `vllm_mlx.memory_cache.MemoryAwarePrefixCache._evict_lru` - Kind: method - Signature: `def _evict_lru(self) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/memory_cache.py#L1100-L1126 - Implementation: Method `MemoryAwarePrefixCache._evict_lru` updates `self._current_memory`, `self._stats.evictions`, `self._stats.entry_count`, `self._stats.current_memory_bytes`; calls `self._entries.popitem`, `self._remove_from_sorted`, `len`, `self._ssd_tier.enqueue_spill`; returns `None`. Evict the least recently used entry. If an SSD tier is attached, the entry is spilled to disk instead of being discarded. - Inputs: none - Return annotation: `None` - Calls: self._entries.popitem, self._remove_from_sorted, len, self._ssd_tier.enqueue_spill, logger.debug - State reads: self._memory_lock, self._entries, self._entries.popitem, self._remove_from_sorted, self._stats, self._current_memory, self._ssd_tier, self._ssd_tier.enqueue_spill - State writes: self._current_memory, self._stats.evictions, self._stats.entry_count, self._stats.current_memory_bytes - Return expressions: None ## `vllm_mlx.memory_cache.MemoryAwarePrefixCache.remove` - Kind: method - Signature: `def remove(self, tokens: list[int]) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/memory_cache.py#L1128-L1147 - Implementation: Method `MemoryAwarePrefixCache.remove` updates `self._current_memory`, `self._stats.entry_count`, `self._stats.current_memory_bytes`; calls `tuple`, `self._entries.pop`, `self._remove_from_sorted`, `len`; has 2 explicit return paths. Remove a specific cache entry. Args: tokens: Token sequence to remove. Returns: True if entry was found and removed. - Inputs: - `tokens` (list[int]; required): Token sequence to remove. - Return annotation: `bool` - Calls: tuple, self._entries.pop, self._remove_from_sorted, len - State reads: self._memory_lock, self._entries.pop, self._entries, self._remove_from_sorted, self._stats, self._current_memory - State writes: self._current_memory, self._stats.entry_count, self._stats.current_memory_bytes - Return expressions: True; False ## `vllm_mlx.memory_cache.MemoryAwarePrefixCache.clear` - Kind: method - Signature: `def clear(self) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/memory_cache.py#L1149-L1156 - Implementation: Method `MemoryAwarePrefixCache.clear` updates `self._current_memory`, `self._stats`; calls `self._entries.clear`, `self._sorted_keys.clear`, `CacheStats`, `logger.debug`. Clear all cached entries. - Inputs: none - Return annotation: `None` - Calls: self._entries.clear, self._sorted_keys.clear, CacheStats, logger.debug - State reads: self._memory_lock, self._entries.clear, self._entries, self._sorted_keys.clear, self._sorted_keys, self._max_memory - State writes: self._current_memory, self._stats ## `vllm_mlx.memory_cache.MemoryAwarePrefixCache.get_stats` - Kind: method - Signature: `def get_stats(self) -> dict[str, Any]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/memory_cache.py#L1158-L1160 - Implementation: Method `MemoryAwarePrefixCache.get_stats` calls `self._stats.to_dict`; returns `self._stats.to_dict()`. Get cache statistics. - Inputs: none - Return annotation: `dict[str, Any]` - Calls: self._stats.to_dict - State reads: self._stats.to_dict, self._stats - Return expressions: self._stats.to_dict() ## `vllm_mlx.memory_cache.MemoryAwarePrefixCache.reset_stats` - Kind: method - Signature: `def reset_stats(self) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/memory_cache.py#L1162-L1169 - Implementation: Method `MemoryAwarePrefixCache.reset_stats` updates `self._stats`; calls `CacheStats`, `len`. Reset statistics while preserving cache contents. - Inputs: none - Return annotation: `None` - Calls: CacheStats, len - State reads: self._memory_lock, self._max_memory, self._current_memory, self._entries - State writes: self._stats ## `vllm_mlx.memory_cache.MemoryAwarePrefixCache.memory_usage_mb` - Kind: method - Signature: `def memory_usage_mb(self) -> float` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/memory_cache.py#L1172-L1174 - Implementation: Method `MemoryAwarePrefixCache.memory_usage_mb` returns `self._current_memory / _BYTES_PER_MB`. Current memory usage in MB. - Inputs: none - Return annotation: `float` - Decorators: property - State reads: self._current_memory - Return expressions: self._current_memory / _BYTES_PER_MB ## `vllm_mlx.memory_cache.MemoryAwarePrefixCache.memory_limit_mb` - Kind: method - Signature: `def memory_limit_mb(self) -> float` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/memory_cache.py#L1177-L1179 - Implementation: Method `MemoryAwarePrefixCache.memory_limit_mb` returns `self._max_memory / _BYTES_PER_MB`. Memory limit in MB. - Inputs: none - Return annotation: `float` - Decorators: property - State reads: self._max_memory - Return expressions: self._max_memory / _BYTES_PER_MB ## `vllm_mlx.memory_cache.MemoryAwarePrefixCache.try_reserve_memory` - Kind: method - Signature: `def try_reserve_memory(self, nbytes: int) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/memory_cache.py#L1181-L1188 - Implementation: Method `MemoryAwarePrefixCache.try_reserve_memory` updates `self._current_memory`, `self._stats.current_memory_bytes`; has 2 explicit return paths. Tentatively reserve cache memory for an upcoming promotion. - Inputs: - `nbytes` (int; required): Required positional or keyword input. - Return annotation: `bool` - State reads: self._memory_lock, self._current_memory, self._max_memory, self._stats - State writes: self._current_memory, self._stats.current_memory_bytes - Return expressions: False; True ## `vllm_mlx.memory_cache.MemoryAwarePrefixCache.release_reserved_memory` - Kind: method - Signature: `def release_reserved_memory(self, nbytes: int) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/memory_cache.py#L1190-L1194 - Implementation: Method `MemoryAwarePrefixCache.release_reserved_memory` updates `self._current_memory`, `self._stats.current_memory_bytes`; calls `max`. Release memory previously reserved by try_reserve_memory(). - Inputs: - `nbytes` (int; required): Required positional or keyword input. - Return annotation: `None` - Calls: max - State reads: self._memory_lock, self._current_memory, self._stats - State writes: self._current_memory, self._stats.current_memory_bytes ## `vllm_mlx.memory_cache.MemoryAwarePrefixCache.__len__` - Kind: method - Signature: `def __len__(self) -> int` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/memory_cache.py#L1196-L1198 - Implementation: Method `MemoryAwarePrefixCache.__len__` calls `len`; returns `len(self._entries)`. Return number of cached entries. - Inputs: none - Return annotation: `int` - Calls: len - State reads: self._entries - Return expressions: len(self._entries) ## `vllm_mlx.memory_cache.MemoryAwarePrefixCache.__contains__` - Kind: method - Signature: `def __contains__(self, tokens: list[int]) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/memory_cache.py#L1200-L1202 - Implementation: Method `MemoryAwarePrefixCache.__contains__` calls `tuple`; returns `tuple(tokens) in self._entries`. Check if tokens are cached. - Inputs: - `tokens` (list[int]; required): Required positional or keyword input. - Return annotation: `bool` - Calls: tuple - State reads: self._entries - Return expressions: tuple(tokens) in self._entries ## `vllm_mlx.memory_cache.MemoryAwarePrefixCache.set_ssd_tier` - Kind: method - Signature: `def set_ssd_tier(self, ssd_tier) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/memory_cache.py#L1204-L1214 - Implementation: Method `MemoryAwarePrefixCache.set_ssd_tier` updates `self._ssd_tier`; calls `logger.info`. Attach an SSD cache tier for eviction spilling. When set, evicted entries are spilled to SSD instead of discarded. Args: ssd_tier: An SSDCacheTier instance (or None to disable). - Inputs: - `ssd_tier` (not annotated; required): An SSDCacheTier instance (or None to disable). - Return annotation: `None` - Calls: logger.info - State writes: self._ssd_tier ## `vllm_mlx.memory_cache.MemoryAwarePrefixCache.check_ssd` - Kind: method - Signature: `def check_ssd(self, tokens: list[int]) -> dict | None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/memory_cache.py#L1216-L1249 - Implementation: Method `MemoryAwarePrefixCache.check_ssd` calls `tuple`, `self._ssd_tier.lookup_ssd`, `len`, `self._ssd_tier.lookup_ssd_prefix`; has 3 explicit return paths. Check if tokens have an SSD cache hit (without reading data). Returns metadata dict with 'match_type' ('exact' or 'prefix') if found in SSD tier, None if not found. For prefix matches, the dict also includes 'matched_tokens' (the count of tokens the SSD entry covers). This is a fast synchronous call (SQLite lookup only). The actual data read happens via the scheduler handoff. - Inputs: - `tokens` (list[int]; required): Required positional or keyword input. - Return annotation: `dict | None` - Calls: tuple, self._ssd_tier.lookup_ssd, len, self._ssd_tier.lookup_ssd_prefix - State reads: self._ssd_tier, self._entries, self._ssd_tier.lookup_ssd, self._ssd_tier.lookup_ssd_prefix - Return expressions: None; candidate; prefix ## `vllm_mlx.memory_cache.MemoryAwarePrefixCache.save_to_disk` - Kind: method - Signature: `def save_to_disk(self, cache_dir: str) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/memory_cache.py#L1255-L1346 - Implementation: Method `MemoryAwarePrefixCache.save_to_disk` calls `logger.info`, `_time.monotonic`, `os.makedirs`, `logger.warning`; has 2 explicit return paths. Save all cache entries to disk using mlx_lm's safetensors format. Directory layout:: cache_dir/ index.json # token keys + metadata per entry entry_0.safetensors # KV arrays for entry 0 entry_1.safetensors ... Returns True if at least one entry was saved. - Inputs: - `cache_dir` (str; required): Required positional or keyword input. - Return annotation: `bool` - Calls: logger.info, _time.monotonic, os.makedirs, logger.warning, len, enumerate, self._entries.items, os.path.join, any, isinstance, _dequantize_cache, save_prompt_cache, str, _array.array, open, arr.tofile, index['entries'].append, json.dump - State reads: self._entries, self._model_fingerprint, self._current_memory, self._entries.items - Return expressions: False; saved > 0 ## `vllm_mlx.memory_cache.MemoryAwarePrefixCache.load_from_disk` - Kind: method - Signature: `def load_from_disk(self, cache_dir: str) -> int` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/memory_cache.py#L1348-L1463 - Implementation: Method `MemoryAwarePrefixCache.load_from_disk` updates `self._current_memory`, `self._stats.entry_count`, `self._stats.current_memory_bytes`; calls `os.path.join`, `os.path.exists`, `logger.info`, `_time.monotonic`; has 2 explicit return paths. Load cache entries from disk. Returns the number of entries successfully loaded. - Inputs: - `cache_dir` (str; required): Required positional or keyword input. - Return annotation: `int` - Calls: os.path.join, os.path.exists, logger.info, _time.monotonic, logger.warning, open, json.load, index.get, _array.array, arr.fromfile, list, len, load_prompt_cache, estimate_kv_cache_memory, tuple, _CacheEntry, bisect.insort - State reads: self._model_fingerprint, self._config.min_prefix_tokens, self._config, self._memory_lock, self._current_memory, self._max_memory, self._entries, self._sorted_keys, self._stats - State writes: self._current_memory, self._stats.entry_count, self._stats.current_memory_bytes - Return expressions: 0; loaded # Module `vllm_mlx.metrics` Prometheus-first server metrics for vllm-mlx. The public surface is a small internal abstraction that keeps instrumentation call sites stable even if we add OpenTelemetry export later. Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/metrics.py#L1-L532 ## `vllm_mlx.metrics._bool_str` - Kind: function - Signature: `def _bool_str(value: bool) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/metrics.py#L17-L18 - Implementation: Function `_bool_str` returns `'true' if value else 'false'`. Function `_bool_str` returns `'true' if value else 'false'`. - Inputs: - `value` (bool; required): Required positional or keyword input. - Return annotation: `str` - Return expressions: 'true' if value else 'false' ## `vllm_mlx.metrics._coerce_float` - Kind: function - Signature: `def _coerce_float(value: Any, default: float=0.0) -> float` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/metrics.py#L21-L27 - Implementation: Function `_coerce_float` calls `float`; has 2 explicit return paths. Function `_coerce_float` calls `float`; has 2 explicit return paths. - Inputs: - `value` (Any; required): Required positional or keyword input. - `default` (float; optional; default `0.0`): Optional positional or keyword input; defaults to `0.0`. - Return annotation: `float` - Calls: float - Return expressions: default; float(value) ## `vllm_mlx.metrics._coerce_int` - Kind: function - Signature: `def _coerce_int(value: Any, default: int=0) -> int` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/metrics.py#L30-L36 - Implementation: Function `_coerce_int` calls `int`; has 2 explicit return paths. Function `_coerce_int` calls `int`; has 2 explicit return paths. - Inputs: - `value` (Any; required): Required positional or keyword input. - `default` (int; optional; default `0`): Optional positional or keyword input; defaults to `0`. - Return annotation: `int` - Calls: int - Return expressions: default; int(value) ## `vllm_mlx.metrics.InferenceTracker` - Kind: class - Signature: `class InferenceTracker` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/metrics.py#L40-L81 - Implementation: Class `InferenceTracker` declares 2 direct member(s). Request-scoped inference timing and token accounting. - Inputs: - `collector` ('MetricsCollector | None'; required): Required constructor field. - `endpoint` (str; required): Required constructor field. - `stream` (bool; required): Required constructor field. - `start_time` (float; optional; default `field(default_factory=time.perf_counter)`): Optional constructor field; defaults to `field(default_factory=time.perf_counter)`. - `_finished` (bool; optional; default `False`): Optional constructor field; defaults to `False`. - `_ttft_observed` (bool; optional; default `False`): Optional constructor field; defaults to `False`. - Constructs: `vllm_mlx.metrics.InferenceTracker` - Decorators: dataclass ## `vllm_mlx.metrics.InferenceTracker.observe_ttft` - Kind: method - Signature: `def observe_ttft(self) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/metrics.py#L50-L60 - Implementation: Method `InferenceTracker.observe_ttft` updates `self._ttft_observed`; calls `self.collector.observe_ttft`, `time.perf_counter`; returns `None`. Record time to first token once for this inference request. - Inputs: none - Return annotation: `None` - Calls: self.collector.observe_ttft, time.perf_counter - State reads: self.collector, self._ttft_observed, self.collector.observe_ttft, self.endpoint, self.stream, self.start_time - State writes: self._ttft_observed - Return expressions: None ## `vllm_mlx.metrics.InferenceTracker.finish` - Kind: method - Signature: `def finish(self, *, result: str, prompt_tokens: int=0, completion_tokens: int=0) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/metrics.py#L62-L81 - Implementation: Method `InferenceTracker.finish` updates `self._finished`; calls `self.collector.observe_inference`, `time.perf_counter`; returns `None`. Record terminal latency and token counts once for this request. - Inputs: - `result` (str; required): Required keyword-only input. - `prompt_tokens` (int; optional; default `0`): Optional keyword-only input; defaults to `0`. - `completion_tokens` (int; optional; default `0`): Optional keyword-only input; defaults to `0`. - Return annotation: `None` - Calls: self.collector.observe_inference, time.perf_counter - State reads: self.collector, self._finished, self.collector.observe_inference, self.endpoint, self.stream, self.start_time - State writes: self._finished - Return expressions: None ## `vllm_mlx.metrics.MetricsCollector` - Kind: class - Signature: `class MetricsCollector` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/metrics.py#L84-L529 - Implementation: Class `MetricsCollector` declares 11 direct member(s). Lazy Prometheus-backed metrics collector. - Inputs: none - Constructs: `vllm_mlx.metrics.MetricsCollector` ## `vllm_mlx.metrics.MetricsCollector.__init__` - Kind: method - Signature: `def __init__(self) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/metrics.py#L87-L90 - Implementation: Method `MetricsCollector.__init__` updates `self._enabled`, `self._lock`, `self._prom`; calls `threading.Lock`. Method `MetricsCollector.__init__` updates `self._enabled`, `self._lock`, `self._prom`; calls `threading.Lock`. - Inputs: none - Return annotation: `None` - Calls: threading.Lock - State writes: self._enabled, self._lock, self._prom ## `vllm_mlx.metrics.MetricsCollector.enabled` - Kind: method - Signature: `def enabled(self) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/metrics.py#L93-L96 - Implementation: Method `MetricsCollector.enabled` returns `self._enabled`. Return whether metric collection is enabled. - Inputs: none - Return annotation: `bool` - Decorators: property - State reads: self._enabled - Return expressions: self._enabled ## `vllm_mlx.metrics.MetricsCollector.configure` - Kind: method - Signature: `def configure(self, *, enabled: bool) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/metrics.py#L98-L105 - Implementation: Method `MetricsCollector.configure` updates `self._enabled`; calls `self._init_prometheus`; returns `None`. Enable or disable collection and lazily initialize Prometheus state. - Inputs: - `enabled` (bool; required): Required keyword-only input. - Return annotation: `None` - Calls: self._init_prometheus - State reads: self._lock, self._prom, self._init_prometheus - State writes: self._enabled - Return expressions: None ## `vllm_mlx.metrics.MetricsCollector._init_prometheus` - Kind: method - Signature: `def _init_prometheus(self) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/metrics.py#L107-L291 - Implementation: Method `MetricsCollector._init_prometheus` updates `self._prom`; calls `CollectorRegistry`, `Counter`, `Histogram`, `Gauge`. Method `MetricsCollector._init_prometheus` updates `self._prom`; calls `CollectorRegistry`, `Counter`, `Histogram`, `Gauge`. - Inputs: none - Return annotation: `None` - Calls: CollectorRegistry, Counter, Histogram, Gauge - State writes: self._prom ## `vllm_mlx.metrics.MetricsCollector.track_inference` - Kind: method - Signature: `def track_inference(self, endpoint: str, *, stream: bool) -> InferenceTracker` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/metrics.py#L293-L298 - Implementation: Method `MetricsCollector.track_inference` calls `InferenceTracker`; has 2 explicit return paths. Create request-scoped inference timing state for an endpoint. - Inputs: - `endpoint` (str; required): Required positional or keyword input. - `stream` (bool; required): Required keyword-only input. - Return annotation: `InferenceTracker` - Calls: InferenceTracker - State reads: self._enabled - Return expressions: InferenceTracker(None, endpoint, stream); InferenceTracker(self, endpoint, stream) ## `vllm_mlx.metrics.MetricsCollector.observe_http_start` - Kind: method - Signature: `def observe_http_start(self, *, method: str, path: str) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/metrics.py#L300-L305 - Implementation: Method `MetricsCollector.observe_http_start` calls `self._prom['http_requests_in_flight'].labels(method=method, path=path).inc`, `self._prom['http_requests_in_flight'].labels`; returns `None`. Increment the in-flight request gauge for a normalized route. - Inputs: - `method` (str; required): Required keyword-only input. - `path` (str; required): Required keyword-only input. - Return annotation: `None` - Calls: self._prom['http_requests_in_flight'].labels(method=method, path=path).inc, self._prom['http_requests_in_flight'].labels - State reads: self._enabled, self._prom - Return expressions: None ## `vllm_mlx.metrics.MetricsCollector.observe_http_finish` - Kind: method - Signature: `def observe_http_finish(self, *, method: str, path: str, status_code: int, duration: float) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/metrics.py#L307-L328 - Implementation: Method `MetricsCollector.observe_http_finish` calls `self._prom['http_requests_in_flight'].labels(method=method, path=path).dec`, `self._prom['http_requests_in_flight'].labels`, `self._prom['http_requests_total'].labels(method=method, path=path, status_code=str(status_code)).inc`, `self._prom['http_requests_total'].labels`; returns `None`. Record an HTTP result and decrement its in-flight gauge. - Inputs: - `method` (str; required): Required keyword-only input. - `path` (str; required): Required keyword-only input. - `status_code` (int; required): Required keyword-only input. - `duration` (float; required): Required keyword-only input. - Return annotation: `None` - Calls: self._prom['http_requests_in_flight'].labels(method=method, path=path).dec, self._prom['http_requests_in_flight'].labels, self._prom['http_requests_total'].labels(method=method, path=path, status_code=str(status_code)).inc, self._prom['http_requests_total'].labels, str, self._prom['http_request_duration_seconds'].labels(method=method, path=path).observe, self._prom['http_request_duration_seconds'].labels - State reads: self._enabled, self._prom - Return expressions: None ## `vllm_mlx.metrics.MetricsCollector.observe_inference` - Kind: method - Signature: `def observe_inference(self, *, endpoint: str, stream: bool, result: str, duration: float, prompt_tokens: int, completion_tokens: int) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/metrics.py#L330-L363 - Implementation: Method `MetricsCollector.observe_inference` calls `_bool_str`, `self._prom['inference_requests_total'].labels(endpoint=endpoint, stream=stream_label, result=result).inc`, `self._prom['inference_requests_total'].labels`, `self._prom['inference_request_duration_seconds'].labels(endpoint=endpoint, stream=stream_label).observe`; returns `None`. Record one terminal inference outcome, latency, and token totals. - Inputs: - `endpoint` (str; required): Required keyword-only input. - `stream` (bool; required): Required keyword-only input. - `result` (str; required): Required keyword-only input. - `duration` (float; required): Required keyword-only input. - `prompt_tokens` (int; required): Required keyword-only input. - `completion_tokens` (int; required): Required keyword-only input. - Return annotation: `None` - Calls: _bool_str, self._prom['inference_requests_total'].labels(endpoint=endpoint, stream=stream_label, result=result).inc, self._prom['inference_requests_total'].labels, self._prom['inference_request_duration_seconds'].labels(endpoint=endpoint, stream=stream_label).observe, self._prom['inference_request_duration_seconds'].labels, self._prom['prompt_tokens_total'].labels(endpoint=endpoint, stream=stream_label).inc, self._prom['prompt_tokens_total'].labels, self._prom['completion_tokens_total'].labels(endpoint=endpoint, stream=stream_label).inc, self._prom['completion_tokens_total'].labels - State reads: self._enabled, self._prom - Return expressions: None ## `vllm_mlx.metrics.MetricsCollector.observe_ttft` - Kind: method - Signature: `def observe_ttft(self, *, endpoint: str, stream: bool, value: float) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/metrics.py#L365-L373 - Implementation: Method `MetricsCollector.observe_ttft` calls `self._prom['inference_ttft_seconds'].labels(endpoint=endpoint, stream=_bool_str(stream)).observe`, `self._prom['inference_ttft_seconds'].labels`, `_bool_str`; returns `None`. Observe time to first token for a streaming or buffered request. - Inputs: - `endpoint` (str; required): Required keyword-only input. - `stream` (bool; required): Required keyword-only input. - `value` (float; required): Required keyword-only input. - Return annotation: `None` - Calls: self._prom['inference_ttft_seconds'].labels(endpoint=endpoint, stream=_bool_str(stream)).observe, self._prom['inference_ttft_seconds'].labels, _bool_str - State reads: self._enabled, self._prom - Return expressions: None ## `vllm_mlx.metrics.MetricsCollector._update_engine_gauges` - Kind: method - Signature: `def _update_engine_gauges(self, *, engine: Any | None, mcp_manager: Any | None) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/metrics.py#L375-L507 - Implementation: Method `MetricsCollector._update_engine_gauges` calls `engine.get_stats`, `self._prom['model_loaded'].set`, `stats.get`, `self._prom['engine_type'].labels(engine_type=engine_type).set`. Method `MetricsCollector._update_engine_gauges` calls `engine.get_stats`, `self._prom['model_loaded'].set`, `stats.get`, `self._prom['engine_type'].labels(engine_type=engine_type).set`. - Inputs: - `engine` (Any | None; required): Required keyword-only input. - `mcp_manager` (Any | None; required): Required keyword-only input. - Return annotation: `None` - Calls: engine.get_stats, self._prom['model_loaded'].set, stats.get, self._prom['engine_type'].labels(engine_type=engine_type).set, self._prom['engine_type'].labels, self._prom['engine_is_mllm'].set, self._prom['scheduler_waiting_requests'].set, _coerce_int, self._prom['scheduler_running_requests'].set, self._prom['engine_steps_executed'].set, self._prom['engine_uptime_seconds'].set, _coerce_float, self._prom['metal_memory_bytes'].labels(kind='active').set, self._prom['metal_memory_bytes'].labels, self._prom['metal_memory_bytes'].labels(kind='peak').set, self._prom['metal_memory_bytes'].labels(kind='cache').set, self._prom['cache_type'].labels(cache_type=candidate).set, self._prom['cache_type'].labels, isinstance, self._prom['cache_entry_count'].set, cache_stats.get, self._prom['cache_hits'].set, self._prom['cache_misses'].set, self._prom['cache_evictions'].set, self._prom['cache_hit_rate'].set, self._prom['cache_utilization_ratio'].set, self._prom['cache_tokens_saved'].set, self._prom['cache_memory_bytes'].set, self._prom['cache_memory_limit_bytes'].set, get_registry().get_stats, get_registry, self._prom['model_registry_entries'].set, registry_stats.get, self._prom['model_registry_active_owners'].set, list, mcp_manager.get_server_status, sum, len, mcp_manager.get_all_tools, self._prom['mcp_connected_servers'].set, self._prom['mcp_total_servers'].set, self._prom['mcp_tools_available'].set - State reads: self._prom ## `vllm_mlx.metrics.MetricsCollector.render_metrics` - Kind: method - Signature: `def render_metrics(self, *, engine: Any | None, mcp_manager: Any | None) -> tuple[bytes, str]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/metrics.py#L509-L529 - Implementation: Method `MetricsCollector.render_metrics` calls `RuntimeError`, `self._init_prometheus`, `self._update_engine_gauges`, `self._prom['generate_latest']`; can raise `RuntimeError`; returns `(self._prom['generate_latest'](self._prom['registry']), self._prom['content_type'])`. Refresh runtime gauges and render Prometheus exposition bytes. Raises: RuntimeError: If metrics are disabled. - Inputs: - `engine` (Any | None; required): Required keyword-only input. - `mcp_manager` (Any | None; required): Required keyword-only input. - Return annotation: `tuple[bytes, str]` - Calls: RuntimeError, self._init_prometheus, self._update_engine_gauges, self._prom['generate_latest'] - State reads: self._enabled, self._prom, self._init_prometheus, self._update_engine_gauges - Raises directly: RuntimeError - Return expressions: (self._prom['generate_latest'](self._prom['registry']), self._prom['content_type']) # Module `vllm_mlx.mllm_batch_generator` MLLM Batch Generator for multimodal continuous batching. This module implements continuous batching for Multimodal Language Models (MLLMs) like Qwen3-VL, following the same architecture as LLM continuous batching but adapted for vision models. Key insight: VLM models have a `model.language_model` which is a standard LLM. After the initial forward pass with vision encoding, text generation uses only the language model - which CAN be batched using the same BatchKVCache pattern. Architecture: 1. Vision inputs are processed per-request (not batched) 2. Initial VLM forward pass extracts cross-attention states / encoder outputs 3. Language model generation is batched using BatchKVCache (like LLM batching) Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L1-L3073 ## `vllm_mlx.mllm_batch_generator._processors_can_retire` - Kind: function - Signature: `def _processors_can_retire(processors: Optional[List[Callable]]) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L37-L43 - Implementation: Function `_processors_can_retire` calls `os.getenv`, `bool`, `any`, `isinstance`; has 2 explicit return paths. True when any processor advertises a retire-to-content transition. - Inputs: - `processors` (Optional[List[Callable]]; required): Required positional or keyword input. - Return annotation: `bool` - Calls: os.getenv, bool, any, isinstance, getattr - Return expressions: False; bool(processors) and any((isinstance(getattr(p, 'is_retired', None), bool) for p in processors)) ## `vllm_mlx.mllm_batch_generator._mark_mtp_attempts_on_primary_responses` - Kind: function - Signature: `def _mark_mtp_attempts_on_primary_responses(responses: List['MLLMBatchResponse'], attempted_drafts_by_uid: Dict[int, int]) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L46-L57 - Implementation: Function `_mark_mtp_attempts_on_primary_responses` calls `attempted_drafts_by_uid.pop`, `attempted_drafts_by_uid.clear`. Mark only responses from steps that actually attempted MTP drafts. - Inputs: - `responses` (List['MLLMBatchResponse']; required): Required positional or keyword input. - `attempted_drafts_by_uid` (Dict[int, int]; required): Required positional or keyword input. - Return annotation: `None` - Calls: attempted_drafts_by_uid.pop, attempted_drafts_by_uid.clear ## `vllm_mlx.mllm_batch_generator._drop_retired_processors` - Kind: function - Signature: `def _drop_retired_processors(processors: Optional[List[Callable]]) -> tuple[Optional[List[Callable]], int]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L60-L74 - Implementation: Function `_drop_retired_processors` calls `getattr`, `remaining.append`; has 2 explicit return paths. Drop retire-capable processors that have completed their work. - Inputs: - `processors` (Optional[List[Callable]]; required): Required positional or keyword input. - Return annotation: `tuple[Optional[List[Callable]], int]` - Calls: getattr, remaining.append - Return expressions: (processors, 0); (remaining or None, retired_count) ## `vllm_mlx.mllm_batch_generator._request_uses_stochastic_sampling` - Kind: function - Signature: `def _request_uses_stochastic_sampling(request: Any) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L77-L92 - Implementation: Function `_request_uses_stochastic_sampling` calls `getattr`; has 2 explicit return paths. Return whether a request needs sampler-aware speculative verification. Greedy (temperature 0) requests are excluded regardless of top_p/top_k/ min_p: _sampling_logprobs() collapses to an argmax delta distribution for temperature 0 and never applies those filters, so a greedy request left at a non-default top_p/top_k/min_p is not actually stochastic. - Inputs: - `request` (Any; required): Required positional or keyword input. - Return annotation: `bool` - Calls: getattr - Return expressions: False; getattr(request, 'top_p', 1.0) < 1.0 or getattr(request, 'top_k', 0) != 0 or getattr(request, 'min_p', 0.0) != 0.0 ## `vllm_mlx.mllm_batch_generator._sampling_logprobs` - Kind: function - Signature: `def _sampling_logprobs(logits: mx.array, request: Any) -> mx.array` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L95-L123 - Implementation: Function `_sampling_logprobs` calls `getattr`, `mx.logsumexp`, `mx.argmax`, `mx.full`; has 2 explicit return paths. Match mlx-lm's request sampler in log-probability space. Speculative decoding compares the post-filter distributions, not the raw target and draft logits. Keep this transformation here rather than reusing a greedy verifier for sampled requests. - Inputs: - `logits` (mx.array; required): Required positional or keyword input. - `request` (Any; required): Required positional or keyword input. - Return annotation: `mx.array` - Calls: getattr, mx.logsumexp, mx.argmax, mx.full, float, mx.put_along_axis, apply_top_p, apply_min_p, apply_top_k - Return expressions: mx.put_along_axis(result, token[:, None], 0.0, axis=-1); logprobs - mx.logsumexp(logprobs, axis=-1, keepdims=True) ## `vllm_mlx.mllm_batch_generator._residual_logprobs` - Kind: function - Signature: `def _residual_logprobs(target_logprobs: mx.array, draft_logprobs: mx.array) -> mx.array` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L126-L139 - Implementation: Function `_residual_logprobs` calls `mx.maximum`, `mx.exp`, `mx.sum`, `mx.where`; returns `mx.where(mass > 1e-12, normalized, fallback)`. Return the normalized residual max(target - draft, 0) distribution. - Inputs: - `target_logprobs` (mx.array; required): Required positional or keyword input. - `draft_logprobs` (mx.array; required): Required positional or keyword input. - Return annotation: `mx.array` - Calls: mx.maximum, mx.exp, mx.sum, mx.where, mx.log, float - Return expressions: mx.where(mass > 1e-12, normalized, fallback) ## `vllm_mlx.mllm_batch_generator._accept_sampled_draft` - Kind: function - Signature: `def _accept_sampled_draft(target_logprob: float, draft_logprob: float, uniform_draw: float) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L142-L149 - Implementation: Function `_accept_sampled_draft` calls `math.log`, `max`; returns `log_acceptance >= 0.0 or math.log(max(uniform_draw, 1e-35)) < log_acceptance`. Apply the exact min(1, p/q) stochastic speculative acceptance rule. - Inputs: - `target_logprob` (float; required): Required positional or keyword input. - `draft_logprob` (float; required): Required positional or keyword input. - `uniform_draw` (float; required): Required positional or keyword input. - Return annotation: `bool` - Calls: math.log, max - Return expressions: log_acceptance >= 0.0 or math.log(max(uniform_draw, 1e-35)) < log_acceptance ## `vllm_mlx.mllm_batch_generator.PrefillAbortedError` - Kind: class - Signature: `class PrefillAbortedError(Exception)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L152-L157 - Implementation: Class `PrefillAbortedError` derives from `Exception` and declares 1 direct member(s). Raised when a prefill is aborted due to client disconnect. - Inputs: - `request_id` (str; required): Required positional or keyword input. - Constructs: `vllm_mlx.mllm_batch_generator.PrefillAbortedError` ## `vllm_mlx.mllm_batch_generator.PrefillAbortedError.__init__` - Kind: method - Signature: `def __init__(self, request_id: str)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L155-L157 - Implementation: Method `PrefillAbortedError.__init__` updates `self.request_id`; calls `super().__init__`, `super`. Method `PrefillAbortedError.__init__` updates `self.request_id`; calls `super().__init__`, `super`. - Inputs: - `request_id` (str; required): Required positional or keyword input. - Return annotation: `not annotated` - Calls: super().__init__, super - State writes: self.request_id ## `vllm_mlx.mllm_batch_generator._cache_eval_tensors` - Kind: function - Signature: `def _cache_eval_tensors(cache: List[Any]) -> List[Any]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L160-L183 - Implementation: Function `_cache_eval_tensors` calls `getattr`, `tensors.append`, `isinstance`, `tensors.extend`; returns `tensors`. Return realized tensors that break lazy cache graphs between chunks. - Inputs: - `cache` (List[Any]; required): Required positional or keyword input. - Return annotation: `List[Any]` - Calls: getattr, tensors.append, isinstance, tensors.extend - Return expressions: tensors ## `vllm_mlx.mllm_batch_generator._eval_prompt_cache` - Kind: function - Signature: `def _eval_prompt_cache(cache: List[Any]) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L186-L190 - Implementation: Function `_eval_prompt_cache` calls `_cache_eval_tensors`, `mx.eval`. Evaluate all cache tensors used by hybrid chunked prefill. - Inputs: - `cache` (List[Any]; required): Required positional or keyword input. - Return annotation: `None` - Calls: _cache_eval_tensors, mx.eval ## `vllm_mlx.mllm_batch_generator.MLLMBatchRequest` - Kind: class - Signature: `class MLLMBatchRequest` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L194-L237 - Implementation: Class `MLLMBatchRequest` declares 0 direct member(s). Request data for MLLM batch processing. Contains all information needed to process a multimodal request within the batch generator. - Inputs: - `uid` (int; required): Required constructor field. - `request_id` (str; required): Required constructor field. - `prompt` (str; required): Required constructor field. - `images` (Optional[List[str]]; optional; default `None`): Optional constructor field; defaults to `None`. - `videos` (Optional[List[str]]; optional; default `None`): Optional constructor field; defaults to `None`. - `audio` (Optional[List[str]]; optional; default `None`): Optional constructor field; defaults to `None`. - `max_tokens` (int; optional; default `256`): Optional constructor field; defaults to `256`. - `temperature` (float; optional; default `0.7`): Optional constructor field; defaults to `0.7`. - `top_p` (float; optional; default `0.9`): Optional constructor field; defaults to `0.9`. - `top_k` (int; optional; default `0`): Optional constructor field; defaults to `0`. - `min_p` (float; optional; default `0.0`): Optional constructor field; defaults to `0.0`. - `presence_penalty` (float; optional; default `0.0`): Optional constructor field; defaults to `0.0`. - `repetition_penalty` (float; optional; default `1.0`): Optional constructor field; defaults to `1.0`. - `logits_processors` (Optional[List[Callable]]; optional; default `None`): Optional constructor field; defaults to `None`. - `input_ids` (Optional[mx.array]; optional; default `None`): Optional constructor field; defaults to `None`. - `pixel_values` (Optional[mx.array]; optional; default `None`): Optional constructor field; defaults to `None`. - `attention_mask` (Optional[mx.array]; optional; default `None`): Optional constructor field; defaults to `None`. - `image_grid_thw` (Optional[mx.array]; optional; default `None`): Optional constructor field; defaults to `None`. - `extra_kwargs` (Dict[str, Any]; optional; default `field(default_factory=dict)`): Optional constructor field; defaults to `field(default_factory=dict)`. - `is_text_only` (bool; optional; default `False`): Optional constructor field; defaults to `False`. - `num_tokens` (int; optional; default `0`): Optional constructor field; defaults to `0`. - `output_tokens` (List[int]; optional; default `field(default_factory=list)`): Optional constructor field; defaults to `field(default_factory=list)`. - `vision_encoded` (bool; optional; default `False`): Optional constructor field; defaults to `False`. - `cross_attention_states` (Optional[Any]; optional; default `None`): Optional constructor field; defaults to `None`. - `encoder_outputs` (Optional[Any]; optional; default `None`): Optional constructor field; defaults to `None`. - Constructs: `vllm_mlx.mllm_batch_generator.MLLMBatchRequest` - Decorators: dataclass ## `vllm_mlx.mllm_batch_generator.MLLMBatchResponse` - Kind: class - Signature: `class MLLMBatchResponse` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L241-L256 - Implementation: Class `MLLMBatchResponse` declares 0 direct member(s). Response from a batch generation step. Contains the generated token and metadata for a single request. - Inputs: - `uid` (int; required): Required constructor field. - `request_id` (str; required): Required constructor field. - `token` (int; required): Required constructor field. - `logprobs` (mx.array; required): Required constructor field. - `finish_reason` (Optional[str]; optional; default `None`): Optional constructor field; defaults to `None`. - `prompt_cache` (Optional[Callable[[], List[Any]]]; optional; default `None`): Optional constructor field; defaults to `None`. - `from_draft` (bool; optional; default `False`): Optional constructor field; defaults to `False`. - `mtp_attempted` (bool; optional; default `False`): Optional constructor field; defaults to `False`. - `mtp_attempted_count` (int; optional; default `0`): Optional constructor field; defaults to `0`. - Constructs: `vllm_mlx.mllm_batch_generator.MLLMBatchResponse` - Decorators: dataclass ## `vllm_mlx.mllm_batch_generator.MLLMBatch` - Kind: class - Signature: `class MLLMBatch` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L260-L392 - Implementation: Class `MLLMBatch` declares 4 direct member(s). Represents an active batch of MLLM requests. Manages the batch state including tokens, caches, and metadata for all requests being processed together. - Inputs: - `uids` (List[int]; required): Required constructor field. - `request_ids` (List[str]; required): Required constructor field. - `y` (mx.array; required): Required constructor field. - `logprobs` (List[mx.array]; required): Required constructor field. - `max_tokens` (List[int]; required): Required constructor field. - `num_tokens` (List[int]; required): Required constructor field. - `cache` (List[Any]; required): Required constructor field. - `requests` (List[MLLMBatchRequest]; required): Required constructor field. - `logits_processors` (Optional[List[Optional[List[Callable]]]]; optional; default `None`): Optional constructor field; defaults to `None`. - `samplers` (Optional[List[Optional[Callable]]]; optional; default `None`): Optional constructor field; defaults to `None`. - Constructs: `vllm_mlx.mllm_batch_generator.MLLMBatch` - Decorators: dataclass ## `vllm_mlx.mllm_batch_generator.MLLMBatch.__len__` - Kind: method - Signature: `def __len__(self) -> int` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L279-L280 - Implementation: Method `MLLMBatch.__len__` calls `len`; returns `len(self.uids)`. Method `MLLMBatch.__len__` calls `len`; returns `len(self.uids)`. - Inputs: none - Return annotation: `int` - Calls: len - State reads: self.uids - Return expressions: len(self.uids) ## `vllm_mlx.mllm_batch_generator.MLLMBatch.filter` - Kind: method - Signature: `def filter(self, keep_idx: List[int]) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L282-L306 - Implementation: Method `MLLMBatch.filter` updates `self.uids`, `self.request_ids`, `self.logprobs`, `self.max_tokens`; calls `mx.array`, `hasattr`, `c.filter`. Filter batch to keep only requests at specified indices. Args: keep_idx: Indices of requests to keep - Inputs: - `keep_idx` (List[int]; required): Indices of requests to keep - Return annotation: `None` - Calls: mx.array, hasattr, c.filter - State reads: self.uids, self.request_ids, self.logprobs, self.max_tokens, self.num_tokens, self.requests, self.logits_processors, self.samplers, self.y, self.cache - State writes: self.uids, self.request_ids, self.logprobs, self.max_tokens, self.num_tokens, self.requests, self.logits_processors, self.samplers, self.y ## `vllm_mlx.mllm_batch_generator.MLLMBatch.extend` - Kind: method - Signature: `def extend(self, other: 'MLLMBatch') -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L308-L351 - Implementation: Method `MLLMBatch.extend` updates `self.y`, `self.logits_processors`, `self.samplers`; calls `self.uids.extend`, `self.request_ids.extend`, `mx.concatenate`, `self.logprobs.extend`. Extend this batch with another batch. Args: other: Batch to merge into this one - Inputs: - `other` ('MLLMBatch'; required): Batch to merge into this one - Return annotation: `None` - Calls: self.uids.extend, self.request_ids.extend, mx.concatenate, self.logprobs.extend, self.num_tokens.extend, self.max_tokens.extend, self.requests.extend, len, list, zip, hasattr, c.empty, c.extend, logger.warning - State reads: self.uids.extend, self.uids, self.request_ids.extend, self.request_ids, self.y, self.logprobs.extend, self.logprobs, self.num_tokens.extend, self.num_tokens, self.max_tokens.extend, self.max_tokens, self.requests.extend, self.requests, self.logits_processors, self.samplers, self.cache - State writes: self.y, self.logits_processors, self.samplers ## `vllm_mlx.mllm_batch_generator.MLLMBatch.extract_cache` - Kind: method - Signature: `def extract_cache(self, idx: int) -> List[Any]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L353-L392 - Implementation: Method `MLLMBatch.extract_cache` calls `hasattr`, `result.append`, `isinstance`, `RotatingKVCache`; returns `result`. Extract cache for a single request (for prefix caching). Handles BatchRotatingKVCache negative left_padding bug: during generation with rotation, left_padding becomes negative, causing extract() to use Python negative indexing and truncate the buffer to only generation tokens instead of the full window. - Inputs: - `idx` (int; required): Required positional or keyword input. - Return annotation: `List[Any]` - Calls: hasattr, result.append, isinstance, RotatingKVCache, max, c.left_padding[idx].item, c.offset[idx].item, mx.roll, mx.contiguous, getattr, c.extract - State reads: self.cache - Return expressions: result ## `vllm_mlx.mllm_batch_generator.MLLMBatchStats` - Kind: class - Signature: `class MLLMBatchStats` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L395-L436 - Implementation: Class `MLLMBatchStats` declares 4 direct member(s). Statistics for MLLM batch generation. - Inputs: none - Constructs: `vllm_mlx.mllm_batch_generator.MLLMBatchStats` ## `vllm_mlx.mllm_batch_generator.MLLMBatchStats.__init__` - Kind: method - Signature: `def __init__(self)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L398-L405 - Implementation: Method `MLLMBatchStats.__init__` updates `self.prompt_tokens`, `self.prompt_time`, `self.generation_tokens`, `self.generation_time`. Method `MLLMBatchStats.__init__` updates `self.prompt_tokens`, `self.prompt_time`, `self.generation_tokens`, `self.generation_time`. - Inputs: none - Return annotation: `not annotated` - State writes: self.prompt_tokens, self.prompt_time, self.generation_tokens, self.generation_time, self.vision_encoding_time, self.num_images_processed, self.peak_memory ## `vllm_mlx.mllm_batch_generator.MLLMBatchStats.prompt_tps` - Kind: method - Signature: `def prompt_tps(self) -> float` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L408-L413 - Implementation: Method `MLLMBatchStats.prompt_tps` has 2 explicit return paths. Return measured multimodal prompt throughput in tokens per second. - Inputs: none - Return annotation: `float` - Decorators: property - State reads: self.prompt_time, self.prompt_tokens - Return expressions: 0; self.prompt_tokens / self.prompt_time ## `vllm_mlx.mllm_batch_generator.MLLMBatchStats.generation_tps` - Kind: method - Signature: `def generation_tps(self) -> float` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L416-L421 - Implementation: Method `MLLMBatchStats.generation_tps` has 2 explicit return paths. Return measured decode throughput in tokens per second. - Inputs: none - Return annotation: `float` - Decorators: property - State reads: self.generation_time, self.generation_tokens - Return expressions: 0; self.generation_tokens / self.generation_time ## `vllm_mlx.mllm_batch_generator.MLLMBatchStats.to_dict` - Kind: method - Signature: `def to_dict(self) -> Dict[str, Any]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L423-L436 - Implementation: Method `MLLMBatchStats.to_dict` returns `{'prompt_tokens': self.prompt_tokens, 'prompt_time': self.prompt_time, 'prompt_tps': self.prompt_tps, 'generation_token…`. Return token, timing, vision, and peak-memory statistics. - Inputs: none - Return annotation: `Dict[str, Any]` - State reads: self.prompt_tokens, self.prompt_time, self.prompt_tps, self.generation_tokens, self.generation_time, self.generation_tps, self.vision_encoding_time, self.num_images_processed, self.peak_memory - Return expressions: {'prompt_tokens': self.prompt_tokens, 'prompt_time': self.prompt_time, 'prompt_tps': self.prompt_tps, 'generation_token… ## `vllm_mlx.mllm_batch_generator._left_pad_prompts` - Kind: function - Signature: `def _left_pad_prompts(prompts: List[List[int]], max_length: Optional[int]=None) -> mx.array` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L439-L454 - Implementation: Function `_left_pad_prompts` calls `max`, `len`, `mx.array`, `list`; returns `mx.array([[0] * (max_length - len(p)) + list(p) for p in prompts])`. Left-pad prompts to uniform length. Args: prompts: List of token lists max_length: Target length (computed if not provided) Returns: Padded prompts as mx.array [batch_size, seq_len] - Inputs: - `prompts` (List[List[int]]; required): List of token lists - `max_length` (Optional[int]; optional; default `None`): Target length (computed if not provided) - Return annotation: `mx.array` - Calls: max, len, mx.array, list - Return expressions: mx.array([[0] * (max_length - len(p)) + list(p) for p in prompts]) ## `vllm_mlx.mllm_batch_generator.MLLMBatchGenerator` - Kind: class - Signature: `class MLLMBatchGenerator` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L457-L2042 - Implementation: Class `MLLMBatchGenerator` declares 26 direct member(s). Batch generator for Vision Language Models. This class manages continuous batching for MLLM requests: 1. Vision Encoding Phase: - Process images/videos through vision encoder (per-request) - Extract vision features and merge with text embeddings - Store cross-attention states for language model 2. Language Generation Phase: - Use language model with BatchKVCache for batched generation - Generate tokens for all requests simultaneously - Same pattern as LLM BatchGenerator Example: >>> generator = MLLMBatchGenerator(model, processor) >>> uids = generator.insert([request1, request2]) >>> while responses := generator.next(): ... for resp in responses: ... print(f"Request {resp.request_id}: token={resp.token}") - Inputs: - `model` (nn.Module; required): The VLM model (must have model.language_model) - `processor` (Any; required): The VLM processor for tokenization and image processing - `mm_processor` (Optional[MultimodalProcessor]; optional; default `None`): Optional MultimodalProcessor for input preparation - `max_tokens` (int; optional; default `256`): Default max tokens per request - `stop_tokens` (Optional[set]; optional; default `None`): Set of stop token IDs - `sampler` (Optional[Callable[[mx.array], mx.array]]; optional; default `None`): Sampling function (default: argmax) - `prefill_batch_size` (int; optional; default `4`): Max requests to prefill together - `completion_batch_size` (int; optional; default `16`): Max requests for completion batching - `prefill_step_size` (int; optional; default `1024`): Tokens to process per prefill step - `enable_vision_cache` (bool; optional; default `True`): Enable vision embedding caching - `vision_cache_size` (int; optional; default `100`): Max entries in vision cache - `prefix_cache_config` (Optional[MemoryCacheConfig]; optional; default `None`): Config for KV prefix cache (text-only requests) - `max_kv_size` (int; optional; default `0`): Maximum KV cache size per sequence (0 = unbounded) - Constructs: `vllm_mlx.mllm_batch_generator.MLLMBatchGenerator` ## `vllm_mlx.mllm_batch_generator.MLLMBatchGenerator.__init__` - Kind: method - Signature: `def __init__(self, model: nn.Module, processor: Any, mm_processor: Optional[MultimodalProcessor]=None, max_tokens: int=256, stop_tokens: Optional[set]=None, sampler: Optional[Callable[[mx.array], mx.array]]=None, prefill_batch_size: int=4, completion_batch_size: int=16, prefill_step_size: int=1024, enable_vision_cache: bool=True, vision_cache_size: int=100, prefix_cache_config: Optional[MemoryCacheConfig]=None, max_kv_size: int=0)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L484-L632 - Implementation: Method `MLLMBatchGenerator.__init__` updates `self.model`, `self.processor`, `self.mm_processor`, `self.max_kv_size`; calls `getattr`, `hasattr`, `logger.info`, `logger.warning`. Initialize MLLM batch generator. Args: model: The VLM model (must have model.language_model) processor: The VLM processor for tokenization and image processing mm_processor: Optional MultimodalProcessor for input preparation max_tokens: Default max tokens per request stop_tokens: Set of stop token IDs sampler: Sampling function (default: argmax) prefill_batch_size: Max requests to prefill together completion_batch_size: Max requests for completion batching prefill_step_size: Tokens to process per prefill step enable_vision_cache: Enable vision embedding caching vision_cache_size: Max entries in vision cache prefix_cache_config: Config for KV prefix cache (text-only requests) max_kv_size: Maximum KV cache size per sequence (0 = unbounded) - Inputs: - `model` (nn.Module; required): The VLM model (must have model.language_model) - `processor` (Any; required): The VLM processor for tokenization and image processing - `mm_processor` (Optional[MultimodalProcessor]; optional; default `None`): Optional MultimodalProcessor for input preparation - `max_tokens` (int; optional; default `256`): Default max tokens per request - `stop_tokens` (Optional[set]; optional; default `None`): Set of stop token IDs - `sampler` (Optional[Callable[[mx.array], mx.array]]; optional; default `None`): Sampling function (default: argmax) - `prefill_batch_size` (int; optional; default `4`): Max requests to prefill together - `completion_batch_size` (int; optional; default `16`): Max requests for completion batching - `prefill_step_size` (int; optional; default `1024`): Tokens to process per prefill step - `enable_vision_cache` (bool; optional; default `True`): Enable vision embedding caching - `vision_cache_size` (int; optional; default `100`): Max entries in vision cache - `prefix_cache_config` (Optional[MemoryCacheConfig]; optional; default `None`): Config for KV prefix cache (text-only requests) - `max_kv_size` (int; optional; default `0`): Maximum KV cache size per sequence (0 = unbounded) - Return annotation: `not annotated` - Calls: getattr, hasattr, logger.info, logger.warning, patch_qwen35_attention_for_batching, patch_gemma4_attention_for_batching, patch_glm4v_moe_for_batching, set, max, MLLMBatchStats, threading.Lock, VisionEmbeddingCache, MemoryAwarePrefixCache, self._normalize_chat_template_for_prefix_cache, self._compute_think_suffix_len, mx.new_stream, mx.default_device, mx.metal.is_available, mx.set_wired_limit, mx.device_info - State reads: self.is_vlm, self.language_model, self._normalize_chat_template_for_prefix_cache, self._compute_think_suffix_len - State writes: self.model, self.processor, self.mm_processor, self.max_kv_size, self.language_model, self.is_vlm, self.max_tokens, self.stop_tokens, self.sampler, self.prefill_batch_size, self.completion_batch_size, self.prefill_step_size, self.unprocessed_requests, self.active_batch, self.uid_counter, self._stats, self._pending_error_responses, self._prefill_progress, self._aborted_request_ids, self._pending_removal_uids, self._pending_removal_lock, self.vision_cache, self.prefix_cache, self._think_suffix_len, self._old_wired_limit ## `vllm_mlx.mllm_batch_generator.MLLMBatchGenerator._normalize_chat_template_for_prefix_cache` - Kind: method - Signature: `def _normalize_chat_template_for_prefix_cache(self) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L634-L697 - Implementation: Method `MLLMBatchGenerator._normalize_chat_template_for_prefix_cache` updates `self.processor.chat_template`; calls `getattr`, `re.sub`, `hasattr`, `logger.info`; returns `None`. Patch chat template so historical assistant turns are prefix-stable. Qwen3.5's chat template computes ``last_query_index`` — the position of the last non-tool-response user message — and conditionally wraps assistant turns after that index in ``...\n\n\n``. When a new user text message is appended, ``last_query_index`` jumps forward, retroactively removing these ```` wrappers from earlier assistant turns. This shifts tokens mid-sequence and breaks prefix cache. Fix: replace the conditional with the plain (ELSE) branch so ALL historical assistant messages use ``<|im_start|>assistant\ncontent`` without any injected ```` block. The generation prompt still adds ``\n`` at the very end, so the model generates thinking. - Inputs: none - Return annotation: `None` - Calls: getattr, re.sub, hasattr, logger.info, logger.debug - State reads: self.prefix_cache, self.processor - State writes: self.processor.chat_template - Return expressions: None ## `vllm_mlx.mllm_batch_generator.MLLMBatchGenerator._compute_think_suffix_len` - Kind: method - Signature: `def _compute_think_suffix_len(self) -> int` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L699-L758 - Implementation: Method `MLLMBatchGenerator._compute_think_suffix_len` calls `getattr`, `hasattr`, `applicator.apply_chat_template`, `text_with.endswith`; has 2 explicit return paths. Compute how many extra tokens enable_thinking=True adds at the END. Compares the generation prompt suffix with and without ``enable_thinking`` to find the think-tag suffix length (typically ``\n`` = 2 tokens for Qwen3/Qwen3.5). Returns 0 if the template doesn't support ``enable_thinking``. - Inputs: none - Return annotation: `int` - Calls: getattr, hasattr, applicator.apply_chat_template, text_with.endswith, text_without.endswith, tokenizer.encode, len, logger.info, tag.strip, max - State reads: self.processor - Return expressions: 0; max(0, suffix_len) ## `vllm_mlx.mllm_batch_generator.MLLMBatchGenerator.close` - Kind: method - Signature: `def close(self) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L760-L765 - Implementation: Method `MLLMBatchGenerator.close` updates `self._old_wired_limit`; calls `mx.synchronize`, `mx.set_wired_limit`. Release resources and reset wired limit. - Inputs: none - Return annotation: `None` - Calls: mx.synchronize, mx.set_wired_limit - State reads: self._old_wired_limit - State writes: self._old_wired_limit ## `vllm_mlx.mllm_batch_generator.MLLMBatchGenerator.abort_prefill` - Kind: method - Signature: `def abort_prefill(self, request_id: str) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L767-L775 - Implementation: Method `MLLMBatchGenerator.abort_prefill` calls `self._aborted_request_ids.add`, `logger.info`. Signal that a request's prefill should be aborted. Called from the event loop thread when a client disconnects. The prefill loop checks this set between chunks and raises PrefillAbortedError to exit early. - Inputs: - `request_id` (str; required): Required positional or keyword input. - Return annotation: `None` - Calls: self._aborted_request_ids.add, logger.info - State reads: self._aborted_request_ids.add, self._aborted_request_ids ## `vllm_mlx.mllm_batch_generator.MLLMBatchGenerator.schedule_removal` - Kind: method - Signature: `def schedule_removal(self, uids: List[int]) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L777-L789 - Implementation: Method `MLLMBatchGenerator.schedule_removal` calls `self._pending_removal_uids.update`. Thread-safe deferred removal of UIDs from the batch. Safe to call from any thread (typically the event loop during client-disconnect cleanup). The actual `remove()`, which creates ``mx.array`` instances and filters the KV cache, runs on the scheduler thread via :meth:`process_pending_removals` at the next batch boundary. This avoids the Metal ``encodeSignalEvent: uncommitted encoder`` crash that occurs when two threads submit GPU work on the same stream concurrently. - Inputs: - `uids` (List[int]; required): Required positional or keyword input. - Return annotation: `None` - Calls: self._pending_removal_uids.update - State reads: self._pending_removal_lock, self._pending_removal_uids.update, self._pending_removal_uids ## `vllm_mlx.mllm_batch_generator.MLLMBatchGenerator.process_pending_removals` - Kind: method - Signature: `def process_pending_removals(self) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L791-L808 - Implementation: Method `MLLMBatchGenerator.process_pending_removals` updates `self._pending_removal_uids`; calls `set`, `list`, `self.remove`; returns `None`. Remove any UIDs enqueued via :meth:`schedule_removal`. MUST be called from the scheduler thread only, at a safe point (e.g. the start of :meth:`MLLMScheduler.step` before any forward pass has been issued). Safe to call even when the queue is empty (no-op). - Inputs: none - Return annotation: `None` - Calls: set, list, self.remove - State reads: self._pending_removal_lock, self._pending_removal_uids, self.remove - State writes: self._pending_removal_uids - Return expressions: None ## `vllm_mlx.mllm_batch_generator.MLLMBatchGenerator.__del__` - Kind: method - Signature: `def __del__(self)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L810-L814 - Implementation: Method `MLLMBatchGenerator.__del__` calls `self.close`. Method `MLLMBatchGenerator.__del__` calls `self.close`. - Inputs: none - Return annotation: `not annotated` - Calls: self.close - State reads: self.close ## `vllm_mlx.mllm_batch_generator.MLLMBatchGenerator.insert` - Kind: method - Signature: `def insert(self, requests: List[MLLMBatchRequest]) -> List[int]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L816-L846 - Implementation: Method `MLLMBatchGenerator.insert` updates `self.uid_counter`, `self.unprocessed_requests`; calls `self.unprocessed_requests.append`, `uids.append`, `sorted`, `logger.debug`; returns `uids`. Insert requests for batch processing. Args: requests: List of MLLMBatchRequest to process Returns: List of UIDs assigned to requests - Inputs: - `requests` (List[MLLMBatchRequest]; required): List of MLLMBatchRequest to process - Return annotation: `List[int]` - Calls: self.unprocessed_requests.append, uids.append, sorted, logger.debug, len - State reads: self.uid_counter, self.unprocessed_requests.append, self.unprocessed_requests - State writes: self.uid_counter, self.unprocessed_requests - Return expressions: uids ## `vllm_mlx.mllm_batch_generator.MLLMBatchGenerator.remove` - Kind: method - Signature: `def remove(self, uids: List[int]) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L848-L870 - Implementation: Method `MLLMBatchGenerator.remove` updates `self.active_batch`, `self.unprocessed_requests`; calls `set`, `enumerate`, `self.active_batch.filter`. Remove requests from processing. Args: uids: List of UIDs to remove - Inputs: - `uids` (List[int]; required): List of UIDs to remove - Return annotation: `None` - Calls: set, enumerate, self.active_batch.filter - State reads: self.active_batch, self.active_batch.uids, self.active_batch.filter, self.unprocessed_requests - State writes: self.active_batch, self.unprocessed_requests ## `vllm_mlx.mllm_batch_generator.MLLMBatchGenerator._preprocess_request` - Kind: method - Signature: `def _preprocess_request(self, request: MLLMBatchRequest) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L872-L1023 - Implementation: Method `MLLMBatchGenerator._preprocess_request` updates `self._stats.num_images_processed`, `self._stats.vision_encoding_time`; calls `time.perf_counter`, `process_image_input`, `all_images.append`, `logger.warning`; returns `None`. Preprocess a single MLLM request (vision encoding). This prepares the inputs by: 1. Processing images/videos through the processor 2. Tokenizing the prompt with image tokens 3. Running vision encoder to get features Uses vision cache to skip processing for repeated images. Idempotent: if input_ids is already set, returns immediately. Args: request: Request to preprocess - Inputs: - `request` (MLLMBatchRequest; required): Request to preprocess - Return annotation: `None` - Calls: time.perf_counter, process_image_input, all_images.append, logger.warning, process_video_input, extract_video_frames_smart, save_frames_to_temp, all_images.extend, process_audio_input, all_audio.append, self.vision_cache.get_pixel_cache, dict, logger.debug, getattr, prepare_inputs, inputs.get, inputs.items, request.extra_kwargs.pop, self.vision_cache.set_pixel_cache, len, bool - State reads: self.vision_cache.get_pixel_cache, self.vision_cache, self.model, self.processor, self.vision_cache.set_pixel_cache, self._stats - State writes: self._stats.num_images_processed, self._stats.vision_encoding_time - Return expressions: None ## `vllm_mlx.mllm_batch_generator.MLLMBatchGenerator._copy_prefix_cache` - Kind: method - Signature: `def _copy_prefix_cache(cache_list)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L1026-L1054 - Implementation: Method `MLLMBatchGenerator._copy_prefix_cache` calls `isinstance`, `RotatingKVCache`, `copies.append`, `KVCache`; returns `copies`. Create shallow copies of cache objects to prevent mutation of stored prefix cache. MLX arrays are immutable and safe to share, but cache objects have mutable Python attributes (offset, _idx) that get modified by update_and_fetch(). Without copying, the stored prefix cache entry is corrupted after each use. - Inputs: - `cache_list` (not annotated; required): Required positional or keyword input. - Return annotation: `not annotated` - Decorators: staticmethod - Calls: isinstance, RotatingKVCache, copies.append, KVCache - Return expressions: copies ## `vllm_mlx.mllm_batch_generator.MLLMBatchGenerator._has_empty_rotating_cache` - Kind: method - Signature: `def _has_empty_rotating_cache(cache_list)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L1057-L1069 - Implementation: Method `MLLMBatchGenerator._has_empty_rotating_cache` calls `isinstance`; has 2 explicit return paths. Check if any RotatingKVCache layer has no data (keys=None). This happens when prefix cache stores a long response where all sliding-window entries were trimmed (entries_to_keep=0). Using such a cache produces garbage — fall through to full prefill. - Inputs: - `cache_list` (not annotated; required): Required positional or keyword input. - Return annotation: `not annotated` - Decorators: staticmethod - Calls: isinstance - Return expressions: True; False ## `vllm_mlx.mllm_batch_generator.MLLMBatchGenerator._trim_rotating_caches` - Kind: method - Signature: `def _trim_rotating_caches(cache_list)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L1072-L1106 - Implementation: Method `MLLMBatchGenerator._trim_rotating_caches` calls `isinstance`, `layer_cache._trim`, `min`, `logger.warning`. Trim RotatingKVCache buffers restored from prefix cache. Prefix cache stores the full KV state (offset may exceed max_size for sliding-window layers). RotatingKVCache._update_in_place computes ``new_size = min(step, max_size - prev)`` which goes negative when ``prev > max_size``, crashing with "Negative dimensions not allowed". Trimming the buffer to max_size and clamping offset/idx prevents this. - Inputs: - `cache_list` (not annotated; required): Required positional or keyword input. - Return annotation: `not annotated` - Decorators: staticmethod - Calls: isinstance, layer_cache._trim, min, logger.warning ## `vllm_mlx.mllm_batch_generator.MLLMBatchGenerator._run_chunked_text_prefill` - Kind: method - Signature: `def _run_chunked_text_prefill(self, request: MLLMBatchRequest, cache: List[Any]) -> mx.array` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L1108-L1206 - Implementation: Method `MLLMBatchGenerator._run_chunked_text_prefill` calls `self.language_model`, `request.extra_kwargs.clear`, `hasattr`, `logger.info`; can raise `PrefillAbortedError`; has 2 explicit return paths. Run prefill in chunks for text-only requests, reporting real progress. Processes input_ids in prefill_step_size chunks through the language model, updating ``_prefill_progress`` after each chunk so the status endpoint can report accurate prefill percentage. Returns: Logits from the last chunk (same contract as _run_vision_encoding). - Inputs: - `request` (MLLMBatchRequest; required): Required positional or keyword input. - `cache` (List[Any]; required): Required positional or keyword input. - Return annotation: `mx.array` - Calls: self.language_model, request.extra_kwargs.clear, hasattr, logger.info, self._aborted_request_ids.discard, PrefillAbortedError, _eval_prompt_cache, mx.clear_cache - State reads: self.prefill_step_size, self._prefill_progress, self.language_model, self._aborted_request_ids, self._aborted_request_ids.discard - Raises directly: PrefillAbortedError - Return expressions: output.logits; output ## `vllm_mlx.mllm_batch_generator.MLLMBatchGenerator._run_vision_encoding` - Kind: method - Signature: `def _run_vision_encoding(self, request: MLLMBatchRequest, cache: Optional[List[Any]]=None) -> mx.array` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L1208-L1258 - Implementation: Method `MLLMBatchGenerator._run_vision_encoding` calls `dict`, `self.model`, `request.extra_kwargs.clear`, `hasattr`; has 2 explicit return paths. Run the initial VLM forward pass to encode vision and get first logits. This runs the full VLM model (vision + language) on the prompt, which encodes the images and fills the provided KV cache. Args: request: Preprocessed request with input_ids and pixel_values cache: KV cache list for the language model. If provided, the language model writes its KV state directly into this cache during the forward pass. Returns: Logits from the forward pass - Inputs: - `request` (MLLMBatchRequest; required): Preprocessed request with input_ids and pixel_values - `cache` (Optional[List[Any]]; optional; default `None`): KV cache list for the language model. If provided, the language model writes its KV state directly into this cache during the forward pass. - Return annotation: `mx.array` - Calls: dict, self.model, request.extra_kwargs.clear, hasattr - State reads: self.model - Return expressions: output.logits; output ## `vllm_mlx.mllm_batch_generator.MLLMBatchGenerator._process_prompts` - Kind: method - Signature: `def _process_prompts(self, requests: List[MLLMBatchRequest]) -> MLLMBatch` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L1260-L1682 - Implementation: Method `MLLMBatchGenerator._process_prompts` updates `self._stats.prompt_tokens`, `self._stats.prompt_time`; calls `time.perf_counter`, `self._preprocess_request`, `logger.error`, `type`; can raise `PrefillAbortedError`; has 2 explicit return paths. Process a batch of requests through vision encoding and initial prefill. For MLLM, this is more complex than LLM: 1. Preprocess each request (tokenize, process images) 2. Run vision encoding per-request with individual KVCache objects 3. Merge individual caches into a BatchKVCache for generation Args: requests: Requests to process Returns: MLLMBatch ready for generation - Inputs: - `requests` (List[MLLMBatchRequest]; required): Requests to process - Return annotation: `MLLMBatch` - Calls: time.perf_counter, self._preprocess_request, logger.error, type, failed_requests.append, requests.remove, self._pending_error_responses.append, MLLMBatchResponse, mx.zeros, combined.extend, make_logits_processors, logger.info, len, make_sampler, sum, logger.warning, self._aborted_request_ids.discard, PrefillAbortedError, req.input_ids.reshape(-1).tolist, req.input_ids.reshape, self.prefix_cache.fetch, list, getattr, self._has_empty_rotating_cache, self._copy_prefix_cache, self._trim_rotating_caches, mx.array, mx.stream, self.language_model, _eval_prompt_cache, mx.clear_cache, hasattr, _sample_first_token, first_tokens.append, sampled.item, all_logprobs.append, logprobs.squeeze, per_request_caches.append, logger.debug, _trim_cache_offset, make_prompt_cache, self._run_chunked_text_prefill, self._run_vision_encoding, aborted_requests.append, self._prefill_progress.pop, isinstance, layer_cache._temporal_order, per_request_caches[0][layer_idx].merge, range, logits_processors_by_request.get, any, samplers_by_request.get, req.extra_kwargs.clear, MLLMBatch - State reads: self._preprocess_request, self._pending_error_responses.append, self._pending_error_responses, self._stats, self.prefill_step_size, self._aborted_request_ids, self._aborted_request_ids.discard, self.prefix_cache, self._think_suffix_len, self.prefix_cache.fetch, self.model, self._has_empty_rotating_cache, self._copy_prefix_cache, self._trim_rotating_caches, self._prefill_progress, self.language_model, self.max_kv_size, self._run_chunked_text_prefill, self._run_vision_encoding, self._prefill_progress.pop - State writes: self._stats.prompt_tokens, self._stats.prompt_time - Raises directly: PrefillAbortedError - Return expressions: None; MLLMBatch(uids=[req.uid for req in requests], request_ids=[req.request_id for req in requests], y=y, logprobs=all_logpr… ## `vllm_mlx.mllm_batch_generator.MLLMBatchGenerator._process_prompts._sample_first_token` - Kind: nested function - Signature: `def _sample_first_token(req: MLLMBatchRequest, logits: mx.array)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L1348-L1362 - Implementation: Nested Function `MLLMBatchGenerator._process_prompts._sample_first_token` calls `logits_processors_by_request.get`, `mx.array`, `processor`, `mx.logsumexp`; returns `(sampled, logprobs)`. Nested Function `MLLMBatchGenerator._process_prompts._sample_first_token` calls `logits_processors_by_request.get`, `mx.array`, `processor`, `mx.logsumexp`; returns `(sampled, logprobs)`. - Inputs: - `req` (MLLMBatchRequest; required): Required positional or keyword input. - `logits` (mx.array; required): Required positional or keyword input. - Return annotation: `not annotated` - Calls: logits_processors_by_request.get, mx.array, processor, mx.logsumexp, samplers_by_request.get, sampler, mx.eval - State reads: self.sampler - Return expressions: (sampled, logprobs) ## `vllm_mlx.mllm_batch_generator.MLLMBatchGenerator._step` - Kind: method - Signature: `def _step(self, input_tokens: mx.array, cache: List[Any], logits_processors: Optional[List[Optional[List[Callable]]]]=None, output_tokens: Optional[List[List[int]]]=None, samplers: Optional[List[Optional[Callable]]]=None) -> Tuple[mx.array, List[mx.array]]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L1684-L1746 - Implementation: Method `MLLMBatchGenerator._step` calls `self.language_model`, `hasattr`, `any`, `range`; returns `(sampled, list(logprobs))`. Run one generation step through the language model. Args: input_tokens: Input tokens [batch_size, 1] or [batch_size] cache: BatchKVCache for the language model logits_processors: Per-request logits processors (e.g. repetition penalty) output_tokens: Per-request generated tokens so far (needed by processors) samplers: Per-request sampler functions (for top_k/min_p) Returns: Tuple of (sampled tokens, logprobs list) - Inputs: - `input_tokens` (mx.array; required): Input tokens [batch_size, 1] or [batch_size] - `cache` (List[Any]; required): BatchKVCache for the language model - `logits_processors` (Optional[List[Optional[List[Callable]]]]; optional; default `None`): Per-request logits processors (e.g. repetition penalty) - `output_tokens` (Optional[List[List[int]]]; optional; default `None`): Per-request generated tokens so far (needed by processors) - `samplers` (Optional[List[Optional[Callable]]]; optional; default `None`): Per-request sampler functions (for top_k/min_p) - Return annotation: `Tuple[mx.array, List[mx.array]]` - Calls: self.language_model, hasattr, any, range, processor, mx.array, processed_logits.append, mx.concatenate, mx.logsumexp, sampled_list.append, s, self.sampler, list - State reads: self.language_model, self.sampler - Return expressions: (sampled, list(logprobs)) ## `vllm_mlx.mllm_batch_generator.MLLMBatchGenerator._next` - Kind: method - Signature: `def _next(self) -> List[MLLMBatchResponse]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L1748-L1964 - Implementation: Method `MLLMBatchGenerator._next` updates `self.active_batch`, `self.unprocessed_requests`, `self._stats.prompt_time`, `self._stats.generation_time`; calls `time.perf_counter`, `len`, `self._process_prompts`, `logger.error`; has 3 explicit return paths. Internal next() implementation. Returns: List of MLLMBatchResponse for this step - Inputs: none - Return annotation: `List[MLLMBatchResponse]` - Calls: time.perf_counter, len, self._process_prompts, logger.error, type, self._pending_error_responses.append, MLLMBatchResponse, mx.zeros, batch.extend, logger.warning, list, self._pending_error_responses.clear, y.tolist, zip, self._step, mx.async_eval, enumerate, req.output_tokens.append, _processors_can_retire, _drop_retired_processors, logger.info, end_idx.append, keep_idx.append, self._prefill_progress.pop, responses.append, self._maybe_store_prefix_cache, batch.filter - State reads: self.active_batch, self.unprocessed_requests, self.completion_batch_size, self._process_prompts, self._pending_error_responses.append, self._pending_error_responses, self._pending_error_responses.clear, self._step, self._stats, self.stop_tokens, self._prefill_progress.pop, self._prefill_progress, self._maybe_store_prefix_cache - State writes: self.active_batch, self.unprocessed_requests, self._stats.prompt_time, self._stats.generation_time, self._stats.generation_tokens - Return expressions: []; error_responses; error_responses + responses ## `vllm_mlx.mllm_batch_generator.MLLMBatchGenerator.next` - Kind: method - Signature: `def next(self) -> List[MLLMBatchResponse]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L1966-L1974 - Implementation: Method `MLLMBatchGenerator.next` calls `mx.stream`, `self._next`; returns `self._next()`. Generate next token for all requests in the batch. Returns: List of MLLMBatchResponse, one per active request - Inputs: none - Return annotation: `List[MLLMBatchResponse]` - Calls: mx.stream, self._next - State reads: self._next - Return expressions: self._next() ## `vllm_mlx.mllm_batch_generator.MLLMBatchGenerator.stats` - Kind: method - Signature: `def stats(self) -> MLLMBatchStats` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L1976-L1984 - Implementation: Method `MLLMBatchGenerator.stats` updates `self._stats.peak_memory`; calls `mx.get_peak_memory`; returns `self._stats`. Get generation statistics. Returns: MLLMBatchStats with timing and token counts - Inputs: none - Return annotation: `MLLMBatchStats` - Calls: mx.get_peak_memory - State reads: self._stats - State writes: self._stats.peak_memory - Return expressions: self._stats ## `vllm_mlx.mllm_batch_generator.MLLMBatchGenerator._maybe_store_prefix_cache` - Kind: method - Signature: `def _maybe_store_prefix_cache(self, batch: MLLMBatch, end_indices: List[int]) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L1986-L2014 - Implementation: Method `MLLMBatchGenerator._maybe_store_prefix_cache` calls `batch.extract_cache`, `req.input_ids.reshape(-1).tolist`, `req.input_ids.reshape`, `_trim_cache_offset`; returns `None`. Store KV caches for finished text-only requests into prefix cache. Must be called BEFORE batch.filter() so that indices are still valid. - Inputs: - `batch` (MLLMBatch; required): Required positional or keyword input. - `end_indices` (List[int]; required): Required positional or keyword input. - Return annotation: `None` - Calls: batch.extract_cache, req.input_ids.reshape(-1).tolist, req.input_ids.reshape, _trim_cache_offset, self.prefix_cache.store, logger.warning, type - State reads: self.prefix_cache, self._think_suffix_len, self.prefix_cache.store - Return expressions: None ## `vllm_mlx.mllm_batch_generator.MLLMBatchGenerator.get_prefill_progress` - Kind: method - Signature: `def get_prefill_progress(self, request_id: str) -> Optional[Tuple[int, int]]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L2016-L2018 - Implementation: Method `MLLMBatchGenerator.get_prefill_progress` calls `self._prefill_progress.get`; returns `self._prefill_progress.get(request_id)`. Return (processed_tokens, total_tokens) or None. - Inputs: - `request_id` (str; required): Required positional or keyword input. - Return annotation: `Optional[Tuple[int, int]]` - Calls: self._prefill_progress.get - State reads: self._prefill_progress.get, self._prefill_progress - Return expressions: self._prefill_progress.get(request_id) ## `vllm_mlx.mllm_batch_generator.MLLMBatchGenerator.get_vision_cache_stats` - Kind: method - Signature: `def get_vision_cache_stats(self) -> Dict[str, Any]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L2020-L2022 - Implementation: Method `MLLMBatchGenerator.get_vision_cache_stats` calls `self.vision_cache.get_stats`; returns `self.vision_cache.get_stats()`. Get vision cache statistics. - Inputs: none - Return annotation: `Dict[str, Any]` - Calls: self.vision_cache.get_stats - State reads: self.vision_cache.get_stats, self.vision_cache - Return expressions: self.vision_cache.get_stats() ## `vllm_mlx.mllm_batch_generator.MLLMBatchGenerator.get_prefix_cache_stats` - Kind: method - Signature: `def get_prefix_cache_stats(self) -> Dict[str, Any]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L2024-L2038 - Implementation: Method `MLLMBatchGenerator.get_prefix_cache_stats` calls `self.prefix_cache.get_stats`; has 2 explicit return paths. Get KV prefix cache statistics. - Inputs: none - Return annotation: `Dict[str, Any]` - Calls: self.prefix_cache.get_stats - State reads: self.prefix_cache, self.prefix_cache.get_stats - Return expressions: self.prefix_cache.get_stats(); {'hits': 0, 'misses': 0, 'hit_rate': 0.0, 'evictions': 0, 'tokens_saved': 0, 'current_memory_mb': 0.0, 'max_memory_mb':… ## `vllm_mlx.mllm_batch_generator.MLLMBatchGenerator.has_pending` - Kind: method - Signature: `def has_pending(self) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L2040-L2042 - Implementation: Method `MLLMBatchGenerator.has_pending` calls `bool`; returns `bool(self.unprocessed_requests or self.active_batch)`. Check if there are pending or active requests. - Inputs: none - Return annotation: `bool` - Calls: bool - State reads: self.unprocessed_requests, self.active_batch - Return expressions: bool(self.unprocessed_requests or self.active_batch) ## `vllm_mlx.mllm_batch_generator.install_mtp_mllm` - Kind: function - Signature: `def install_mtp_mllm(batch_gen: 'MLLMBatchGenerator', language_model: Any, num_draft_tokens: int=1) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L2045-L2590 - Implementation: Function `install_mtp_mllm` calls `make_sampler`, `threading.Lock`, `logger.warning`, `logger.info`. Install MTP (Multi-Token Prediction) on an MLLMBatchGenerator. Adapts the always-advance MTP strategy from scheduler._install_mtp for the MLLM batched generation path. Handles hybrid model caches (BatchKVCache for attention + ArraysCache for recurrent layers). Flow per generation step: 1. Use skip_state logits/hidden OR run model forward -> sample primary 2. MTP head drafts one token 3. Verify [primary, draft] in one model call (always advances cache) 4. Accept: skip_state from pos 1, defer draft for next step emission Reject: trim KV by 2 + restore RNN state + re-advance with primary 5. Draft is emitted in the NEXT generation step after primary - Inputs: - `batch_gen` ('MLLMBatchGenerator'; required): Required positional or keyword input. - `language_model` (Any; required): Required positional or keyword input. - `num_draft_tokens` (int; optional; default `1`): Optional positional or keyword input; defaults to `1`. - Return annotation: `None` - Calls: make_sampler, threading.Lock, logger.warning, logger.info ## `vllm_mlx.mllm_batch_generator.install_mtp_mllm._get_mtp_stats` - Kind: nested function - Signature: `def _get_mtp_stats() -> Dict[str, Any]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L2089-L2110 - Implementation: Nested Function `install_mtp_mllm._get_mtp_stats` calls `dict`; returns `{'enabled': True, 'requested_draft_tokens': num_draft_tokens, 'effective_draft_tokens': 1, 'mode': 'request_local_sampl…`. Nested Function `install_mtp_mllm._get_mtp_stats` calls `dict`; returns `{'enabled': True, 'requested_draft_tokens': num_draft_tokens, 'effective_draft_tokens': 1, 'mode': 'request_local_sampl…`. - Inputs: none - Return annotation: `Dict[str, Any]` - Calls: dict - Return expressions: {'enabled': True, 'requested_draft_tokens': num_draft_tokens, 'effective_draft_tokens': 1, 'mode': 'request_local_sampl… ## `vllm_mlx.mllm_batch_generator.install_mtp_mllm._mtp_step` - Kind: nested function - Signature: `def _mtp_step(input_tokens: mx.array, cache: List[Any], logits_processors: Optional[List[Optional[List[Callable]]]]=None, output_tokens: Optional[List[List[int]]]=None, samplers: Optional[List[Optional[Callable]]]=None) -> Tuple[mx.array, List[mx.array]]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L2114-L2455 - Implementation: Nested Function `install_mtp_mllm._mtp_step` calls `list`, `any`, `_skip_state_by_uid.clear`, `_orig_step`; has 2 explicit return paths. Extended _step with MTP always-advance strategy. - Inputs: - `input_tokens` (mx.array; required): Required positional or keyword input. - `cache` (List[Any]; required): Required positional or keyword input. - `logits_processors` (Optional[List[Optional[List[Callable]]]]; optional; default `None`): Optional positional or keyword input; defaults to `None`. - `output_tokens` (Optional[List[List[int]]]; optional; default `None`): Optional positional or keyword input; defaults to `None`. - `samplers` (Optional[List[Optional[Callable]]]; optional; default `None`): Optional positional or keyword input; defaults to `None`. - Return annotation: `Tuple[mx.array, List[mx.array]]` - Calls: list, any, _skip_state_by_uid.clear, _orig_step, _skip_state_by_uid.pop, all, logger.debug, mx.concatenate, language_model, isinstance, range, processor, mx.array, processed_logits.append, mx.logsumexp, sampled_list.append, s, batch_gen.sampler, language_model.mtp_forward, _request_uses_stochastic_sampling, _sampling_logprobs, enumerate, mx.random.categorical, _draft_sampler, hasattr, _c.is_trimmable, draft_tokens.tolist, mx.random.uniform, mx.eval, int, _accept_sampled_draft, float, verify_distribution[row, draft_token].item, draft_distribution[row, draft_token].item, draws[row].item, _residual_logprobs, residual_token.item, mx.argmax, verify_pred.tolist, mx.async_eval, bool, c.is_trimmable, c.trim, _rnn_snapshots.items, _deferred_drafts.pop, logger.warning, logger.info - Return expressions: _orig_step(input_tokens, cache, logits_processors, output_tokens, samplers); (primary_tokens, list(logprobs)) ## `vllm_mlx.mllm_batch_generator.install_mtp_mllm._mtp_next` - Kind: nested function - Signature: `def _mtp_next() -> List[MLLMBatchResponse]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L2460-L2576 - Implementation: Nested Function `install_mtp_mllm._mtp_next` calls `_skip_state_by_uid.clear`, `_deferred_drafts.clear`, `_attempted_drafts_by_uid.clear`, `_deferred_drafts.pop`; returns `augmented`. Wrapper around _next that emits deferred MTP draft tokens. - Inputs: none - Return annotation: `List[MLLMBatchResponse]` - Calls: _skip_state_by_uid.clear, _deferred_drafts.clear, _attempted_drafts_by_uid.clear, _deferred_drafts.pop, batch_gen._inner_next, _mark_mtp_attempts_on_primary_responses, set, augmented.append, _skip_state_by_uid.pop, prev_deferred.pop, MLLMBatchResponse, draft_end_uids.add, enumerate, batch.requests[e].output_tokens.append, batch_gen._maybe_store_prefix_cache, batch_gen.active_batch.filter, list - Return expressions: augmented ## `vllm_mlx.mllm_batch_generator.install_chunked_prefill_mllm` - Kind: function - Signature: `def install_chunked_prefill_mllm(batch_gen: 'MLLMBatchGenerator', budget: int=1024) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L2593-L3073 - Implementation: Function `install_chunked_prefill_mllm` calls `logger.info`. Install interleaved prefill/decode on an MLLMBatchGenerator. When a long text-only request arrives, instead of blocking the entire event loop for 20-60+ seconds during prefill, this processes ONE chunk of the new request's prefill per ``step()`` call. Between steps the scheduler yields to the event loop (``await asyncio.sleep(0)``), so health/status/metrics endpoints remain responsive. When an active batch is generating, prefill chunks are interleaved with generation steps to keep throughput for existing requests at 30-50 tok/s. Args: batch_gen: The MLLMBatchGenerator to patch. budget: Max tokens to prefill per step (chunk size). - Inputs: - `batch_gen` ('MLLMBatchGenerator'; required): The MLLMBatchGenerator to patch. - `budget` (int; optional; default `1024`): Max tokens to prefill per step (chunk size). - Return annotation: `None` - Calls: logger.info ## `vllm_mlx.mllm_batch_generator.install_chunked_prefill_mllm._generation_step` - Kind: nested function - Signature: `def _generation_step() -> List[MLLMBatchResponse]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L2623-L2713 - Implementation: Nested Function `install_chunked_prefill_mllm._generation_step` calls `list`, `batch_gen._pending_error_responses.clear`, `time.perf_counter`, `batch_gen._step`; has 2 explicit return paths. Run one generation step for the active batch. Returns responses. - Inputs: none - Return annotation: `List[MLLMBatchResponse]` - Calls: list, batch_gen._pending_error_responses.clear, time.perf_counter, batch_gen._step, mx.eval, y.tolist, enumerate, zip, req.output_tokens.append, end_idx.append, keep_idx.append, batch_gen._prefill_progress.pop, responses.append, MLLMBatchResponse, batch_gen._maybe_store_prefix_cache, batch.filter, len - Return expressions: error_responses; error_responses + responses ## `vllm_mlx.mllm_batch_generator.install_chunked_prefill_mllm._chunked_next` - Kind: nested function - Signature: `def _chunked_next() -> List[MLLMBatchResponse]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L2715-L3058 - Implementation: Nested Function `install_chunked_prefill_mllm._chunked_next` calls `batch_gen._aborted_request_ids.discard`, `mx.clear_cache`, `batch_gen._prefill_progress.pop`, `batch_gen._pending_error_responses.append`; has 3 explicit return paths. Interleaved prefill/decode: one prefill chunk + one gen step. - Inputs: none - Return annotation: `List[MLLMBatchResponse]` - Calls: batch_gen._aborted_request_ids.discard, mx.clear_cache, batch_gen._prefill_progress.pop, batch_gen._pending_error_responses.append, MLLMBatchResponse, mx.zeros, _generation_step, time.perf_counter, batch_gen.language_model, _eval_prompt_cache, batch_gen._preprocess_request, short_reqs.append, batch_gen._process_prompts, batch_gen.active_batch.extend, logger.warning, hasattr, getattr, mx.array, processor, mx.logsumexp, batch_gen.sampler, mx.eval, req_lp.extend, make_logits_processors, make_sampler, MLLMBatch, logprobs.squeeze, batch_gen._trim_rotating_caches, isinstance, layer_cache._temporal_order, request_cache[layer_idx].merge, range, len, req.input_ids.reshape(-1).tolist, req.input_ids.reshape, _trim_cache_offset, batch_gen.prefix_cache.store, logger.info, logger.error, batch_gen.unprocessed_requests.remove, input_ids.reshape(-1).tolist, input_ids.reshape, batch_gen.prefix_cache.fetch, list, batch_gen._has_empty_rotating_cache, batch_gen._copy_prefix_cache, make_prompt_cache, _orig_next - Return expressions: _generation_step(); []; _orig_next() ## `vllm_mlx.mllm_batch_generator.install_chunked_prefill_mllm._patched_remove` - Kind: nested function - Signature: `def _patched_remove(uids: List[int]) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L3063-L3068 - Implementation: Nested Function `install_chunked_prefill_mllm._patched_remove` calls `set`, `mx.clear_cache`, `_orig_remove`. Nested Function `install_chunked_prefill_mllm._patched_remove` calls `set`, `mx.clear_cache`, `_orig_remove`. - Inputs: - `uids` (List[int]; required): Required positional or keyword input. - Return annotation: `None` - Calls: set, mx.clear_cache, _orig_remove # Module `vllm_mlx.mllm_cache` MLLM (Multimodal Language Model) Prefix Cache Manager. This module provides advanced caching for MLLM inference, implementing the LMCache-style approach for multimodal prefix caching: Features: - Image content hashing for cache keys (LMCache style) - Vision embedding caching (skip encoder on hit) - KV cache state caching with prefix matching - Token ID tracking for partial prefix reuse - LRU eviction policy with memory limits - Stats tracking (hits, misses, tokens saved, encoder skips) Based on research from: - LMCache: https://blog.lmcache.ai/2025-07-03-multimodal-models/ - vLLM Prefix Caching: https://docs.vllm.ai/en/stable/design/prefix_caching/ - mlx-lm cache_prompt: https://github.com/ml-explore/mlx-lm Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_cache.py#L1-L459 ## `vllm_mlx.mllm_cache.MLLMCacheStats` - Kind: class - Signature: `class MLLMCacheStats` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_cache.py#L34-L65 - Implementation: Class `MLLMCacheStats` declares 2 direct member(s). Statistics for MLLM cache performance. - Inputs: - `hits` (int; optional; default `0`): Optional constructor field; defaults to `0`. - `misses` (int; optional; default `0`): Optional constructor field; defaults to `0`. - `partial_hits` (int; optional; default `0`): Optional constructor field; defaults to `0`. - `tokens_saved` (int; optional; default `0`): Optional constructor field; defaults to `0`. - `image_cache_hits` (int; optional; default `0`): Optional constructor field; defaults to `0`. - `vision_encoder_skips` (int; optional; default `0`): Optional constructor field; defaults to `0`. - `total_queries` (int; optional; default `0`): Optional constructor field; defaults to `0`. - `evictions` (int; optional; default `0`): Optional constructor field; defaults to `0`. - Constructs: `vllm_mlx.mllm_cache.MLLMCacheStats` - Decorators: dataclass ## `vllm_mlx.mllm_cache.MLLMCacheStats.hit_rate` - Kind: method - Signature: `def hit_rate(self) -> float` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_cache.py#L47-L51 - Implementation: Method `MLLMCacheStats.hit_rate` has 2 explicit return paths. Calculate cache hit rate. - Inputs: none - Return annotation: `float` - Decorators: property - State reads: self.total_queries, self.hits - Return expressions: 0.0; self.hits / self.total_queries ## `vllm_mlx.mllm_cache.MLLMCacheStats.to_dict` - Kind: method - Signature: `def to_dict(self) -> dict` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_cache.py#L53-L65 - Implementation: Method `MLLMCacheStats.to_dict` returns `{'hits': self.hits, 'misses': self.misses, 'partial_hits': self.partial_hits, 'hit_rate': self.hit_rate, 'tokens_saved'…`. Convert stats to dictionary. - Inputs: none - Return annotation: `dict` - State reads: self.hits, self.misses, self.partial_hits, self.hit_rate, self.tokens_saved, self.image_cache_hits, self.vision_encoder_skips, self.total_queries, self.evictions - Return expressions: {'hits': self.hits, 'misses': self.misses, 'partial_hits': self.partial_hits, 'hit_rate': self.hit_rate, 'tokens_saved'… ## `vllm_mlx.mllm_cache.MLLMPrefixCacheEntry` - Kind: class - Signature: `class MLLMPrefixCacheEntry` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_cache.py#L69-L133 - Implementation: Class `MLLMPrefixCacheEntry` declares 3 direct member(s). Enhanced cache entry storing vision embeddings, KV cache, and token IDs. This enables: 1. Skipping vision encoder on image cache hit (saves ~1-2s per image) 2. Skipping prefix computation on token match (saves ~0.5s per 1k tokens) 3. Partial prefix reuse for multi-turn conversations - Inputs: - `image_hash` (str; required): Required constructor field. - `prompt_hash` (str; required): Required constructor field. - `vision_embeddings` (Any; optional; default `None`): Optional constructor field; defaults to `None`. - `kv_cache` (list[Any]; optional; default `field(default_factory=list)`): Optional constructor field; defaults to `field(default_factory=list)`. - `token_ids` (list[int]; optional; default `field(default_factory=list)`): Optional constructor field; defaults to `field(default_factory=list)`. - `num_image_tokens` (int; optional; default `0`): Optional constructor field; defaults to `0`. - `num_text_tokens` (int; optional; default `0`): Optional constructor field; defaults to `0`. - `prompt_tokens` (int; optional; default `0`): Optional constructor field; defaults to `0`. - `created_at` (float; optional; default `field(default_factory=time.time)`): Optional constructor field; defaults to `field(default_factory=time.time)`. - `hit_count` (int; optional; default `0`): Optional constructor field; defaults to `0`. - `model_name` (str; optional; default `''`): Optional constructor field; defaults to `''`. - Constructs: `vllm_mlx.mllm_cache.MLLMPrefixCacheEntry` - Decorators: dataclass ## `vllm_mlx.mllm_cache.MLLMPrefixCacheEntry.total_tokens` - Kind: method - Signature: `def total_tokens(self) -> int` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_cache.py#L99-L102 - Implementation: Method `MLLMPrefixCacheEntry.total_tokens` calls `len`; returns `len(self.token_ids)`. Return the number of token IDs represented by this cache entry. - Inputs: none - Return annotation: `int` - Decorators: property - Calls: len - State reads: self.token_ids - Return expressions: len(self.token_ids) ## `vllm_mlx.mllm_cache.MLLMPrefixCacheEntry.memory_size` - Kind: method - Signature: `def memory_size(self) -> int` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_cache.py#L105-L119 - Implementation: Method `MLLMPrefixCacheEntry.memory_size` calls `hasattr`; returns `size`. Estimate memory usage in bytes. - Inputs: none - Return annotation: `int` - Decorators: property - Calls: hasattr - State reads: self.vision_embeddings, self.vision_embeddings.nbytes, self.kv_cache - Return expressions: size ## `vllm_mlx.mllm_cache.MLLMPrefixCacheEntry.get_prefix_match_length` - Kind: method - Signature: `def get_prefix_match_length(self, new_token_ids: list[int]) -> int` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_cache.py#L121-L133 - Implementation: Method `MLLMPrefixCacheEntry.get_prefix_match_length` calls `enumerate`, `zip`; returns `match_length`. Find how many tokens match between cached prefix and new input. This is the key to prefix caching - if the first N tokens match, we can skip computing KV states for those N tokens. - Inputs: - `new_token_ids` (list[int]; required): Required positional or keyword input. - Return annotation: `int` - Calls: enumerate, zip - State reads: self.token_ids - Return expressions: match_length ## `vllm_mlx.mllm_cache.compute_image_hash` - Kind: function - Signature: `def compute_image_hash(image_path: str) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_cache.py#L136-L161 - Implementation: Function `compute_image_hash` calls `Path`, `path.exists`, `path.read_bytes`, `hashlib.sha256(content).hexdigest`; has 3 explicit return paths. Compute hash of image content for cache key. Following LMCache approach: hash the actual image bytes, not the path. This ensures cache hits even when the same image is loaded from different paths or as base64. Args: image_path: Path to image file Returns: SHA256 hash of image content (first 16 chars) - Inputs: - `image_path` (str; required): Path to image file - Return annotation: `str` - Calls: Path, path.exists, path.read_bytes, hashlib.sha256(content).hexdigest, hashlib.sha256, hashlib.sha256(image_path.encode()).hexdigest, image_path.encode, logger.warning, hashlib.sha256(str(image_path).encode()).hexdigest, str(image_path).encode, str - Return expressions: hashlib.sha256(content).hexdigest()[:16]; hashlib.sha256(image_path.encode()).hexdigest()[:16]; hashlib.sha256(str(image_path).encode()).hexdigest()[:16] ## `vllm_mlx.mllm_cache.compute_images_hash` - Kind: function - Signature: `def compute_images_hash(images: list[str]) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_cache.py#L164-L179 - Implementation: Function `compute_images_hash` calls `compute_image_hash`, `'_'.join`, `sorted`, `hashlib.sha256(combined.encode()).hexdigest`; has 2 explicit return paths. Compute combined hash for multiple images. Args: images: List of image paths/URLs Returns: Combined hash string - Inputs: - `images` (list[str]; required): List of image paths/URLs - Return annotation: `str` - Calls: compute_image_hash, '_'.join, sorted, hashlib.sha256(combined.encode()).hexdigest, hashlib.sha256, combined.encode - Return expressions: 'no_images'; hashlib.sha256(combined.encode()).hexdigest()[:16] ## `vllm_mlx.mllm_cache.MLLMPrefixCacheManager` - Kind: class - Signature: `class MLLMPrefixCacheManager` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_cache.py#L182-L448 - Implementation: Class `MLLMPrefixCacheManager` declares 14 direct member(s). LRU Cache manager for MLLM prefix states with vision embedding caching. Implements the LMCache approach for multimodal caching: 1. Hash-based identification of image+prompt combinations 2. Vision embedding caching (skip encoder on hit - saves 1-2s!) 3. KV cache reuse for matching prefixes 4. Token ID tracking for partial prefix matching 5. Memory-based eviction (configurable limit) Example: >>> cache = MLLMPrefixCacheManager(max_memory_mb=2048) >>> # First request - cache miss, full computation >>> entry, match_len = cache.fetch(["image.jpg"], prompt, token_ids) >>> # ... run full forward pass ... >>> cache.store(["image.jpg"], prompt, vision_emb, kv_cache, token_ids) >>> >>> # Second request with same image - cache hit! >>> entry, match_len = cache.fetch(["image.jpg"], prompt, token_ids) >>> # entry.vision_embeddings available - skip encoder! >>> # match_len > 0 - skip prefix computation! Performance (Gemma 3 27B, 256 image tokens): - Vision encoder: ~1.5s -> 0s (skip on hit) - Prefix computation: ~0.5s/1k tokens -> 0s (skip on match) - Multi-turn speedup: 8-12x for subsequent turns - Inputs: - `max_entries` (int; optional; default `50`): Maximum number of cache entries (default: 50) - `max_memory_mb` (int; optional; default `2048`): Maximum memory in MB (default: 2048) - Constructs: `vllm_mlx.mllm_cache.MLLMPrefixCacheManager` ## `vllm_mlx.mllm_cache.MLLMPrefixCacheManager.__init__` - Kind: method - Signature: `def __init__(self, max_entries: int=50, max_memory_mb: int=2048)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_cache.py#L211-L227 - Implementation: Method `MLLMPrefixCacheManager.__init__` updates `self.max_size`, `self.max_memory`, `self._cache`, `self._current_memory`; calls `OrderedDict`, `MLLMCacheStats`. Initialize MLLM prefix cache manager. Args: max_entries: Maximum number of cache entries (default: 50) max_memory_mb: Maximum memory in MB (default: 2048) - Inputs: - `max_entries` (int; optional; default `50`): Maximum number of cache entries (default: 50) - `max_memory_mb` (int; optional; default `2048`): Maximum memory in MB (default: 2048) - Return annotation: `not annotated` - Calls: OrderedDict, MLLMCacheStats - State writes: self.max_size, self.max_memory, self._cache, self._current_memory, self.stats ## `vllm_mlx.mllm_cache.MLLMPrefixCacheManager._make_cache_key` - Kind: method - Signature: `def _make_cache_key(self, images: list[str], prompt: str) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_cache.py#L229-L233 - Implementation: Method `MLLMPrefixCacheManager._make_cache_key` calls `compute_images_hash`, `hashlib.sha256(prompt.encode()).hexdigest`, `hashlib.sha256`, `prompt.encode`; returns `f'{image_hash}_{prompt_hash}'`. Create cache key from images and prompt. - Inputs: - `images` (list[str]; required): Required positional or keyword input. - `prompt` (str; required): Required positional or keyword input. - Return annotation: `str` - Calls: compute_images_hash, hashlib.sha256(prompt.encode()).hexdigest, hashlib.sha256, prompt.encode - Return expressions: f'{image_hash}_{prompt_hash}' ## `vllm_mlx.mllm_cache.MLLMPrefixCacheManager._make_image_only_key` - Kind: method - Signature: `def _make_image_only_key(self, images: list[str]) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_cache.py#L235-L237 - Implementation: Method `MLLMPrefixCacheManager._make_image_only_key` calls `compute_images_hash`; returns `compute_images_hash(images)`. Create cache key for image-only lookup (vision embedding reuse). - Inputs: - `images` (list[str]; required): Required positional or keyword input. - Return annotation: `str` - Calls: compute_images_hash - Return expressions: compute_images_hash(images) ## `vllm_mlx.mllm_cache.MLLMPrefixCacheManager._evict_by_memory` - Kind: method - Signature: `def _evict_by_memory(self, required_size: int) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_cache.py#L239-L246 - Implementation: Method `MLLMPrefixCacheManager._evict_by_memory` updates `self._current_memory`, `self.stats.evictions`; calls `next`, `iter`, `self._cache.pop`, `logger.debug`. Evict entries until we have enough memory. - Inputs: - `required_size` (int; required): Required positional or keyword input. - Return annotation: `None` - Calls: next, iter, self._cache.pop, logger.debug - State reads: self._current_memory, self.max_memory, self._cache, self._cache.pop, self.stats - State writes: self._current_memory, self.stats.evictions ## `vllm_mlx.mllm_cache.MLLMPrefixCacheManager._evict_by_count` - Kind: method - Signature: `def _evict_by_count(self) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_cache.py#L248-L255 - Implementation: Method `MLLMPrefixCacheManager._evict_by_count` updates `self._current_memory`, `self.stats.evictions`; calls `len`, `next`, `iter`, `self._cache.pop`. Evict entries until we're under max_size. - Inputs: none - Return annotation: `None` - Calls: len, next, iter, self._cache.pop, logger.debug - State reads: self._cache, self.max_size, self._cache.pop, self.stats - State writes: self._current_memory, self.stats.evictions ## `vllm_mlx.mllm_cache.MLLMPrefixCacheManager.fetch` - Kind: method - Signature: `def fetch(self, images: list[str], prompt: str, token_ids: list[int] | None=None) -> tuple[MLLMPrefixCacheEntry | None, int]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_cache.py#L257-L329 - Implementation: Method `MLLMPrefixCacheManager.fetch` updates `self.stats.total_queries`, `self.stats.hits`, `self.stats.image_cache_hits`, `self.stats.vision_encoder_skips`; calls `self._make_cache_key`, `self._cache.pop`, `entry.get_prefix_match_length`, `logger.debug`; has 3 explicit return paths. Fetch cached prefix state with prefix matching. This is the main entry point for cache lookups. Returns both the cache entry (if found) and the prefix match length. Args: images: List of image paths prompt: Text prompt token_ids: Optional token IDs for prefix matching Returns: Tuple of (entry, prefix_match_length) where: - entry: The cache entry if found, None otherwise - prefix_match_length: Number of tokens that match (0 if miss) - Inputs: - `images` (list[str]; required): List of image paths - `prompt` (str; required): Text prompt - `token_ids` (list[int] | None; optional; default `None`): Optional token IDs for prefix matching - Return annotation: `tuple[MLLMPrefixCacheEntry | None, int]` - Calls: self._make_cache_key, self._cache.pop, entry.get_prefix_match_length, logger.debug, self._make_image_only_key, self._cache.items - State reads: self.stats, self._make_cache_key, self._cache, self._cache.pop, self._make_image_only_key, self._cache.items - State writes: self.stats.total_queries, self.stats.hits, self.stats.image_cache_hits, self.stats.vision_encoder_skips, self.stats.partial_hits, self.stats.tokens_saved, self.stats.misses - Return expressions: (entry, match_length); (entry, 0); (None, 0) ## `vllm_mlx.mllm_cache.MLLMPrefixCacheManager.fetch_cache` - Kind: method - Signature: `def fetch_cache(self, images: list[str], prompt: str) -> tuple[list[Any] | None, bool]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_cache.py#L331-L345 - Implementation: Method `MLLMPrefixCacheManager.fetch_cache` calls `self.fetch`; has 2 explicit return paths. Legacy API: Fetch cached KV state for image+prompt combination. For backwards compatibility with existing code. - Inputs: - `images` (list[str]; required): Required positional or keyword input. - `prompt` (str; required): Required positional or keyword input. - Return annotation: `tuple[list[Any] | None, bool]` - Calls: self.fetch - State reads: self.fetch - Return expressions: (entry.kv_cache, True); (None, False) ## `vllm_mlx.mllm_cache.MLLMPrefixCacheManager.store` - Kind: method - Signature: `def store(self, images: list[str], prompt: str, vision_embeddings: Any, kv_cache: list[Any], token_ids: list[int], num_image_tokens: int=0, model_name: str='') -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_cache.py#L347-L396 - Implementation: Method `MLLMPrefixCacheManager.store` updates `self._current_memory`; calls `self._make_cache_key`, `MLLMPrefixCacheEntry`, `compute_images_hash`, `hashlib.sha256(prompt.encode()).hexdigest`. Store prefix state in cache. Args: images: List of image paths prompt: Text prompt vision_embeddings: Output of vision encoder (can be None for text-only) kv_cache: Language model KV cache states token_ids: Full token sequence num_image_tokens: Number of image tokens (e.g., 256 for Gemma 3) model_name: Model name for validation - Inputs: - `images` (list[str]; required): List of image paths - `prompt` (str; required): Text prompt - `vision_embeddings` (Any; required): Output of vision encoder (can be None for text-only) - `kv_cache` (list[Any]; required): Language model KV cache states - `token_ids` (list[int]; required): Full token sequence - `num_image_tokens` (int; optional; default `0`): Number of image tokens (e.g., 256 for Gemma 3) - `model_name` (str; optional; default `''`): Model name for validation - Return annotation: `None` - Calls: self._make_cache_key, MLLMPrefixCacheEntry, compute_images_hash, hashlib.sha256(prompt.encode()).hexdigest, hashlib.sha256, prompt.encode, len, self._evict_by_memory, self._evict_by_count, logger.debug - State reads: self._make_cache_key, self._evict_by_memory, self._evict_by_count, self._cache - State writes: self._current_memory ## `vllm_mlx.mllm_cache.MLLMPrefixCacheManager.store_cache` - Kind: method - Signature: `def store_cache(self, images: list[str], prompt: str, cache: list[Any] | None, num_tokens: int=0) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_cache.py#L398-L421 - Implementation: Method `MLLMPrefixCacheManager.store_cache` calls `isinstance`, `len`, `self.store`; returns `None`. Legacy API: Store KV cache for future reuse. For backwards compatibility with existing code. - Inputs: - `images` (list[str]; required): Required positional or keyword input. - `prompt` (str; required): Required positional or keyword input. - `cache` (list[Any] | None; required): Required positional or keyword input. - `num_tokens` (int; optional; default `0`): Optional positional or keyword input; defaults to `0`. - Return annotation: `None` - Calls: isinstance, len, self.store - State reads: self.store - Return expressions: None ## `vllm_mlx.mllm_cache.MLLMPrefixCacheManager.get_stats` - Kind: method - Signature: `def get_stats(self) -> dict[str, Any]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_cache.py#L423-L430 - Implementation: Method `MLLMPrefixCacheManager.get_stats` calls `self.stats.to_dict`, `len`; returns `stats`. Get cache statistics. - Inputs: none - Return annotation: `dict[str, Any]` - Calls: self.stats.to_dict, len - State reads: self.stats.to_dict, self.stats, self._cache, self.max_size, self._current_memory, self.max_memory - Return expressions: stats ## `vllm_mlx.mllm_cache.MLLMPrefixCacheManager.reset_stats` - Kind: method - Signature: `def reset_stats(self) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_cache.py#L432-L434 - Implementation: Method `MLLMPrefixCacheManager.reset_stats` updates `self.stats`; calls `MLLMCacheStats`. Reset statistics counters. - Inputs: none - Return annotation: `None` - Calls: MLLMCacheStats - State writes: self.stats ## `vllm_mlx.mllm_cache.MLLMPrefixCacheManager.clear` - Kind: method - Signature: `def clear(self) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_cache.py#L436-L440 - Implementation: Method `MLLMPrefixCacheManager.clear` updates `self._current_memory`; calls `self._cache.clear`, `self.reset_stats`. Clear all cached entries and reset stats. - Inputs: none - Return annotation: `None` - Calls: self._cache.clear, self.reset_stats - State reads: self._cache.clear, self._cache, self.reset_stats - State writes: self._current_memory ## `vllm_mlx.mllm_cache.MLLMPrefixCacheManager.__len__` - Kind: method - Signature: `def __len__(self) -> int` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_cache.py#L442-L444 - Implementation: Method `MLLMPrefixCacheManager.__len__` calls `len`; returns `len(self._cache)`. Return number of cached entries. - Inputs: none - Return annotation: `int` - Calls: len - State reads: self._cache - Return expressions: len(self._cache) ## `vllm_mlx.mllm_cache.MLLMPrefixCacheManager.__repr__` - Kind: method - Signature: `def __repr__(self) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_cache.py#L446-L448 - Implementation: Method `MLLMPrefixCacheManager.__repr__` calls `len`; returns `f''`. Method `MLLMPrefixCacheManager.__repr__` calls `len`; returns `f''`. - Inputs: none - Return annotation: `str` - Calls: len - State reads: self._current_memory - Return expressions: f'' # Module `vllm_mlx.mllm_scheduler` MLLM Scheduler for multimodal continuous batching. This scheduler handles Multimodal Language Model requests with continuous batching support, following the same architecture as the LLM scheduler. Key features: - Batch processing of multiple MLLM requests - Vision embedding caching for repeated images - Step-based generation loop (like LLM scheduler) - Support for both streaming and non-streaming generation Architecture: 1. Requests arrive via add_request() -> waiting queue 2. Scheduler moves requests from waiting to running (via MLLMBatchGenerator) 3. step() method generates one token for ALL running requests 4. Finished requests are removed and outputs returned Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_scheduler.py#L1-L1242 ## `vllm_mlx.mllm_scheduler.MLLMSchedulerConfig` - Kind: class - Signature: `class MLLMSchedulerConfig` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_scheduler.py#L46-L92 - Implementation: Class `MLLMSchedulerConfig` declares 0 direct member(s). Configuration for MLLM scheduler. - Inputs: - `max_num_seqs` (int; optional; default `16`): Optional constructor field; defaults to `16`. - `prefill_batch_size` (int; optional; default `16`): Optional constructor field; defaults to `16`. - `completion_batch_size` (int; optional; default `16`): Optional constructor field; defaults to `16`. - `prefill_step_size` (int; optional; default `1024`): Optional constructor field; defaults to `1024`. - `enable_vision_cache` (bool; optional; default `True`): Optional constructor field; defaults to `True`. - `vision_cache_size` (int; optional; default `100`): Optional constructor field; defaults to `100`. - `default_max_tokens` (int; optional; default `256`): Optional constructor field; defaults to `256`. - `default_video_fps` (float; optional; default `2.0`): Optional constructor field; defaults to `2.0`. - `cache_memory_mb` (Optional[int]; optional; default `None`): Optional constructor field; defaults to `None`. - `max_video_frames` (int; optional; default `128`): Optional constructor field; defaults to `128`. - `enable_mtp` (bool; optional; default `False`): Optional constructor field; defaults to `False`. - `mtp_num_draft_tokens` (int; optional; default `1`): Optional constructor field; defaults to `1`. - `enable_prefix_cache` (bool; optional; default `True`): Optional constructor field; defaults to `True`. - `use_memory_aware_cache` (bool; optional; default `True`): Optional constructor field; defaults to `True`. - `prefix_cache_memory_mb` (Optional[int]; optional; default `None`): Optional constructor field; defaults to `None`. - `kv_cache_quantization` (bool; optional; default `False`): Optional constructor field; defaults to `False`. - `kv_cache_quantization_bits` (int; optional; default `8`): Optional constructor field; defaults to `8`. - `kv_cache_quantization_group_size` (int; optional; default `64`): Optional constructor field; defaults to `64`. - `chunked_prefill_tokens` (int; optional; default `0`): Optional constructor field; defaults to `0`. - `max_kv_size` (int; optional; default `0`): Optional constructor field; defaults to `0`. - `ssd_cache_dir` (Optional[str]; optional; default `None`): Optional constructor field; defaults to `None`. - `ssd_cache_max_gb` (float; optional; default `10.0`): Optional constructor field; defaults to `10.0`. - Constructs: `vllm_mlx.mllm_scheduler.MLLMSchedulerConfig` - Decorators: dataclass ## `vllm_mlx.mllm_scheduler.MLLMRequest` - Kind: class - Signature: `class MLLMRequest` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_scheduler.py#L96-L127 - Implementation: Class `MLLMRequest` declares 0 direct member(s). Extended request for MLLM processing. Includes all multimodal data needed for generation. - Inputs: - `request_id` (str; required): Required constructor field. - `prompt` (str; required): Required constructor field. - `images` (Optional[List[str]]; optional; default `None`): Optional constructor field; defaults to `None`. - `videos` (Optional[List[str]]; optional; default `None`): Optional constructor field; defaults to `None`. - `audio` (Optional[List[str]]; optional; default `None`): Optional constructor field; defaults to `None`. - `sampling_params` (SamplingParams; optional; default `field(default_factory=SamplingParams)`): Optional constructor field; defaults to `field(default_factory=SamplingParams)`. - `arrival_time` (float; optional; default `field(default_factory=time.time)`): Optional constructor field; defaults to `field(default_factory=time.time)`. - `batch_uid` (Optional[int]; optional; default `None`): Optional constructor field; defaults to `None`. - `status` (RequestStatus; optional; default `RequestStatus.WAITING`): Optional constructor field; defaults to `RequestStatus.WAITING`. - `output_text` (str; optional; default `''`): Optional constructor field; defaults to `''`. - `output_tokens` (List[int]; optional; default `field(default_factory=list)`): Optional constructor field; defaults to `field(default_factory=list)`. - `finish_reason` (Optional[str]; optional; default `None`): Optional constructor field; defaults to `None`. - `num_prompt_tokens` (int; optional; default `0`): Optional constructor field; defaults to `0`. - `num_output_tokens` (int; optional; default `0`): Optional constructor field; defaults to `0`. - `mtp_drafts` (int; optional; default `0`): Optional constructor field; defaults to `0`. - `mtp_accepted` (int; optional; default `0`): Optional constructor field; defaults to `0`. - `first_token_time` (Optional[float]; optional; default `None`): Optional constructor field; defaults to `None`. - Constructs: `vllm_mlx.mllm_scheduler.MLLMRequest` - Decorators: dataclass ## `vllm_mlx.mllm_scheduler.MLLMSchedulerOutput` - Kind: class - Signature: `class MLLMSchedulerOutput` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_scheduler.py#L131-L147 - Implementation: Class `MLLMSchedulerOutput` declares 0 direct member(s). Output from a scheduling step. Contains information about what was scheduled and results. - Inputs: - `scheduled_request_ids` (List[str]; optional; default `field(default_factory=list)`): Optional constructor field; defaults to `field(default_factory=list)`. - `num_scheduled_tokens` (int; optional; default `0`): Optional constructor field; defaults to `0`. - `finished_request_ids` (Set[str]; optional; default `field(default_factory=set)`): Optional constructor field; defaults to `field(default_factory=set)`. - `outputs` (List[RequestOutput]; optional; default `field(default_factory=list)`): Optional constructor field; defaults to `field(default_factory=list)`. - `has_work` (bool; optional; default `False`): Optional constructor field; defaults to `False`. - Constructs: `vllm_mlx.mllm_scheduler.MLLMSchedulerOutput` - Decorators: dataclass ## `vllm_mlx.mllm_scheduler.MLLMScheduler` - Kind: class - Signature: `class MLLMScheduler` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_scheduler.py#L150-L1242 - Implementation: Class `MLLMScheduler` declares 24 direct member(s). Scheduler for Vision Language Model requests with continuous batching. This scheduler manages the lifecycle of MLLM requests using the MLLMBatchGenerator for efficient batch processing: 1. Requests arrive and are added to the waiting queue 2. Scheduler moves requests from waiting to running (via batch generator) 3. step() generates one token for ALL running requests simultaneously 4. Finished requests are removed and outputs returned Example: >>> scheduler = MLLMScheduler(model, processor, config) >>> # Add requests >>> request_id = scheduler.add_request( ... prompt="What's in this image?", ... images=["photo.jpg"] ... ) >>> # Run generation loop >>> while scheduler.has_requests(): ... output = scheduler.step() ... for req_output in output.outputs: ... if req_output.finished: ... print(f"Finished: {req_output.output_text}") For async usage with streaming: >>> await scheduler.start() >>> request_id = await scheduler.add_request_async(...) >>> async for output in scheduler.stream_outputs(request_id): ... print(output.new_text, end="") - Inputs: - `model` (Any; required): The VLM model - `processor` (Any; required): The VLM processor - `config` (Optional[MLLMSchedulerConfig]; optional; default `None`): Scheduler configuration - Constructs: `vllm_mlx.mllm_scheduler.MLLMScheduler` ## `vllm_mlx.mllm_scheduler.MLLMScheduler.__init__` - Kind: method - Signature: `def __init__(self, model: Any, processor: Any, config: Optional[MLLMSchedulerConfig]=None)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_scheduler.py#L183-L248 - Implementation: Method `MLLMScheduler.__init__` updates `self.model`, `self.processor`, `self.config`, `self.model_config`; calls `MLLMSchedulerConfig`, `getattr`, `MultimodalProcessor`, `self._get_stop_tokens`. Initialize MLLM scheduler. Args: model: The VLM model processor: The VLM processor config: Scheduler configuration - Inputs: - `model` (Any; required): The VLM model - `processor` (Any; required): The VLM processor - `config` (Optional[MLLMSchedulerConfig]; optional; default `None`): Scheduler configuration - Return annotation: `not annotated` - Calls: MLLMSchedulerConfig, getattr, MultimodalProcessor, self._get_stop_tokens, deque, set - State reads: self.model_config, self._get_stop_tokens - State writes: self.model, self.processor, self.config, self.model_config, self.mm_processor, self.stop_tokens, self.batch_generator, self.waiting, self.running, self.requests, self.finished_req_ids, self.request_id_to_uid, self.uid_to_request_id, self._detokenizer_pool, self.output_queues, self._running, self._processing_task, self._step_count, self._clear_cache_interval, self.num_requests_processed, self.total_prompt_tokens, self.total_completion_tokens ## `vllm_mlx.mllm_scheduler.MLLMScheduler._get_stop_tokens` - Kind: method - Signature: `def _get_stop_tokens(self) -> Set[int]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_scheduler.py#L250-L290 - Implementation: Method `MLLMScheduler._get_stop_tokens` calls `set`, `hasattr`, `isinstance`, `stop_tokens.update`; returns `stop_tokens`. Get stop token IDs from tokenizer and generation_config.json. - Inputs: none - Return annotation: `Set[int]` - Calls: set, hasattr, isinstance, stop_tokens.update, stop_tokens.add, getattr, Path, gc_path.exists, json.loads, gc_path.read_text, gc.get - State reads: self.processor, self.processor.tokenizer - Return expressions: stop_tokens ## `vllm_mlx.mllm_scheduler.MLLMScheduler._ensure_batch_generator` - Kind: method - Signature: `def _ensure_batch_generator(self) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_scheduler.py#L292-L374 - Implementation: Method `MLLMScheduler._ensure_batch_generator` updates `self.batch_generator`, `self._ssd_tier`; calls `make_sampler`, `MemoryCacheConfig`, `MLLMBatchGenerator`, `getattr`. Ensure batch generator exists. - Inputs: none - Return annotation: `None` - Calls: make_sampler, MemoryCacheConfig, MLLMBatchGenerator, getattr, SSDCacheConfig, SSDCacheTier, self._ssd_tier.start_writer, self._ssd_tier.reconcile, prefix_cache.set_ssd_tier, logger.info, install_chunked_prefill_mllm, hasattr, install_mtp_mllm - State reads: self.batch_generator, self.config.enable_prefix_cache, self.config, self.config.use_memory_aware_cache, self.config.prefix_cache_memory_mb, self.config.kv_cache_quantization, self.config.kv_cache_quantization_bits, self.config.kv_cache_quantization_group_size, self.model, self.processor, self.mm_processor, self.config.default_max_tokens, self.stop_tokens, self.config.prefill_batch_size, self.config.completion_batch_size, self.config.prefill_step_size, self.config.max_kv_size, self.config.ssd_cache_dir, self.config.ssd_cache_max_gb, self._ssd_tier.start_writer, self._ssd_tier, self._ssd_tier.reconcile, self.config.chunked_prefill_tokens, self.config.enable_mtp, self.batch_generator.language_model, self.config.mtp_num_draft_tokens - State writes: self.batch_generator, self._ssd_tier ## `vllm_mlx.mllm_scheduler.MLLMScheduler.add_request` - Kind: method - Signature: `def add_request(self, prompt: str, images: Optional[List[str]]=None, videos: Optional[List[str]]=None, audio: Optional[List[str]]=None, max_tokens: int=256, temperature: float=0.7, top_p: float=0.9, request_id: Optional[str]=None, **kwargs) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_scheduler.py#L378-L453 - Implementation: Method `MLLMScheduler.add_request` calls `str`, `uuid.uuid4`, `SamplingParams`, `kwargs.pop`; returns `request_id`. Add a multimodal request to the scheduler (sync version). Args: prompt: Text prompt (should be formatted with chat template) images: List of image inputs (paths, URLs, base64) videos: List of video inputs audio: List of audio inputs max_tokens: Maximum tokens to generate temperature: Sampling temperature top_p: Top-p sampling request_id: Optional custom request ID **kwargs: Additional generation parameters. ``logits_processors`` — list of callables ``(tokens, logits) -> logits`` applied during sampling (e.g. constrained JSON decoding). Returns: Request ID for tracking - Inputs: - `prompt` (str; required): Text prompt (should be formatted with chat template) - `images` (Optional[List[str]]; optional; default `None`): List of image inputs (paths, URLs, base64) - `videos` (Optional[List[str]]; optional; default `None`): List of video inputs - `audio` (Optional[List[str]]; optional; default `None`): List of audio inputs - `max_tokens` (int; optional; default `256`): Maximum tokens to generate - `temperature` (float; optional; default `0.7`): Sampling temperature - `top_p` (float; optional; default `0.9`): Top-p sampling - `request_id` (Optional[str]; optional; default `None`): Optional custom request ID - `**kwargs` (not annotated; optional): Additional generation parameters. ``logits_processors`` — list of callables ``(tokens, logits) -> logits`` applied during sampling (e.g. constrained JSON decoding). - Return annotation: `str` - Calls: str, uuid.uuid4, SamplingParams, kwargs.pop, MLLMRequest, hasattr, len, tokenizer.encode, self.waiting.append, logger.debug - State reads: self.processor, self.processor.tokenizer, self.requests, self.waiting.append, self.waiting - Return expressions: request_id ## `vllm_mlx.mllm_scheduler.MLLMScheduler.abort_request` - Kind: method - Signature: `def abort_request(self, request_id: str) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_scheduler.py#L455-L532 - Implementation: Method `MLLMScheduler.abort_request` updates `self.total_completion_tokens`, `self.total_prompt_tokens`; calls `self.requests.get`, `self.batch_generator.abort_prefill`, `self.waiting.remove`, `self.batch_generator.schedule_removal`; has 2 explicit return paths. Abort a request. Args: request_id: The request ID to abort Returns: True if request was found and aborted - Inputs: - `request_id` (str; required): The request ID to abort - Return annotation: `bool` - Calls: self.requests.get, self.batch_generator.abort_prefill, self.waiting.remove, self.batch_generator.schedule_removal, self.finished_req_ids.add, self.requests.pop, self._detokenizer_pool.pop, self.output_queues[request_id].put_nowait, logger.debug - State reads: self.requests.get, self.requests, self.batch_generator, self.batch_generator.abort_prefill, self.waiting.remove, self.waiting, self.request_id_to_uid, self.batch_generator.schedule_removal, self.uid_to_request_id, self.running, self.finished_req_ids.add, self.finished_req_ids, self.requests.pop, self._detokenizer_pool.pop, self._detokenizer_pool, self.output_queues - State writes: self.total_completion_tokens, self.total_prompt_tokens - Return expressions: False; True ## `vllm_mlx.mllm_scheduler.MLLMScheduler.has_requests` - Kind: method - Signature: `def has_requests(self) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_scheduler.py#L534-L536 - Implementation: Method `MLLMScheduler.has_requests` calls `bool`; returns `bool(self.waiting or self.running)`. Check if there are any pending or running requests. - Inputs: none - Return annotation: `bool` - Calls: bool - State reads: self.waiting, self.running - Return expressions: bool(self.waiting or self.running) ## `vllm_mlx.mllm_scheduler.MLLMScheduler.get_num_waiting` - Kind: method - Signature: `def get_num_waiting(self) -> int` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_scheduler.py#L538-L540 - Implementation: Method `MLLMScheduler.get_num_waiting` calls `len`; returns `len(self.waiting)`. Get number of waiting requests. - Inputs: none - Return annotation: `int` - Calls: len - State reads: self.waiting - Return expressions: len(self.waiting) ## `vllm_mlx.mllm_scheduler.MLLMScheduler.get_num_running` - Kind: method - Signature: `def get_num_running(self) -> int` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_scheduler.py#L542-L544 - Implementation: Method `MLLMScheduler.get_num_running` calls `len`; returns `len(self.running)`. Get number of running requests. - Inputs: none - Return annotation: `int` - Calls: len - State reads: self.running - Return expressions: len(self.running) ## `vllm_mlx.mllm_scheduler.MLLMScheduler._schedule_waiting` - Kind: method - Signature: `def _schedule_waiting(self) -> List[MLLMRequest]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_scheduler.py#L546-L597 - Implementation: Method `MLLMScheduler._schedule_waiting` updates `self.total_prompt_tokens`; calls `self._ensure_batch_generator`, `len`, `self.waiting.popleft`, `MLLMBatchRequest`; returns `scheduled`. Move requests from waiting queue to running. Returns: List of requests that were scheduled - Inputs: none - Return annotation: `List[MLLMRequest]` - Calls: self._ensure_batch_generator, len, self.waiting.popleft, MLLMBatchRequest, batch_requests.append, scheduled.append, self.batch_generator.insert, zip, logger.debug - State reads: self._ensure_batch_generator, self.waiting, self.running, self.config.max_num_seqs, self.config, self.waiting.popleft, self.batch_generator, self.batch_generator.insert, self.request_id_to_uid, self.uid_to_request_id - State writes: self.total_prompt_tokens - Return expressions: scheduled ## `vllm_mlx.mllm_scheduler.MLLMScheduler._process_batch_responses` - Kind: method - Signature: `def _process_batch_responses(self, responses: List[MLLMBatchResponse]) -> Tuple[List[RequestOutput], Set[str]]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_scheduler.py#L599-L716 - Implementation: Method `MLLMScheduler._process_batch_responses` updates `self.num_requests_processed`, `self.total_completion_tokens`; calls `set`, `hasattr`, `self.uid_to_request_id.get`, `self.running.get`; returns `(outputs, finished_ids)`. Process responses from batch generator. Args: responses: List of MLLMBatchResponse objects Returns: Tuple of (outputs, finished_request_ids) - Inputs: - `responses` (List[MLLMBatchResponse]; required): List of MLLMBatchResponse objects - Return annotation: `Tuple[List[RequestOutput], Set[str]]` - Calls: set, hasattr, self.uid_to_request_id.get, self.running.get, RequestOutput, finished_ids.add, logger.warning, outputs.append, request.output_tokens.append, len, time.time, NaiveStreamingDetokenizer, detok.add_token, self._detokenizer_pool.pop, detok.finalize, tokenizer.decode, logger.debug - State reads: self.processor, self.processor.tokenizer, self.uid_to_request_id.get, self.uid_to_request_id, self.running.get, self.running, self._detokenizer_pool, self._detokenizer_pool.pop - State writes: self.num_requests_processed, self.total_completion_tokens - Return expressions: (outputs, finished_ids) ## `vllm_mlx.mllm_scheduler.MLLMScheduler._cleanup_finished` - Kind: method - Signature: `def _cleanup_finished(self, finished_ids: Set[str]) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_scheduler.py#L718-L744 - Implementation: Method `MLLMScheduler._cleanup_finished` calls `self.requests.pop`, `self._detokenizer_pool.pop`, `self.finished_req_ids.add`, `mx.clear_cache`. Clean up finished requests. - Inputs: - `finished_ids` (Set[str]; required): Required positional or keyword input. - Return annotation: `None` - Calls: self.requests.pop, self._detokenizer_pool.pop, self.finished_req_ids.add, mx.clear_cache - State reads: self.running, self.requests.pop, self.requests, self.request_id_to_uid, self.uid_to_request_id, self._detokenizer_pool.pop, self._detokenizer_pool, self.finished_req_ids.add, self.finished_req_ids ## `vllm_mlx.mllm_scheduler.MLLMScheduler.step` - Kind: method - Signature: `def step(self) -> MLLMSchedulerOutput` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_scheduler.py#L746-L813 - Implementation: Method `MLLMScheduler.step` updates `self._step_count`, `self.finished_req_ids`; calls `MLLMSchedulerOutput`, `self.batch_generator.process_pending_removals`, `self._schedule_waiting`, `sum`; returns `output`. Execute one scheduling step. This method: 1. Schedules waiting requests into the batch 2. Runs one generation step via MLLMBatchGenerator 3. Processes outputs and handles finished requests Returns: MLLMSchedulerOutput with results of this step - Inputs: none - Return annotation: `MLLMSchedulerOutput` - Calls: MLLMSchedulerOutput, self.batch_generator.process_pending_removals, self._schedule_waiting, sum, self.batch_generator.next, self._process_batch_responses, self.output_queues.get, queue.put_nowait, self._cleanup_finished, mx.clear_cache, len, max, set - State reads: self.batch_generator, self.batch_generator.process_pending_removals, self._schedule_waiting, self.running, self.batch_generator.next, self._process_batch_responses, self.output_queues.get, self.output_queues, self._cleanup_finished, self._clear_cache_interval, self._step_count - State writes: self._step_count, self.finished_req_ids - Return expressions: output ## `vllm_mlx.mllm_scheduler.MLLMScheduler.get_request` - Kind: method - Signature: `def get_request(self, request_id: str) -> Optional[MLLMRequest]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_scheduler.py#L815-L817 - Implementation: Method `MLLMScheduler.get_request` calls `self.requests.get`; returns `self.requests.get(request_id)`. Get a request by ID. - Inputs: - `request_id` (str; required): Required positional or keyword input. - Return annotation: `Optional[MLLMRequest]` - Calls: self.requests.get - State reads: self.requests.get, self.requests - Return expressions: self.requests.get(request_id) ## `vllm_mlx.mllm_scheduler.MLLMScheduler.remove_finished_request` - Kind: method - Signature: `def remove_finished_request(self, request_id: str) -> Optional[MLLMRequest]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_scheduler.py#L819-L821 - Implementation: Method `MLLMScheduler.remove_finished_request` calls `self.requests.pop`; returns `self.requests.pop(request_id, None)`. Remove a finished request from tracking. - Inputs: - `request_id` (str; required): Required positional or keyword input. - Return annotation: `Optional[MLLMRequest]` - Calls: self.requests.pop - State reads: self.requests.pop, self.requests - Return expressions: self.requests.pop(request_id, None) ## `vllm_mlx.mllm_scheduler.MLLMScheduler.start` - Kind: method - Signature: `async def start(self) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_scheduler.py#L825-L834 - Implementation: Method `MLLMScheduler.start` updates `self._running`, `self._processing_task`; calls `asyncio.create_task`, `self._process_loop`, `logger.info`; returns `None`. Start the async scheduler processing loop. - Inputs: none - Return annotation: `None` - Calls: asyncio.create_task, self._process_loop, logger.info - State reads: self._running, self._process_loop, self.config.max_num_seqs, self.config - State writes: self._running, self._processing_task - Return expressions: None ## `vllm_mlx.mllm_scheduler.MLLMScheduler.stop` - Kind: method - Signature: `async def stop(self) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_scheduler.py#L836-L850 - Implementation: Method `MLLMScheduler.stop` updates `self._running`, `self.batch_generator`; calls `self._processing_task.cancel`, `self.batch_generator.close`, `logger.info`; awaits asynchronous work. Stop the scheduler. - Inputs: none - Return annotation: `None` - Calls: self._processing_task.cancel, self.batch_generator.close, logger.info - State reads: self._processing_task, self._processing_task.cancel, self.batch_generator, self.batch_generator.close - State writes: self._running, self.batch_generator ## `vllm_mlx.mllm_scheduler.MLLMScheduler._process_loop` - Kind: method - Signature: `async def _process_loop(self) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_scheduler.py#L852-L947 - Implementation: Method `MLLMScheduler._process_loop` calls `asyncio.get_running_loop`, `list`, `getattr`, `time.perf_counter`; awaits asynchronous work. Main async processing loop. MLLM models are loaded on the server/event-loop thread, so their MLX arrays and cache state must be consumed on that same thread. Unlike the text-only EngineCore path, moving MLLM prefill to a worker crosses MLX stream ownership and can fail with "no Stream in current thread". Text-only preprocessing (Jinja2 template rendering + tokenization) is run BEFORE ``step()`` with ``await asyncio.sleep(0)`` yields between each request. This prevents long preprocessing (10-30+ s for 40K+ token conversations) from blocking health checks and new connections. - Inputs: none - Return annotation: `None` - Calls: asyncio.get_running_loop, list, getattr, time.perf_counter, loop.run_in_executor, logger.info, logger.error, self.has_requests, _ensure_streams_bound, self.step, logger.warning, len, range, asyncio.sleep - State reads: self._running, self.batch_generator, self.has_requests, self.step, self.waiting, self.running ## `vllm_mlx.mllm_scheduler.MLLMScheduler._process_loop._ensure_streams_bound` - Kind: nested function - Signature: `def _ensure_streams_bound() -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_scheduler.py#L867-L871 - Implementation: Nested Function `MLLMScheduler._process_loop._ensure_streams_bound` calls `bind_generation_streams`. Nested Function `MLLMScheduler._process_loop._ensure_streams_bound` calls `bind_generation_streams`. - Inputs: none - Return annotation: `None` - Calls: bind_generation_streams ## `vllm_mlx.mllm_scheduler.MLLMScheduler.add_request_async` - Kind: method - Signature: `async def add_request_async(self, prompt: str, images: Optional[List[str]]=None, videos: Optional[List[str]]=None, audio: Optional[List[str]]=None, max_tokens: int=256, temperature: float=0.7, top_p: float=0.9, **kwargs) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_scheduler.py#L949-L990 - Implementation: Method `MLLMScheduler.add_request_async` calls `self.add_request`, `asyncio.Queue`; returns `request_id`. Add a multimodal request (async version with output queue). Args: prompt: Text prompt images: List of image inputs videos: List of video inputs audio: List of audio inputs max_tokens: Maximum tokens to generate temperature: Sampling temperature top_p: Top-p sampling **kwargs: Additional parameters Returns: Request ID for tracking - Inputs: - `prompt` (str; required): Text prompt - `images` (Optional[List[str]]; optional; default `None`): List of image inputs - `videos` (Optional[List[str]]; optional; default `None`): List of video inputs - `audio` (Optional[List[str]]; optional; default `None`): List of audio inputs - `max_tokens` (int; optional; default `256`): Maximum tokens to generate - `temperature` (float; optional; default `0.7`): Sampling temperature - `top_p` (float; optional; default `0.9`): Top-p sampling - `**kwargs` (not annotated; optional): Additional parameters - Return annotation: `str` - Calls: self.add_request, asyncio.Queue - State reads: self.add_request, self.output_queues - Return expressions: request_id ## `vllm_mlx.mllm_scheduler.MLLMScheduler.stream_outputs` - Kind: method - Signature: `async def stream_outputs(self, request_id: str) -> AsyncIterator[RequestOutput]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_scheduler.py#L992-L1027 - Implementation: Method `MLLMScheduler.stream_outputs` calls `self.output_queues.get`, `output_queue.get`, `logger.info`, `self.abort_request`; awaits asynchronous work; yields values incrementally; returns `None`. Stream outputs for a request. Args: request_id: The request ID to stream Yields: RequestOutput objects as tokens are generated - Inputs: - `request_id` (str; required): The request ID to stream - Return annotation: `AsyncIterator[RequestOutput]` - Calls: self.output_queues.get, output_queue.get, logger.info, self.abort_request - State reads: self.output_queues.get, self.output_queues, self.abort_request - Return expressions: None ## `vllm_mlx.mllm_scheduler.MLLMScheduler.generate` - Kind: method - Signature: `async def generate(self, prompt: str, images: Optional[List[str]]=None, videos: Optional[List[str]]=None, audio: Optional[List[str]]=None, **kwargs) -> RequestOutput` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_scheduler.py#L1029-L1078 - Implementation: Method `MLLMScheduler.generate` calls `self.add_request_async`, `self.stream_outputs`, `RequestOutput`; awaits asynchronous work; returns `final_output`. Generate complete output for a request (non-streaming). Args: prompt: Text prompt images: Image inputs videos: Video inputs audio: Audio inputs **kwargs: Generation parameters Returns: Final RequestOutput - Inputs: - `prompt` (str; required): Text prompt - `images` (Optional[List[str]]; optional; default `None`): Image inputs - `videos` (Optional[List[str]]; optional; default `None`): Video inputs - `audio` (Optional[List[str]]; optional; default `None`): Audio inputs - `**kwargs` (not annotated; optional): Generation parameters - Return annotation: `RequestOutput` - Calls: self.add_request_async, self.stream_outputs, RequestOutput - State reads: self.add_request_async, self.stream_outputs, self.requests - Return expressions: final_output ## `vllm_mlx.mllm_scheduler.MLLMScheduler.get_running_requests_info` - Kind: method - Signature: `def get_running_requests_info(self) -> List[Dict[str, Any]]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_scheduler.py#L1082-L1151 - Implementation: Method `MLLMScheduler.get_running_requests_info` calls `time.time`, `result.append`, `round`, `self.running.values`; returns `result`. Per-request details for status endpoint. - Inputs: none - Return annotation: `List[Dict[str, Any]]` - Calls: time.time, result.append, round, self.running.values, self.batch_generator.get_prefill_progress, min - State reads: self.waiting, self.running.values, self.running, self.batch_generator, self.batch_generator.get_prefill_progress - Return expressions: result ## `vllm_mlx.mllm_scheduler.MLLMScheduler.get_stats` - Kind: method - Signature: `def get_stats(self) -> Dict[str, Any]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_scheduler.py#L1153-L1204 - Implementation: Method `MLLMScheduler.get_stats` calls `len`, `self.get_running_requests_info`, `self.batch_generator.stats`, `batch_stats.to_dict`; returns `stats`. Get scheduler statistics. - Inputs: none - Return annotation: `Dict[str, Any]` - Calls: len, self.get_running_requests_info, self.batch_generator.stats, batch_stats.to_dict, self.batch_generator.get_vision_cache_stats, hasattr, self.batch_generator.get_mtp_stats, mx.metal.is_available, round, mx.get_active_memory, mx.get_peak_memory, mx.get_cache_memory, self.batch_generator.get_prefix_cache_stats - State reads: self.waiting, self.running, self.finished_req_ids, self.num_requests_processed, self.total_prompt_tokens, self.total_completion_tokens, self.get_running_requests_info, self.batch_generator, self.batch_generator.stats, self.batch_generator.get_vision_cache_stats, self.batch_generator.get_mtp_stats, self.batch_generator.get_prefix_cache_stats - Return expressions: stats ## `vllm_mlx.mllm_scheduler.MLLMScheduler.clear_runtime_caches` - Kind: method - Signature: `def clear_runtime_caches(self) -> Dict[str, bool]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_scheduler.py#L1206-L1221 - Implementation: Method `MLLMScheduler.clear_runtime_caches` calls `self.vision_cache.clear`, `self.batch_generator.prefix_cache.clear`; returns `cleared`. Clear runtime caches without resetting scheduler/request state. - Inputs: none - Return annotation: `Dict[str, bool]` - Calls: self.vision_cache.clear, self.batch_generator.prefix_cache.clear - State reads: self.vision_cache, self.vision_cache.clear, self.batch_generator, self.batch_generator.prefix_cache, self.batch_generator.prefix_cache.clear - Return expressions: cleared ## `vllm_mlx.mllm_scheduler.MLLMScheduler.reset` - Kind: method - Signature: `def reset(self) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_scheduler.py#L1223-L1242 - Implementation: Method `MLLMScheduler.reset` updates `self.batch_generator`; calls `list`, `self.requests.keys`, `self.abort_request`, `self.waiting.clear`. Reset the scheduler state. - Inputs: none - Return annotation: `None` - Calls: list, self.requests.keys, self.abort_request, self.waiting.clear, self.running.clear, self.requests.clear, self.finished_req_ids.clear, self.request_id_to_uid.clear, self.uid_to_request_id.clear, self._detokenizer_pool.clear, self.batch_generator.close, self.vision_cache.clear - State reads: self.requests.keys, self.requests, self.abort_request, self.waiting.clear, self.waiting, self.running.clear, self.running, self.requests.clear, self.finished_req_ids.clear, self.finished_req_ids, self.request_id_to_uid.clear, self.request_id_to_uid, self.uid_to_request_id.clear, self.uid_to_request_id, self._detokenizer_pool.clear, self._detokenizer_pool, self.batch_generator, self.batch_generator.close, self.vision_cache, self.vision_cache.clear - State writes: self.batch_generator # Module `vllm_mlx.mlx_streams` Helpers for binding MLX generation streams to worker threads. Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mlx_streams.py#L1-L39 ## `vllm_mlx.mlx_streams.bind_generation_streams` - Kind: function - Signature: `def bind_generation_streams(module_names: Iterable[str]=('mlx_lm.generate', 'mlx_vlm.generate')) -> object` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mlx_streams.py#L15-L39 - Implementation: Function `bind_generation_streams` calls `mx.new_stream`, `mx.default_device`, `mx.set_default_stream`, `importlib.import_module`; returns `default_stream`. Bind mlx-lm/mlx-vlm generation streams to the current thread. MLX streams are thread-local. If a model is loaded on one thread and generation runs on another, module-level generation streams created during import can point at a stream that does not exist in the worker thread. This intentionally creates a fresh stream for the current worker call and replaces module-level generation_stream handles under a process-local lock. It is an admission/ownership fix, not a batching optimization; callers should invoke it at worker-entry boundaries rather than inside token loops. - Inputs: - `module_names` (Iterable[str]; optional; default `('mlx_lm.generate', 'mlx_vlm.generate')`): Optional positional or keyword input; defaults to `('mlx_lm.generate', 'mlx_vlm.generate')`. - Return annotation: `object` - Calls: mx.new_stream, mx.default_device, mx.set_default_stream, importlib.import_module, hasattr, setattr - Return expressions: default_stream # Module `vllm_mlx.model_registry` Registry-backed multi-model serving with memory-budget eviction. The registry maps OpenAI-compatible ``model`` names to concrete local paths or declared HuggingFace IDs. Models are loaded lazily, optionally preloaded, and evicted according to a memory-budget policy with configurable wait/fail/preempt behaviour. Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/model_registry.py#L1-L1201 ## `vllm_mlx.model_registry.ModelOwnershipError` - Kind: class - Signature: `class ModelOwnershipError(RuntimeError)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/model_registry.py#L38-L39 - Implementation: Class `ModelOwnershipError` derives from `RuntimeError` and declares 0 direct member(s). Raised when an EngineCore attempts to use a model already in use. - Inputs: none - Constructs: `vllm_mlx.model_registry.ModelOwnershipError` ## `vllm_mlx.model_registry._ModelOwnershipRegistry` - Kind: class - Signature: `class _ModelOwnershipRegistry` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/model_registry.py#L42-L82 - Implementation: Class `_ModelOwnershipRegistry` declares 5 direct member(s). Process-local model ownership guard used by EngineCore. - Inputs: none - Constructs: `vllm_mlx.model_registry._ModelOwnershipRegistry` ## `vllm_mlx.model_registry._ModelOwnershipRegistry.__init__` - Kind: method - Signature: `def __init__(self) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/model_registry.py#L45-L46 - Implementation: Method `_ModelOwnershipRegistry.__init__` updates `self._owners`. Method `_ModelOwnershipRegistry.__init__` updates `self._owners`. - Inputs: none - Return annotation: `None` - State writes: self._owners ## `vllm_mlx.model_registry._ModelOwnershipRegistry.acquire` - Kind: method - Signature: `def acquire(self, *, model: Any, engine: Any, engine_id: str, force: bool=True) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/model_registry.py#L48-L63 - Implementation: Method `_ModelOwnershipRegistry.acquire` calls `id`, `self._owners.get`, `ModelOwnershipError`; can raise `ModelOwnershipError`. Method `_ModelOwnershipRegistry.acquire` calls `id`, `self._owners.get`, `ModelOwnershipError`; can raise `ModelOwnershipError`. - Inputs: - `model` (Any; required): Required keyword-only input. - `engine` (Any; required): Required keyword-only input. - `engine_id` (str; required): Required keyword-only input. - `force` (bool; optional; default `True`): Optional keyword-only input; defaults to `True`. - Return annotation: `None` - Calls: id, self._owners.get, ModelOwnershipError - State reads: self._owners.get, self._owners - Raises directly: ModelOwnershipError ## `vllm_mlx.model_registry._ModelOwnershipRegistry.release` - Kind: method - Signature: `def release(self, model: Any, engine_id: str) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/model_registry.py#L65-L69 - Implementation: Method `_ModelOwnershipRegistry.release` calls `id`, `self._owners.get`, `self._owners.pop`. Method `_ModelOwnershipRegistry.release` calls `id`, `self._owners.get`, `self._owners.pop`. - Inputs: - `model` (Any; required): Required positional or keyword input. - `engine_id` (str; required): Required positional or keyword input. - Return annotation: `None` - Calls: id, self._owners.get, self._owners.pop - State reads: self._owners.get, self._owners, self._owners.pop ## `vllm_mlx.model_registry._ModelOwnershipRegistry.is_owned` - Kind: method - Signature: `def is_owned(self, model: Any) -> tuple[bool, str | None]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/model_registry.py#L71-L76 - Implementation: Method `_ModelOwnershipRegistry.is_owned` calls `id`, `self._owners.get`; has 2 explicit return paths. Method `_ModelOwnershipRegistry.is_owned` calls `id`, `self._owners.get`; has 2 explicit return paths. - Inputs: - `model` (Any; required): Required positional or keyword input. - Return annotation: `tuple[bool, str | None]` - Calls: id, self._owners.get - State reads: self._owners.get, self._owners - Return expressions: (True, owner); (False, None) ## `vllm_mlx.model_registry._ModelOwnershipRegistry.get_stats` - Kind: method - Signature: `def get_stats(self) -> dict[str, Any]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/model_registry.py#L78-L82 - Implementation: Method `_ModelOwnershipRegistry.get_stats` calls `len`; returns `{'total_entries': len(self._owners), 'active_owners': len(self._owners)}`. Method `_ModelOwnershipRegistry.get_stats` calls `len`; returns `{'total_entries': len(self._owners), 'active_owners': len(self._owners)}`. - Inputs: none - Return annotation: `dict[str, Any]` - Calls: len - State reads: self._owners - Return expressions: {'total_entries': len(self._owners), 'active_owners': len(self._owners)} ## `vllm_mlx.model_registry.get_registry` - Kind: function - Signature: `def get_registry() -> _ModelOwnershipRegistry` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/model_registry.py#L88-L90 - Implementation: Function `get_registry` returns `_ownership_registry`. Return the global model ownership registry used by EngineCore. - Inputs: none - Return annotation: `_ModelOwnershipRegistry` - Return expressions: _ownership_registry ## `vllm_mlx.model_registry.RegistryServeDefaults` - Kind: class - Signature: `class RegistryServeDefaults` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/model_registry.py#L109-L125 - Implementation: Class `RegistryServeDefaults` declares 0 direct member(s). Global serve defaults inherited by registry entries. - Inputs: - `continuous_batching` (bool; required): Required constructor field. - `force_mllm` (bool; required): Required constructor field. - `enable_mtp` (bool; required): Required constructor field. - `prefill_step_size` (int; required): Required constructor field. - `specprefill_enabled` (bool; required): Required constructor field. - `specprefill_threshold` (int; required): Required constructor field. - `specprefill_keep_pct` (float; required): Required constructor field. - `specprefill_backbone_pct` (float; required): Required constructor field. - `specprefill_draft_model` (str | None; required): Required constructor field. - `stream_interval` (int; required): Required constructor field. - `gpu_memory_utilization` (float; required): Required constructor field. - `scheduler_config` (SchedulerConfig | None; required): Required constructor field. - `max_tokens` (int; required): Required constructor field. - `download_config` (DownloadConfig; required): Required constructor field. - Constructs: `vllm_mlx.model_registry.RegistryServeDefaults` - Decorators: dataclass(frozen=True) ## `vllm_mlx.model_registry.ContentionPolicy` - Kind: class - Signature: `class ContentionPolicy` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/model_registry.py#L129-L134 - Implementation: Class `ContentionPolicy` declares 0 direct member(s). Policy used when a new model cannot fit inside the memory budget. - Inputs: - `strategy` (ContentionStrategy; optional; default `'wait_then_fail'`): Optional constructor field; defaults to `'wait_then_fail'`. - `wait_timeout_s` (float | None; optional; default `30.0`): Optional constructor field; defaults to `30.0`. - `preempt_after_s` (float | None; optional; default `None`): Optional constructor field; defaults to `None`. - Constructs: `vllm_mlx.model_registry.ContentionPolicy` - Decorators: dataclass(frozen=True) ## `vllm_mlx.model_registry.RegistryManagerConfig` - Kind: class - Signature: `class RegistryManagerConfig` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/model_registry.py#L138-L142 - Implementation: Class `RegistryManagerConfig` declares 0 direct member(s). Global registry manager configuration. - Inputs: - `memory_budget_bytes` (int; required): Required constructor field. - `policy` (ContentionPolicy; required): Required constructor field. - Constructs: `vllm_mlx.model_registry.RegistryManagerConfig` - Decorators: dataclass(frozen=True) ## `vllm_mlx.model_registry.RegisteredModel` - Kind: class - Signature: `class RegisteredModel` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/model_registry.py#L146-L163 - Implementation: Class `RegisteredModel` declares 0 direct member(s). One configured model entry. - Inputs: - `name` (str; required): Required constructor field. - `source` (str; required): Required constructor field. - `preload` (bool; optional; default `False`): Optional constructor field; defaults to `False`. - `continuous_batching` (bool | None; optional; default `None`): Optional constructor field; defaults to `None`. - `force_mllm` (bool | None; optional; default `None`): Optional constructor field; defaults to `None`. - `enable_mtp` (bool | None; optional; default `None`): Optional constructor field; defaults to `None`. - `prefill_step_size` (int | None; optional; default `None`): Optional constructor field; defaults to `None`. - `specprefill_enabled` (bool | None; optional; default `None`): Optional constructor field; defaults to `None`. - `specprefill_threshold` (int | None; optional; default `None`): Optional constructor field; defaults to `None`. - `specprefill_keep_pct` (float | None; optional; default `None`): Optional constructor field; defaults to `None`. - `specprefill_backbone_pct` (float | None; optional; default `None`): Optional constructor field; defaults to `None`. - `specprefill_draft_model` (str | None; optional; default `None`): Optional constructor field; defaults to `None`. - `stream_interval` (int | None; optional; default `None`): Optional constructor field; defaults to `None`. - `gpu_memory_utilization` (float | None; optional; default `None`): Optional constructor field; defaults to `None`. - `estimated_memory_bytes` (int | None; optional; default `None`): Optional constructor field; defaults to `None`. - Constructs: `vllm_mlx.model_registry.RegisteredModel` - Decorators: dataclass(frozen=True) ## `vllm_mlx.model_registry.ResolvedModelConfig` - Kind: class - Signature: `class ResolvedModelConfig` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/model_registry.py#L167-L184 - Implementation: Class `ResolvedModelConfig` declares 0 direct member(s). Effective configuration for a loaded model. - Inputs: - `entry` (RegisteredModel; required): Required constructor field. - `resolved_source` (str; required): Required constructor field. - `continuous_batching` (bool; required): Required constructor field. - `force_mllm` (bool; required): Required constructor field. - `enable_mtp` (bool; required): Required constructor field. - `prefill_step_size` (int; required): Required constructor field. - `specprefill_enabled` (bool; required): Required constructor field. - `specprefill_threshold` (int; required): Required constructor field. - `specprefill_keep_pct` (float; required): Required constructor field. - `specprefill_backbone_pct` (float; required): Required constructor field. - `specprefill_draft_model` (str | None; required): Required constructor field. - `stream_interval` (int; required): Required constructor field. - `gpu_memory_utilization` (float; required): Required constructor field. - `scheduler_config` (SchedulerConfig | None; required): Required constructor field. - `estimated_memory_bytes` (int; required): Required constructor field. - Constructs: `vllm_mlx.model_registry.ResolvedModelConfig` - Decorators: dataclass(frozen=True) ## `vllm_mlx.model_registry.LoadedModel` - Kind: class - Signature: `class LoadedModel` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/model_registry.py#L188-L197 - Implementation: Class `LoadedModel` declares 0 direct member(s). Runtime state for a loaded engine. - Inputs: - `config` (ResolvedModelConfig; required): Required constructor field. - `engine` (BaseEngine; required): Required constructor field. - `loaded_at` (float; optional; default `field(default_factory=time.time)`): Optional constructor field; defaults to `field(default_factory=time.time)`. - `last_used_at` (float; optional; default `field(default_factory=time.time)`): Optional constructor field; defaults to `field(default_factory=time.time)`. - `active_requests` (int; optional; default `0`): Optional constructor field; defaults to `0`. - `active_tasks` (set[asyncio.Task[Any]]; optional; default `field(default_factory=set)`): Optional constructor field; defaults to `field(default_factory=set)`. - `preempting` (bool; optional; default `False`): Optional constructor field; defaults to `False`. - Constructs: `vllm_mlx.model_registry.LoadedModel` - Decorators: dataclass ## `vllm_mlx.model_registry.PendingLoad` - Kind: class - Signature: `class PendingLoad` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/model_registry.py#L201-L206 - Implementation: Class `PendingLoad` declares 0 direct member(s). A reserved model load in progress. - Inputs: - `model_name` (str; required): Required constructor field. - `required_bytes` (int; required): Required constructor field. - `future` (asyncio.Future[LoadedModel]; required): Required constructor field. - Constructs: `vllm_mlx.model_registry.PendingLoad` - Decorators: dataclass ## `vllm_mlx.model_registry.ModelLease` - Kind: class - Signature: `class ModelLease` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/model_registry.py#L210-L231 - Implementation: Class `ModelLease` declares 3 direct member(s). Active lease for a loaded model. - Inputs: - `manager` ('ModelManager | None'; required): Required constructor field. - `model_name` (str; required): Required constructor field. - `engine` (BaseEngine; required): Required constructor field. - `release_cb` (Callable[[], Awaitable[None]]; required): Required constructor field. - Constructs: `vllm_mlx.model_registry.ModelLease` - Decorators: dataclass ## `vllm_mlx.model_registry.ModelLease.release` - Kind: method - Signature: `async def release(self) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/model_registry.py#L218-L225 - Implementation: Method `ModelLease.release` updates `self.manager`; calls `self.release_cb`; awaits asynchronous work; returns `None`. Release this lease once and allow the model to become evictable. - Inputs: none - Return annotation: `None` - Calls: self.release_cb - State reads: self.manager, self.release_cb - State writes: self.manager - Return expressions: None ## `vllm_mlx.model_registry.ModelLease.__aenter__` - Kind: method - Signature: `async def __aenter__(self) -> 'ModelLease'` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/model_registry.py#L227-L228 - Implementation: Method `ModelLease.__aenter__` returns `self`. Method `ModelLease.__aenter__` returns `self`. - Inputs: none - Return annotation: `'ModelLease'` - Return expressions: self ## `vllm_mlx.model_registry.ModelLease.__aexit__` - Kind: method - Signature: `async def __aexit__(self, exc_type, exc, tb) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/model_registry.py#L230-L231 - Implementation: Method `ModelLease.__aexit__` calls `self.release`; awaits asynchronous work. Method `ModelLease.__aexit__` calls `self.release`; awaits asynchronous work. - Inputs: - `exc_type` (not annotated; required): Required positional or keyword input. - `exc` (not annotated; required): Required positional or keyword input. - `tb` (not annotated; required): Required positional or keyword input. - Return annotation: `None` - Calls: self.release - State reads: self.release ## `vllm_mlx.model_registry._clone_scheduler_config` - Kind: function - Signature: `def _clone_scheduler_config(config: SchedulerConfig | None) -> SchedulerConfig | None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/model_registry.py#L234-L238 - Implementation: Function `_clone_scheduler_config` calls `SchedulerConfig`, `vars`; has 2 explicit return paths. Clone a SchedulerConfig so per-model overrides do not mutate globals. - Inputs: - `config` (SchedulerConfig | None; required): Required positional or keyword input. - Return annotation: `SchedulerConfig | None` - Calls: SchedulerConfig, vars - Return expressions: None; SchedulerConfig(**vars(config)) ## `vllm_mlx.model_registry._parse_memory_budget_bytes` - Kind: function - Signature: `def _parse_memory_budget_bytes(value: Any) -> int` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/model_registry.py#L241-L256 - Implementation: Function `_parse_memory_budget_bytes` calls `ValueError`, `isinstance`, `int`, `float`; can raise `ValueError`, `TypeError`; has 5 explicit return paths. Parse a memory budget from bytes, MB, or GB. - Inputs: - `value` (Any; required): Required positional or keyword input. - Return annotation: `int` - Calls: ValueError, isinstance, int, float, value.strip().lower, value.strip, raw.endswith, TypeError - Raises directly: ValueError, TypeError - Return expressions: int(float(value) * 1024 ** 3); int(float(raw[:-2]) * 1024 ** 3); int(float(raw[:-2]) * 1024 ** 2); int(float(raw[:-1])); int(float(raw) * 1024 ** 3) ## `vllm_mlx.model_registry._safe_available_memory_bytes` - Kind: function - Signature: `def _safe_available_memory_bytes() -> int` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/model_registry.py#L259-L263 - Implementation: Function `_safe_available_memory_bytes` calls `int`, `psutil.virtual_memory`; has 2 explicit return paths. Best-effort available system memory. - Inputs: none - Return annotation: `int` - Calls: int, psutil.virtual_memory - Return expressions: 0; int(psutil.virtual_memory().available) ## `vllm_mlx.model_registry._device_working_set_bytes` - Kind: function - Signature: `def _device_working_set_bytes() -> int | None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/model_registry.py#L266-L282 - Implementation: Function `_device_working_set_bytes` calls `mx.metal.is_available`, `mx.device_info`, `info.get`, `int`; has 2 explicit return paths. Best-effort Metal recommended working-set size, or None when unavailable. - Inputs: none - Return annotation: `int | None` - Calls: mx.metal.is_available, mx.device_info, info.get, int, logger.debug - Return expressions: None; working_set or None ## `vllm_mlx.model_registry.MemoryBudgetReport` - Kind: class - Signature: `class MemoryBudgetReport` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/model_registry.py#L286-L339 - Implementation: Class `MemoryBudgetReport` declares 3 direct member(s). Reconciliation of the manager weight budget with the Metal ceiling. The manager budget counts model *weights* only, and the Metal allocation ceiling (``gpu_memory_utilization`` x device working set) is process-wide. Those two are directly comparable, so a budget above the ceiling is a deterministic conflict: the manager will keep models resident that MLX cannot allocate, and the load fails instead of evicting. The prefix-cache limit is deliberately *not* folded into that comparison. ``cache_memory_mb`` is a per-engine maximum — it is cloned into each resident continuous-batching engine and allocated lazily, and simple-mode entries never receive it at all — so it is neither a single process-wide reservation nor a bound that can be subtracted once. It is reported alongside the ceiling instead, with its own conflict check. - Inputs: - `budget_bytes` (int; required): Required constructor field. - `device_working_set_bytes` (int | None; required): Required constructor field. - `gpu_memory_utilization` (float | None; required): Required constructor field. - `gpu_memory_utilization_source` (str | None; required): Required constructor field. - `per_engine_cache_limit_bytes` (int | None; required): Required constructor field. - `per_engine_cache_percent` (float | None; required): Required constructor field. - `continuous_batching_entries` (int; required): Required constructor field. - `total_entries` (int; required): Required constructor field. - Constructs: `vllm_mlx.model_registry.MemoryBudgetReport` - Decorators: dataclass(frozen=True) ## `vllm_mlx.model_registry.MemoryBudgetReport.allocation_ceiling_bytes` - Kind: method - Signature: `def allocation_ceiling_bytes(self) -> int | None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/model_registry.py#L313-L322 - Implementation: Method `MemoryBudgetReport.allocation_ceiling_bytes` calls `int`; has 2 explicit return paths. Metal soft allocation limit that will be installed at engine start. ``None`` when no ceiling can be attributed: either MLX cannot report a device working set, or no entry will install one (only ``BatchedEngine`` calls ``mx.set_memory_limit``). - Inputs: none - Return annotation: `int | None` - Decorators: property - Calls: int - State reads: self.device_working_set_bytes, self.gpu_memory_utilization - Return expressions: None; int(self.device_working_set_bytes * self.gpu_memory_utilization) ## `vllm_mlx.model_registry.MemoryBudgetReport.exceeds_ceiling` - Kind: method - Signature: `def exceeds_ceiling(self) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/model_registry.py#L325-L331 - Implementation: Method `MemoryBudgetReport.exceeds_ceiling` returns `ceiling is not None and self.budget_bytes > ceiling`. True when the weights budget alone cannot fit under the ceiling. Both sides are process-wide totals, so this is the deterministic check. - Inputs: none - Return annotation: `bool` - Decorators: property - State reads: self.allocation_ceiling_bytes, self.budget_bytes - Return expressions: ceiling is not None and self.budget_bytes > ceiling ## `vllm_mlx.model_registry.MemoryBudgetReport.cache_limit_exceeds_ceiling` - Kind: method - Signature: `def cache_limit_exceeds_ceiling(self) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/model_registry.py#L334-L339 - Implementation: Method `MemoryBudgetReport.cache_limit_exceeds_ceiling` has 2 explicit return paths. True when one engine's prefix cache could alone fill the ceiling. - Inputs: none - Return annotation: `bool` - Decorators: property - State reads: self.allocation_ceiling_bytes, self.per_engine_cache_limit_bytes - Return expressions: False; self.per_engine_cache_limit_bytes >= ceiling ## `vllm_mlx.model_registry.build_memory_budget_report` - Kind: function - Signature: `def build_memory_budget_report(manager_config: RegistryManagerConfig, registry: dict[str, RegisteredModel], defaults: RegistryServeDefaults, *, device_working_set_bytes: int | None=None) -> MemoryBudgetReport` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/model_registry.py#L342-L421 - Implementation: Function `build_memory_budget_report` calls `_device_working_set_bytes`, `sorted`, `candidates.append`, `len`; returns `MemoryBudgetReport(budget_bytes=manager_config.memory_budget_bytes, device_working_set_bytes=device_working_set_bytes, …`. Reconcile the manager weight budget against the Metal allocation ceiling. The Metal limit is process-wide but is re-installed by every ``BatchedEngine`` start, so the ceiling the manager has to live under is the *lowest* utilization among the entries that actually install one. Only continuous-batching entries qualify: ``SimpleEngine`` never calls ``mx.set_memory_limit`` and is not even given a ``gpu_memory_utilization``. A registry with no continuous-batching entries therefore gets no attributed ceiling rather than one derived from a value nothing installs. - Inputs: - `manager_config` (RegistryManagerConfig; required): Required positional or keyword input. - `registry` (dict[str, RegisteredModel]; required): Required positional or keyword input. - `defaults` (RegistryServeDefaults; required): Required positional or keyword input. - `device_working_set_bytes` (int | None; optional; default `None`): Optional keyword-only input; defaults to `None`. - Return annotation: `MemoryBudgetReport` - Calls: _device_working_set_bytes, sorted, candidates.append, len, min, getattr, int, float, MemoryBudgetReport - Return expressions: MemoryBudgetReport(budget_bytes=manager_config.memory_budget_bytes, device_working_set_bytes=device_working_set_bytes, … ## `vllm_mlx.model_registry.log_memory_budget_report` - Kind: function - Signature: `def log_memory_budget_report(report: MemoryBudgetReport) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/model_registry.py#L424-L502 - Implementation: Function `log_memory_budget_report` calls `logger.info`, `logger.warning`; returns `None`. Log the budget/ceiling reconciliation, warning when they conflict. - Inputs: - `report` (MemoryBudgetReport; required): Required positional or keyword input. - Return annotation: `None` - Calls: logger.info, logger.warning - Return expressions: None ## `vllm_mlx.model_registry._estimate_model_bytes_from_source` - Kind: function - Signature: `def _estimate_model_bytes_from_source(source: str) -> int` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/model_registry.py#L505-L521 - Implementation: Function `_estimate_model_bytes_from_source` calls `Path`, `path.exists`, `path.is_file`, `path.stat`; has 3 explicit return paths. Estimate model footprint from local artifact size when possible. - Inputs: - `source` (str; required): Required positional or keyword input. - Return annotation: `int` - Calls: Path, path.exists, path.is_file, path.stat, path.rglob, fp.stat - Return expressions: 0; path.stat().st_size if path.suffix in {'.safetensors', '.gguf'} else 0; total ## `vllm_mlx.model_registry.load_registry_config` - Kind: function - Signature: `def load_registry_config(config_path: str | os.PathLike[str], defaults: RegistryServeDefaults) -> tuple[RegistryManagerConfig, dict[str, RegisteredModel]]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/model_registry.py#L524-L621 - Implementation: Function `load_registry_config` calls `yaml.safe_load`, `Path(config_path).read_text`, `Path`, `raw.get`; can raise `ValueError`; returns `(manager, registry)`. Load and validate the models registry YAML file. - Inputs: - `config_path` (str | os.PathLike[str]; required): Required positional or keyword input. - `defaults` (RegistryServeDefaults; required): Required positional or keyword input. - Return annotation: `tuple[RegistryManagerConfig, dict[str, RegisteredModel]]` - Calls: yaml.safe_load, Path(config_path).read_text, Path, raw.get, isinstance, ValueError, manager_raw.get, ContentionPolicy, policy_raw.get, float, RegistryManagerConfig, _parse_memory_budget_bytes, item.get, int, math.isfinite, RegisteredModel, str, bool - Raises directly: ValueError - Return expressions: (manager, registry) ## `vllm_mlx.model_registry.ModelManager` - Kind: class - Signature: `class ModelManager` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/model_registry.py#L624-L1201 - Implementation: Class `ModelManager` declares 27 direct member(s). Registry-backed model manager with lazy load and memory-budget eviction. - Inputs: - `manager_config` (RegistryManagerConfig; required): Required positional or keyword input. - `registry` (dict[str, RegisteredModel]; required): Required positional or keyword input. - `defaults` (RegistryServeDefaults; required): Required positional or keyword input. - `engine_factory` (EngineFactory | None; optional; default `None`): Optional keyword-only input; defaults to `None`. - Constructs: `vllm_mlx.model_registry.ModelManager` ## `vllm_mlx.model_registry.ModelManager.__init__` - Kind: method - Signature: `def __init__(self, manager_config: RegistryManagerConfig, registry: dict[str, RegisteredModel], defaults: RegistryServeDefaults, *, engine_factory: EngineFactory | None=None) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/model_registry.py#L627-L643 - Implementation: Method `ModelManager.__init__` updates `self._config`, `self._registry`, `self._defaults`, `self._engine_factory`; calls `asyncio.Condition`. Method `ModelManager.__init__` updates `self._config`, `self._registry`, `self._defaults`, `self._engine_factory`; calls `asyncio.Condition`. - Inputs: - `manager_config` (RegistryManagerConfig; required): Required positional or keyword input. - `registry` (dict[str, RegisteredModel]; required): Required positional or keyword input. - `defaults` (RegistryServeDefaults; required): Required positional or keyword input. - `engine_factory` (EngineFactory | None; optional; default `None`): Optional keyword-only input; defaults to `None`. - Return annotation: `None` - Calls: asyncio.Condition - State writes: self._config, self._registry, self._defaults, self._engine_factory, self._loaded, self._loading, self._unloading, self._condition, self._shutting_down ## `vllm_mlx.model_registry.ModelManager.memory_budget_bytes` - Kind: method - Signature: `def memory_budget_bytes(self) -> int` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/model_registry.py#L646-L649 - Implementation: Method `ModelManager.memory_budget_bytes` returns `self._config.memory_budget_bytes`. Return the registry's configured resident-model memory budget. - Inputs: none - Return annotation: `int` - Decorators: property - State reads: self._config.memory_budget_bytes, self._config - Return expressions: self._config.memory_budget_bytes ## `vllm_mlx.model_registry.ModelManager.registered_model_names` - Kind: method - Signature: `def registered_model_names(self) -> list[str]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/model_registry.py#L652-L654 - Implementation: Method `ModelManager.registered_model_names` calls `sorted`, `self._registry.keys`; returns `sorted(self._registry.keys())`. Return sorted list of all registered model names. - Inputs: none - Return annotation: `list[str]` - Decorators: property - Calls: sorted, self._registry.keys - State reads: self._registry.keys, self._registry - Return expressions: sorted(self._registry.keys()) ## `vllm_mlx.model_registry.ModelManager.has_model` - Kind: method - Signature: `def has_model(self, model_name: str) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/model_registry.py#L656-L659 - Implementation: Method `ModelManager.has_model` returns `model_name in self._registry`. Return whether a model name is present in the serving registry. - Inputs: - `model_name` (str; required): Required positional or keyword input. - Return annotation: `bool` - State reads: self._registry - Return expressions: model_name in self._registry ## `vllm_mlx.model_registry.ModelManager.list_models` - Kind: method - Signature: `def list_models(self) -> list[dict[str, Any]]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/model_registry.py#L661-L699 - Implementation: Method `ModelManager.list_models` calls `self._registry.items`, `self._loaded.get`, `self._unloading.get`, `self._loading.get`; returns `data`. Return registry state for /v1/models. - Inputs: none - Return annotation: `list[dict[str, Any]]` - Calls: self._registry.items, self._loaded.get, self._unloading.get, self._loading.get, self._resolve_estimated_bytes, data.append, round - State reads: self._registry.items, self._registry, self._loaded.get, self._loaded, self._unloading.get, self._unloading, self._loading.get, self._loading, self._resolve_estimated_bytes - Return expressions: data ## `vllm_mlx.model_registry.ModelManager.preload` - Kind: method - Signature: `async def preload(self) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/model_registry.py#L701-L706 - Implementation: Method `ModelManager.preload` calls `self._registry.values`, `self.acquire`, `lease.release`; awaits asynchronous work. Preload any entries marked preload=true. - Inputs: none - Return annotation: `None` - Calls: self._registry.values, self.acquire, lease.release - State reads: self._registry.values, self._registry, self.acquire ## `vllm_mlx.model_registry.ModelManager.shutdown` - Kind: method - Signature: `async def shutdown(self) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/model_registry.py#L708-L739 - Implementation: Method `ModelManager.shutdown` updates `self._shutting_down`; calls `set`, `self._loading.values`, `self._loaded.values`, `cancel_tasks.update`; awaits asynchronous work. Stop and unload every loaded engine. - Inputs: none - Return annotation: `None` - Calls: set, self._loading.values, self._loaded.values, cancel_tasks.update, list, self._loaded.keys, unloads.append, self._begin_unload_locked, self._condition.notify_all, task.cancel, self._run_unloads, asyncio.gather, remaining.append - State reads: self._condition, self._loading.values, self._loading, self._loaded.values, self._loaded, self._loaded.keys, self._begin_unload_locked, self._condition.notify_all, self._run_unloads - State writes: self._shutting_down ## `vllm_mlx.model_registry.ModelManager.acquire` - Kind: method - Signature: `async def acquire(self, model_name: str) -> ModelLease` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/model_registry.py#L741-L819 - Implementation: Method `ModelManager.acquire` calls `KeyError`, `time.monotonic`, `set`, `RuntimeError`; awaits asynchronous work; can raise `KeyError`, `RuntimeError`; returns `claimed`. Acquire a lease for a configured model. - Inputs: - `model_name` (str; required): Required positional or keyword input. - Return annotation: `ModelLease` - Calls: KeyError, time.monotonic, set, RuntimeError, self._claim_loaded_locked, self._loading.get, self._remaining_wait_timeout, self._resolve_estimated_bytes, self._collect_idle_unloads_locked, self._can_reserve_locked, self._reserve_load_locked, self._maybe_preempt_locked, self._should_wait_locked, self._run_unloads, task.cancel, self._wait_for_change, self._execute_load - State reads: self._registry, self._condition, self._shutting_down, self._claim_loaded_locked, self._loading.get, self._loading, self._unloading, self._remaining_wait_timeout, self._resolve_estimated_bytes, self._collect_idle_unloads_locked, self._can_reserve_locked, self._reserve_load_locked, self._maybe_preempt_locked, self._should_wait_locked, self._config.memory_budget_bytes, self._config, self._run_unloads, self._wait_for_change, self._execute_load - Raises directly: KeyError, RuntimeError - Return expressions: claimed ## `vllm_mlx.model_registry.ModelManager.release` - Kind: method - Signature: `async def release(self, model_name: str) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/model_registry.py#L821-L842 - Implementation: Method `ModelManager.release` calls `self._loaded.get`, `max`, `time.time`, `asyncio.current_task`; awaits asynchronous work; returns `None`. Release a previously acquired model lease. - Inputs: - `model_name` (str; required): Required positional or keyword input. - Return annotation: `None` - Calls: self._loaded.get, max, time.time, asyncio.current_task, loaded.active_tasks.discard, self._begin_unload_locked, self._condition.notify_all, self._run_unloads - State reads: self._condition, self._loaded.get, self._loaded, self._begin_unload_locked, self._condition.notify_all, self._run_unloads - Return expressions: None ## `vllm_mlx.model_registry.ModelManager._claim_loaded_locked` - Kind: method - Signature: `def _claim_loaded_locked(self, model_name: str, *, loaded_override: LoadedModel | None=None) -> ModelLease | None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/model_registry.py#L844-L874 - Implementation: Method `ModelManager._claim_loaded_locked` calls `self._loaded.get`, `time.time`, `asyncio.current_task`, `loaded.active_tasks.add`; has 2 explicit return paths. Method `ModelManager._claim_loaded_locked` calls `self._loaded.get`, `time.time`, `asyncio.current_task`, `loaded.active_tasks.add`; has 2 explicit return paths. - Inputs: - `model_name` (str; required): Required positional or keyword input. - `loaded_override` (LoadedModel | None; optional; default `None`): Optional keyword-only input; defaults to `None`. - Return annotation: `ModelLease | None` - Calls: self._loaded.get, time.time, asyncio.current_task, loaded.active_tasks.add, ModelLease - State reads: self._loaded.get, self._loaded - Return expressions: None; ModelLease(manager=self, model_name=model_name, engine=loaded.engine, release_cb=_release) ## `vllm_mlx.model_registry.ModelManager._claim_loaded_locked._release` - Kind: nested function - Signature: `async def _release() -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/model_registry.py#L866-L867 - Implementation: Nested Function `ModelManager._claim_loaded_locked._release` calls `self.release`; awaits asynchronous work. Nested Function `ModelManager._claim_loaded_locked._release` calls `self.release`; awaits asynchronous work. - Inputs: none - Return annotation: `None` - Calls: self.release - State reads: self.release ## `vllm_mlx.model_registry.ModelManager._execute_load` - Kind: method - Signature: `async def _execute_load(self, pending: PendingLoad) -> LoadedModel` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/model_registry.py#L876-L913 - Implementation: Method `ModelManager._execute_load` calls `self._resolve_source`, `self._instantiate_model`, `self._loading.pop`, `current.future.done`; awaits asynchronous work; can raise `RuntimeError`; returns `loaded`. Instantiate a reserved model load outside the manager lock. - Inputs: - `pending` (PendingLoad; required): Required positional or keyword input. - Return annotation: `LoadedModel` - Calls: self._resolve_source, self._instantiate_model, self._loading.pop, current.future.done, current.future.set_exception, self._condition.notify_all, RuntimeError, current.future.set_result, unload_after_load.engine.stop - State reads: self._registry, self._resolve_source, self._instantiate_model, self._condition, self._loading.pop, self._loading, self._condition.notify_all, self._shutting_down, self._loaded - Raises directly: RuntimeError - Return expressions: loaded ## `vllm_mlx.model_registry.ModelManager._wait_for_change` - Kind: method - Signature: `async def _wait_for_change(self, timeout: float | None) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/model_registry.py#L915-L922 - Implementation: Method `ModelManager._wait_for_change` calls `self._condition.wait`, `RuntimeError`, `asyncio.wait_for`; awaits asynchronous work; can raise `RuntimeError`; returns `None`. Method `ModelManager._wait_for_change` calls `self._condition.wait`, `RuntimeError`, `asyncio.wait_for`; awaits asynchronous work; can raise `RuntimeError`; returns `None`. - Inputs: - `timeout` (float | None; required): Required positional or keyword input. - Return annotation: `None` - Calls: self._condition.wait, RuntimeError, asyncio.wait_for - State reads: self._condition, self._condition.wait - Raises directly: RuntimeError - Return expressions: None ## `vllm_mlx.model_registry.ModelManager._run_unloads` - Kind: method - Signature: `async def _run_unloads(self, unloads: list[LoadedModel]) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/model_registry.py#L924-L931 - Implementation: Method `ModelManager._run_unloads` calls `loaded.engine.stop`, `self._unloading.pop`, `self._condition.notify_all`; awaits asynchronous work. Method `ModelManager._run_unloads` calls `loaded.engine.stop`, `self._unloading.pop`, `self._condition.notify_all`; awaits asynchronous work. - Inputs: - `unloads` (list[LoadedModel]; required): Required positional or keyword input. - Return annotation: `None` - Calls: loaded.engine.stop, self._unloading.pop, self._condition.notify_all - State reads: self._condition, self._unloading.pop, self._unloading, self._condition.notify_all ## `vllm_mlx.model_registry.ModelManager._reserve_load_locked` - Kind: method - Signature: `def _reserve_load_locked(self, model_name: str, required_bytes: int) -> PendingLoad` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/model_registry.py#L933-L941 - Implementation: Method `ModelManager._reserve_load_locked` calls `asyncio.get_running_loop().create_future`, `asyncio.get_running_loop`, `PendingLoad`; returns `pending`. Method `ModelManager._reserve_load_locked` calls `asyncio.get_running_loop().create_future`, `asyncio.get_running_loop`, `PendingLoad`; returns `pending`. - Inputs: - `model_name` (str; required): Required positional or keyword input. - `required_bytes` (int; required): Required positional or keyword input. - Return annotation: `PendingLoad` - Calls: asyncio.get_running_loop().create_future, asyncio.get_running_loop, PendingLoad - State reads: self._loading - Return expressions: pending ## `vllm_mlx.model_registry.ModelManager._begin_unload_locked` - Kind: method - Signature: `def _begin_unload_locked(self, model_name: str) -> LoadedModel` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/model_registry.py#L943-L946 - Implementation: Method `ModelManager._begin_unload_locked` calls `self._loaded.pop`; returns `loaded`. Method `ModelManager._begin_unload_locked` calls `self._loaded.pop`; returns `loaded`. - Inputs: - `model_name` (str; required): Required positional or keyword input. - Return annotation: `LoadedModel` - Calls: self._loaded.pop - State reads: self._loaded.pop, self._loaded, self._unloading - Return expressions: loaded ## `vllm_mlx.model_registry.ModelManager._collect_idle_unloads_locked` - Kind: method - Signature: `def _collect_idle_unloads_locked(self, requested_model: str, required_bytes: int) -> list[LoadedModel]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/model_registry.py#L948-L968 - Implementation: Method `ModelManager._collect_idle_unloads_locked` calls `self._committed_bytes_locked`, `sorted`, `self._loaded.items`, `selected.append`; returns `selected`. Method `ModelManager._collect_idle_unloads_locked` calls `self._committed_bytes_locked`, `sorted`, `self._loaded.items`, `selected.append`; returns `selected`. - Inputs: - `requested_model` (str; required): Required positional or keyword input. - `required_bytes` (int; required): Required positional or keyword input. - Return annotation: `list[LoadedModel]` - Calls: self._committed_bytes_locked, sorted, self._loaded.items, selected.append, self._begin_unload_locked - State reads: self._committed_bytes_locked, self._loaded.items, self._loaded, self._config.memory_budget_bytes, self._config, self._begin_unload_locked - Return expressions: selected ## `vllm_mlx.model_registry.ModelManager._maybe_preempt_locked` - Kind: method - Signature: `def _maybe_preempt_locked(self, *, model_name: str, required_bytes: int, start: float) -> set[asyncio.Task[Any]]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/model_registry.py#L970-L1003 - Implementation: Method `ModelManager._maybe_preempt_locked` calls `self._should_preempt_locked`, `set`, `self._committed_bytes_locked`, `sorted`; has 2 explicit return paths. Method `ModelManager._maybe_preempt_locked` calls `self._should_preempt_locked`, `set`, `self._committed_bytes_locked`, `sorted`; has 2 explicit return paths. - Inputs: - `model_name` (str; required): Required keyword-only input. - `required_bytes` (int; required): Required keyword-only input. - `start` (float; required): Required keyword-only input. - Return annotation: `set[asyncio.Task[Any]]` - Calls: self._should_preempt_locked, set, self._committed_bytes_locked, sorted, self._loaded.items, cancel_tasks.update, self._condition.notify_all - State reads: self._should_preempt_locked, self._committed_bytes_locked, self._loaded.items, self._loaded, self._config.memory_budget_bytes, self._config, self._condition.notify_all, self._condition - Return expressions: set(); cancel_tasks ## `vllm_mlx.model_registry.ModelManager._should_wait_locked` - Kind: method - Signature: `def _should_wait_locked(self, start: float) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/model_registry.py#L1005-L1010 - Implementation: Method `ModelManager._should_wait_locked` calls `self._remaining_wait_timeout`; has 2 explicit return paths. Method `ModelManager._should_wait_locked` calls `self._remaining_wait_timeout`; has 2 explicit return paths. - Inputs: - `start` (float; required): Required positional or keyword input. - Return annotation: `bool` - Calls: self._remaining_wait_timeout - State reads: self._config.policy.strategy, self._config.policy, self._config, self._remaining_wait_timeout - Return expressions: False; timeout is None or timeout > 0 ## `vllm_mlx.model_registry.ModelManager._should_preempt_locked` - Kind: method - Signature: `def _should_preempt_locked(self, start: float) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/model_registry.py#L1012-L1020 - Implementation: Method `ModelManager._should_preempt_locked` calls `time.monotonic`; has 3 explicit return paths. Method `ModelManager._should_preempt_locked` calls `time.monotonic`; has 3 explicit return paths. - Inputs: - `start` (float; required): Required positional or keyword input. - Return annotation: `bool` - Calls: time.monotonic - State reads: self._config.policy, self._config - Return expressions: True; False; elapsed >= trigger ## `vllm_mlx.model_registry.ModelManager._remaining_wait_timeout` - Kind: method - Signature: `def _remaining_wait_timeout(self, start: float) -> float | None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/model_registry.py#L1022-L1026 - Implementation: Method `ModelManager._remaining_wait_timeout` calls `max`, `time.monotonic`; has 2 explicit return paths. Method `ModelManager._remaining_wait_timeout` calls `max`, `time.monotonic`; has 2 explicit return paths. - Inputs: - `start` (float; required): Required positional or keyword input. - Return annotation: `float | None` - Calls: max, time.monotonic - State reads: self._config.policy.wait_timeout_s, self._config.policy, self._config - Return expressions: None; max(timeout - (time.monotonic() - start), 0.0) ## `vllm_mlx.model_registry.ModelManager._can_reserve_locked` - Kind: method - Signature: `def _can_reserve_locked(self, required_bytes: int) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/model_registry.py#L1028-L1032 - Implementation: Method `ModelManager._can_reserve_locked` calls `self._committed_bytes_locked`; returns `self._committed_bytes_locked() + required_bytes <= self._config.memory_budget_bytes`. Method `ModelManager._can_reserve_locked` calls `self._committed_bytes_locked`; returns `self._committed_bytes_locked() + required_bytes <= self._config.memory_budget_bytes`. - Inputs: - `required_bytes` (int; required): Required positional or keyword input. - Return annotation: `bool` - Calls: self._committed_bytes_locked - State reads: self._committed_bytes_locked, self._config.memory_budget_bytes, self._config - Return expressions: self._committed_bytes_locked() + required_bytes <= self._config.memory_budget_bytes ## `vllm_mlx.model_registry.ModelManager._committed_bytes_locked` - Kind: method - Signature: `def _committed_bytes_locked(self) -> int` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/model_registry.py#L1034-L1042 - Implementation: Method `ModelManager._committed_bytes_locked` calls `sum`, `self._loaded.values`, `self._loading.values`, `self._unloading.values`; returns `loaded_bytes + loading_bytes + unloading_bytes`. Method `ModelManager._committed_bytes_locked` calls `sum`, `self._loaded.values`, `self._loading.values`, `self._unloading.values`; returns `loaded_bytes + loading_bytes + unloading_bytes`. - Inputs: none - Return annotation: `int` - Calls: sum, self._loaded.values, self._loading.values, self._unloading.values - State reads: self._loaded.values, self._loaded, self._loading.values, self._loading, self._unloading.values, self._unloading - Return expressions: loaded_bytes + loading_bytes + unloading_bytes ## `vllm_mlx.model_registry.ModelManager._instantiate_model` - Kind: method - Signature: `async def _instantiate_model(self, entry: RegisteredModel, resolved_source: str) -> LoadedModel` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/model_registry.py#L1044-L1073 - Implementation: Method `ModelManager._instantiate_model` calls `self._resolve_model_config`, `self._engine_factory`, `BatchedEngine`, `SimpleEngine`; awaits asynchronous work; returns `LoadedModel(config=config, engine=engine)`. Method `ModelManager._instantiate_model` calls `self._resolve_model_config`, `self._engine_factory`, `BatchedEngine`, `SimpleEngine`; awaits asynchronous work; returns `LoadedModel(config=config, engine=engine)`. - Inputs: - `entry` (RegisteredModel; required): Required positional or keyword input. - `resolved_source` (str; required): Required positional or keyword input. - Return annotation: `LoadedModel` - Calls: self._resolve_model_config, self._engine_factory, BatchedEngine, SimpleEngine, engine.start, LoadedModel - State reads: self._resolve_model_config, self._engine_factory - Return expressions: LoadedModel(config=config, engine=engine) ## `vllm_mlx.model_registry.ModelManager._resolve_source` - Kind: method - Signature: `async def _resolve_source(self, entry: RegisteredModel) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/model_registry.py#L1075-L1076 - Implementation: Method `ModelManager._resolve_source` calls `asyncio.to_thread`; awaits asynchronous work; returns `await asyncio.to_thread(self._resolve_source_sync, entry)`. Method `ModelManager._resolve_source` calls `asyncio.to_thread`; awaits asynchronous work; returns `await asyncio.to_thread(self._resolve_source_sync, entry)`. - Inputs: - `entry` (RegisteredModel; required): Required positional or keyword input. - Return annotation: `str` - Calls: asyncio.to_thread - State reads: self._resolve_source_sync - Return expressions: await asyncio.to_thread(self._resolve_source_sync, entry) ## `vllm_mlx.model_registry.ModelManager._resolve_source_sync` - Kind: method - Signature: `def _resolve_source_sync(self, entry: RegisteredModel) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/model_registry.py#L1078-L1087 - Implementation: Method `ModelManager._resolve_source_sync` calls `Path(source).exists`, `Path`, `ensure_model_downloaded`, `is_mllm_model`; has 2 explicit return paths. Method `ModelManager._resolve_source_sync` calls `Path(source).exists`, `Path`, `ensure_model_downloaded`, `is_mllm_model`; has 2 explicit return paths. - Inputs: - `entry` (RegisteredModel; required): Required positional or keyword input. - Return annotation: `str` - Calls: Path(source).exists, Path, ensure_model_downloaded, is_mllm_model, bool, str - State reads: self._defaults.download_config, self._defaults - Return expressions: source; str(downloaded) ## `vllm_mlx.model_registry.ModelManager._resolve_estimated_bytes` - Kind: method - Signature: `def _resolve_estimated_bytes(self, entry: RegisteredModel, resolved_source: str) -> int` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/model_registry.py#L1089-L1121 - Implementation: Method `ModelManager._resolve_estimated_bytes` calls `_estimate_model_bytes_from_source`, `Path`, `source_path.exists`, `ValueError`; can raise `ValueError`; has 3 explicit return paths. Method `ModelManager._resolve_estimated_bytes` calls `_estimate_model_bytes_from_source`, `Path`, `source_path.exists`, `ValueError`; can raise `ValueError`; has 3 explicit return paths. - Inputs: - `entry` (RegisteredModel; required): Required positional or keyword input. - `resolved_source` (str; required): Required positional or keyword input. - Return annotation: `int` - Calls: _estimate_model_bytes_from_source, Path, source_path.exists, ValueError, _safe_available_memory_bytes, logger.warning, max - Raises directly: ValueError - Return expressions: entry.estimated_memory_bytes; estimated; max(available // 8, 1) ## `vllm_mlx.model_registry.ModelManager._resolve_model_config` - Kind: method - Signature: `def _resolve_model_config(self, entry: RegisteredModel, resolved_source: str) -> ResolvedModelConfig` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/model_registry.py#L1123-L1201 - Implementation: Method `ModelManager._resolve_model_config` calls `_clone_scheduler_config`, `self._resolve_estimated_bytes`, `ResolvedModelConfig`; returns `ResolvedModelConfig(entry=entry, resolved_source=resolved_source, continuous_batching=continuous_batching, force_mllm=f…`. Method `ModelManager._resolve_model_config` calls `_clone_scheduler_config`, `self._resolve_estimated_bytes`, `ResolvedModelConfig`; returns `ResolvedModelConfig(entry=entry, resolved_source=resolved_source, continuous_batching=continuous_batching, force_mllm=f…`. - Inputs: - `entry` (RegisteredModel; required): Required positional or keyword input. - `resolved_source` (str; required): Required positional or keyword input. - Return annotation: `ResolvedModelConfig` - Calls: _clone_scheduler_config, self._resolve_estimated_bytes, ResolvedModelConfig - State reads: self._defaults.scheduler_config, self._defaults, self._defaults.continuous_batching, self._defaults.force_mllm, self._defaults.enable_mtp, self._defaults.prefill_step_size, self._defaults.specprefill_enabled, self._defaults.specprefill_threshold, self._defaults.specprefill_keep_pct, self._defaults.specprefill_backbone_pct, self._defaults.specprefill_draft_model, self._defaults.stream_interval, self._defaults.gpu_memory_utilization, self._resolve_estimated_bytes - Return expressions: ResolvedModelConfig(entry=entry, resolved_source=resolved_source, continuous_batching=continuous_batching, force_mllm=f… # Module `vllm_mlx.model_runner` MLX Model Runner for vLLM. This module implements the model runner that bridges vLLM's request handling with mlx-lm's inference capabilities. Includes low-level optimizations: - mx.compile() for kernel fusion - Memory bandwidth optimization - Prefill chunking for L2 cache efficiency Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/model_runner.py#L1-L476 ## `vllm_mlx.model_runner.SamplerOutput` - Kind: class - Signature: `class SamplerOutput` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/model_runner.py#L29-L33 - Implementation: Class `SamplerOutput` declares 0 direct member(s). Output from sampling. - Inputs: - `token_ids` (list[int]; required): Required constructor field. - `logprobs` (list[dict] | None; optional; default `None`): Optional constructor field; defaults to `None`. - Constructs: `vllm_mlx.model_runner.SamplerOutput` - Decorators: dataclass ## `vllm_mlx.model_runner.MLXModelRunnerOutput` - Kind: class - Signature: `class MLXModelRunnerOutput` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/model_runner.py#L37-L50 - Implementation: Class `MLXModelRunnerOutput` declares 0 direct member(s). Output from MLX model runner, compatible with vLLM's ModelRunnerOutput. - Inputs: - `req_id_to_token_ids` (dict[str, list[int]]; required): Required constructor field. - `req_id_to_logprobs` (dict[str, list[dict]] | None; optional; default `None`): Optional constructor field; defaults to `None`. - `num_tokens_generated` (int; optional; default `0`): Optional constructor field; defaults to `0`. - `generation_time_s` (float; optional; default `0.0`): Optional constructor field; defaults to `0.0`. - Constructs: `vllm_mlx.model_runner.MLXModelRunnerOutput` - Decorators: dataclass ## `vllm_mlx.model_runner.MLXModelRunner` - Kind: class - Signature: `class MLXModelRunner` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/model_runner.py#L53-L476 - Implementation: Class `MLXModelRunner` declares 16 direct member(s). Model runner that uses mlx-lm for inference. This class handles: - Model loading via mlx-lm - Converting vLLM requests to mlx-lm format - Running inference and returning results in vLLM format - KV cache management (delegated to mlx-lm) Optimizations: - mx.compile() for kernel fusion (fuses multiple ops into single Metal kernel) - Memory optimization for bandwidth efficiency - Prefill chunking for L2 cache utilization - Inputs: - `vllm_config` ('VllmConfig'; required): vLLM configuration - `enable_optimizations` (bool; optional; default `True`): Whether to enable low-level optimizations - Constructs: `vllm_mlx.model_runner.MLXModelRunner` ## `vllm_mlx.model_runner.MLXModelRunner.__init__` - Kind: method - Signature: `def __init__(self, vllm_config: 'VllmConfig', enable_optimizations: bool=True)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/model_runner.py#L69-L104 - Implementation: Method `MLXModelRunner.__init__` updates `self.vllm_config`, `self.model_config`, `self.cache_config`, `self.scheduler_config`; calls `logger.info`. Initialize MLX model runner. Args: vllm_config: vLLM configuration enable_optimizations: Whether to enable low-level optimizations - Inputs: - `vllm_config` ('VllmConfig'; required): vLLM configuration - `enable_optimizations` (bool; optional; default `True`): Whether to enable low-level optimizations - Return annotation: `not annotated` - Calls: logger.info - State reads: self.model_config.model, self.model_config - State writes: self.vllm_config, self.model_config, self.cache_config, self.scheduler_config, self.model, self.tokenizer, self._loaded, self._sampler, self._prompt_cache, self._num_cache_blocks, self._enable_optimizations, self._compiled_forward, self._hardware_info ## `vllm_mlx.model_runner.MLXModelRunner.load_model` - Kind: method - Signature: `def load_model(self) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/model_runner.py#L106-L145 - Implementation: Method `MLXModelRunner.load_model` updates `self.model`, `self.tokenizer`, `self._loaded`; calls `logger.info`, `time.time`, `load`, `self._create_default_sampler`; can raise `ImportError`; returns `None`. Load model using mlx-lm with optimizations. - Inputs: none - Return annotation: `None` - Calls: logger.info, time.time, load, self._create_default_sampler, self._apply_optimizations, ImportError, logger.error - State reads: self._loaded, self.model_config.model, self.model_config, self.model_config.trust_remote_code, self._create_default_sampler, self._enable_optimizations, self._apply_optimizations - State writes: self.model, self.tokenizer, self._loaded - Raises directly: ImportError - Return expressions: None ## `vllm_mlx.model_runner.MLXModelRunner._apply_optimizations` - Kind: method - Signature: `def _apply_optimizations(self) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/model_runner.py#L147-L168 - Implementation: Method `MLXModelRunner._apply_optimizations` updates `self._hardware_info`; calls `detect_hardware`, `logger.info`, `configure_memory_optimization`, `self._setup_compiled_forward`. Apply low-level optimizations for maximum performance. - Inputs: none - Return annotation: `None` - Calls: detect_hardware, logger.info, configure_memory_optimization, self._setup_compiled_forward, logger.warning - State reads: self._hardware_info.chip_name, self._hardware_info, self._hardware_info.total_memory_gb, self._hardware_info.memory_bandwidth_gbs, self._setup_compiled_forward - State writes: self._hardware_info ## `vllm_mlx.model_runner.MLXModelRunner._setup_compiled_forward` - Kind: method - Signature: `def _setup_compiled_forward(self) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/model_runner.py#L170-L193 - Implementation: Method `MLXModelRunner._setup_compiled_forward` updates `self._compiled_forward`; calls `hasattr`, `mx.compile`, `logger.info`, `logger.warning`; returns `None`. Setup compiled forward pass using mx.compile() for kernel fusion. This fuses multiple operations into single Metal kernels, reducing kernel launch overhead and improving throughput. - Inputs: none - Return annotation: `None` - Calls: hasattr, mx.compile, logger.info, logger.warning - State reads: self.model, self.model.__call__ - State writes: self._compiled_forward - Return expressions: None ## `vllm_mlx.model_runner.MLXModelRunner._create_default_sampler` - Kind: method - Signature: `def _create_default_sampler(self) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/model_runner.py#L195-L205 - Implementation: Method `MLXModelRunner._create_default_sampler` updates `self._sampler`; calls `make_sampler`, `logger.warning`. Create default sampler for generation. - Inputs: none - Return annotation: `None` - Calls: make_sampler, logger.warning - State writes: self._sampler ## `vllm_mlx.model_runner.MLXModelRunner.initialize_cache` - Kind: method - Signature: `def initialize_cache(self, num_blocks: int) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/model_runner.py#L207-L210 - Implementation: Method `MLXModelRunner.initialize_cache` updates `self._num_cache_blocks`; calls `logger.info`. Initialize KV cache. - Inputs: - `num_blocks` (int; required): Required positional or keyword input. - Return annotation: `None` - Calls: logger.info - State writes: self._num_cache_blocks ## `vllm_mlx.model_runner.MLXModelRunner.get_kv_cache_spec` - Kind: method - Signature: `def get_kv_cache_spec(self) -> dict` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/model_runner.py#L215-L220 - Implementation: Method `MLXModelRunner.get_kv_cache_spec` returns `{'num_blocks': self._num_cache_blocks, 'block_size': self.cache_config.block_size}`. Get KV cache specification. - Inputs: none - Return annotation: `dict` - State reads: self._num_cache_blocks, self.cache_config.block_size, self.cache_config - Return expressions: {'num_blocks': self._num_cache_blocks, 'block_size': self.cache_config.block_size} ## `vllm_mlx.model_runner.MLXModelRunner.get_cache_block_size_bytes` - Kind: method - Signature: `def get_cache_block_size_bytes(self) -> int` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/model_runner.py#L222-L240 - Implementation: Method `MLXModelRunner.get_cache_block_size_bytes` calls `getattr`; has 2 explicit return paths. Calculate cache block size in bytes. - Inputs: none - Return annotation: `int` - Calls: getattr - State reads: self._loaded, self.model, self.cache_config.block_size, self.cache_config - Return expressions: 0; 2 * block_size * num_layers * num_kv_heads * head_size * 2 ## `vllm_mlx.model_runner.MLXModelRunner.warm_up` - Kind: method - Signature: `def warm_up(self) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/model_runner.py#L242-L263 - Implementation: Method `MLXModelRunner.warm_up` calls `self.load_model`, `logger.info`, `generate`, `logger.warning`. Warm up model with a test generation. - Inputs: none - Return annotation: `None` - Calls: self.load_model, logger.info, generate, logger.warning - State reads: self._loaded, self.load_model, self.model, self.tokenizer ## `vllm_mlx.model_runner.MLXModelRunner.execute_model` - Kind: method - Signature: `def execute_model(self, scheduler_output: 'SchedulerOutput') -> MLXModelRunnerOutput` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/model_runner.py#L265-L315 - Implementation: Method `MLXModelRunner.execute_model` calls `RuntimeError`, `time.time`, `self._generate_for_request`, `len`; can raise `RuntimeError`; returns `MLXModelRunnerOutput(req_id_to_token_ids=req_id_to_token_ids, num_tokens_generated=total_tokens, generation_time_s=gene…`. Execute model inference for scheduled requests. Args: scheduler_output: Contains requests to process Returns: MLXModelRunnerOutput with generated tokens - Inputs: - `scheduler_output` ('SchedulerOutput'; required): Contains requests to process - Return annotation: `MLXModelRunnerOutput` - Calls: RuntimeError, time.time, self._generate_for_request, len, self._continue_generation, MLXModelRunnerOutput - State reads: self._loaded, self._generate_for_request, self._continue_generation - Raises directly: RuntimeError - Return expressions: MLXModelRunnerOutput(req_id_to_token_ids=req_id_to_token_ids, num_tokens_generated=total_tokens, generation_time_s=gene… ## `vllm_mlx.model_runner.MLXModelRunner._prefill_with_chunking` - Kind: method - Signature: `def _prefill_with_chunking(self, input_ids: mx.array, cache: Optional[Any]=None) -> tuple[mx.array, Any]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/model_runner.py#L317-L362 - Implementation: Method `MLXModelRunner._prefill_with_chunking` calls `len`, `get_optimal_prefill_size`, `input_ids.reshape`, `forward_fn`; has 2 explicit return paths. Process prompt with optimal chunking for L2 cache efficiency. Long prompts are broken into chunks that fit in L2 cache, maximizing memory bandwidth utilization during prefill. Args: input_ids: Input token IDs [1, seq_len] cache: Optional existing KV cache Returns: Tuple of (logits, updated_cache) - Inputs: - `input_ids` (mx.array; required): Input token IDs [1, seq_len] - `cache` (Optional[Any]; optional; default `None`): Optional existing KV cache - Return annotation: `tuple[mx.array, Any]` - Calls: len, get_optimal_prefill_size, input_ids.reshape, forward_fn, range, mx.eval - State reads: self._compiled_forward, self.model - Return expressions: forward_fn(input_ids, cache=cache); (logits, cache) ## `vllm_mlx.model_runner.MLXModelRunner._prefill_with_chunking.get_optimal_prefill_size` - Kind: nested function - Signature: `def get_optimal_prefill_size(seq_len)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/model_runner.py#L339-L340 - Implementation: Nested Function `MLXModelRunner._prefill_with_chunking.get_optimal_prefill_size` calls `min`; returns `min(512, seq_len)`. Nested Function `MLXModelRunner._prefill_with_chunking.get_optimal_prefill_size` calls `min`; returns `min(512, seq_len)`. - Inputs: - `seq_len` (not annotated; required): Required positional or keyword input. - Return annotation: `not annotated` - Calls: min - Return expressions: min(512, seq_len) ## `vllm_mlx.model_runner.MLXModelRunner._generate_for_request` - Kind: method - Signature: `def _generate_for_request(self, prompt_token_ids: list[int], sampling_params: Any, max_tokens: int=1) -> list[int]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/model_runner.py#L364-L418 - Implementation: Method `MLXModelRunner._generate_for_request` calls `getattr`, `make_sampler`, `mx.array`, `generate_step`; has 2 explicit return paths. Generate tokens for a single request. Uses optimizations when enabled: - Compiled forward pass (kernel fusion) - Prefill chunking for long prompts Args: prompt_token_ids: Input token IDs sampling_params: Sampling parameters max_tokens: Maximum tokens to generate Returns: List of generated token IDs - Inputs: - `prompt_token_ids` (list[int]; required): Input token IDs - `sampling_params` (Any; required): Sampling parameters - `max_tokens` (int; optional; default `1`): Maximum tokens to generate - Return annotation: `list[int]` - Calls: getattr, make_sampler, mx.array, generate_step, hasattr, generated_ids.append, isinstance, len, logger.error - State reads: self.model - Return expressions: generated_ids; [] ## `vllm_mlx.model_runner.MLXModelRunner._continue_generation` - Kind: method - Signature: `def _continue_generation(self, req_id: str) -> list[int]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/model_runner.py#L420-L428 - Implementation: Method `MLXModelRunner._continue_generation` returns `[]`. Continue generation for an existing request. This is a placeholder - in a full implementation, we would use cached KV states to continue generation efficiently. - Inputs: - `req_id` (str; required): Required positional or keyword input. - Return annotation: `list[int]` - Return expressions: [] ## `vllm_mlx.model_runner.MLXModelRunner.decode_tokens` - Kind: method - Signature: `def decode_tokens(self, token_ids: list[int]) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/model_runner.py#L430-L434 - Implementation: Method `MLXModelRunner.decode_tokens` calls `self.tokenizer.decode`; has 2 explicit return paths. Decode token IDs to text. - Inputs: - `token_ids` (list[int]; required): Required positional or keyword input. - Return annotation: `str` - Calls: self.tokenizer.decode - State reads: self.tokenizer, self.tokenizer.decode - Return expressions: ''; self.tokenizer.decode(token_ids) ## `vllm_mlx.model_runner.MLXModelRunner.get_model_info` - Kind: method - Signature: `def get_model_info(self) -> dict` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/model_runner.py#L436-L471 - Implementation: Method `MLXModelRunner.get_model_info` calls `getattr`, `info.update`; returns `info`. Get information about the loaded model and optimizations. - Inputs: none - Return annotation: `dict` - Calls: getattr, info.update - State reads: self._loaded, self.model_config.model, self.model_config, self._enable_optimizations, self.model, self._compiled_forward, self._hardware_info, self._hardware_info.chip_name, self._hardware_info.total_memory_gb, self._hardware_info.memory_bandwidth_gbs, self._hardware_info.gpu_cores, self._hardware_info.optimal_prefill_size - Return expressions: info ## `vllm_mlx.model_runner.MLXModelRunner.__repr__` - Kind: method - Signature: `def __repr__(self) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/model_runner.py#L473-L476 - Implementation: Method `MLXModelRunner.__repr__` returns `f''`. Method `MLXModelRunner.__repr__` returns `f''`. - Inputs: none - Return annotation: `str` - State reads: self._loaded, self._compiled_forward, self.model_config.model, self.model_config - Return expressions: f'' # Module `vllm_mlx.model_workflow` Model acquisition, inspection, and conversion workflow helpers. The functions in this module intentionally avoid loading model weights. They collect repository/file metadata, download artifacts, and record manifests so a model can be qualified before it is served. Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/model_workflow.py#L1-L661 ## `vllm_mlx.model_workflow.AcquisitionOptions` - Kind: class - Signature: `class AcquisitionOptions` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/model_workflow.py#L38-L46 - Implementation: Class `AcquisitionOptions` declares 0 direct member(s). Options for Hugging Face model acquisition. - Inputs: - `revision` (str | None; optional; default `None`): Optional constructor field; defaults to `None`. - `target_dir` (str | None; optional; default `None`): Optional constructor field; defaults to `None`. - `staging_dir` (str | None; optional; default `None`): Optional constructor field; defaults to `None`. - `is_mllm` (bool; optional; default `False`): Optional constructor field; defaults to `False`. - `fast_transfer` (bool; optional; default `True`): Optional constructor field; defaults to `True`. - `local_files_only` (bool; optional; default `False`): Optional constructor field; defaults to `False`. - Constructs: `vllm_mlx.model_workflow.AcquisitionOptions` - Decorators: dataclass(frozen=True) ## `vllm_mlx.model_workflow.ConversionOptions` - Kind: class - Signature: `class ConversionOptions` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/model_workflow.py#L50-L62 - Implementation: Class `ConversionOptions` declares 0 direct member(s). Options for the mlx-lm conversion backend. - Inputs: - `source_path` (str; required): Required constructor field. - `output_path` (str; required): Required constructor field. - `quantize` (bool; optional; default `False`): Optional constructor field; defaults to `False`. - `q_bits` (int | None; optional; default `None`): Optional constructor field; defaults to `None`. - `q_group_size` (int | None; optional; default `None`): Optional constructor field; defaults to `None`. - `q_mode` (str | None; optional; default `None`): Optional constructor field; defaults to `None`. - `quant_predicate` (str | None; optional; default `None`): Optional constructor field; defaults to `None`. - `dtype` (str | None; optional; default `None`): Optional constructor field; defaults to `None`. - `trust_remote_code` (bool; optional; default `False`): Optional constructor field; defaults to `False`. - `dry_run` (bool; optional; default `False`): Optional constructor field; defaults to `False`. - Constructs: `vllm_mlx.model_workflow.ConversionOptions` - Decorators: dataclass(frozen=True) ## `vllm_mlx.model_workflow.RegistrationOptions` - Kind: class - Signature: `class RegistrationOptions` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/model_workflow.py#L66-L84 - Implementation: Class `RegistrationOptions` declares 0 direct member(s). Options for generating a portable model registration manifest. - Inputs: - `artifact_path` (str; required): Required constructor field. - `model_id` (str | None; optional; default `None`): Optional constructor field; defaults to `None`. - `served_model_name` (str | None; optional; default `None`): Optional constructor field; defaults to `None`. - `preset_alias` (str | None; optional; default `None`): Optional constructor field; defaults to `None`. - `output_path` (str | None; optional; default `None`): Optional constructor field; defaults to `None`. - `mllm` (bool | None; optional; default `None`): Optional constructor field; defaults to `None`. - `tool_call_parser` (str | None; optional; default `None`): Optional constructor field; defaults to `None`. - `reasoning_parser` (str | None; optional; default `None`): Optional constructor field; defaults to `None`. - `default_temperature` (float | None; optional; default `None`): Optional constructor field; defaults to `None`. - `default_top_p` (float | None; optional; default `None`): Optional constructor field; defaults to `None`. - `default_top_k` (int | None; optional; default `None`): Optional constructor field; defaults to `None`. - `default_min_p` (float | None; optional; default `None`): Optional constructor field; defaults to `None`. - `default_presence_penalty` (float | None; optional; default `None`): Optional constructor field; defaults to `None`. - `default_repetition_penalty` (float | None; optional; default `None`): Optional constructor field; defaults to `None`. - `chat_template_kwargs` (dict[str, Any] | None; optional; default `None`): Optional constructor field; defaults to `None`. - `feature_flags` (list[str] | None; optional; default `None`): Optional constructor field; defaults to `None`. - Constructs: `vllm_mlx.model_workflow.RegistrationOptions` - Decorators: dataclass(frozen=True) ## `vllm_mlx.model_workflow.QualificationOptions` - Kind: class - Signature: `class QualificationOptions` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/model_workflow.py#L88-L98 - Implementation: Class `QualificationOptions` declares 0 direct member(s). Options for creating or running a bench-serve qualification handoff. - Inputs: - `model_id` (str; required): Required constructor field. - `server_url` (str; optional; default `'http://127.0.0.1:8080'`): Optional constructor field; defaults to `'http://127.0.0.1:8080'`. - `workload_path` (str | None; optional; default `None`): Optional constructor field; defaults to `None`. - `output_path` (str | None; optional; default `None`): Optional constructor field; defaults to `None`. - `result_path` (str | None; optional; default `None`): Optional constructor field; defaults to `None`. - `repetitions` (int | None; optional; default `None`): Optional constructor field; defaults to `None`. - `dry_run` (bool; optional; default `False`): Optional constructor field; defaults to `False`. - `extra_args` (list[str] | None; optional; default `None`): Optional constructor field; defaults to `None`. - Constructs: `vllm_mlx.model_workflow.QualificationOptions` - Decorators: dataclass(frozen=True) ## `vllm_mlx.model_workflow._now_iso` - Kind: function - Signature: `def _now_iso() -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/model_workflow.py#L101-L102 - Implementation: Function `_now_iso` calls `datetime.now(timezone.utc).isoformat`, `datetime.now`; returns `datetime.now(timezone.utc).isoformat()`. Function `_now_iso` calls `datetime.now(timezone.utc).isoformat`, `datetime.now`; returns `datetime.now(timezone.utc).isoformat()`. - Inputs: none - Return annotation: `str` - Calls: datetime.now(timezone.utc).isoformat, datetime.now - Return expressions: datetime.now(timezone.utc).isoformat() ## `vllm_mlx.model_workflow._bytes_to_gb` - Kind: function - Signature: `def _bytes_to_gb(size: int | float | None) -> float | None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/model_workflow.py#L105-L108 - Implementation: Function `_bytes_to_gb` calls `round`, `float`; has 2 explicit return paths. Function `_bytes_to_gb` calls `round`, `float`; has 2 explicit return paths. - Inputs: - `size` (int | float | None; required): Required positional or keyword input. - Return annotation: `float | None` - Calls: round, float - Return expressions: None; round(float(size) / 1024 ** 3, 3) ## `vllm_mlx.model_workflow._read_json` - Kind: function - Signature: `def _read_json(path: Path) -> dict[str, Any]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/model_workflow.py#L111-L115 - Implementation: Function `_read_json` calls `json.loads`, `path.read_text`; has 2 explicit return paths. Function `_read_json` calls `json.loads`, `path.read_text`; has 2 explicit return paths. - Inputs: - `path` (Path; required): Required positional or keyword input. - Return annotation: `dict[str, Any]` - Calls: json.loads, path.read_text - Return expressions: json.loads(path.read_text()); {} ## `vllm_mlx.model_workflow._write_json` - Kind: function - Signature: `def _write_json(path: Path, payload: dict[str, Any]) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/model_workflow.py#L118-L120 - Implementation: Function `_write_json` calls `path.parent.mkdir`, `path.write_text`, `json.dumps`. Function `_write_json` calls `path.parent.mkdir`, `path.write_text`, `json.dumps`. - Inputs: - `path` (Path; required): Required positional or keyword input. - `payload` (dict[str, Any]; required): Required positional or keyword input. - Return annotation: `None` - Calls: path.parent.mkdir, path.write_text, json.dumps ## `vllm_mlx.model_workflow._local_file_inventory` - Kind: function - Signature: `def _local_file_inventory(path: Path) -> tuple[list[dict[str, Any]], int]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/model_workflow.py#L123-L135 - Implementation: Function `_local_file_inventory` calls `sorted`, `path.rglob`, `item.is_file`, `item.stat`; returns `(files, total)`. Function `_local_file_inventory` calls `sorted`, `path.rglob`, `item.is_file`, `item.stat`; returns `(files, total)`. - Inputs: - `path` (Path; required): Required positional or keyword input. - Return annotation: `tuple[list[dict[str, Any]], int]` - Calls: sorted, path.rglob, item.is_file, item.stat, files.append, str, item.relative_to - Return expressions: (files, total) ## `vllm_mlx.model_workflow._hf_file_inventory` - Kind: function - Signature: `def _hf_file_inventory(model_id: str, *, revision: str | None, local_files_only: bool) -> tuple[list[dict[str, Any]], int | None, str | None]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/model_workflow.py#L138-L158 - Implementation: Function `_hf_file_inventory` calls `HfApi().model_info`, `HfApi`, `getattr`, `int`; has 2 explicit return paths. Function `_hf_file_inventory` calls `HfApi().model_info`, `HfApi`, `getattr`, `int`; has 2 explicit return paths. - Inputs: - `model_id` (str; required): Required positional or keyword input. - `revision` (str | None; required): Required keyword-only input. - `local_files_only` (bool; required): Required keyword-only input. - Return annotation: `tuple[list[dict[str, Any]], int | None, str | None]` - Calls: HfApi().model_info, HfApi, getattr, int, files.append - Return expressions: ([], None, revision); (files, total if total_known else None, getattr(info, 'sha', revision)) ## `vllm_mlx.model_workflow._hf_config` - Kind: function - Signature: `def _hf_config(model_id: str, *, revision: str | None, local_files_only: bool) -> dict[str, Any]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/model_workflow.py#L161-L170 - Implementation: Function `_hf_config` calls `hf_hub_download`, `_read_json`, `Path`; returns `_read_json(Path(config_path))`. Function `_hf_config` calls `hf_hub_download`, `_read_json`, `Path`; returns `_read_json(Path(config_path))`. - Inputs: - `model_id` (str; required): Required positional or keyword input. - `revision` (str | None; required): Required keyword-only input. - `local_files_only` (bool; required): Required keyword-only input. - Return annotation: `dict[str, Any]` - Calls: hf_hub_download, _read_json, Path - Return expressions: _read_json(Path(config_path)) ## `vllm_mlx.model_workflow._config_value` - Kind: function - Signature: `def _config_value(config: dict[str, Any], key: str) -> Any` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/model_workflow.py#L173-L179 - Implementation: Function `_config_value` calls `config.get`, `isinstance`, `text_config.get`; has 3 explicit return paths. Function `_config_value` calls `config.get`, `isinstance`, `text_config.get`; has 3 explicit return paths. - Inputs: - `config` (dict[str, Any]; required): Required positional or keyword input. - `key` (str; required): Required positional or keyword input. - Return annotation: `Any` - Calls: config.get, isinstance, text_config.get - Return expressions: config[key]; text_config.get(key); None ## `vllm_mlx.model_workflow._model_family` - Kind: function - Signature: `def _model_family(config: dict[str, Any]) -> dict[str, Any]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/model_workflow.py#L182-L201 - Implementation: Function `_model_family` calls `_config_value`, `isinstance`, `config.get`; returns `{'model_type': _config_value(config, 'model_type'), 'architectures': architectures, 'torch_dtype': _config_value(config…`. Function `_model_family` calls `_config_value`, `isinstance`, `config.get`; returns `{'model_type': _config_value(config, 'model_type'), 'architectures': architectures, 'torch_dtype': _config_value(config…`. - Inputs: - `config` (dict[str, Any]; required): Required positional or keyword input. - Return annotation: `dict[str, Any]` - Calls: _config_value, isinstance, config.get - Return expressions: {'model_type': _config_value(config, 'model_type'), 'architectures': architectures, 'torch_dtype': _config_value(config… ## `vllm_mlx.model_workflow._estimate_fit` - Kind: function - Signature: `def _estimate_fit(*, total_bytes: int | None, model_files_bytes: int | None, config: dict[str, Any]) -> dict[str, Any]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/model_workflow.py#L204-L231 - Implementation: Function `_estimate_fit` calls `_model_family(config).get`, `_model_family`, `isinstance`, `warnings.append`; returns `{'download_size_gb': _bytes_to_gb(total_bytes), 'model_file_size_gb': _bytes_to_gb(model_files_bytes), 'estimated_conve…`. Function `_estimate_fit` calls `_model_family(config).get`, `_model_family`, `isinstance`, `warnings.append`; returns `{'download_size_gb': _bytes_to_gb(total_bytes), 'model_file_size_gb': _bytes_to_gb(model_files_bytes), 'estimated_conve…`. - Inputs: - `total_bytes` (int | None; required): Required keyword-only input. - `model_files_bytes` (int | None; required): Required keyword-only input. - `config` (dict[str, Any]; required): Required keyword-only input. - Return annotation: `dict[str, Any]` - Calls: _model_family(config).get, _model_family, isinstance, warnings.append, int, _bytes_to_gb - Return expressions: {'download_size_gb': _bytes_to_gb(total_bytes), 'model_file_size_gb': _bytes_to_gb(model_files_bytes), 'estimated_conve… ## `vllm_mlx.model_workflow._model_file_bytes` - Kind: function - Signature: `def _model_file_bytes(files: list[dict[str, Any]]) -> int | None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/model_workflow.py#L234-L246 - Implementation: Function `_model_file_bytes` calls `str`, `entry.get`, `path.endswith`, `int`; has 2 explicit return paths. Function `_model_file_bytes` calls `str`, `entry.get`, `path.endswith`, `int`; has 2 explicit return paths. - Inputs: - `files` (list[dict[str, Any]]; required): Required positional or keyword input. - Return annotation: `int | None` - Calls: str, entry.get, path.endswith, int - Return expressions: None; total if known else None ## `vllm_mlx.model_workflow._is_mlx_quantization` - Kind: function - Signature: `def _is_mlx_quantization(quant: Any) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/model_workflow.py#L252-L265 - Implementation: Function `_is_mlx_quantization` calls `isinstance`, `str(quant.get('quant_method', '')).lower`, `str`, `quant.get`; has 2 explicit return paths. Return True only when *quant* looks like an mlx-lm quantization config. PyTorch quantization configs (GPTQ, AWQ, ...) carry a ``quant_method`` key that MLX configs never set. Treating those as MLX-ready is a false positive reported in review. - Inputs: - `quant` (Any; required): Required positional or keyword input. - Return annotation: `bool` - Calls: isinstance, str(quant.get('quant_method', '')).lower, str, quant.get - Return expressions: False; 'bits' in quant ## `vllm_mlx.model_workflow._looks_like_mlx_name` - Kind: function - Signature: `def _looks_like_mlx_name(model: str, *, source: str) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/model_workflow.py#L268-L275 - Implementation: Function `_looks_like_mlx_name` calls `model.lower`, `Path(model).name.lower`, `Path`, `name.startswith`; returns `name.startswith('mlx-community/') or '-mlx' in name or '_mlx' in name or name.endswith('mlx')`. Function `_looks_like_mlx_name` calls `model.lower`, `Path(model).name.lower`, `Path`, `name.startswith`; returns `name.startswith('mlx-community/') or '-mlx' in name or '_mlx' in name or name.endswith('mlx')`. - Inputs: - `model` (str; required): Required positional or keyword input. - `source` (str; required): Required keyword-only input. - Return annotation: `bool` - Calls: model.lower, Path(model).name.lower, Path, name.startswith, name.endswith - Return expressions: name.startswith('mlx-community/') or '-mlx' in name or '_mlx' in name or name.endswith('mlx') ## `vllm_mlx.model_workflow._is_model_id` - Kind: function - Signature: `def _is_model_id(value: str) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/model_workflow.py#L278-L279 - Implementation: Function `_is_model_id` calls `bool`, `_MODEL_ID_RE.fullmatch`; returns `bool(_MODEL_ID_RE.fullmatch(value))`. Function `_is_model_id` calls `bool`, `_MODEL_ID_RE.fullmatch`; returns `bool(_MODEL_ID_RE.fullmatch(value))`. - Inputs: - `value` (str; required): Required positional or keyword input. - Return annotation: `bool` - Calls: bool, _MODEL_ID_RE.fullmatch - Return expressions: bool(_MODEL_ID_RE.fullmatch(value)) ## `vllm_mlx.model_workflow._fast_transfer_env` - Kind: function - Signature: `def _fast_transfer_env(requested: bool) -> tuple[dict[str, str], dict[str, Any]]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/model_workflow.py#L282-L297 - Implementation: Function `_fast_transfer_env` calls `find_spec`; has 3 explicit return paths. Function `_fast_transfer_env` calls `find_spec`; has 3 explicit return paths. - Inputs: - `requested` (bool; required): Required positional or keyword input. - Return annotation: `tuple[dict[str, str], dict[str, Any]]` - Calls: find_spec - Return expressions: ({}, {'requested': False, 'enabled': False, 'reason': 'disabled'}); ({}, {'requested': True, 'enabled': False, 'reason': 'hf_transfer package is not installed'}); ({'HF_HUB_ENABLE_HF_TRANSFER': '1'}, {'requested': True, 'enabled': True, 'reason': 'enabled'}) ## `vllm_mlx.model_workflow.inspect_model` - Kind: function - Signature: `def inspect_model(model: str, *, revision: str | None=None, local_files_only: bool=False) -> dict[str, Any]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/model_workflow.py#L300-L366 - Implementation: Function `inspect_model` calls `Path(model).expanduser`, `Path`, `model_path.exists`, `_local_file_inventory`; can raise `ValueError`; returns `{'model': model, 'source': source, 'location': location, 'revision': resolved_revision or revision, 'inspected_at': _no…`. Inspect a local model path or Hugging Face model id without loading weights. - Inputs: - `model` (str; required): Required positional or keyword input. - `revision` (str | None; optional; default `None`): Optional keyword-only input; defaults to `None`. - `local_files_only` (bool; optional; default `False`): Optional keyword-only input; defaults to `False`. - Return annotation: `dict[str, Any]` - Calls: Path(model).expanduser, Path, model_path.exists, _local_file_inventory, _read_json, str, _is_model_id, ValueError, _hf_file_inventory, _hf_config, warnings.append, _model_file_bytes, _model_family, _estimate_fit, warnings.extend, estimate.pop, _looks_like_mlx_name, bool, _is_mlx_quantization, family.get, _now_iso, len, _bytes_to_gb - Raises directly: ValueError - Return expressions: {'model': model, 'source': source, 'location': location, 'revision': resolved_revision or revision, 'inspected_at': _no… ## `vllm_mlx.model_workflow.acquire_model` - Kind: function - Signature: `def acquire_model(model_id: str, *, options: AcquisitionOptions | None=None) -> dict[str, Any]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/model_workflow.py#L369-L446 - Implementation: Function `acquire_model` calls `AcquisitionOptions`, `_is_model_id`, `ValueError`, `_fast_transfer_env`; can raise `ValueError`, `FileExistsError`; returns `manifest`. Download a model repository and write a finalized artifact manifest. - Inputs: - `model_id` (str; required): Required positional or keyword input. - `options` (AcquisitionOptions | None; optional; default `None`): Optional keyword-only input; defaults to `None`. - Return annotation: `dict[str, Any]` - Calls: AcquisitionOptions, _is_model_id, ValueError, _fast_transfer_env, os.environ.get, os.environ.update, Path(options.target_dir).expanduser, Path, target.exists, FileExistsError, Path(options.staging_dir).expanduser, staging_root.mkdir, tempfile.mkdtemp, snapshot_download, str, target.parent.mkdir, shutil.move, shutil.rmtree, old_env.items, os.environ.pop, inspect_model, _now_iso, _write_json - Raises directly: ValueError, FileExistsError - Return expressions: manifest ## `vllm_mlx.model_workflow._conversion_command` - Kind: function - Signature: `def _conversion_command(options: ConversionOptions) -> list[str]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/model_workflow.py#L449-L474 - Implementation: Function `_conversion_command` calls `command.append`, `command.extend`, `str`; returns `command`. Function `_conversion_command` calls `command.append`, `command.extend`, `str`; returns `command`. - Inputs: - `options` (ConversionOptions; required): Required positional or keyword input. - Return annotation: `list[str]` - Calls: command.append, command.extend, str - Return expressions: command ## `vllm_mlx.model_workflow.convert_model` - Kind: function - Signature: `def convert_model(options: ConversionOptions) -> dict[str, Any]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/model_workflow.py#L477-L525 - Implementation: Function `convert_model` calls `_conversion_command`, `_now_iso`, `inspect_model`, `sys.version.split`; returns `result`. Run mlx-lm conversion and record the exact recipe. - Inputs: - `options` (ConversionOptions; required): Required positional or keyword input. - Return annotation: `dict[str, Any]` - Calls: _conversion_command, _now_iso, inspect_model, sys.version.split, platform.platform, subprocess.run, Path(options.output_path).expanduser, Path, str, _write_json - Return expressions: result ## `vllm_mlx.model_workflow._existing_manifests` - Kind: function - Signature: `def _existing_manifests(path: Path) -> dict[str, Any]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/model_workflow.py#L528-L540 - Implementation: Function `_existing_manifests` calls `manifest_path.exists`, `str`, `_read_json`; returns `manifests`. Function `_existing_manifests` calls `manifest_path.exists`, `str`, `_read_json`; returns `manifests`. - Inputs: - `path` (Path; required): Required positional or keyword input. - Return annotation: `dict[str, Any]` - Calls: manifest_path.exists, str, _read_json - Return expressions: manifests ## `vllm_mlx.model_workflow._drop_none` - Kind: function - Signature: `def _drop_none(payload: dict[str, Any]) -> dict[str, Any]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/model_workflow.py#L543-L544 - Implementation: Function `_drop_none` calls `payload.items`; returns `{key: value for key, value in payload.items() if value is not None}`. Function `_drop_none` calls `payload.items`; returns `{key: value for key, value in payload.items() if value is not None}`. - Inputs: - `payload` (dict[str, Any]; required): Required positional or keyword input. - Return annotation: `dict[str, Any]` - Calls: payload.items - Return expressions: {key: value for key, value in payload.items() if value is not None} ## `vllm_mlx.model_workflow.register_model` - Kind: function - Signature: `def register_model(options: RegistrationOptions) -> dict[str, Any]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/model_workflow.py#L547-L603 - Implementation: Function `register_model` calls `Path(options.artifact_path).expanduser`, `Path`, `artifact.exists`, `FileNotFoundError`; can raise `FileNotFoundError`, `NotADirectoryError`; returns `payload`. Write a portable registration manifest for a finalized local artifact. This deliberately does not mutate a production registry. The manifest is a handoff artifact that Ops or a deployment tool can apply after qualification. - Inputs: - `options` (RegistrationOptions; required): Required positional or keyword input. - Return annotation: `dict[str, Any]` - Calls: Path(options.artifact_path).expanduser, Path, artifact.exists, FileNotFoundError, artifact.is_dir, NotADirectoryError, inspect_model, str, _drop_none, _now_iso, _existing_manifests, Path(options.output_path).expanduser, _write_json - Raises directly: FileNotFoundError, NotADirectoryError - Return expressions: payload ## `vllm_mlx.model_workflow._qualification_command` - Kind: function - Signature: `def _qualification_command(options: QualificationOptions) -> list[str]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/model_workflow.py#L606-L627 - Implementation: Function `_qualification_command` calls `command.extend`, `str`; returns `command`. Function `_qualification_command` calls `command.extend`, `str`; returns `command`. - Inputs: - `options` (QualificationOptions; required): Required positional or keyword input. - Return annotation: `list[str]` - Calls: command.extend, str - Return expressions: command ## `vllm_mlx.model_workflow.qualify_model` - Kind: function - Signature: `def qualify_model(options: QualificationOptions) -> dict[str, Any]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/model_workflow.py#L630-L661 - Implementation: Function `qualify_model` calls `_qualification_command`, `_now_iso`, `subprocess.run`, `Path(options.output_path).expanduser`; returns `payload`. Create or run a bench-serve qualification handoff. - Inputs: - `options` (QualificationOptions; required): Required positional or keyword input. - Return annotation: `dict[str, Any]` - Calls: _qualification_command, _now_iso, subprocess.run, Path(options.output_path).expanduser, Path, _write_json, str - Return expressions: payload # Module `vllm_mlx.models` MLX Model wrappers for vLLM. This module provides wrappers around mlx-lm and mlx-vlm for integration with vLLM's model execution system. Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/models/__init__.py#L1-L15 # Module `vllm_mlx.models.llm` MLX Language Model wrapper. This module provides a wrapper around mlx-lm for LLM inference, integrating with vLLM's model execution system. Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/models/llm.py#L1-L422 ## `vllm_mlx.models.llm.GenerationOutput` - Kind: class - Signature: `class GenerationOutput` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/models/llm.py#L21-L26 - Implementation: Class `GenerationOutput` declares 0 direct member(s). Output from text generation. - Inputs: - `text` (str; required): Required constructor field. - `tokens` (list[int]; required): Required constructor field. - `finish_reason` (str | None; optional; default `None`): Optional constructor field; defaults to `None`. - Constructs: `vllm_mlx.models.llm.GenerationOutput` - Decorators: dataclass ## `vllm_mlx.models.llm.StreamingOutput` - Kind: class - Signature: `class StreamingOutput` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/models/llm.py#L30-L37 - Implementation: Class `StreamingOutput` declares 0 direct member(s). Streaming output chunk. - Inputs: - `text` (str; required): Required constructor field. - `token` (int; required): Required constructor field. - `finished` (bool; optional; default `False`): Optional constructor field; defaults to `False`. - `finish_reason` (str | None; optional; default `None`): Optional constructor field; defaults to `None`. - `prompt_tokens` (int; optional; default `0`): Optional constructor field; defaults to `0`. - Constructs: `vllm_mlx.models.llm.StreamingOutput` - Decorators: dataclass ## `vllm_mlx.models.llm.MLXLanguageModel` - Kind: class - Signature: `class MLXLanguageModel` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/models/llm.py#L40-L422 - Implementation: Class `MLXLanguageModel` declares 9 direct member(s). Wrapper around mlx-lm for LLM inference. This class provides a unified interface for loading and running inference on language models using Apple's MLX framework. Example: >>> model = MLXLanguageModel("mlx-community/Llama-3.2-3B-Instruct-4bit") >>> output = model.generate("Hello, how are you?", max_tokens=100) >>> print(output.text) - Inputs: - `model_name` (str; required): HuggingFace model name or local path - `tokenizer_name` (str | None; optional; default `None`): Optional separate tokenizer name - `trust_remote_code` (bool; optional; default `False`): Whether to trust remote code - `mtp` (bool; optional; default `False`): Enable native MTP speculative decoding (model must have MTP head) - `mtp_num_draft_tokens` (int; optional; default `1`): Draft tokens per speculative MTP step - Constructs: `vllm_mlx.models.llm.MLXLanguageModel` ## `vllm_mlx.models.llm.MLXLanguageModel.__init__` - Kind: method - Signature: `def __init__(self, model_name: str, tokenizer_name: str | None=None, trust_remote_code: bool=False, mtp: bool=False, mtp_num_draft_tokens: int=1)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/models/llm.py#L53-L79 - Implementation: Method `MLXLanguageModel.__init__` updates `self.model_name`, `self.tokenizer_name`, `self.trust_remote_code`, `self._mtp`. Initialize the MLX language model. Args: model_name: HuggingFace model name or local path tokenizer_name: Optional separate tokenizer name trust_remote_code: Whether to trust remote code mtp: Enable native MTP speculative decoding (model must have MTP head) mtp_num_draft_tokens: Draft tokens per speculative MTP step - Inputs: - `model_name` (str; required): HuggingFace model name or local path - `tokenizer_name` (str | None; optional; default `None`): Optional separate tokenizer name - `trust_remote_code` (bool; optional; default `False`): Whether to trust remote code - `mtp` (bool; optional; default `False`): Enable native MTP speculative decoding (model must have MTP head) - `mtp_num_draft_tokens` (int; optional; default `1`): Draft tokens per speculative MTP step - Return annotation: `not annotated` - State writes: self.model_name, self.tokenizer_name, self.trust_remote_code, self._mtp, self._mtp_num_draft_tokens, self.model, self.tokenizer, self._loaded ## `vllm_mlx.models.llm.MLXLanguageModel.load` - Kind: method - Signature: `def load(self) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/models/llm.py#L81-L114 - Implementation: Method `MLXLanguageModel.load` updates `self.model`, `self.tokenizer`, `self._loaded`; calls `logger.info`, `self.model_name.lower`, `load_model_with_fallback`, `ImportError`; can raise `ImportError`; returns `None`. Load the model and tokenizer. - Inputs: none - Return annotation: `None` - Calls: logger.info, self.model_name.lower, load_model_with_fallback, ImportError, logger.error - State reads: self._loaded, self.model_name, self.trust_remote_code, self.model_name.lower - State writes: self.model, self.tokenizer, self._loaded - Raises directly: ImportError - Return expressions: None ## `vllm_mlx.models.llm.MLXLanguageModel._create_sampler` - Kind: method - Signature: `def _create_sampler(self, temperature: float=0.7, top_p: float=0.9, top_k: int=0, min_p: float=0.0)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/models/llm.py#L116-L131 - Implementation: Method `MLXLanguageModel._create_sampler` calls `make_sampler`; returns `make_sampler(temp=temperature, top_p=top_p, top_k=top_k, min_p=min_p)`. Create a sampler for text generation. - Inputs: - `temperature` (float; optional; default `0.7`): Optional positional or keyword input; defaults to `0.7`. - `top_p` (float; optional; default `0.9`): Optional positional or keyword input; defaults to `0.9`. - `top_k` (int; optional; default `0`): Optional positional or keyword input; defaults to `0`. - `min_p` (float; optional; default `0.0`): Optional positional or keyword input; defaults to `0.0`. - Return annotation: `not annotated` - Calls: make_sampler - Return expressions: make_sampler(temp=temperature, top_p=top_p, top_k=top_k, min_p=min_p) ## `vllm_mlx.models.llm.MLXLanguageModel._create_logits_processors` - Kind: method - Signature: `def _create_logits_processors(self, presence_penalty: float=0.0, repetition_penalty: float=1.0)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/models/llm.py#L133-L147 - Implementation: Method `MLXLanguageModel._create_logits_processors` calls `make_logits_processors`; returns `processors if processors else None`. Create logits processors for penalty-based sampling. - Inputs: - `presence_penalty` (float; optional; default `0.0`): Optional positional or keyword input; defaults to `0.0`. - `repetition_penalty` (float; optional; default `1.0`): Optional positional or keyword input; defaults to `1.0`. - Return annotation: `not annotated` - Calls: make_logits_processors - Return expressions: processors if processors else None ## `vllm_mlx.models.llm.MLXLanguageModel.generate` - Kind: method - Signature: `def generate(self, prompt: str, max_tokens: int=256, temperature: float=0.7, top_p: float=0.9, top_k: int=0, min_p: float=0.0, presence_penalty: float=0.0, repetition_penalty: float=1.0, stop: list[str] | None=None, logits_processors: list | None=None, **kwargs) -> GenerationOutput` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/models/llm.py#L149-L219 - Implementation: Method `MLXLanguageModel.generate` calls `self.load`, `self._create_sampler`, `self._create_logits_processors`, `list`; returns `GenerationOutput(text=output_text, tokens=tokens, finish_reason=finish_reason)`. Generate text from a prompt. Args: prompt: Input prompt text max_tokens: Maximum number of tokens to generate temperature: Sampling temperature (0 = greedy) top_p: Top-p (nucleus) sampling parameter top_k: Top-k sampling (0 = disabled) min_p: Minimum probability threshold presence_penalty: Additive penalty for token presence repetition_penalty: Multiplicative penalty for repeating tokens stop: List of stop sequences logits_processors: Optional externally-supplied logits processors (e.g. JSON schema constrained decoding). Merged with built-in penalty processors. Returns: GenerationOutput with generated text and tokens - Inputs: - `prompt` (str; required): Input prompt text - `max_tokens` (int; optional; default `256`): Maximum number of tokens to generate - `temperature` (float; optional; default `0.7`): Sampling temperature (0 = greedy) - `top_p` (float; optional; default `0.9`): Top-p (nucleus) sampling parameter - `top_k` (int; optional; default `0`): Top-k sampling (0 = disabled) - `min_p` (float; optional; default `0.0`): Minimum probability threshold - `presence_penalty` (float; optional; default `0.0`): Additive penalty for token presence - `repetition_penalty` (float; optional; default `1.0`): Multiplicative penalty for repeating tokens - `stop` (list[str] | None; optional; default `None`): List of stop sequences - `logits_processors` (list | None; optional; default `None`): Optional externally-supplied logits processors (e.g. JSON schema constrained decoding). Merged with built-in penalty processors. - `**kwargs` (not annotated; optional): Additional variadic keyword inputs accepted by this callable. - Return annotation: `GenerationOutput` - Calls: self.load, self._create_sampler, self._create_logits_processors, list, generate, self.tokenizer.encode, len, GenerationOutput - State reads: self._loaded, self.load, self._create_sampler, self._create_logits_processors, self.model, self.tokenizer, self.tokenizer.encode - Return expressions: GenerationOutput(text=output_text, tokens=tokens, finish_reason=finish_reason) ## `vllm_mlx.models.llm.MLXLanguageModel.stream_generate` - Kind: method - Signature: `def stream_generate(self, prompt: Union[str, 'mx.array', list[int]], max_tokens: int=256, temperature: float=0.7, top_p: float=0.9, top_k: int=0, min_p: float=0.0, presence_penalty: float=0.0, repetition_penalty: float=1.0, stop: list[str] | None=None, logits_processors: list | None=None, prompt_cache=None, **kwargs) -> Iterator[StreamingOutput]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/models/llm.py#L221-L325 - Implementation: Method `MLXLanguageModel.stream_generate` calls `self.load`, `self._create_sampler`, `self._create_logits_processors`, `isinstance`; yields values incrementally. Stream text generation token by token. Args: prompt: Input prompt text, token array, or token id list max_tokens: Maximum number of tokens to generate temperature: Sampling temperature (0 = greedy) top_p: Top-p (nucleus) sampling parameter top_k: Top-k sampling (0 = disabled) min_p: Minimum probability threshold presence_penalty: Additive penalty for token presence repetition_penalty: Multiplicative penalty for repeating tokens stop: List of stop sequences prompt_cache: Pre-populated KV cache (e.g. from SpecPrefill) Yields: StreamingOutput for each generated token - Inputs: - `prompt` (Union[str, 'mx.array', list[int]]; required): Input prompt text, token array, or token id list - `max_tokens` (int; optional; default `256`): Maximum number of tokens to generate - `temperature` (float; optional; default `0.7`): Sampling temperature (0 = greedy) - `top_p` (float; optional; default `0.9`): Top-p (nucleus) sampling parameter - `top_k` (int; optional; default `0`): Top-k sampling (0 = disabled) - `min_p` (float; optional; default `0.0`): Minimum probability threshold - `presence_penalty` (float; optional; default `0.0`): Additive penalty for token presence - `repetition_penalty` (float; optional; default `1.0`): Multiplicative penalty for repeating tokens - `stop` (list[str] | None; optional; default `None`): List of stop sequences - `logits_processors` (list | None; optional; default `None`): Optional positional or keyword input; defaults to `None`. - `prompt_cache` (not annotated; optional; default `None`): Pre-populated KV cache (e.g. from SpecPrefill) - `**kwargs` (not annotated; optional): Additional variadic keyword inputs accepted by this callable. - Return annotation: `Iterator[StreamingOutput]` - Calls: self.load, self._create_sampler, self._create_logits_processors, isinstance, len, self.tokenizer.encode, max, enumerate, stream_generate, StreamingOutput, hasattr - State reads: self._loaded, self.load, self._create_sampler, self._create_logits_processors, self.tokenizer.encode, self.tokenizer, self._mtp, self._mtp_num_draft_tokens, self.model ## `vllm_mlx.models.llm.MLXLanguageModel.chat` - Kind: method - Signature: `def chat(self, messages: list[dict], max_tokens: int=256, temperature: float=0.7, top_p: float=0.9, tools: list | None=None, chat_template_kwargs: dict | None=None, **kwargs) -> GenerationOutput` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/models/llm.py#L327-L393 - Implementation: Method `MLXLanguageModel.chat` calls `self.load`, `hasattr`, `template_kwargs.update`, `self.tokenizer.apply_chat_template`; returns `self.generate(prompt=prompt, max_tokens=max_tokens, temperature=temperature, top_p=top_p, **kwargs)`. Generate a chat response. Args: messages: List of chat messages [{"role": "user", "content": "..."}] max_tokens: Maximum tokens to generate temperature: Sampling temperature top_p: Top-p sampling parameter tools: Optional list of tools for function calling **kwargs: Additional generation parameters Returns: GenerationOutput with the assistant's response - Inputs: - `messages` (list[dict]; required): List of chat messages [{"role": "user", "content": "..."}] - `max_tokens` (int; optional; default `256`): Maximum tokens to generate - `temperature` (float; optional; default `0.7`): Sampling temperature - `top_p` (float; optional; default `0.9`): Top-p sampling parameter - `tools` (list | None; optional; default `None`): Optional list of tools for function calling - `chat_template_kwargs` (dict | None; optional; default `None`): Optional positional or keyword input; defaults to `None`. - `**kwargs` (not annotated; optional): Additional generation parameters - Return annotation: `GenerationOutput` - Calls: self.load, hasattr, template_kwargs.update, self.tokenizer.apply_chat_template, template_kwargs.pop, (chat_template_kwargs or {}).keys, '\n'.join, self.generate - State reads: self._loaded, self.load, self.tokenizer, self.tokenizer.apply_chat_template, self.generate - Return expressions: self.generate(prompt=prompt, max_tokens=max_tokens, temperature=temperature, top_p=top_p, **kwargs) ## `vllm_mlx.models.llm.MLXLanguageModel.get_model_info` - Kind: method - Signature: `def get_model_info(self) -> dict` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/models/llm.py#L395-L418 - Implementation: Method `MLXLanguageModel.get_model_info` calls `hasattr`, `info.update`, `getattr`; has 2 explicit return paths. Get information about the loaded model. - Inputs: none - Return annotation: `dict` - Calls: hasattr, info.update, getattr - State reads: self._loaded, self.model_name, self.tokenizer_name, self.model, self.model.config - Return expressions: {'loaded': False, 'model_name': self.model_name}; info ## `vllm_mlx.models.llm.MLXLanguageModel.__repr__` - Kind: method - Signature: `def __repr__(self) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/models/llm.py#L420-L422 - Implementation: Method `MLXLanguageModel.__repr__` returns `f''`. Method `MLXLanguageModel.__repr__` returns `f''`. - Inputs: none - Return annotation: `str` - State reads: self._loaded, self.model_name - Return expressions: f'' # Module `vllm_mlx.models.mllm` MLX Multimodal Language Model (MLLM) wrapper. This module provides a wrapper around mlx-vlm for multimodal inference, supporting vision, audio, and video understanding on Apple Silicon. Features: - OpenAI-compatible API format for images and video - Smart video frame extraction with configurable FPS - Base64 and URL image support - Streaming generation - MLLM KV cache for repeated image/video+prompt combinations Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/models/mllm.py#L1-L2944 ## `vllm_mlx.models.mllm.TempFileManager` - Kind: class - Signature: `class TempFileManager` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/models/mllm.py#L41-L86 - Implementation: Class `TempFileManager` declares 4 direct member(s). Thread-safe manager for tracking and cleaning up temporary files. - Inputs: none - Constructs: `vllm_mlx.models.mllm.TempFileManager` ## `vllm_mlx.models.mllm.TempFileManager.__init__` - Kind: method - Signature: `def __init__(self)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/models/mllm.py#L44-L47 - Implementation: Method `TempFileManager.__init__` updates `self._files`, `self._lock`; calls `set`, `threading.Lock`, `atexit.register`. Method `TempFileManager.__init__` updates `self._files`, `self._lock`; calls `set`, `threading.Lock`, `atexit.register`. - Inputs: none - Return annotation: `not annotated` - Calls: set, threading.Lock, atexit.register - State reads: self.cleanup_all - State writes: self._files, self._lock ## `vllm_mlx.models.mllm.TempFileManager.register` - Kind: method - Signature: `def register(self, path: str) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/models/mllm.py#L49-L53 - Implementation: Method `TempFileManager.register` calls `self._files.add`; returns `path`. Register a temp file for tracking. Returns the path for convenience. - Inputs: - `path` (str; required): Required positional or keyword input. - Return annotation: `str` - Calls: self._files.add - State reads: self._lock, self._files.add, self._files - Return expressions: path ## `vllm_mlx.models.mllm.TempFileManager.cleanup` - Kind: method - Signature: `def cleanup(self, path: str) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/models/mllm.py#L55-L67 - Implementation: Method `TempFileManager.cleanup` calls `self._files.discard`, `os.path.exists`, `os.unlink`, `logger.debug`; has 2 explicit return paths. Clean up a specific temp file. Returns True if successful. - Inputs: - `path` (str; required): Required positional or keyword input. - Return annotation: `bool` - Calls: self._files.discard, os.path.exists, os.unlink, logger.debug, logger.warning - State reads: self._lock, self._files, self._files.discard - Return expressions: True; False ## `vllm_mlx.models.mllm.TempFileManager.cleanup_all` - Kind: method - Signature: `def cleanup_all(self) -> int` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/models/mllm.py#L69-L86 - Implementation: Method `TempFileManager.cleanup_all` calls `list`, `self._files.clear`, `os.path.exists`, `os.unlink`; returns `cleaned`. Clean up all tracked temp files. Returns count of cleaned files. - Inputs: none - Return annotation: `int` - Calls: list, self._files.clear, os.path.exists, os.unlink, logger.info - State reads: self._lock, self._files, self._files.clear - Return expressions: cleaned ## `vllm_mlx.models.mllm.cleanup_temp_file` - Kind: function - Signature: `def cleanup_temp_file(path: str) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/models/mllm.py#L93-L95 - Implementation: Function `cleanup_temp_file` calls `_temp_manager.cleanup`; returns `_temp_manager.cleanup(path)`. Clean up a specific temporary file. - Inputs: - `path` (str; required): Required positional or keyword input. - Return annotation: `bool` - Calls: _temp_manager.cleanup - Return expressions: _temp_manager.cleanup(path) ## `vllm_mlx.models.mllm.cleanup_all_temp_files` - Kind: function - Signature: `def cleanup_all_temp_files() -> int` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/models/mllm.py#L98-L100 - Implementation: Function `cleanup_all_temp_files` calls `_temp_manager.cleanup_all`; returns `_temp_manager.cleanup_all()`. Clean up all tracked temporary files. Returns count of cleaned files. - Inputs: none - Return annotation: `int` - Calls: _temp_manager.cleanup_all - Return expressions: _temp_manager.cleanup_all() ## `vllm_mlx.models.mllm.FileSizeExceededError` - Kind: class - Signature: `class FileSizeExceededError(Exception)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/models/mllm.py#L119-L122 - Implementation: Class `FileSizeExceededError` derives from `Exception` and declares 0 direct member(s). Raised when a downloaded file exceeds the size limit. - Inputs: none - Constructs: `vllm_mlx.models.mllm.FileSizeExceededError` ## `vllm_mlx.models.mllm.UnsafeRemoteURLError` - Kind: class - Signature: `class UnsafeRemoteURLError(ValueError)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/models/mllm.py#L125-L135 - Implementation: Class `UnsafeRemoteURLError` derives from `ValueError` and declares 1 direct member(s). Raised when a remote media URL targets an unsafe destination. - Inputs: - `message` (str; required): Required positional or keyword input. - `public_message` (str; optional; default `'Remote media URL is not allowed'`): Optional keyword-only input; defaults to `'Remote media URL is not allowed'`. - Constructs: `vllm_mlx.models.mllm.UnsafeRemoteURLError` ## `vllm_mlx.models.mllm.UnsafeRemoteURLError.__init__` - Kind: method - Signature: `def __init__(self, message: str, *, public_message: str='Remote media URL is not allowed') -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/models/mllm.py#L128-L135 - Implementation: Method `UnsafeRemoteURLError.__init__` updates `self.public_message`; calls `super().__init__`, `super`. Method `UnsafeRemoteURLError.__init__` updates `self.public_message`; calls `super().__init__`, `super`. - Inputs: - `message` (str; required): Required positional or keyword input. - `public_message` (str; optional; default `'Remote media URL is not allowed'`): Optional keyword-only input; defaults to `'Remote media URL is not allowed'`. - Return annotation: `None` - Calls: super().__init__, super - State writes: self.public_message ## `vllm_mlx.models.mllm._normalize_content_part` - Kind: function - Signature: `def _normalize_content_part(item: object) -> object` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/models/mllm.py#L138-L144 - Implementation: Function `_normalize_content_part` calls `hasattr`, `item.model_dump`, `item.dict().items`, `item.dict`; has 3 explicit return paths. Convert Pydantic content parts into plain Python objects. - Inputs: - `item` (object; required): Required positional or keyword input. - Return annotation: `object` - Calls: hasattr, item.model_dump, item.dict().items, item.dict - Return expressions: item.model_dump(exclude_none=True); {k: v for k, v in item.dict().items() if v is not None}; item ## `vllm_mlx.models.mllm._extract_media_url` - Kind: function - Signature: `def _extract_media_url(item: dict, item_type: str) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/models/mllm.py#L147-L161 - Implementation: Function `_extract_media_url` calls `item.get`, `isinstance`, `media_value.get`; has 2 explicit return paths. Function `_extract_media_url` calls `item.get`, `isinstance`, `media_value.get`; has 2 explicit return paths. - Inputs: - `item` (dict; required): Required positional or keyword input. - `item_type` (str; required): Required positional or keyword input. - Return annotation: `str` - Calls: item.get, isinstance, media_value.get - Return expressions: ''; media_value if isinstance(media_value, str) else '' ## `vllm_mlx.models.mllm._text_content_part` - Kind: function - Signature: `def _text_content_part(text: str) -> dict[str, str]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/models/mllm.py#L164-L165 - Implementation: Function `_text_content_part` returns `{'type': 'text', 'text': text, 'content': text}`. Function `_text_content_part` returns `{'type': 'text', 'text': text, 'content': text}`. - Inputs: - `text` (str; required): Required positional or keyword input. - Return annotation: `dict[str, str]` - Return expressions: {'type': 'text', 'text': text, 'content': text} ## `vllm_mlx.models.mllm._append_text_content_part` - Kind: function - Signature: `def _append_text_content_part(built_parts: list[dict[str, str]], text_parts: list[str], text: str) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/models/mllm.py#L168-L174 - Implementation: Function `_append_text_content_part` calls `built_parts.append`, `_text_content_part`, `text_parts.append`; returns `None`. Function `_append_text_content_part` calls `built_parts.append`, `_text_content_part`, `text_parts.append`; returns `None`. - Inputs: - `built_parts` (list[dict[str, str]]; required): Required positional or keyword input. - `text_parts` (list[str]; required): Required positional or keyword input. - `text` (str; required): Required positional or keyword input. - Return annotation: `None` - Calls: built_parts.append, _text_content_part, text_parts.append - Return expressions: None ## `vllm_mlx.models.mllm._build_string_mllm_message_content` - Kind: function - Signature: `def _build_string_mllm_message_content(content: str, role: str) -> tuple[object, bool]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/models/mllm.py#L177-L182 - Implementation: Function `_build_string_mllm_message_content` calls `_text_content_part`; has 3 explicit return paths. Function `_build_string_mllm_message_content` calls `_text_content_part`; has 3 explicit return paths. - Inputs: - `content` (str; required): Required positional or keyword input. - `role` (str; required): Required positional or keyword input. - Return annotation: `tuple[object, bool]` - Calls: _text_content_part - Return expressions: ('', False); (content, True); ([_text_content_part(content)], True) ## `vllm_mlx.models.mllm._append_ordered_mllm_content_part` - Kind: function - Signature: `def _append_ordered_mllm_content_part(raw_item: object, *, built_parts: list[dict[str, str]], text_parts: list[str], all_image_urls: list[str], video_frame_count: int) -> int` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/models/mllm.py#L185-L220 - Implementation: Function `_append_ordered_mllm_content_part` calls `_normalize_content_part`, `isinstance`, `_append_text_content_part`, `item.get`; has 2 explicit return paths. Function `_append_ordered_mllm_content_part` calls `_normalize_content_part`, `isinstance`, `_append_text_content_part`, `item.get`; has 2 explicit return paths. - Inputs: - `raw_item` (object; required): Required positional or keyword input. - `built_parts` (list[dict[str, str]]; required): Required keyword-only input. - `text_parts` (list[str]; required): Required keyword-only input. - `all_image_urls` (list[str]; required): Required keyword-only input. - `video_frame_count` (int; required): Required keyword-only input. - Return annotation: `int` - Calls: _normalize_content_part, isinstance, _append_text_content_part, item.get, _extract_media_url, all_image_urls.append, built_parts.append, built_parts.extend, range - Return expressions: video_frame_count; 0 ## `vllm_mlx.models.mllm._build_ordered_mllm_message_content` - Kind: function - Signature: `def _build_ordered_mllm_message_content(content: object, *, role: str, all_image_urls: list[str], video_frame_count: int=0) -> tuple[object, bool]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/models/mllm.py#L223-L254 - Implementation: Function `_build_ordered_mllm_message_content` calls `isinstance`, `_build_string_mllm_message_content`, `_append_ordered_mllm_content_part`, `''.join`; has 4 explicit return paths. Build template content while preserving OpenAI media/text part order. - Inputs: - `content` (object; required): Required positional or keyword input. - `role` (str; required): Required keyword-only input. - `all_image_urls` (list[str]; required): Required keyword-only input. - `video_frame_count` (int; optional; default `0`): Optional keyword-only input; defaults to `0`. - Return annotation: `tuple[object, bool]` - Calls: isinstance, _build_string_mllm_message_content, _append_ordered_mllm_content_part, ''.join, bool - Return expressions: _build_string_mllm_message_content(content, role); ('', False); (text, bool(text)); (built_parts, bool(built_parts)) ## `vllm_mlx.models.mllm._normalize_mllm_tool_calls` - Kind: function - Signature: `def _normalize_mllm_tool_calls(tool_calls: list) -> list` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/models/mllm.py#L257-L268 - Implementation: Function `_normalize_mllm_tool_calls` calls `_normalize_content_part`, `normalize_messages_for_chat_template`, `normalized[0].get`; returns `normalized[0].get('tool_calls', plain_calls)`. Normalize replayed assistant tool calls for chat templates. Mirrors ``_normalize_tool_call_arguments_for_template`` in ``vllm_mlx/engine/batched.py``: JSON argument strings become mappings so templates that iterate argument keys render correctly. - Inputs: - `tool_calls` (list; required): Required positional or keyword input. - Return annotation: `list` - Calls: _normalize_content_part, normalize_messages_for_chat_template, normalized[0].get - Return expressions: normalized[0].get('tool_calls', plain_calls) ## `vllm_mlx.models.mllm._build_mllm_chat_messages` - Kind: function - Signature: `def _build_mllm_chat_messages(messages: list[dict], *, all_image_urls: list[str], video_frame_counts: dict[int, int]) -> list[dict]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/models/mllm.py#L271-L315 - Implementation: Function `_build_mllm_chat_messages` calls `enumerate`, `msg.get`, `isinstance`, `str`; returns `chat_messages`. Build chat-template messages without reordering multimodal content parts. - Inputs: - `messages` (list[dict]; required): Required positional or keyword input. - `all_image_urls` (list[str]; required): Required keyword-only input. - `video_frame_counts` (dict[int, int]; required): Required keyword-only input. - Return annotation: `list[dict]` - Calls: enumerate, msg.get, isinstance, str, _build_ordered_mllm_message_content, video_frame_counts.get, _normalize_mllm_tool_calls, chat_messages.append - Return expressions: chat_messages ## `vllm_mlx.models.mllm.MultimodalInput` - Kind: class - Signature: `class MultimodalInput` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/models/mllm.py#L319-L325 - Implementation: Class `MultimodalInput` declares 0 direct member(s). Input for multimodal generation. - Inputs: - `prompt` (str; required): Required constructor field. - `images` (list[str]; optional; default `field(default_factory=list)`): Optional constructor field; defaults to `field(default_factory=list)`. - `videos` (list[str]; optional; default `field(default_factory=list)`): Optional constructor field; defaults to `field(default_factory=list)`. - `audio` (list[str]; optional; default `field(default_factory=list)`): Optional constructor field; defaults to `field(default_factory=list)`. - Constructs: `vllm_mlx.models.mllm.MultimodalInput` - Decorators: dataclass ## `vllm_mlx.models.mllm.MLLMOutput` - Kind: class - Signature: `class MLLMOutput` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/models/mllm.py#L329-L337 - Implementation: Class `MLLMOutput` declares 0 direct member(s). Output from multimodal language model. - Inputs: - `text` (str; required): Required constructor field. - `finish_reason` (str | None; optional; default `None`): Optional constructor field; defaults to `None`. - `prompt_tokens` (int; optional; default `0`): Optional constructor field; defaults to `0`. - `completion_tokens` (int; optional; default `0`): Optional constructor field; defaults to `0`. - `mtp_drafts` (int; optional; default `0`): Optional constructor field; defaults to `0`. - `mtp_accepted` (int; optional; default `0`): Optional constructor field; defaults to `0`. - Constructs: `vllm_mlx.models.mllm.MLLMOutput` - Decorators: dataclass ## `vllm_mlx.models.mllm.load_gemma4_assistant_drafter` - Kind: function - Signature: `def load_gemma4_assistant_drafter(model_path: str)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/models/mllm.py#L340-L381 - Implementation: Function `load_gemma4_assistant_drafter` calls `ImportError`, `version`, `logger.info`, `Path`; can raise `ImportError`, `FileNotFoundError`; returns `model`. Load a Gemma 4 assistant drafter for mlx-vlm speculative decoding. - Inputs: - `model_path` (str; required): Required positional or keyword input. - Return annotation: `not annotated` - Calls: ImportError, version, logger.info, Path, sorted, path.glob, config_path.exists, FileNotFoundError, arch.ModelConfig.from_dict, json.loads, config_path.read_text, arch.Model, weights.update, mx.load, str, hasattr, model.sanitize, model.load_weights, list, weights.items, mx.eval, model.parameters, model.eval - Raises directly: ImportError, FileNotFoundError - Return expressions: model ## `vllm_mlx.models.mllm._count_draft_tokens` - Kind: function - Signature: `def _count_draft_tokens(draft_tokens) -> int` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/models/mllm.py#L387-L398 - Implementation: Function `_count_draft_tokens` calls `getattr`, `max`, `int`, `len`; has 3 explicit return paths. Best-effort drafted-token count for an mlx-vlm drafter output. - Inputs: - `draft_tokens` (not annotated; required): Required positional or keyword input. - Return annotation: `int` - Calls: getattr, max, int, len - Return expressions: max(int(shape[-1]), 0); max(len(draft_tokens), 0); 0 ## `vllm_mlx.models.mllm._install_draft_metrics_hooks` - Kind: function - Signature: `def _install_draft_metrics_hooks(draft_model) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/models/mllm.py#L401-L428 - Implementation: Function `_install_draft_metrics_hooks` calls `getattr`, `hasattr`, `callable`; returns `None`. Record actual drafted token counts from mlx-vlm assistant drafters. - Inputs: - `draft_model` (not annotated; required): Required positional or keyword input. - Return annotation: `None` - Calls: getattr, hasattr, callable - Return expressions: None ## `vllm_mlx.models.mllm._install_draft_metrics_hooks.draft_block_with_metrics` - Kind: nested function - Signature: `def draft_block_with_metrics(*args, **kwargs)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/models/mllm.py#L412-L415 - Implementation: Nested Function `_install_draft_metrics_hooks.draft_block_with_metrics` calls `draft_block`, `draft_model._vllm_mlx_draft_counts.append`, `_count_draft_tokens`; returns `draft_tokens`. Nested Function `_install_draft_metrics_hooks.draft_block_with_metrics` calls `draft_block`, `draft_model._vllm_mlx_draft_counts.append`, `_count_draft_tokens`; returns `draft_tokens`. - Inputs: - `*args` (not annotated; optional): Additional variadic positional inputs accepted by this callable. - `**kwargs` (not annotated; optional): Additional variadic keyword inputs accepted by this callable. - Return annotation: `not annotated` - Calls: draft_block, draft_model._vllm_mlx_draft_counts.append, _count_draft_tokens - Return expressions: draft_tokens ## `vllm_mlx.models.mllm._install_draft_metrics_hooks.reset_with_metrics` - Kind: nested function - Signature: `def reset_with_metrics(*args, **kwargs)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/models/mllm.py#L422-L424 - Implementation: Nested Function `_install_draft_metrics_hooks.reset_with_metrics` calls `reset`; returns `reset(*args, **kwargs)`. Nested Function `_install_draft_metrics_hooks.reset_with_metrics` calls `reset`; returns `reset(*args, **kwargs)`. - Inputs: - `*args` (not annotated; optional): Additional variadic positional inputs accepted by this callable. - `**kwargs` (not annotated; optional): Additional variadic keyword inputs accepted by this callable. - Return annotation: `not annotated` - Calls: reset - Return expressions: reset(*args, **kwargs) ## `vllm_mlx.models.mllm.is_base64_image` - Kind: function - Signature: `def is_base64_image(s: str) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/models/mllm.py#L431-L435 - Implementation: Function `is_base64_image` calls `s.startswith`, `len`; returns `s.startswith('data:image/') or (len(s) > 100 and (not s.startswith(('http://', 'https://', '/'))))`. Check if string is base64-encoded image data. - Inputs: - `s` (str; required): Required positional or keyword input. - Return annotation: `bool` - Calls: s.startswith, len - Return expressions: s.startswith('data:image/') or (len(s) > 100 and (not s.startswith(('http://', 'https://', '/')))) ## `vllm_mlx.models.mllm.is_url` - Kind: function - Signature: `def is_url(s: str) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/models/mllm.py#L438-L440 - Implementation: Function `is_url` calls `s.startswith`; returns `s.startswith(('http://', 'https://'))`. Check if string is a URL. - Inputs: - `s` (str; required): Required positional or keyword input. - Return annotation: `bool` - Calls: s.startswith - Return expressions: s.startswith(('http://', 'https://')) ## `vllm_mlx.models.mllm.is_base64_video` - Kind: function - Signature: `def is_base64_video(s: str) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/models/mllm.py#L443-L445 - Implementation: Function `is_base64_video` calls `s.startswith`; returns `s.startswith('data:video/')`. Check if string is base64-encoded video data. - Inputs: - `s` (str; required): Required positional or keyword input. - Return annotation: `bool` - Calls: s.startswith - Return expressions: s.startswith('data:video/') ## `vllm_mlx.models.mllm.is_base64_audio` - Kind: function - Signature: `def is_base64_audio(s: str) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/models/mllm.py#L448-L450 - Implementation: Function `is_base64_audio` calls `s.startswith`; returns `s.startswith('data:audio/')`. Check if string is base64-encoded audio data. - Inputs: - `s` (str; required): Required positional or keyword input. - Return annotation: `bool` - Calls: s.startswith - Return expressions: s.startswith('data:audio/') ## `vllm_mlx.models.mllm.decode_base64_image` - Kind: function - Signature: `def decode_base64_image(base64_string: str, max_length: int=MAX_BASE64_IMAGE_LENGTH) -> bytes` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/models/mllm.py#L453-L480 - Implementation: Function `decode_base64_image` calls `len`, `FileSizeExceededError`, `base64_string.startswith`, `base64_string.split`; can raise `FileSizeExceededError`; has 2 explicit return paths. Decode base64 image to bytes. Args: base64_string: Base64 encoded image (optionally with data URL prefix) max_length: Maximum allowed length of base64 string Returns: Decoded image bytes Raises: FileSizeExceededError: If base64 string exceeds max_length - Inputs: - `base64_string` (str; required): Base64 encoded image (optionally with data URL prefix) - `max_length` (int; optional; default `MAX_BASE64_IMAGE_LENGTH`): Maximum allowed length of base64 string - Return annotation: `bytes` - Calls: len, FileSizeExceededError, base64_string.startswith, base64_string.split, base64.b64decode - Raises directly: FileSizeExceededError - Return expressions: base64.b64decode(data); base64.b64decode(base64_string) ## `vllm_mlx.models.mllm._validate_url_safety` - Kind: function - Signature: `def _validate_url_safety(url: str) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/models/mllm.py#L483-L519 - Implementation: Function `_validate_url_safety` calls `urlparse`, `UnsafeRemoteURLError`, `hostname.endswith`, `ipaddress.ip_address`; can raise `UnsafeRemoteURLError`. Reject remote URLs that target local or private network resources. - Inputs: - `url` (str; required): Required positional or keyword input. - Return annotation: `None` - Calls: urlparse, UnsafeRemoteURLError, hostname.endswith, ipaddress.ip_address, socket.getaddrinfo, str, ', '.join, sorted, set - Raises directly: UnsafeRemoteURLError ## `vllm_mlx.models.mllm._request_with_safe_redirects` - Kind: function - Signature: `def _request_with_safe_redirects(method: str, url: str, *, timeout: int, headers: dict[str, str], stream: bool=False, max_redirects: int=5)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/models/mllm.py#L522-L557 - Implementation: Function `_request_with_safe_redirects` calls `range`, `_validate_url_safety`, `requests.request`, `response.headers.get`; can raise `UnsafeRemoteURLError`; returns `response`. Issue a requests call while validating every redirect target. - Inputs: - `method` (str; required): Required positional or keyword input. - `url` (str; required): Required positional or keyword input. - `timeout` (int; required): Required keyword-only input. - `headers` (dict[str, str]; required): Required keyword-only input. - `stream` (bool; optional; default `False`): Optional keyword-only input; defaults to `False`. - `max_redirects` (int; optional; default `5`): Optional keyword-only input; defaults to `5`. - Return annotation: `not annotated` - Calls: range, _validate_url_safety, requests.request, response.headers.get, response.close, UnsafeRemoteURLError, urljoin - Raises directly: UnsafeRemoteURLError - Return expressions: response ## `vllm_mlx.models.mllm.download_image` - Kind: function - Signature: `def download_image(url: str, timeout: int=30, max_size: int=MAX_IMAGE_SIZE) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/models/mllm.py#L560-L645 - Implementation: Function `download_image` calls `_request_with_safe_redirects`, `head_response.headers.get`, `int`, `FileSizeExceededError`; can raise `FileSizeExceededError`; returns `_temp_manager.register(temp_file.name)`. Download image from URL and return local path. Args: url: Image URL timeout: Download timeout in seconds max_size: Maximum allowed file size in bytes Returns: Local file path to downloaded image Raises: FileSizeExceededError: If image exceeds max_size - Inputs: - `url` (str; required): Image URL - `timeout` (int; optional; default `30`): Download timeout in seconds - `max_size` (int; optional; default `MAX_IMAGE_SIZE`): Maximum allowed file size in bytes - Return annotation: `str` - Calls: _request_with_safe_redirects, head_response.headers.get, int, FileSizeExceededError, response.raise_for_status, response.headers.get, urlparse, Path, tempfile.NamedTemporaryFile, response.iter_content, len, temp_file.close, os.unlink, temp_file.write, os.path.exists, _temp_manager.register - Raises directly: FileSizeExceededError - Return expressions: _temp_manager.register(temp_file.name) ## `vllm_mlx.models.mllm._download_media` - Kind: function - Signature: `def _download_media(url: str, media_type: str, ext_map: dict[str, str], default_ext: str, timeout: int, max_size: int) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/models/mllm.py#L670-L748 - Implementation: Function `_download_media` calls `logger.info`, `_request_with_safe_redirects`, `head_response.headers.get`, `int`; can raise `FileSizeExceededError`; returns `_temp_manager.register(temp_file.name)`. Download media from URL, enforce size limits, and return a local temp path. - Inputs: - `url` (str; required): Required positional or keyword input. - `media_type` (str; required): Required positional or keyword input. - `ext_map` (dict[str, str]; required): Required positional or keyword input. - `default_ext` (str; required): Required positional or keyword input. - `timeout` (int; required): Required positional or keyword input. - `max_size` (int; required): Required positional or keyword input. - Return annotation: `str` - Calls: logger.info, _request_with_safe_redirects, head_response.headers.get, int, FileSizeExceededError, media_type.capitalize, response.raise_for_status, response.headers.get, response.headers.get('content-type', '').lower, ext_map.items, Path, urlparse, tempfile.NamedTemporaryFile, response.iter_content, len, temp_file.close, os.unlink, temp_file.write, os.path.exists, Path(temp_file.name).stat, _temp_manager.register - Raises directly: FileSizeExceededError - Return expressions: _temp_manager.register(temp_file.name) ## `vllm_mlx.models.mllm.download_video` - Kind: function - Signature: `def download_video(url: str, timeout: int=120, max_size: int=MAX_VIDEO_SIZE) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/models/mllm.py#L751-L753 - Implementation: Function `download_video` calls `_download_media`; returns `_download_media(url, 'video', _VIDEO_EXT_MAP, '.mp4', timeout, max_size)`. Download video from URL and return local path. - Inputs: - `url` (str; required): Required positional or keyword input. - `timeout` (int; optional; default `120`): Optional positional or keyword input; defaults to `120`. - `max_size` (int; optional; default `MAX_VIDEO_SIZE`): Optional positional or keyword input; defaults to `MAX_VIDEO_SIZE`. - Return annotation: `str` - Calls: _download_media - Return expressions: _download_media(url, 'video', _VIDEO_EXT_MAP, '.mp4', timeout, max_size) ## `vllm_mlx.models.mllm.download_audio` - Kind: function - Signature: `def download_audio(url: str, timeout: int=120, max_size: int=MAX_AUDIO_SIZE) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/models/mllm.py#L756-L758 - Implementation: Function `download_audio` calls `_download_media`; returns `_download_media(url, 'audio', _AUDIO_EXT_MAP, '.wav', timeout, max_size)`. Download audio from URL and return local path. - Inputs: - `url` (str; required): Required positional or keyword input. - `timeout` (int; optional; default `120`): Optional positional or keyword input; defaults to `120`. - `max_size` (int; optional; default `MAX_AUDIO_SIZE`): Optional positional or keyword input; defaults to `MAX_AUDIO_SIZE`. - Return annotation: `str` - Calls: _download_media - Return expressions: _download_media(url, 'audio', _AUDIO_EXT_MAP, '.wav', timeout, max_size) ## `vllm_mlx.models.mllm.decode_base64_video` - Kind: function - Signature: `def decode_base64_video(base64_string: str, max_length: int=MAX_BASE64_VIDEO_LENGTH) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/models/mllm.py#L761-L807 - Implementation: Function `decode_base64_video` calls `len`, `FileSizeExceededError`, `base64_string.startswith`, `base64_string.split`; can raise `FileSizeExceededError`; returns `_temp_manager.register(temp_file.name)`. Decode base64 video to temp file and return path. Supports format: data:video/mp4;base64,AAAA... Args: base64_string: Base64-encoded video with data URL prefix max_length: Maximum allowed length of base64 string Returns: Local file path to decoded video Raises: FileSizeExceededError: If base64 string exceeds max_length - Inputs: - `base64_string` (str; required): Base64-encoded video with data URL prefix - `max_length` (int; optional; default `MAX_BASE64_VIDEO_LENGTH`): Maximum allowed length of base64 string - Return annotation: `str` - Calls: len, FileSizeExceededError, base64_string.startswith, base64_string.split, header.split, format_part.split, base64.b64decode, tempfile.NamedTemporaryFile, temp_file.write, temp_file.close, logger.info, _temp_manager.register - Raises directly: FileSizeExceededError - Return expressions: _temp_manager.register(temp_file.name) ## `vllm_mlx.models.mllm.decode_base64_audio` - Kind: function - Signature: `def decode_base64_audio(base64_string: str, max_length: int=MAX_BASE64_AUDIO_LENGTH) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/models/mllm.py#L810-L836 - Implementation: Function `decode_base64_audio` calls `len`, `FileSizeExceededError`, `base64_string.startswith`, `base64_string.split`; can raise `FileSizeExceededError`; returns `_temp_manager.register(temp_file.name)`. Decode base64 audio to temp file and return path. Supports format: data:audio/wav;base64,AAAA... - Inputs: - `base64_string` (str; required): Required positional or keyword input. - `max_length` (int; optional; default `MAX_BASE64_AUDIO_LENGTH`): Optional positional or keyword input; defaults to `MAX_BASE64_AUDIO_LENGTH`. - Return annotation: `str` - Calls: len, FileSizeExceededError, base64_string.startswith, base64_string.split, header.split, format_part.split, base64.b64decode, tempfile.NamedTemporaryFile, temp_file.write, temp_file.close, _temp_manager.register - Raises directly: FileSizeExceededError - Return expressions: _temp_manager.register(temp_file.name) ## `vllm_mlx.models.mllm.process_video_input` - Kind: function - Signature: `def process_video_input(video: str | dict) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/models/mllm.py#L839-L874 - Implementation: Function `process_video_input` calls `isinstance`, `video.get`, `url.get`, `ValueError`; can raise `ValueError`; has 2 explicit return paths. Process video input in various formats and return local path. Supports: - URL (http/https) - Base64 encoded string (data:video/mp4;base64,...) - OpenAI format dict: {"url": "..."} or {"url": "data:video/...;base64,..."} Args: video: Video input in any supported format Returns: Local file path to video - Inputs: - `video` (str | dict; required): Video input in any supported format - Return annotation: `str` - Calls: isinstance, video.get, url.get, ValueError, is_url, download_video, is_base64_video, decode_base64_video - Raises directly: ValueError - Return expressions: download_video(video); decode_base64_video(video) ## `vllm_mlx.models.mllm.process_audio_input` - Kind: function - Signature: `def process_audio_input(audio: str | dict) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/models/mllm.py#L877-L905 - Implementation: Function `process_audio_input` calls `isinstance`, `audio.get`, `url.get`, `ValueError`; can raise `ValueError`; has 3 explicit return paths. Process audio input in various formats and return local path. Supports: - Local file path - URL (http/https) - Base64 encoded string (data:audio/wav;base64,...) - OpenAI format dict: {"url": "..."} or {"audio_url": {"url": "..."}} - Inputs: - `audio` (str | dict; required): Required positional or keyword input. - Return annotation: `str` - Calls: isinstance, audio.get, url.get, ValueError, is_base64_audio, decode_base64_audio, is_url, download_audio, len, Path(audio).exists, Path - Raises directly: ValueError - Return expressions: decode_base64_audio(audio); download_audio(audio); audio ## `vllm_mlx.models.mllm._video_has_audio_track` - Kind: function - Signature: `def _video_has_audio_track(video_path: str) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/models/mllm.py#L908-L935 - Implementation: Function `_video_has_audio_track` calls `shutil.which`, `subprocess.run`, `bool`, `r.stdout.strip`; has 2 explicit return paths. Return True if ffprobe finds an audio stream in the video. - Inputs: - `video_path` (str; required): Required positional or keyword input. - Return annotation: `bool` - Calls: shutil.which, subprocess.run, bool, r.stdout.strip - Return expressions: True; bool(r.stdout.strip()) ## `vllm_mlx.models.mllm._model_has_sound_encoder` - Kind: function - Signature: `def _model_has_sound_encoder(model) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/models/mllm.py#L938-L947 - Implementation: Function `_model_has_sound_encoder` calls `getattr`; returns `getattr(model, 'sound_encoder', None) is not None`. Whether a loaded model exposes a usable sound encoder. Uses ``getattr(..., None) is not None`` rather than ``hasattr`` so model wrappers that declare ``sound_encoder`` in ``__init__`` but leave it as ``None`` until the first encoder pass are correctly treated as not yet enabled. A bare ``hasattr`` check would spuriously enable A/V fusion against a missing encoder and crash the processor downstream. - Inputs: - `model` (not annotated; required): Required positional or keyword input. - Return annotation: `bool` - Calls: getattr - Return expressions: getattr(model, 'sound_encoder', None) is not None ## `vllm_mlx.models.mllm.extract_audio_from_video` - Kind: function - Signature: `def extract_audio_from_video(video_path: str) -> str | None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/models/mllm.py#L950-L1005 - Implementation: Function `extract_audio_from_video` calls `shutil.which`, `logger.warning`, `_video_has_audio_track`, `tempfile.mkstemp`; has 2 explicit return paths. Extract the audio track from a video file as 16 kHz mono WAV. Returns the path to the WAV (registered with the temp manager so it's cleaned up automatically), or None if the video has no audio or ffmpeg is unavailable. - Inputs: - `video_path` (str; required): Required positional or keyword input. - Return annotation: `str | None` - Calls: shutil.which, logger.warning, _video_has_audio_track, tempfile.mkstemp, os.close, subprocess.run, os.path.getsize, os.unlink, _temp_manager.register - Return expressions: None; _temp_manager.register(out_path) ## `vllm_mlx.models.mllm.save_base64_image` - Kind: function - Signature: `def save_base64_image(base64_string: str) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/models/mllm.py#L1012-L1048 - Implementation: Function `save_base64_image` calls `hashlib.sha256(base64_string.encode()).hexdigest`, `hashlib.sha256`, `base64_string.encode`, `Path(cached_path).exists`; has 2 explicit return paths. Save base64 image to temp file and return path. Caches identical images. - Inputs: - `base64_string` (str; required): Required positional or keyword input. - Return annotation: `str` - Calls: hashlib.sha256(base64_string.encode()).hexdigest, hashlib.sha256, base64_string.encode, Path(cached_path).exists, Path, decode_base64_image, tempfile.NamedTemporaryFile, temp_file.write, temp_file.close, _temp_manager.register - Return expressions: cached_path; path ## `vllm_mlx.models.mllm.process_image_input` - Kind: function - Signature: `def process_image_input(image: str | dict) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/models/mllm.py#L1051-L1080 - Implementation: Function `process_image_input` calls `isinstance`, `image.get`, `url.get`, `ValueError`; can raise `ValueError`; has 2 explicit return paths. Process image input in various formats and return local path. Supports: - URL (http/https) - Base64 encoded string - OpenAI format dict: {"url": "..."} or {"url": "data:image/...;base64,..."} - Inputs: - `image` (str | dict; required): Required positional or keyword input. - Return annotation: `str` - Calls: isinstance, image.get, url.get, ValueError, is_base64_image, save_base64_image, is_url, download_image - Raises directly: ValueError - Return expressions: save_base64_image(image); download_image(image) ## `vllm_mlx.models.mllm.round_by_factor` - Kind: function - Signature: `def round_by_factor(x: int, factor: int) -> int` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/models/mllm.py#L1083-L1085 - Implementation: Function `round_by_factor` calls `round`; returns `round(x / factor) * factor`. Round to nearest multiple of factor. - Inputs: - `x` (int; required): Required positional or keyword input. - `factor` (int; required): Required positional or keyword input. - Return annotation: `int` - Calls: round - Return expressions: round(x / factor) * factor ## `vllm_mlx.models.mllm.ceil_by_factor` - Kind: function - Signature: `def ceil_by_factor(x: float, factor: int) -> int` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/models/mllm.py#L1088-L1090 - Implementation: Function `ceil_by_factor` calls `math.ceil`; returns `math.ceil(x / factor) * factor`. Ceiling to next multiple of factor. - Inputs: - `x` (float; required): Required positional or keyword input. - `factor` (int; required): Required positional or keyword input. - Return annotation: `int` - Calls: math.ceil - Return expressions: math.ceil(x / factor) * factor ## `vllm_mlx.models.mllm.floor_by_factor` - Kind: function - Signature: `def floor_by_factor(x: float, factor: int) -> int` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/models/mllm.py#L1093-L1095 - Implementation: Function `floor_by_factor` calls `math.floor`; returns `math.floor(x / factor) * factor`. Floor to previous multiple of factor. - Inputs: - `x` (float; required): Required positional or keyword input. - `factor` (int; required): Required positional or keyword input. - Return annotation: `int` - Calls: math.floor - Return expressions: math.floor(x / factor) * factor ## `vllm_mlx.models.mllm.smart_nframes` - Kind: function - Signature: `def smart_nframes(total_frames: int, video_fps: float, target_fps: float=DEFAULT_FPS, min_frames: int=MIN_FRAMES, max_frames: int=MAX_FRAMES) -> int` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/models/mllm.py#L1098-L1120 - Implementation: Function `smart_nframes` calls `max`, `min`, `floor_by_factor`, `int`; returns `int(nframes)`. Calculate optimal number of frames to extract from video. Uses smart sampling based on video length and target FPS. - Inputs: - `total_frames` (int; required): Required positional or keyword input. - `video_fps` (float; required): Required positional or keyword input. - `target_fps` (float; optional; default `DEFAULT_FPS`): Optional positional or keyword input; defaults to `DEFAULT_FPS`. - `min_frames` (int; optional; default `MIN_FRAMES`): Optional positional or keyword input; defaults to `MIN_FRAMES`. - `max_frames` (int; optional; default `MAX_FRAMES`): Optional positional or keyword input; defaults to `MAX_FRAMES`. - Return annotation: `int` - Calls: max, min, floor_by_factor, int - Return expressions: int(nframes) ## `vllm_mlx.models.mllm.extract_video_frames_smart` - Kind: function - Signature: `def extract_video_frames_smart(video_path: str, fps: float=DEFAULT_FPS, max_frames: int=MAX_FRAMES, resize: tuple[int, int] | None=None) -> list[np.ndarray]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/models/mllm.py#L1123-L1187 - Implementation: Function `extract_video_frames_smart` calls `ImportError`, `cv2.VideoCapture`, `cap.isOpened`, `ValueError`; can raise `ImportError`, `ValueError`; returns `frames`. Extract frames from video with smart sampling. Args: video_path: Path to video file fps: Target frames per second (default: 2.0) max_frames: Maximum frames to extract resize: Optional (width, height) to resize frames Returns: List of frame arrays (RGB format) - Inputs: - `video_path` (str; required): Path to video file - `fps` (float; optional; default `DEFAULT_FPS`): Target frames per second (default: 2.0) - `max_frames` (int; optional; default `MAX_FRAMES`): Maximum frames to extract - `resize` (tuple[int, int] | None; optional; default `None`): Optional (width, height) to resize frames - Return annotation: `list[np.ndarray]` - Calls: ImportError, cv2.VideoCapture, cap.isOpened, ValueError, int, cap.get, smart_nframes, np.linspace(0, total_frames - 1, nframes).round().astype, np.linspace(0, total_frames - 1, nframes).round, np.linspace, logger.info, cap.set, cap.read, cv2.cvtColor, cv2.resize, frames.append, cap.release - Raises directly: ImportError, ValueError - Return expressions: frames ## `vllm_mlx.models.mllm.save_frames_to_temp` - Kind: function - Signature: `def save_frames_to_temp(frames: list[np.ndarray]) -> list[str]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/models/mllm.py#L1190-L1204 - Implementation: Function `save_frames_to_temp` calls `ImportError`, `enumerate`, `Image.fromarray`, `tempfile.NamedTemporaryFile`; can raise `ImportError`; returns `paths`. Save frame arrays to temporary files and return paths. - Inputs: - `frames` (list[np.ndarray]; required): Required positional or keyword input. - Return annotation: `list[str]` - Calls: ImportError, enumerate, Image.fromarray, tempfile.NamedTemporaryFile, img.save, paths.append, _temp_manager.register - Raises directly: ImportError - Return expressions: paths ## `vllm_mlx.models.mllm.MLXMultimodalLM` - Kind: class - Signature: `class MLXMultimodalLM` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/models/mllm.py#L1207-L2938 - Implementation: Class `MLXMultimodalLM` declares 29 direct member(s). Wrapper around mlx-vlm for multimodal inference. This class provides a unified interface for multimodal language models using Apple's MLX framework. Supports: - Image understanding (single and multi-image) - Video understanding (smart frame extraction) - Audio understanding (for supported models) - OpenAI-compatible API format Supported models include: - Qwen2-VL / Qwen2.5-VL / Qwen3-VL - LLaVA - Idefics3 - PaliGemma - And more via mlx-vlm Example: >>> model = MLXMultimodalLM("mlx-community/Qwen2-VL-2B-Instruct-4bit") >>> model.load() >>> output = model.generate( ... prompt="What's in this image?", ... images=["photo.jpg"] ... ) >>> print(output.text) - Inputs: - `model_name` (str; required): HuggingFace model name or local path - `trust_remote_code` (bool; optional; default `False`): Whether to trust remote code - `enable_cache` (bool; optional; default `True`): Enable KV cache for repeated image/video+prompt (default: True) - `cache_size` (int; optional; default `50`): Maximum cache entries (default: 50) - `max_kv_size` (int; optional; default `0`): Maximum KV cache size per sequence (0 = unbounded) - `draft_model` (str | None; optional; default `None`): Optional MLLM speculative draft/assistant model path. - `draft_kind` (str | None; optional; default `None`): Optional mlx-vlm draft kind, for example "mtp". - `draft_block_size` (int | None; optional; default `None`): Optional speculative block size passed to mlx-vlm. - Constructs: `vllm_mlx.models.mllm.MLXMultimodalLM` ## `vllm_mlx.models.mllm.MLXMultimodalLM.__init__` - Kind: method - Signature: `def __init__(self, model_name: str, trust_remote_code: bool=False, enable_cache: bool=True, cache_size: int=50, max_kv_size: int=0, draft_model: str | None=None, draft_kind: str | None=None, draft_block_size: int | None=None)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/models/mllm.py#L1235-L1278 - Implementation: Method `MLXMultimodalLM.__init__` updates `self.model_name`, `self.trust_remote_code`, `self.enable_cache`, `self.max_kv_size`; calls `MLLMPrefixCacheManager`. Initialize the MLX multimodal language model. Args: model_name: HuggingFace model name or local path trust_remote_code: Whether to trust remote code enable_cache: Enable KV cache for repeated image/video+prompt (default: True) cache_size: Maximum cache entries (default: 50) max_kv_size: Maximum KV cache size per sequence (0 = unbounded) draft_model: Optional MLLM speculative draft/assistant model path. draft_kind: Optional mlx-vlm draft kind, for example "mtp". draft_block_size: Optional speculative block size passed to mlx-vlm. - Inputs: - `model_name` (str; required): HuggingFace model name or local path - `trust_remote_code` (bool; optional; default `False`): Whether to trust remote code - `enable_cache` (bool; optional; default `True`): Enable KV cache for repeated image/video+prompt (default: True) - `cache_size` (int; optional; default `50`): Maximum cache entries (default: 50) - `max_kv_size` (int; optional; default `0`): Maximum KV cache size per sequence (0 = unbounded) - `draft_model` (str | None; optional; default `None`): Optional MLLM speculative draft/assistant model path. - `draft_kind` (str | None; optional; default `None`): Optional mlx-vlm draft kind, for example "mtp". - `draft_block_size` (int | None; optional; default `None`): Optional speculative block size passed to mlx-vlm. - Return annotation: `not annotated` - Calls: MLLMPrefixCacheManager - State writes: self.model_name, self.trust_remote_code, self.enable_cache, self.max_kv_size, self.draft_model_path, self.draft_kind, self.draft_block_size, self.model, self.processor, self.config, self._draft_model, self._loaded, self._video_native, self._video_native_with_audio, self._cache_manager ## `vllm_mlx.models.mllm.MLXMultimodalLM.load` - Kind: method - Signature: `def load(self) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/models/mllm.py#L1280-L1323 - Implementation: Method `MLXMultimodalLM.load` updates `self.model`, `self.processor`, `self.config`, `self._draft_model`; calls `logger.info`, `load`, `load_config`, `self._load_draft_model`; can raise `ImportError`; returns `None`. Load the model and processor. - Inputs: none - Return annotation: `None` - Calls: logger.info, load, load_config, self._load_draft_model, _install_draft_metrics_hooks, hasattr, _model_has_sound_encoder, ImportError, logger.error - State reads: self._loaded, self.model_name, self.draft_model_path, self._load_draft_model, self._draft_model, self.model.config, self.model, self._video_native, self._video_native_with_audio - State writes: self.model, self.processor, self.config, self._draft_model, self._loaded, self._video_native, self._video_native_with_audio - Raises directly: ImportError - Return expressions: None ## `vllm_mlx.models.mllm.MLXMultimodalLM._load_draft_model` - Kind: method - Signature: `def _load_draft_model(self)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/models/mllm.py#L1325-L1332 - Implementation: Method `MLXMultimodalLM._load_draft_model` calls `load_gemma4_assistant_drafter`, `load`; has 2 explicit return paths. Method `MLXMultimodalLM._load_draft_model` calls `load_gemma4_assistant_drafter`, `load`; has 2 explicit return paths. - Inputs: none - Return annotation: `not annotated` - Calls: load_gemma4_assistant_drafter, load - State reads: self.draft_kind, self.draft_model_path - Return expressions: load_gemma4_assistant_drafter(self.draft_model_path); draft_model ## `vllm_mlx.models.mllm.MLXMultimodalLM._draft_generation_kwargs` - Kind: method - Signature: `def _draft_generation_kwargs(self, call_kwargs: dict | None=None) -> dict` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/models/mllm.py#L1334-L1355 - Implementation: Method `MLXMultimodalLM._draft_generation_kwargs` calls `bool`, `call_kwargs.pop`, `_install_draft_metrics_hooks`; has 2 explicit return paths. Return mlx-vlm drafter kwargs when the request explicitly opts in. ``call_kwargs`` is the outbound mlx-vlm kwargs dict. This method removes vllm-mlx drafter control keys before the dict is forwarded so caller passthrough values cannot conflict with the configured server drafter. - Inputs: - `call_kwargs` (dict | None; optional; default `None`): Optional positional or keyword input; defaults to `None`. - Return annotation: `dict` - Calls: bool, call_kwargs.pop, _install_draft_metrics_hooks - State reads: self._draft_model, self.draft_kind, self.draft_block_size - Return expressions: {}; kwargs ## `vllm_mlx.models.mllm.MLXMultimodalLM._reset_draft_metrics` - Kind: method - Signature: `def _reset_draft_metrics(self) -> int` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/models/mllm.py#L1357-L1364 - Implementation: Method `MLXMultimodalLM._reset_draft_metrics` updates `self._draft_model.accept_lens`, `self._draft_model._vllm_mlx_draft_counts`; calls `hasattr`; returns `0`. Method `MLXMultimodalLM._reset_draft_metrics` updates `self._draft_model.accept_lens`, `self._draft_model._vllm_mlx_draft_counts`; calls `hasattr`; returns `0`. - Inputs: none - Return annotation: `int` - Calls: hasattr - State reads: self._draft_model - State writes: self._draft_model.accept_lens, self._draft_model._vllm_mlx_draft_counts - Return expressions: 0 ## `vllm_mlx.models.mllm.MLXMultimodalLM._draft_metrics_since` - Kind: method - Signature: `def _draft_metrics_since(self, start_accept_lens: int) -> dict[str, int]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/models/mllm.py#L1366-L1395 - Implementation: Method `MLXMultimodalLM._draft_metrics_since` calls `list`, `getattr`, `len`, `int`; has 2 explicit return paths. Method `MLXMultimodalLM._draft_metrics_since` calls `list`, `getattr`, `len`, `int`; has 2 explicit return paths. - Inputs: - `start_accept_lens` (int; required): Required positional or keyword input. - Return annotation: `dict[str, int]` - Calls: list, getattr, len, int, max, sum - State reads: self._draft_model, self.draft_block_size - Return expressions: {'mtp_drafts': 0, 'mtp_accepted': 0}; {'mtp_drafts': mtp_drafts, 'mtp_accepted': sum((int(value) for value in new_accept_lens))} ## `vllm_mlx.models.mllm.MLXMultimodalLM.get_language_model` - Kind: method - Signature: `def get_language_model(self)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/models/mllm.py#L1397-L1399 - Implementation: Method `MLXMultimodalLM.get_language_model` returns `self.model.language_model`. Extract the underlying language model for mlx_lm TextModel construction. - Inputs: none - Return annotation: `not annotated` - State reads: self.model.language_model, self.model - Return expressions: self.model.language_model ## `vllm_mlx.models.mllm.MLXMultimodalLM.get_tokenizer` - Kind: method - Signature: `def get_tokenizer(self)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/models/mllm.py#L1401-L1403 - Implementation: Method `MLXMultimodalLM.get_tokenizer` returns `self.processor.tokenizer`. Get the text tokenizer (not the multimodal processor). - Inputs: none - Return annotation: `not annotated` - State reads: self.processor.tokenizer, self.processor - Return expressions: self.processor.tokenizer ## `vllm_mlx.models.mllm.MLXMultimodalLM._prepare_images` - Kind: method - Signature: `def _prepare_images(self, images: list) -> list[str]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/models/mllm.py#L1405-L1414 - Implementation: Method `MLXMultimodalLM._prepare_images` calls `process_image_input`, `processed.append`, `logger.warning`; returns `processed`. Process remote/base64 image inputs into local temp file paths. - Inputs: - `images` (list; required): Required positional or keyword input. - Return annotation: `list[str]` - Calls: process_image_input, processed.append, logger.warning - Return expressions: processed ## `vllm_mlx.models.mllm.MLXMultimodalLM._prepare_audio` - Kind: method - Signature: `def _prepare_audio(self, audio_inputs: list) -> list[str]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/models/mllm.py#L1416-L1425 - Implementation: Method `MLXMultimodalLM._prepare_audio` calls `process_audio_input`, `processed.append`, `logger.warning`; returns `processed`. Process audio inputs and return local file paths. - Inputs: - `audio_inputs` (list; required): Required positional or keyword input. - Return annotation: `list[str]` - Calls: process_audio_input, processed.append, logger.warning - Return expressions: processed ## `vllm_mlx.models.mllm.MLXMultimodalLM._prepare_video` - Kind: method - Signature: `def _prepare_video(self, video_input: str | dict, fps: float=DEFAULT_FPS, max_frames: int=MAX_FRAMES, resolved_path: str | None=None) -> list[str]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/models/mllm.py#L1427-L1463 - Implementation: Method `MLXMultimodalLM._prepare_video` calls `process_video_input`, `extract_video_frames_smart`, `save_frames_to_temp`; returns `save_frames_to_temp(frames)`. Process video input and extract frames. Supports: - URLs (http/https) - will be downloaded - Base64 encoded videos (data:video/mp4;base64,...) - OpenAI format dicts: {"url": "..."} or {"video_url": {"url": "..."}} Args: video_input: Video in any supported format fps: Frames per second to extract max_frames: Maximum frames to extract resolved_path: Optional pre-resolved local path. Callers that already ran process_video_input (e.g. for parallel audio extraction) pass it here to avoid re-downloading / re-decoding. Returns: List of paths to extracted frame images - Inputs: - `video_input` (str | dict; required): Video in any supported format - `fps` (float; optional; default `DEFAULT_FPS`): Frames per second to extract - `max_frames` (int; optional; default `MAX_FRAMES`): Maximum frames to extract - `resolved_path` (str | None; optional; default `None`): Optional pre-resolved local path. Callers that already ran process_video_input (e.g. for parallel audio extraction) pass it here to avoid re-downloading / re-decoding. - Return annotation: `list[str]` - Calls: process_video_input, extract_video_frames_smart, save_frames_to_temp - Return expressions: save_frames_to_temp(frames) ## `vllm_mlx.models.mllm.MLXMultimodalLM._collect_video_inputs` - Kind: method - Signature: `def _collect_video_inputs(self, messages: list[dict]) -> dict[int, list]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/models/mllm.py#L1465-L1497 - Implementation: Method `MLXMultimodalLM._collect_video_inputs` calls `enumerate`, `msg.get`, `isinstance`, `hasattr`; returns `video_inputs`. Collect video inputs from messages, keyed by message index. Handles both 'video' and 'video_url' content types, including Pydantic model conversion. - Inputs: - `messages` (list[dict]; required): Required positional or keyword input. - Return annotation: `dict[int, list]` - Calls: enumerate, msg.get, isinstance, hasattr, item.model_dump, item.dict().items, item.dict, item.get, video_inputs.setdefault(msg_idx, []).append, video_inputs.setdefault, vid_url.get - Return expressions: video_inputs ## `vllm_mlx.models.mllm.MLXMultimodalLM._collect_audio_inputs` - Kind: method - Signature: `def _collect_audio_inputs(self, messages: list[dict]) -> dict[int, list]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/models/mllm.py#L1499-L1528 - Implementation: Method `MLXMultimodalLM._collect_audio_inputs` calls `enumerate`, `msg.get`, `isinstance`, `hasattr`; returns `audio_inputs`. Collect audio inputs from messages, keyed by message index. - Inputs: - `messages` (list[dict]; required): Required positional or keyword input. - Return annotation: `dict[int, list]` - Calls: enumerate, msg.get, isinstance, hasattr, item.model_dump, item.dict().items, item.dict, item.get, audio_inputs.setdefault(msg_idx, []).append, audio_inputs.setdefault, audio_url.get - Return expressions: audio_inputs ## `vllm_mlx.models.mllm.MLXMultimodalLM._prepare_native_video_inputs` - Kind: method - Signature: `def _prepare_native_video_inputs(self, messages: list[dict], video_fps: float=DEFAULT_FPS, video_max_frames: int=MAX_FRAMES, tools: list | None=None) -> tuple[str, dict]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/models/mllm.py#L1530-L1648 - Implementation: Method `MLXMultimodalLM._prepare_native_video_inputs` calls `ImportError`, `self._translate_messages_for_native_video`, `self.processor.apply_chat_template`, `process_vision_info`; can raise `ImportError`; returns `(text, gen_kwargs)`. Preprocess messages into prompt + generation kwargs for native video. Mirrors the preprocessing in mlx_vlm.video_generate.main() so that upstream improvements are easy to adopt. Returns the formatted prompt text and a dict of kwargs ready to pass to video_generate.generate(). Currently Qwen-family-specific (video_token_id / video_token_index). - Inputs: - `messages` (list[dict]; required): Required positional or keyword input. - `video_fps` (float; optional; default `DEFAULT_FPS`): Optional positional or keyword input; defaults to `DEFAULT_FPS`. - `video_max_frames` (int; optional; default `MAX_FRAMES`): Optional positional or keyword input; defaults to `MAX_FRAMES`. - `tools` (list | None; optional; default `None`): Optional positional or keyword input; defaults to `None`. - Return annotation: `tuple[str, dict]` - Calls: ImportError, self._translate_messages_for_native_video, self.processor.apply_chat_template, process_vision_info, nmsg.get, isinstance, nitem.get, audio_inputs.append, self.processor, mx.array, inputs.get, logger.info, len, gen_kwargs.get, grid_thw_info.tolist - State reads: self._translate_messages_for_native_video, self.processor.apply_chat_template, self.processor - Raises directly: ImportError - Return expressions: (text, gen_kwargs) ## `vllm_mlx.models.mllm.MLXMultimodalLM._generate_native_video` - Kind: method - Signature: `def _generate_native_video(self, messages: list[dict], max_tokens: int=256, temperature: float=0.7, video_fps: float=DEFAULT_FPS, video_max_frames: int=MAX_FRAMES, tools: list | None=None, **kwargs) -> MLLMOutput` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/models/mllm.py#L1650-L1695 - Implementation: Method `MLXMultimodalLM._generate_native_video` calls `ImportError`, `self._prepare_native_video_inputs`, `generate`, `hasattr`; can raise `ImportError`; has 2 explicit return paths. Generate using native video pipeline (Qwen-family models). Delegates preprocessing to _prepare_native_video_inputs and generation to mlx_vlm.video_generate.generate(), keeping our code aligned with upstream's video pipeline so improvements are easy to adopt. - Inputs: - `messages` (list[dict]; required): Required positional or keyword input. - `max_tokens` (int; optional; default `256`): Optional positional or keyword input; defaults to `256`. - `temperature` (float; optional; default `0.7`): Optional positional or keyword input; defaults to `0.7`. - `video_fps` (float; optional; default `DEFAULT_FPS`): Optional positional or keyword input; defaults to `DEFAULT_FPS`. - `video_max_frames` (int; optional; default `MAX_FRAMES`): Optional positional or keyword input; defaults to `MAX_FRAMES`. - `tools` (list | None; optional; default `None`): Optional positional or keyword input; defaults to `None`. - `**kwargs` (not annotated; optional): Additional variadic keyword inputs accepted by this callable. - Return annotation: `MLLMOutput` - Calls: ImportError, self._prepare_native_video_inputs, generate, hasattr, MLLMOutput, getattr, str - State reads: self._prepare_native_video_inputs, self.model, self.processor - Raises directly: ImportError - Return expressions: MLLMOutput(text=result.text, finish_reason='stop', prompt_tokens=getattr(result, 'prompt_tokens', 0), completion_tokens…; MLLMOutput(text=str(result), finish_reason='stop') ## `vllm_mlx.models.mllm.MLXMultimodalLM._translate_messages_for_native_video` - Kind: method - Signature: `def _translate_messages_for_native_video(self, messages: list[dict], video_fps: float, video_max_frames: int) -> list[dict]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/models/mllm.py#L1697-L1832 - Implementation: Method `MLXMultimodalLM._translate_messages_for_native_video` calls `msg.get`, `isinstance`, `translated.append`, `str`; returns `translated`. Translate OpenAI API format messages to process_vision_info format. Converts video_url/video types and resolves remote/base64 inputs to local paths. Images are preserved as-is (process_vision_info handles them). - Inputs: - `messages` (list[dict]; required): Required positional or keyword input. - `video_fps` (float; required): Required positional or keyword input. - `video_max_frames` (int; required): Required positional or keyword input. - Return annotation: `list[dict]` - Calls: msg.get, isinstance, translated.append, str, hasattr, item.model_dump, item.dict().items, item.dict, probe.get, new_content.append, item.get, img_url.get, process_image_input, vid_url.get, process_video_input, getattr, extract_audio_from_video, aud_url.get, process_audio_input - Return expressions: translated ## `vllm_mlx.models.mllm.MLXMultimodalLM.generate` - Kind: method - Signature: `def generate(self, prompt: str, images: list | None=None, videos: list | None=None, audio: list[str] | None=None, max_tokens: int=256, temperature: float=0.7, top_p: float=0.9, video_fps: float=DEFAULT_FPS, video_max_frames: int=MAX_FRAMES, use_cache: bool=True, **kwargs) -> MLLMOutput` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/models/mllm.py#L1834-L2002 - Implementation: Method `MLXMultimodalLM.generate` calls `self.load`, `all_images.extend`, `self._prepare_images`, `all_sources.extend`; returns `MLLMOutput(text=output_text, finish_reason='stop', prompt_tokens=prompt_tokens, completion_tokens=generation_tokens, **…`. Generate text from multimodal input. Args: prompt: Text prompt/question images: List of image URLs or base64 strings videos: List of video inputs (URLs, base64, or OpenAI format dicts) audio: List of audio file paths max_tokens: Maximum tokens to generate temperature: Sampling temperature top_p: Top-p sampling parameter video_fps: FPS for video frame extraction (default: 2.0) video_max_frames: Max frames to extract from video use_cache: Whether to use KV cache (default: True) **kwargs: Additional generation parameters Returns: MLLMOutput with generated text Example: # With local video output = model.generate("Describe this video", videos=["video.mp4"]) # With video URL output = model.generate("What happens?", videos=["https://example.com/video.mp4"]) # With base64 video output = model.generate("Describe", videos=["data:video/mp4;base64,AAAA..."]) - Inputs: - `prompt` (str; required): Text prompt/question - `images` (list | None; optional; default `None`): List of image URLs or base64 strings - `videos` (list | None; optional; default `None`): List of video inputs (URLs, base64, or OpenAI format dicts) - `audio` (list[str] | None; optional; default `None`): List of audio file paths - `max_tokens` (int; optional; default `256`): Maximum tokens to generate - `temperature` (float; optional; default `0.7`): Sampling temperature - `top_p` (float; optional; default `0.9`): Top-p sampling parameter - `video_fps` (float; optional; default `DEFAULT_FPS`): FPS for video frame extraction (default: 2.0) - `video_max_frames` (int; optional; default `MAX_FRAMES`): Max frames to extract from video - `use_cache` (bool; optional; default `True`): Whether to use KV cache (default: True) - `**kwargs` (not annotated; optional): Additional generation parameters - Return annotation: `MLLMOutput` - Calls: self.load, all_images.extend, self._prepare_images, all_sources.extend, self._prepare_video, isinstance, str, all_sources.append, logger.info, len, all_audio.extend, self._prepare_audio, hasattr, apply_chat_template, self._cache_manager.fetch_cache, vlm_cache.make_prompt_cache, self._reset_draft_metrics, generate, self._draft_generation_kwargs, self._draft_metrics_since, getattr, self._cache_manager.store_cache, logger.debug, MLLMOutput - State reads: self._loaded, self.load, self._prepare_images, self._prepare_video, self._prepare_audio, self.processor, self.config, self._cache_manager, self._cache_manager.fetch_cache, self.model, self.model.language_model, self.max_kv_size, self._reset_draft_metrics, self._draft_generation_kwargs, self._draft_metrics_since, self._cache_manager.store_cache - Return expressions: MLLMOutput(text=output_text, finish_reason='stop', prompt_tokens=prompt_tokens, completion_tokens=generation_tokens, **… ## `vllm_mlx.models.mllm.MLXMultimodalLM.stream_generate` - Kind: method - Signature: `def stream_generate(self, prompt: str, images: list | None=None, videos: list[str] | None=None, audio: list[str] | None=None, max_tokens: int=256, temperature: float=0.7, video_fps: float=DEFAULT_FPS, **kwargs) -> Iterator[str]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/models/mllm.py#L2004-L2093 - Implementation: Method `MLXMultimodalLM.stream_generate` calls `self.load`, `self.generate`, `all_images.extend`, `self._prepare_images`; yields values incrementally; returns `None`. Stream text generation for multimodal input. Args: prompt: Text prompt images: List of image inputs videos: List of video paths audio: List of audio inputs max_tokens: Maximum tokens to generate temperature: Sampling temperature video_fps: FPS for video frame extraction **kwargs: Additional parameters Yields: Generated text chunks - Inputs: - `prompt` (str; required): Text prompt - `images` (list | None; optional; default `None`): List of image inputs - `videos` (list[str] | None; optional; default `None`): List of video paths - `audio` (list[str] | None; optional; default `None`): List of audio inputs - `max_tokens` (int; optional; default `256`): Maximum tokens to generate - `temperature` (float; optional; default `0.7`): Sampling temperature - `video_fps` (float; optional; default `DEFAULT_FPS`): FPS for video frame extraction - `**kwargs` (not annotated; optional): Additional parameters - Return annotation: `Iterator[str]` - Calls: self.load, self.generate, all_images.extend, self._prepare_images, self._prepare_video, all_audio.extend, self._prepare_audio, apply_chat_template, len, stream_generate, self._draft_generation_kwargs - State reads: self._loaded, self.load, self.generate, self._prepare_images, self._prepare_video, self._prepare_audio, self.processor, self.config, self.model, self._draft_generation_kwargs - Return expressions: None ## `vllm_mlx.models.mllm.MLXMultimodalLM.chat` - Kind: method - Signature: `def chat(self, messages: list[dict], max_tokens: int=256, temperature: float=0.7, **kwargs) -> MLLMOutput` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/models/mllm.py#L2095-L2487 - Implementation: Method `MLXMultimodalLM.chat` calls `self.load`, `logger.info`, `len`, `kwargs.pop`; has 2 explicit return paths. Chat with OpenAI-compatible message format. Supports multimodal content in messages: - {"type": "text", "text": "..."} - {"type": "image_url", "image_url": {"url": "..."}} - {"type": "image_url", "image_url": {"url": "data:image/...;base64,..."}} Args: messages: List of chat messages (OpenAI format) max_tokens: Maximum tokens to generate temperature: Sampling temperature **kwargs: Additional parameters Returns: MLLMOutput with assistant's response - Inputs: - `messages` (list[dict]; required): List of chat messages (OpenAI format) - `max_tokens` (int; optional; default `256`): Maximum tokens to generate - `temperature` (float; optional; default `0.7`): Sampling temperature - `**kwargs` (not annotated; optional): Additional parameters - Return annotation: `MLLMOutput` - Calls: self.load, logger.info, len, kwargs.pop, chat_template_kwargs.pop, self._collect_video_inputs, self._collect_audio_inputs, self._generate_native_video, _msg_video_inputs.items, bool, _msg_audio_inputs.get, process_video_input, logger.warning, extract_audio_from_video, _msg_extra_audio.setdefault(msg_idx, []).append, _msg_extra_audio.setdefault, self._prepare_video, all_video_frames.extend, _msg_extra_audio.items, _msg_audio_inputs.setdefault(msg_idx, []).extend, _msg_audio_inputs.setdefault, _msg_audio_inputs.values, all_audio_inputs.extend, _build_mllm_chat_messages, all_images.extend, self._prepare_images, self._prepare_audio, enumerate, str, cm.get, template_extra_kwargs.update, get_chat_template, template_extra_kwargs.pop, reversed, m.get, isinstance, item.get, hasattr, tokenizer.encode, self._cache_manager.fetch, time.time, copy.copy, mx.array, prompt_cache.append, vlm_cache.make_prompt_cache, self._reset_draft_metrics, generate, self._draft_generation_kwargs, self._draft_metrics_since, getattr, min, cache_to_store.append, self._cache_manager.store, MLLMOutput - State reads: self._loaded, self.load, self._collect_video_inputs, self._collect_audio_inputs, self._video_native, self._generate_native_video, self._video_native_with_audio, self._prepare_video, self._prepare_images, self._prepare_audio, self.processor, self.processor.tokenizer, self._cache_manager, self._cache_manager.fetch, self.model, self.model.language_model, self.max_kv_size, self._reset_draft_metrics, self._draft_generation_kwargs, self._draft_metrics_since, self._cache_manager.store, self.model_name - Return expressions: self._generate_native_video(messages=messages, max_tokens=max_tokens, temperature=temperature, video_fps=video_fps, vid…; MLLMOutput(text=output_text, finish_reason='stop', prompt_tokens=prompt_tokens, completion_tokens=generation_tokens, **… ## `vllm_mlx.models.mllm.MLXMultimodalLM.stream_chat` - Kind: method - Signature: `def stream_chat(self, messages: list[dict], max_tokens: int=256, temperature: float=0.7, **kwargs) -> Iterator[MLLMOutput]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/models/mllm.py#L2489-L2737 - Implementation: Method `MLXMultimodalLM.stream_chat` calls `self.load`, `self.chat`, `kwargs.pop`, `chat_template_kwargs.pop`; yields values incrementally; returns `None`. Stream chat with OpenAI-compatible message format. Supports multimodal content in messages: - {"type": "text", "text": "..."} - {"type": "image_url", "image_url": {"url": "..."}} - {"type": "image_url", "image_url": {"url": "data:image/...;base64,..."}} Args: messages: List of chat messages (OpenAI format) max_tokens: Maximum tokens to generate temperature: Sampling temperature **kwargs: Additional parameters Yields: MLLMOutput with incremental text chunks - Inputs: - `messages` (list[dict]; required): List of chat messages (OpenAI format) - `max_tokens` (int; optional; default `256`): Maximum tokens to generate - `temperature` (float; optional; default `0.7`): Sampling temperature - `**kwargs` (not annotated; optional): Additional parameters - Return annotation: `Iterator[MLLMOutput]` - Calls: self.load, self.chat, kwargs.pop, chat_template_kwargs.pop, self._collect_video_inputs, self._collect_audio_inputs, self._generate_native_video, _msg_video_inputs.items, bool, _msg_audio_inputs.get, process_video_input, logger.warning, extract_audio_from_video, _msg_extra_audio.setdefault(msg_idx, []).append, _msg_extra_audio.setdefault, self._prepare_video, all_video_frames.extend, len, logger.info, _msg_extra_audio.items, _msg_audio_inputs.setdefault(msg_idx, []).extend, _msg_audio_inputs.setdefault, _msg_audio_inputs.values, all_audio_inputs.extend, _build_mllm_chat_messages, all_images.extend, self._prepare_images, self._prepare_audio, template_extra_kwargs.update, get_chat_template, template_extra_kwargs.pop, reversed, m.get, isinstance, item.get, self._cache_manager.fetch_cache, logger.debug, vlm_cache.make_prompt_cache, self._reset_draft_metrics, stream_generate, self._draft_generation_kwargs, hasattr, str, MLLMOutput, getattr, dir, self._draft_metrics_since - State reads: self._loaded, self.load, self.chat, self._collect_video_inputs, self._collect_audio_inputs, self._video_native, self._generate_native_video, self._video_native_with_audio, self._prepare_video, self._prepare_images, self._prepare_audio, self.processor, self._cache_manager, self._cache_manager.fetch_cache, self.model, self.model.language_model, self.max_kv_size, self._reset_draft_metrics, self._draft_generation_kwargs, self._draft_metrics_since - Return expressions: None ## `vllm_mlx.models.mllm.MLXMultimodalLM.describe_image` - Kind: method - Signature: `def describe_image(self, image: str, prompt: str='Describe this image in detail.', max_tokens: int=512, **kwargs) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/models/mllm.py#L2739-L2764 - Implementation: Method `MLXMultimodalLM.describe_image` calls `self.generate`; returns `output.text`. Convenience method to describe an image. Args: image: Image path, URL, or base64 string prompt: Description prompt max_tokens: Maximum tokens **kwargs: Additional parameters Returns: Image description text - Inputs: - `image` (str; required): Image path, URL, or base64 string - `prompt` (str; optional; default `'Describe this image in detail.'`): Description prompt - `max_tokens` (int; optional; default `512`): Maximum tokens - `**kwargs` (not annotated; optional): Additional parameters - Return annotation: `str` - Calls: self.generate - State reads: self.generate - Return expressions: output.text ## `vllm_mlx.models.mllm.MLXMultimodalLM.answer_about_image` - Kind: method - Signature: `def answer_about_image(self, image: str, question: str, max_tokens: int=256, **kwargs) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/models/mllm.py#L2766-L2791 - Implementation: Method `MLXMultimodalLM.answer_about_image` calls `self.generate`; returns `output.text`. Answer a question about an image. Args: image: Image path, URL, or base64 string question: Question about the image max_tokens: Maximum tokens **kwargs: Additional parameters Returns: Answer text - Inputs: - `image` (str; required): Image path, URL, or base64 string - `question` (str; required): Question about the image - `max_tokens` (int; optional; default `256`): Maximum tokens - `**kwargs` (not annotated; optional): Additional parameters - Return annotation: `str` - Calls: self.generate - State reads: self.generate - Return expressions: output.text ## `vllm_mlx.models.mllm.MLXMultimodalLM.describe_video` - Kind: method - Signature: `def describe_video(self, video: str | dict, prompt: str='Describe what happens in this video.', fps: float=2.0, max_frames: int=32, max_tokens: int=512, **kwargs) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/models/mllm.py#L2793-L2830 - Implementation: Method `MLXMultimodalLM.describe_video` calls `self.generate`; returns `output.text`. Describe a video using frame extraction. Args: video: Video file path, URL, base64, or OpenAI format dict prompt: Description prompt fps: Frames per second to extract max_frames: Maximum frames to extract max_tokens: Maximum tokens to generate Returns: Video description text Example: # URL model.describe_video("https://example.com/video.mp4") # OpenAI format model.describe_video({"url": "https://example.com/video.mp4"}) - Inputs: - `video` (str | dict; required): Video file path, URL, base64, or OpenAI format dict - `prompt` (str; optional; default `'Describe what happens in this video.'`): Description prompt - `fps` (float; optional; default `2.0`): Frames per second to extract - `max_frames` (int; optional; default `32`): Maximum frames to extract - `max_tokens` (int; optional; default `512`): Maximum tokens to generate - `**kwargs` (not annotated; optional): Additional variadic keyword inputs accepted by this callable. - Return annotation: `str` - Calls: self.generate - State reads: self.generate - Return expressions: output.text ## `vllm_mlx.models.mllm.MLXMultimodalLM.get_cache_stats` - Kind: method - Signature: `def get_cache_stats(self) -> dict` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/models/mllm.py#L2832-L2846 - Implementation: Method `MLXMultimodalLM.get_cache_stats` calls `self._cache_manager.get_stats`, `len`; has 2 explicit return paths. Get MLLM cache statistics. Returns: Dictionary with cache stats (hits, misses, hit_rate, tokens_saved, etc.) - Inputs: none - Return annotation: `dict` - Calls: self._cache_manager.get_stats, len - State reads: self._cache_manager, self._cache_manager.get_stats, self._cache_manager.max_size - Return expressions: {'enabled': False}; stats ## `vllm_mlx.models.mllm.MLXMultimodalLM.clear_cache` - Kind: method - Signature: `def clear_cache(self) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/models/mllm.py#L2848-L2852 - Implementation: Method `MLXMultimodalLM.clear_cache` calls `self._cache_manager.clear`, `logger.info`. Clear the MLLM KV cache. - Inputs: none - Return annotation: `None` - Calls: self._cache_manager.clear, logger.info - State reads: self._cache_manager, self._cache_manager.clear ## `vllm_mlx.models.mllm.MLXMultimodalLM.get_model_info` - Kind: method - Signature: `def get_model_info(self) -> dict` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/models/mllm.py#L2854-L2874 - Implementation: Method `MLXMultimodalLM.get_model_info` calls `getattr`, `self._cache_manager.get_stats`; has 2 explicit return paths. Get information about the loaded model. - Inputs: none - Return annotation: `dict` - Calls: getattr, self._cache_manager.get_stats - State reads: self._loaded, self.model_name, self.enable_cache, self.config, self._cache_manager, self._cache_manager.get_stats - Return expressions: {'loaded': False, 'model_name': self.model_name}; info ## `vllm_mlx.models.mllm.MLXMultimodalLM.list_supported_model_families` - Kind: method - Signature: `def list_supported_model_families() -> dict[str, str]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/models/mllm.py#L2877-L2897 - Implementation: Method `MLXMultimodalLM.list_supported_model_families` returns `{'Qwen-VL': 'Qwen VL models (Qwen2-VL, Qwen2.5-VL, Qwen3-VL, etc.)', 'LLaVA': 'LLaVA vision-language models', 'Idefics'…`. List supported model families and their patterns. Any model on HuggingFace containing these patterns in the name is likely compatible with mlx-vlm. - Inputs: none - Return annotation: `dict[str, str]` - Decorators: staticmethod - Return expressions: {'Qwen-VL': 'Qwen VL models (Qwen2-VL, Qwen2.5-VL, Qwen3-VL, etc.)', 'LLaVA': 'LLaVA vision-language models', 'Idefics'… ## `vllm_mlx.models.mllm.MLXMultimodalLM.is_mllm_model` - Kind: method - Signature: `def is_mllm_model(model_name: str) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/models/mllm.py#L2900-L2934 - Implementation: Method `MLXMultimodalLM.is_mllm_model` calls `model_name.lower`, `any`, `pattern.lower`; returns `any((pattern.lower() in model_lower for pattern in mllm_patterns))`. Check if a model name indicates an MLLM model. - Inputs: - `model_name` (str; required): Required positional or keyword input. - Return annotation: `bool` - Decorators: staticmethod - Calls: model_name.lower, any, pattern.lower - Return expressions: any((pattern.lower() in model_lower for pattern in mllm_patterns)) ## `vllm_mlx.models.mllm.MLXMultimodalLM.__repr__` - Kind: method - Signature: `def __repr__(self) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/models/mllm.py#L2936-L2938 - Implementation: Method `MLXMultimodalLM.__repr__` returns `f''`. Method `MLXMultimodalLM.__repr__` returns `f''`. - Inputs: none - Return annotation: `str` - State reads: self._loaded, self.model_name - Return expressions: f'' # Module `vllm_mlx.multimodal_processor` Multimodal processor for VLM continuous batching. This module handles preprocessing of multimodal inputs (images, videos) for use with the continuous batching scheduler. It extracts processed inputs that can be batched together efficiently. Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/multimodal_processor.py#L1-L431 ## `vllm_mlx.multimodal_processor.ProcessedMultimodalInput` - Kind: class - Signature: `class ProcessedMultimodalInput` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/multimodal_processor.py#L29-L49 - Implementation: Class `ProcessedMultimodalInput` declares 0 direct member(s). Container for processed multimodal inputs ready for batching. Attributes: input_ids: Tokenized text with image/video tokens (mx.array) pixel_values: Processed image tensors (mx.array) attention_mask: Attention mask for the input (mx.array) image_grid_thw: Grid info for Qwen-VL models (mx.array) num_images: Number of images in this input num_tokens: Number of tokens in input_ids extra_kwargs: Additional model-specific kwargs - Inputs: - `input_ids` (mx.array; required): Required constructor field. - `pixel_values` (Optional[mx.array]; optional; default `None`): Optional constructor field; defaults to `None`. - `attention_mask` (Optional[mx.array]; optional; default `None`): Optional constructor field; defaults to `None`. - `image_grid_thw` (Optional[mx.array]; optional; default `None`): Optional constructor field; defaults to `None`. - `num_images` (int; optional; default `0`): Optional constructor field; defaults to `0`. - `num_tokens` (int; optional; default `0`): Optional constructor field; defaults to `0`. - `extra_kwargs` (Dict[str, Any]; optional; default `field(default_factory=dict)`): Optional constructor field; defaults to `field(default_factory=dict)`. - Constructs: `vllm_mlx.multimodal_processor.ProcessedMultimodalInput` - Decorators: dataclass ## `vllm_mlx.multimodal_processor.MultimodalProcessor` - Kind: class - Signature: `class MultimodalProcessor` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/multimodal_processor.py#L52-L431 - Implementation: Class `MultimodalProcessor` declares 8 direct member(s). Processor for preparing multimodal inputs for VLM batching. This class wraps mlx_vlm's prepare_inputs function and provides a clean interface for the scheduler to preprocess requests. Example: >>> processor = MultimodalProcessor(model, vlm_processor) >>> processed = processor.process( ... prompt="What's in this image?", ... images=["photo.jpg"] ... ) >>> # processed.input_ids, processed.pixel_values ready for batching - Inputs: - `model` (Any; required): The VLM model (for config access) - `processor` (Any; required): The VLM processor (tokenizer + image processor) - `config` (Optional[Any]; optional; default `None`): Optional model config - Constructs: `vllm_mlx.multimodal_processor.MultimodalProcessor` ## `vllm_mlx.multimodal_processor.MultimodalProcessor.__init__` - Kind: method - Signature: `def __init__(self, model: Any, processor: Any, config: Optional[Any]=None)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/multimodal_processor.py#L68-L94 - Implementation: Method `MultimodalProcessor.__init__` updates `self.model`, `self.processor`, `self.config`, `self.tokenizer`; calls `getattr`, `hasattr`. Initialize the multimodal processor. Args: model: The VLM model (for config access) processor: The VLM processor (tokenizer + image processor) config: Optional model config - Inputs: - `model` (Any; required): The VLM model (for config access) - `processor` (Any; required): The VLM processor (tokenizer + image processor) - `config` (Optional[Any]; optional; default `None`): Optional model config - Return annotation: `not annotated` - Calls: getattr, hasattr - State reads: self.config - State writes: self.model, self.processor, self.config, self.tokenizer, self.image_token_index ## `vllm_mlx.multimodal_processor.MultimodalProcessor.process` - Kind: method - Signature: `def process(self, prompt: str, images: Optional[List[str]]=None, videos: Optional[List[str]]=None, video_fps: float=DEFAULT_FPS, video_max_frames: int=MAX_FRAMES, add_special_tokens: bool=True, **kwargs) -> ProcessedMultimodalInput` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/multimodal_processor.py#L96-L186 - Implementation: Method `MultimodalProcessor.process` calls `process_image_input`, `all_images.append`, `logger.warning`, `process_video_input`; returns `ProcessedMultimodalInput(input_ids=input_ids, pixel_values=pixel_values, attention_mask=attention_mask, image_grid_thw=…`. Process multimodal inputs for batching. Args: prompt: Text prompt (already formatted with chat template) images: List of image URLs or base64 strings videos: List of video URLs or base64 inputs video_fps: FPS for video frame extraction video_max_frames: Max frames per video add_special_tokens: Whether to add special tokens **kwargs: Additional model-specific parameters Returns: ProcessedMultimodalInput with all processed tensors - Inputs: - `prompt` (str; required): Text prompt (already formatted with chat template) - `images` (Optional[List[str]]; optional; default `None`): List of image URLs or base64 strings - `videos` (Optional[List[str]]; optional; default `None`): List of video URLs or base64 inputs - `video_fps` (float; optional; default `DEFAULT_FPS`): FPS for video frame extraction - `video_max_frames` (int; optional; default `MAX_FRAMES`): Max frames per video - `add_special_tokens` (bool; optional; default `True`): Whether to add special tokens - `**kwargs` (not annotated; optional): Additional model-specific parameters - Return annotation: `ProcessedMultimodalInput` - Calls: process_image_input, all_images.append, logger.warning, process_video_input, extract_video_frames_smart, save_frames_to_temp, all_images.extend, logger.debug, len, hasattr, prepare_inputs, inputs.get, inputs.items, extra_kwargs.pop, ProcessedMultimodalInput - State reads: self.config, self.config.model_type, self.processor, self.image_token_index - Return expressions: ProcessedMultimodalInput(input_ids=input_ids, pixel_values=pixel_values, attention_mask=attention_mask, image_grid_thw=… ## `vllm_mlx.multimodal_processor.MultimodalProcessor.process_for_request` - Kind: method - Signature: `def process_for_request(self, prompt: str, images: Optional[List[str]]=None, videos: Optional[List[str]]=None, **kwargs) -> Dict[str, Any]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/multimodal_processor.py#L188-L224 - Implementation: Method `MultimodalProcessor.process_for_request` calls `self.process`, `processed.input_ids.tolist`; returns `{'prompt_token_ids': processed.input_ids.tolist() if processed.input_ids is not None else None, 'num_prompt_tokens': pr…`. Process inputs and return a dict suitable for Request fields. This is a convenience method that returns the processed data in a format that can be directly assigned to Request fields. Args: prompt: Text prompt images: List of image inputs videos: List of video inputs **kwargs: Additional parameters Returns: Dict with keys matching Request multimodal fields - Inputs: - `prompt` (str; required): Text prompt - `images` (Optional[List[str]]; optional; default `None`): List of image inputs - `videos` (Optional[List[str]]; optional; default `None`): List of video inputs - `**kwargs` (not annotated; optional): Additional parameters - Return annotation: `Dict[str, Any]` - Calls: self.process, processed.input_ids.tolist - State reads: self.process - Return expressions: {'prompt_token_ids': processed.input_ids.tolist() if processed.input_ids is not None else None, 'num_prompt_tokens': pr… ## `vllm_mlx.multimodal_processor.MultimodalProcessor.batch_pixel_values` - Kind: method - Signature: `def batch_pixel_values(self, pixel_values_list: List[Optional[mx.array]]) -> Optional[mx.array]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/multimodal_processor.py#L226-L255 - Implementation: Method `MultimodalProcessor.batch_pixel_values` calls `mx.concatenate`, `logger.warning`; has 3 explicit return paths. Batch multiple pixel_values tensors together. For VLM batching, we need to concatenate pixel values from multiple requests. This handles the case where some requests may not have images. Args: pixel_values_list: List of pixel_values from multiple requests Returns: Batched pixel_values or None if no images - Inputs: - `pixel_values_list` (List[Optional[mx.array]]; required): List of pixel_values from multiple requests - Return annotation: `Optional[mx.array]` - Calls: mx.concatenate, logger.warning - Return expressions: None; mx.concatenate(valid_pixels, axis=0); valid_pixels[0] if valid_pixels else None ## `vllm_mlx.multimodal_processor.MultimodalProcessor.batch_image_grid_thw` - Kind: method - Signature: `def batch_image_grid_thw(self, grid_thw_list: List[Optional[mx.array]]) -> Optional[mx.array]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/multimodal_processor.py#L257-L279 - Implementation: Method `MultimodalProcessor.batch_image_grid_thw` calls `mx.concatenate`, `logger.warning`; has 3 explicit return paths. Batch multiple image_grid_thw tensors together. Args: grid_thw_list: List of image_grid_thw from multiple requests Returns: Batched image_grid_thw or None - Inputs: - `grid_thw_list` (List[Optional[mx.array]]; required): List of image_grid_thw from multiple requests - Return annotation: `Optional[mx.array]` - Calls: mx.concatenate, logger.warning - Return expressions: None; mx.concatenate(valid_grids, axis=0); valid_grids[0] if valid_grids else None ## `vllm_mlx.multimodal_processor.MultimodalProcessor.prepare_for_batch` - Kind: method - Signature: `def prepare_for_batch(self, processed_inputs: List[ProcessedMultimodalInput]) -> Tuple[mx.array, Dict[str, Any], List[int]]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/multimodal_processor.py#L281-L366 - Implementation: Method `MultimodalProcessor.prepare_for_batch` calls `mx.array`, `max`, `zip`, `padded_ids.append`; has 2 explicit return paths. Prepare multiple processed inputs for batch generation. This method takes a list of ProcessedMultimodalInput objects and combines them into batched tensors suitable for the MLLMBatchGenerator. Args: processed_inputs: List of ProcessedMultimodalInput from process() Returns: Tuple of: - input_ids: Left-padded input tokens [batch_size, max_seq_len] - batch_kwargs: Dict with batched pixel_values, attention_mask, etc. - padding_amounts: List of padding amounts for each request - Inputs: - `processed_inputs` (List[ProcessedMultimodalInput]; required): List of ProcessedMultimodalInput from process() - Return annotation: `Tuple[mx.array, Dict[str, Any], List[int]]` - Calls: mx.array, max, zip, padded_ids.append, hasattr, ids.tolist, list, self.batch_pixel_values, self.batch_image_grid_thw, padded_masks.append, mx.ones, mask.reshape, mx.zeros, mx.concatenate, mx.stack, logger.warning, merged_extra.update, batch_kwargs.items - State reads: self.batch_pixel_values, self.batch_image_grid_thw - Return expressions: (mx.array([]), {}, []); (input_ids, batch_kwargs, padding_amounts) ## `vllm_mlx.multimodal_processor.MultimodalProcessor.extract_vision_embeddings` - Kind: method - Signature: `def extract_vision_embeddings(self, pixel_values: mx.array, image_grid_thw: Optional[mx.array]=None) -> mx.array` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/multimodal_processor.py#L368-L409 - Implementation: Method `MultimodalProcessor.extract_vision_embeddings` calls `hasattr`, `ValueError`, `getattr`, `vision_encoder`; can raise `ValueError`; returns `embeddings`. Extract vision embeddings from pixel values. This runs the vision encoder part of the VLM to get embeddings that can be cached and reused. Args: pixel_values: Processed image tensors image_grid_thw: Optional grid info for Qwen-VL models Returns: Vision embeddings tensor - Inputs: - `pixel_values` (mx.array; required): Processed image tensors - `image_grid_thw` (Optional[mx.array]; optional; default `None`): Optional grid info for Qwen-VL models - Return annotation: `mx.array` - Calls: hasattr, ValueError, getattr, vision_encoder - State reads: self.model - Raises directly: ValueError - Return expressions: embeddings ## `vllm_mlx.multimodal_processor.MultimodalProcessor.compute_vision_hash` - Kind: method - Signature: `def compute_vision_hash(self, pixel_values: mx.array) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/multimodal_processor.py#L411-L431 - Implementation: Method `MultimodalProcessor.compute_vision_hash` calls `str`, `pixel_values.reshape(-1)[:100].tolist`, `pixel_values.reshape`, `hashlib.sha256(hash_input.encode()).hexdigest`; returns `hashlib.sha256(hash_input.encode()).hexdigest()[:16]`. Compute a hash for pixel values for caching purposes. Args: pixel_values: Processed image tensors Returns: Hash string for the vision inputs - Inputs: - `pixel_values` (mx.array; required): Processed image tensors - Return annotation: `str` - Calls: str, pixel_values.reshape(-1)[:100].tolist, pixel_values.reshape, hashlib.sha256(hash_input.encode()).hexdigest, hashlib.sha256, hash_input.encode - Return expressions: hashlib.sha256(hash_input.encode()).hexdigest()[:16] # Module `vllm_mlx.optimizations` Hardware detection and system information for vllm-mlx. This module provides: - Hardware detection for Apple Silicon (M1, M2, M3, M4 series) - System memory detection - Memory bandwidth benchmarking Note: mlx-lm already includes optimized implementations internally: - Flash Attention via mx.fast.scaled_dot_product_attention - Efficient memory management - Optimized Metal kernels No additional optimization is needed - mlx-lm is already fast out of the box. Usage: from vllm_mlx.optimizations import ( detect_hardware, get_optimization_status, benchmark_memory_bandwidth, ) Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/optimizations.py#L1-L209 ## `vllm_mlx.optimizations.HardwareInfo` - Kind: class - Signature: `class HardwareInfo` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/optimizations.py#L34-L40 - Implementation: Class `HardwareInfo` declares 0 direct member(s). Hardware information for Apple Silicon. - Inputs: - `chip_name` (str; required): Required constructor field. - `total_memory_gb` (float; required): Required constructor field. - `memory_bandwidth_gbs` (float; required): Required constructor field. - `gpu_cores` (int; required): Required constructor field. - Constructs: `vllm_mlx.optimizations.HardwareInfo` - Decorators: dataclass ## `vllm_mlx.optimizations.get_system_memory_gb` - Kind: function - Signature: `def get_system_memory_gb() -> float` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/optimizations.py#L68-L94 - Implementation: Function `get_system_memory_gb` calls `subprocess.run`, `int`, `result.stdout.strip`, `mx.device_info`; has 3 explicit return paths. Get actual system memory in GB. Returns: Total system memory in GB (unified memory on Apple Silicon) - Inputs: none - Return annotation: `float` - Calls: subprocess.run, int, result.stdout.strip, mx.device_info - Return expressions: mem_bytes / 1024 ** 3; device_info['memory_size'] / 1024 ** 3; 16.0 ## `vllm_mlx.optimizations.detect_hardware` - Kind: function - Signature: `def detect_hardware() -> HardwareInfo` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/optimizations.py#L97-L141 - Implementation: Function `detect_hardware` calls `mx.device_info`, `device_info.get`, `get_system_memory_gb`, `sorted`; has 3 explicit return paths. Detect Apple Silicon hardware and return info. Memory is detected dynamically from the system. Other specs (bandwidth, GPU cores) come from known chip profiles. Returns: HardwareInfo with detected hardware specifications - Inputs: none - Return annotation: `HardwareInfo` - Calls: mx.device_info, device_info.get, get_system_memory_gb, sorted, HARDWARE_PROFILES.items, HardwareInfo, logger.warning - Return expressions: HardwareInfo(chip_name=chip_name, total_memory_gb=actual_memory_gb, memory_bandwidth_gbs=profile['bandwidth'], gpu_core…; HardwareInfo(chip_name='Unknown', total_memory_gb=actual_memory_gb, memory_bandwidth_gbs=200, gpu_cores=16); HardwareInfo(chip_name='Unknown', total_memory_gb=get_system_memory_gb(), memory_bandwidth_gbs=200, gpu_cores=16) ## `vllm_mlx.optimizations.benchmark_memory_bandwidth` - Kind: function - Signature: `def benchmark_memory_bandwidth() -> dict` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/optimizations.py#L144-L174 - Implementation: Function `benchmark_memory_bandwidth` calls `mx.random.normal`, `mx.eval`, `time.perf_counter`, `range`; returns `results`. Benchmark actual memory bandwidth achieved. Returns: dict with bandwidth measurements for different array sizes - Inputs: none - Return annotation: `dict` - Calls: mx.random.normal, mx.eval, time.perf_counter, range - Return expressions: results ## `vllm_mlx.optimizations.get_optimization_status` - Kind: function - Signature: `def get_optimization_status() -> dict` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/optimizations.py#L177-L209 - Implementation: Function `get_optimization_status` calls `detect_hardware`, `mx.device_info`, `hasattr`, `device_info.get`; returns `{'hardware': {'chip': hw.chip_name, 'total_memory_gb': hw.total_memory_gb, 'memory_bandwidth_gbs': hw.memory_bandwidth_…`. Get current hardware and MLX status. Returns: dict with hardware info and MLX configuration - Inputs: none - Return annotation: `dict` - Calls: detect_hardware, mx.device_info, hasattr, device_info.get, mx.get_active_memory, mx.get_cache_memory, mx.get_peak_memory - Return expressions: {'hardware': {'chip': hw.chip_name, 'total_memory_gb': hw.total_memory_gb, 'memory_bandwidth_gbs': hw.memory_bandwidth_… # Module `vllm_mlx.output_collector` Output collector for streaming with low-latency optimizations. This module implements the RequestOutputCollector pattern from vLLM, providing non-blocking output collection with intelligent aggregation. Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/output_collector.py#L1-L212 ## `vllm_mlx.output_collector.RequestOutputCollector` - Kind: class - Signature: `class RequestOutputCollector` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/output_collector.py#L17-L170 - Implementation: Class `RequestOutputCollector` declares 7 direct member(s). Per-request output collector with smart buffering. This class implements the vLLM pattern for efficient streaming: - Non-blocking get_nowait() to avoid unnecessary task switches - Output aggregation when producer is faster than consumer - Event-based signaling for efficient waiting - Tracking of active consumers for yield optimization Usage: collector = RequestOutputCollector() # Producer side (engine loop) collector.put(output) # Consumer side (streaming generator) output = collector.get_nowait() or await collector.get() - Inputs: - `aggregate` (bool; optional; default `True`): If True, merge outputs when producer gets ahead. This prevents buffer explosion under load. - Constructs: `vllm_mlx.output_collector.RequestOutputCollector` ## `vllm_mlx.output_collector.RequestOutputCollector.__init__` - Kind: method - Signature: `def __init__(self, aggregate: bool=True)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/output_collector.py#L42-L53 - Implementation: Method `RequestOutputCollector.__init__` updates `self.output`, `self.ready`, `self.aggregate`, `self._is_waiting`; calls `asyncio.Event`. Initialize the collector. Args: aggregate: If True, merge outputs when producer gets ahead. This prevents buffer explosion under load. - Inputs: - `aggregate` (bool; optional; default `True`): If True, merge outputs when producer gets ahead. This prevents buffer explosion under load. - Return annotation: `not annotated` - Calls: asyncio.Event - State writes: self.output, self.ready, self.aggregate, self._is_waiting ## `vllm_mlx.output_collector.RequestOutputCollector.put` - Kind: method - Signature: `def put(self, output: RequestOutput) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/output_collector.py#L55-L73 - Implementation: Method `RequestOutputCollector.put` updates `self.output`; calls `self._merge_outputs`, `self.ready.set`. Put an output into the collector (non-blocking). If aggregation is enabled and an output already exists, the new output is merged with the existing one. Args: output: The RequestOutput to store - Inputs: - `output` (RequestOutput; required): The RequestOutput to store - Return annotation: `None` - Calls: self._merge_outputs, self.ready.set - State reads: self.output, self.aggregate, self._merge_outputs, self.ready.set, self.ready - State writes: self.output ## `vllm_mlx.output_collector.RequestOutputCollector.get_nowait` - Kind: method - Signature: `def get_nowait(self) -> Optional[RequestOutput]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/output_collector.py#L75-L89 - Implementation: Method `RequestOutputCollector.get_nowait` updates `self.output`; calls `self.ready.clear`; returns `output`. Get output without blocking. This avoids task switching when output is available, reducing latency under load. Returns: The output if available, None otherwise - Inputs: none - Return annotation: `Optional[RequestOutput]` - Calls: self.ready.clear - State reads: self.output, self.ready.clear, self.ready - State writes: self.output - Return expressions: output ## `vllm_mlx.output_collector.RequestOutputCollector.get` - Kind: method - Signature: `async def get(self) -> RequestOutput` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/output_collector.py#L91-L118 - Implementation: Method `RequestOutputCollector.get` updates `self._is_waiting`; calls `self.ready.wait`, `self.get_nowait`; awaits asynchronous work; returns `output`. Get output, blocking only if none available. This method blocks until an output is available. For low-latency streaming, prefer: output = collector.get_nowait() or await collector.get() Returns: The RequestOutput - Inputs: none - Return annotation: `RequestOutput` - Calls: self.ready.wait, self.get_nowait - State reads: self._is_waiting, self.output, self.ready.wait, self.ready, self.get_nowait - State writes: self._is_waiting - Return expressions: output ## `vllm_mlx.output_collector.RequestOutputCollector._merge_outputs` - Kind: method - Signature: `def _merge_outputs(self, existing: RequestOutput, new: RequestOutput) -> RequestOutput` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/output_collector.py#L120-L152 - Implementation: Method `RequestOutputCollector._merge_outputs` calls `RequestOutput`; returns `RequestOutput(request_id=new.request_id, new_token_ids=merged_new_token_ids, new_text=merged_new_text, output_token_ids…`. Merge two outputs when producer gets ahead of consumer. This combines the token lists and text, keeping the latest status information. Args: existing: The existing output in the buffer new: The new output to merge Returns: Merged RequestOutput - Inputs: - `existing` (RequestOutput; required): The existing output in the buffer - `new` (RequestOutput; required): The new output to merge - Return annotation: `RequestOutput` - Calls: RequestOutput - Return expressions: RequestOutput(request_id=new.request_id, new_token_ids=merged_new_token_ids, new_text=merged_new_text, output_token_ids… ## `vllm_mlx.output_collector.RequestOutputCollector.clear` - Kind: method - Signature: `def clear(self) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/output_collector.py#L154-L161 - Implementation: Method `RequestOutputCollector.clear` updates `self.output`, `self._is_waiting`; calls `self.ready.clear`. Clear any pending output. - Inputs: none - Return annotation: `None` - Calls: self.ready.clear - State reads: self.ready.clear, self.ready, self._is_waiting - State writes: self.output, self._is_waiting ## `vllm_mlx.output_collector.RequestOutputCollector.has_waiting_consumers` - Kind: method - Signature: `def has_waiting_consumers(cls) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/output_collector.py#L164-L170 - Implementation: Method `RequestOutputCollector.has_waiting_consumers` returns `cls._waiting_consumers > 0`. Check if any collector has waiting consumers. Used by engine to optimize: only yield when someone is waiting. - Inputs: none - Return annotation: `bool` - Decorators: classmethod - State reads: cls._waiting_lock, cls._waiting_consumers - Return expressions: cls._waiting_consumers > 0 ## `vllm_mlx.output_collector.RequestStreamState` - Kind: class - Signature: `class RequestStreamState` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/output_collector.py#L174-L212 - Implementation: Class `RequestStreamState` declares 2 direct member(s). Tracks streaming state for a request. This is used to implement stream_interval batching, allowing tokens to be accumulated before sending. - Inputs: - `stream_interval` (int; optional; default `1`): Optional constructor field; defaults to `1`. - `sent_tokens` (int; optional; default `0`): Optional constructor field; defaults to `0`. - Constructs: `vllm_mlx.output_collector.RequestStreamState` - Decorators: dataclass ## `vllm_mlx.output_collector.RequestStreamState.should_send` - Kind: method - Signature: `def should_send(self, total_tokens: int, finished: bool) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/output_collector.py#L185-L203 - Implementation: Method `RequestStreamState.should_send` has 2 explicit return paths. Determine if output should be sent based on stream_interval. Args: total_tokens: Total tokens generated so far finished: Whether generation is complete Returns: True if output should be sent - Inputs: - `total_tokens` (int; required): Total tokens generated so far - `finished` (bool; required): Whether generation is complete - Return annotation: `bool` - State reads: self.sent_tokens, self.stream_interval - Return expressions: True; total_tokens - self.sent_tokens >= self.stream_interval ## `vllm_mlx.output_collector.RequestStreamState.mark_sent` - Kind: method - Signature: `def mark_sent(self, total_tokens: int) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/output_collector.py#L205-L212 - Implementation: Method `RequestStreamState.mark_sent` updates `self.sent_tokens`. Update state after sending output. Args: total_tokens: Total tokens at time of send - Inputs: - `total_tokens` (int; required): Total tokens at time of send - Return annotation: `None` - State writes: self.sent_tokens # Module `vllm_mlx.paged_cache` Paged KV Cache Manager for vllm-mlx. This module implements block-based paged KV cache management following vLLM's architecture (vllm/v1/core/block_pool.py), adapted for MLX on Apple Silicon. Key components: - KVCacheBlock: Metadata for each cache block with doubly linked list pointers - FreeKVCacheBlockQueue: O(1) doubly linked list for LRU block allocation - BlockHashToBlockMap: Hash-to-block cache for prefix caching - PagedCacheManager: Main manager with block allocation, prefix caching, and COW Features: - Block-based allocation (configurable tokens per block) - Reference counting for shared blocks - Copy-on-Write (COW) for efficient prefix sharing - LRU eviction using doubly linked list (O(1) operations) - Chain hashing for prefix caching (hash depends on parent block) Reference: vLLM v1 - vllm/v1/core/block_pool.py, vllm/v1/core/kv_cache_utils.py Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/paged_cache.py#L1-L1197 ## `vllm_mlx.paged_cache.compute_block_hash` - Kind: function - Signature: `def compute_block_hash(parent_hash: Optional[BlockHash], token_ids: List[int], extra_keys: Optional[Tuple[Any, ...]]=None) -> BlockHash` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/paged_cache.py#L40-L75 - Implementation: Function `compute_block_hash` calls `hashlib.sha256`, `hasher.update`, `bytes`, `str`; returns `BlockHash(hasher.digest())`. Compute hash for a block based on its content and parent block. This enables prefix caching by creating a chain of hashes where each block's hash depends on all previous blocks (similar to vLLM). Args: parent_hash: Hash of the previous block, or None for first block token_ids: Token IDs in this block extra_keys: Additional keys (e.g., LoRA, multimodal) Returns: Content-based hash for this block - Inputs: - `parent_hash` (Optional[BlockHash]; required): Hash of the previous block, or None for first block - `token_ids` (List[int]; required): Token IDs in this block - `extra_keys` (Optional[Tuple[Any, ...]]; optional; default `None`): Additional keys (e.g., LoRA, multimodal) - Return annotation: `BlockHash` - Calls: hashlib.sha256, hasher.update, bytes, str, tuple, BlockHash, hasher.digest - Return expressions: BlockHash(hasher.digest()) ## `vllm_mlx.paged_cache.CacheBlock` - Kind: class - Signature: `class CacheBlock` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/paged_cache.py#L84-L146 - Implementation: Class `CacheBlock` declares 5 direct member(s). KV cache block metadata following vLLM's design. Each block represents a fixed number of tokens (block_size) worth of KV cache data. Blocks can be shared across requests via reference counting for prefix caching. Attributes: block_id: Physical block index (0 to num_blocks - 1) ref_count: Reference count for sharing (0 = can be evicted) block_hash: Content hash for prefix caching (None if not cached) prev_free_block: Previous block in free list (doubly linked) next_free_block: Next block in free list (doubly linked) is_null: True if this is the null/placeholder block cache_data: Actual KV tensor data stored in this block token_count: Number of tokens stored in this block - Inputs: - `block_id` (int; required): Required constructor field. - `ref_count` (int; optional; default `0`): Optional constructor field; defaults to `0`. - `block_hash` (Optional[BlockHash]; optional; default `None`): Optional constructor field; defaults to `None`. - `prev_free_block` (Optional['CacheBlock']; optional; default `None`): Optional constructor field; defaults to `None`. - `next_free_block` (Optional['CacheBlock']; optional; default `None`): Optional constructor field; defaults to `None`. - `is_null` (bool; optional; default `False`): Optional constructor field; defaults to `False`. - `cache_data` (Optional[List[Tuple[Any, Any]]]; optional; default `None`): Optional constructor field; defaults to `None`. - `token_count` (int; optional; default `0`): Optional constructor field; defaults to `0`. - `hash_value` (Optional[str]; optional; default `None`): Optional constructor field; defaults to `None`. - `last_access` (float; optional; default `field(default_factory=time.time)`): Optional constructor field; defaults to `field(default_factory=time.time)`. - Constructs: `vllm_mlx.paged_cache.CacheBlock` - Decorators: dataclass ## `vllm_mlx.paged_cache.CacheBlock.is_full` - Kind: method - Signature: `def is_full(self, block_size: int) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/paged_cache.py#L123-L125 - Implementation: Method `CacheBlock.is_full` returns `self.token_count >= block_size`. Check if block is at capacity. - Inputs: - `block_size` (int; required): Required positional or keyword input. - Return annotation: `bool` - State reads: self.token_count - Return expressions: self.token_count >= block_size ## `vllm_mlx.paged_cache.CacheBlock.is_shared` - Kind: method - Signature: `def is_shared(self) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/paged_cache.py#L127-L129 - Implementation: Method `CacheBlock.is_shared` returns `self.ref_count > 1`. Check if block is shared (ref_count > 1). - Inputs: none - Return annotation: `bool` - State reads: self.ref_count - Return expressions: self.ref_count > 1 ## `vllm_mlx.paged_cache.CacheBlock.reset_hash` - Kind: method - Signature: `def reset_hash(self) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/paged_cache.py#L131-L134 - Implementation: Method `CacheBlock.reset_hash` updates `self.block_hash`, `self.hash_value`. Reset block hash when evicted from cache. - Inputs: none - Return annotation: `None` - State writes: self.block_hash, self.hash_value ## `vllm_mlx.paged_cache.CacheBlock.touch` - Kind: method - Signature: `def touch(self) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/paged_cache.py#L136-L138 - Implementation: Method `CacheBlock.touch` updates `self.last_access`; calls `time.time`. Update last access time. - Inputs: none - Return annotation: `None` - Calls: time.time - State writes: self.last_access ## `vllm_mlx.paged_cache.CacheBlock.__repr__` - Kind: method - Signature: `def __repr__(self) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/paged_cache.py#L140-L146 - Implementation: Method `CacheBlock.__repr__` returns `f'CacheBlock(id={self.block_id}, ref={self.ref_count}, tokens={self.token_count}, prev={prev_id}, next={next_id})'`. Method `CacheBlock.__repr__` returns `f'CacheBlock(id={self.block_id}, ref={self.ref_count}, tokens={self.token_count}, prev={prev_id}, next={next_id})'`. - Inputs: none - Return annotation: `str` - State reads: self.prev_free_block, self.prev_free_block.block_id, self.next_free_block, self.next_free_block.block_id, self.block_id, self.ref_count, self.token_count - Return expressions: f'CacheBlock(id={self.block_id}, ref={self.ref_count}, tokens={self.token_count}, prev={prev_id}, next={next_id})' ## `vllm_mlx.paged_cache.FreeKVCacheBlockQueue` - Kind: class - Signature: `class FreeKVCacheBlockQueue` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/paged_cache.py#L158-L337 - Implementation: Class `FreeKVCacheBlockQueue` declares 7 direct member(s). Doubly linked list of free blocks following vLLM's design. Provides O(1) operations for: - popleft(): Allocate block from front (LRU order) - remove(): Remove block from middle (when touched by cache hit) - append(): Return block to end (when freed) The queue maintains LRU eviction order: - Front = least recently used (evict first) - Back = most recently used (evict last) Uses fake head/tail sentinels to simplify edge cases. - Inputs: - `blocks` (List[CacheBlock]; required): List of all CacheBlock objects - Constructs: `vllm_mlx.paged_cache.FreeKVCacheBlockQueue` ## `vllm_mlx.paged_cache.FreeKVCacheBlockQueue.__init__` - Kind: method - Signature: `def __init__(self, blocks: List[CacheBlock]) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/paged_cache.py#L174-L201 - Implementation: Method `FreeKVCacheBlockQueue.__init__` updates `self.num_free_blocks`, `self.fake_head`, `self.fake_tail`, `self.fake_head.next_free_block`; calls `len`, `range`, `CacheBlock`. Initialize queue with all blocks as free. Args: blocks: List of all CacheBlock objects - Inputs: - `blocks` (List[CacheBlock]; required): List of all CacheBlock objects - Return annotation: `None` - Calls: len, range, CacheBlock - State reads: self.fake_head, self.fake_tail - State writes: self.num_free_blocks, self.fake_head, self.fake_tail, self.fake_head.next_free_block, self.fake_tail.prev_free_block ## `vllm_mlx.paged_cache.FreeKVCacheBlockQueue.popleft` - Kind: method - Signature: `def popleft(self) -> CacheBlock` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/paged_cache.py#L203-L225 - Implementation: Method `FreeKVCacheBlockQueue.popleft` updates `self.fake_head.next_free_block`, `self.num_free_blocks`; calls `ValueError`; can raise `ValueError`; returns `block`. Pop and return the first (LRU) free block. Raises: ValueError: If no free blocks available - Inputs: none - Return annotation: `CacheBlock` - Calls: ValueError - State reads: self.fake_head.next_free_block, self.fake_head, self.fake_tail - State writes: self.fake_head.next_free_block, self.num_free_blocks - Raises directly: ValueError - Return expressions: block ## `vllm_mlx.paged_cache.FreeKVCacheBlockQueue.popleft_n` - Kind: method - Signature: `def popleft_n(self, n: int) -> List[CacheBlock]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/paged_cache.py#L227-L265 - Implementation: Method `FreeKVCacheBlockQueue.popleft_n` updates `self.fake_head.next_free_block`, `self.num_free_blocks`; calls `range`, `result.append`; has 2 explicit return paths. Pop n blocks from the front. Args: n: Number of blocks to allocate Returns: List of n free blocks Raises: AssertionError: If not enough free blocks - Inputs: - `n` (int; required): Number of blocks to allocate - Return annotation: `List[CacheBlock]` - Calls: range, result.append - State reads: self.num_free_blocks, self.fake_head.next_free_block, self.fake_head, self.fake_tail - State writes: self.fake_head.next_free_block, self.num_free_blocks - Return expressions: []; result ## `vllm_mlx.paged_cache.FreeKVCacheBlockQueue.remove` - Kind: method - Signature: `def remove(self, block: CacheBlock) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/paged_cache.py#L267-L288 - Implementation: Method `FreeKVCacheBlockQueue.remove` updates `self.num_free_blocks`; calls `RuntimeError`; can raise `RuntimeError`. Remove a block from the middle of the queue. Used when a free block is "touched" (reused by prefix cache hit). Args: block: Block to remove Raises: RuntimeError: If block not in queue - Inputs: - `block` (CacheBlock; required): Block to remove - Return annotation: `None` - Calls: RuntimeError - State writes: self.num_free_blocks - Raises directly: RuntimeError ## `vllm_mlx.paged_cache.FreeKVCacheBlockQueue.append` - Kind: method - Signature: `def append(self, block: CacheBlock) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/paged_cache.py#L290-L305 - Implementation: Method `FreeKVCacheBlockQueue.append` updates `self.fake_tail.prev_free_block`, `self.num_free_blocks`. Append a block to the end (MRU position). Args: block: Block to append - Inputs: - `block` (CacheBlock; required): Block to append - Return annotation: `None` - State reads: self.fake_tail.prev_free_block, self.fake_tail - State writes: self.fake_tail.prev_free_block, self.num_free_blocks ## `vllm_mlx.paged_cache.FreeKVCacheBlockQueue.append_n` - Kind: method - Signature: `def append_n(self, blocks: List[CacheBlock]) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/paged_cache.py#L307-L328 - Implementation: Method `FreeKVCacheBlockQueue.append_n` updates `self.fake_tail.prev_free_block`, `self.num_free_blocks`; calls `len`; returns `None`. Append multiple blocks to the end. Args: blocks: Blocks to append (in order) - Inputs: - `blocks` (List[CacheBlock]; required): Blocks to append (in order) - Return annotation: `None` - Calls: len - State reads: self.fake_tail.prev_free_block, self.fake_tail - State writes: self.fake_tail.prev_free_block, self.num_free_blocks - Return expressions: None ## `vllm_mlx.paged_cache.FreeKVCacheBlockQueue.get_all_free_blocks` - Kind: method - Signature: `def get_all_free_blocks(self) -> List[CacheBlock]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/paged_cache.py#L330-L337 - Implementation: Method `FreeKVCacheBlockQueue.get_all_free_blocks` calls `result.append`; returns `result`. Get all free blocks (for testing). - Inputs: none - Return annotation: `List[CacheBlock]` - Calls: result.append - State reads: self.fake_head.next_free_block, self.fake_head, self.fake_tail - Return expressions: result ## `vllm_mlx.paged_cache.BlockHashToBlockMap` - Kind: class - Signature: `class BlockHashToBlockMap` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/paged_cache.py#L345-L407 - Implementation: Class `BlockHashToBlockMap` declares 6 direct member(s). Cache mapping block hashes to blocks for prefix caching. Follows vLLM's design where the same hash can map to multiple blocks (for different KV cache groups in hybrid models). - Inputs: none - Constructs: `vllm_mlx.paged_cache.BlockHashToBlockMap` ## `vllm_mlx.paged_cache.BlockHashToBlockMap.__init__` - Kind: method - Signature: `def __init__(self) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/paged_cache.py#L353-L354 - Implementation: Method `BlockHashToBlockMap.__init__` updates `self._cache`. Method `BlockHashToBlockMap.__init__` updates `self._cache`. - Inputs: none - Return annotation: `None` - State writes: self._cache ## `vllm_mlx.paged_cache.BlockHashToBlockMap.get_block` - Kind: method - Signature: `def get_block(self, block_hash: BlockHash) -> Optional[CacheBlock]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/paged_cache.py#L356-L365 - Implementation: Method `BlockHashToBlockMap.get_block` calls `self._cache.get`, `isinstance`, `next`, `iter`; has 3 explicit return paths. Get any block with the given hash. - Inputs: - `block_hash` (BlockHash; required): Required positional or keyword input. - Return annotation: `Optional[CacheBlock]` - Calls: self._cache.get, isinstance, next, iter, blocks.values - State reads: self._cache.get, self._cache - Return expressions: None; blocks; next(iter(blocks.values())) ## `vllm_mlx.paged_cache.BlockHashToBlockMap.insert` - Kind: method - Signature: `def insert(self, block_hash: BlockHash, block: CacheBlock) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/paged_cache.py#L367-L378 - Implementation: Method `BlockHashToBlockMap.insert` calls `self._cache.get`, `isinstance`. Insert a block into the cache. - Inputs: - `block_hash` (BlockHash; required): Required positional or keyword input. - `block` (CacheBlock; required): Required positional or keyword input. - Return annotation: `None` - Calls: self._cache.get, isinstance - State reads: self._cache.get, self._cache ## `vllm_mlx.paged_cache.BlockHashToBlockMap.pop` - Kind: method - Signature: `def pop(self, block_hash: BlockHash, block_id: int) -> Optional[CacheBlock]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/paged_cache.py#L380-L399 - Implementation: Method `BlockHashToBlockMap.pop` calls `self._cache.pop`, `isinstance`, `blocks.pop`; has 3 explicit return paths. Remove and return a specific block from the cache. - Inputs: - `block_hash` (BlockHash; required): Required positional or keyword input. - `block_id` (int; required): Required positional or keyword input. - Return annotation: `Optional[CacheBlock]` - Calls: self._cache.pop, isinstance, blocks.pop - State reads: self._cache.pop, self._cache - Return expressions: None; blocks; block ## `vllm_mlx.paged_cache.BlockHashToBlockMap.__len__` - Kind: method - Signature: `def __len__(self) -> int` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/paged_cache.py#L401-L402 - Implementation: Method `BlockHashToBlockMap.__len__` calls `len`; returns `len(self._cache)`. Method `BlockHashToBlockMap.__len__` calls `len`; returns `len(self._cache)`. - Inputs: none - Return annotation: `int` - Calls: len - State reads: self._cache - Return expressions: len(self._cache) ## `vllm_mlx.paged_cache.BlockHashToBlockMap.clear` - Kind: method - Signature: `def clear(self) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/paged_cache.py#L404-L407 - Implementation: Method `BlockHashToBlockMap.clear` calls `self._cache.clear`. Remove every block-hash mapping without mutating the blocks. - Inputs: none - Return annotation: `None` - Calls: self._cache.clear - State reads: self._cache.clear, self._cache ## `vllm_mlx.paged_cache.BlockTable` - Kind: class - Signature: `class BlockTable` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/paged_cache.py#L416-L447 - Implementation: Class `BlockTable` declares 3 direct member(s). Per-request block table mapping logical to physical blocks. Similar to vLLM's block table, this maps a request's token positions to physical cache blocks. Attributes: request_id: Unique request identifier block_ids: List of physical block IDs num_tokens: Total number of cached tokens - Inputs: - `request_id` (str; required): Required constructor field. - `block_ids` (List[int]; optional; default `field(default_factory=list)`): Optional constructor field; defaults to `field(default_factory=list)`. - `num_tokens` (int; optional; default `0`): Optional constructor field; defaults to `0`. - Constructs: `vllm_mlx.paged_cache.BlockTable` - Decorators: dataclass ## `vllm_mlx.paged_cache.BlockTable.add_block` - Kind: method - Signature: `def add_block(self, block_id: int, num_tokens: int) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/paged_cache.py#L433-L436 - Implementation: Method `BlockTable.add_block` updates `self.num_tokens`; calls `self.block_ids.append`. Add a block to the table. - Inputs: - `block_id` (int; required): Required positional or keyword input. - `num_tokens` (int; required): Required positional or keyword input. - Return annotation: `None` - Calls: self.block_ids.append - State reads: self.block_ids.append, self.block_ids - State writes: self.num_tokens ## `vllm_mlx.paged_cache.BlockTable.__len__` - Kind: method - Signature: `def __len__(self) -> int` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/paged_cache.py#L438-L439 - Implementation: Method `BlockTable.__len__` calls `len`; returns `len(self.block_ids)`. Method `BlockTable.__len__` calls `len`; returns `len(self.block_ids)`. - Inputs: none - Return annotation: `int` - Calls: len - State reads: self.block_ids - Return expressions: len(self.block_ids) ## `vllm_mlx.paged_cache.BlockTable.copy` - Kind: method - Signature: `def copy(self, new_request_id: str) -> 'BlockTable'` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/paged_cache.py#L441-L447 - Implementation: Method `BlockTable.copy` calls `BlockTable`, `self.block_ids.copy`; returns `BlockTable(request_id=new_request_id, block_ids=self.block_ids.copy(), num_tokens=self.num_tokens)`. Create a copy with new request ID. - Inputs: - `new_request_id` (str; required): Required positional or keyword input. - Return annotation: `'BlockTable'` - Calls: BlockTable, self.block_ids.copy - State reads: self.block_ids.copy, self.block_ids, self.num_tokens - Return expressions: BlockTable(request_id=new_request_id, block_ids=self.block_ids.copy(), num_tokens=self.num_tokens) ## `vllm_mlx.paged_cache.CacheStats` - Kind: class - Signature: `class CacheStats` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/paged_cache.py#L456-L467 - Implementation: Class `CacheStats` declares 0 direct member(s). Statistics for cache monitoring. - Inputs: - `total_blocks` (int; optional; default `0`): Optional constructor field; defaults to `0`. - `allocated_blocks` (int; optional; default `0`): Optional constructor field; defaults to `0`. - `free_blocks` (int; optional; default `0`): Optional constructor field; defaults to `0`. - `shared_blocks` (int; optional; default `0`): Optional constructor field; defaults to `0`. - `total_tokens_cached` (int; optional; default `0`): Optional constructor field; defaults to `0`. - `cache_hits` (int; optional; default `0`): Optional constructor field; defaults to `0`. - `cache_misses` (int; optional; default `0`): Optional constructor field; defaults to `0`. - `cow_copies` (int; optional; default `0`): Optional constructor field; defaults to `0`. - `evictions` (int; optional; default `0`): Optional constructor field; defaults to `0`. - Constructs: `vllm_mlx.paged_cache.CacheStats` - Decorators: dataclass ## `vllm_mlx.paged_cache.PagedCacheManager` - Kind: class - Signature: `class PagedCacheManager` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/paged_cache.py#L475-L1197 - Implementation: Class `PagedCacheManager` declares 34 direct member(s). Paged KV cache manager following vLLM's BlockPool architecture. Features: - Block allocation/deallocation with reference counting - Prefix sharing via chain-based hash deduplication - Copy-on-Write for efficient forking - O(1) LRU eviction using doubly linked list Args: block_size: Number of tokens per block (default: 64) max_blocks: Maximum number of blocks to allocate (default: 1000) enable_caching: Whether to enable prefix caching (default: True) - Inputs: - `block_size` (int; optional; default `64`): Number of tokens per block (default: 64) - `max_blocks` (int; optional; default `1000`): Maximum number of blocks to allocate (default: 1000) - `enable_caching` (bool; optional; default `True`): Whether to enable prefix caching (default: True) - Constructs: `vllm_mlx.paged_cache.PagedCacheManager` ## `vllm_mlx.paged_cache.PagedCacheManager.__init__` - Kind: method - Signature: `def __init__(self, block_size: int=64, max_blocks: int=1000, enable_caching: bool=True)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/paged_cache.py#L491-L540 - Implementation: Method `PagedCacheManager.__init__` updates `self.block_size`, `self.max_blocks`, `self.enable_caching`, `self.blocks`; calls `CacheBlock`, `range`, `FreeKVCacheBlockQueue`, `BlockHashToBlockMap`. Method `PagedCacheManager.__init__` updates `self.block_size`, `self.max_blocks`, `self.enable_caching`, `self.blocks`; calls `CacheBlock`, `range`, `FreeKVCacheBlockQueue`, `BlockHashToBlockMap`. - Inputs: - `block_size` (int; optional; default `64`): Optional positional or keyword input; defaults to `64`. - `max_blocks` (int; optional; default `1000`): Optional positional or keyword input; defaults to `1000`. - `enable_caching` (bool; optional; default `True`): Optional positional or keyword input; defaults to `True`. - Return annotation: `not annotated` - Calls: CacheBlock, range, FreeKVCacheBlockQueue, BlockHashToBlockMap, self.free_block_queue.popleft, CacheStats, threading.RLock, logger.info - State reads: self.blocks, self.free_block_queue.popleft, self.free_block_queue, self.null_block, self.allocated_blocks, self.null_block.block_id - State writes: self.block_size, self.max_blocks, self.enable_caching, self.blocks, self.free_block_queue, self.cached_block_hash_to_block, self.hash_to_block, self.request_tables, self.allocated_blocks, self.null_block, self.null_block.is_null, self.null_block.ref_count, self.stats, self._lock ## `vllm_mlx.paged_cache.PagedCacheManager.allocate_block` - Kind: method - Signature: `def allocate_block(self) -> Optional[CacheBlock]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/paged_cache.py#L546-L571 - Implementation: Method `PagedCacheManager.allocate_block` updates `self.stats.allocated_blocks`, `self.stats.free_blocks`; calls `logger.warning`, `self.free_block_queue.popleft`, `self._maybe_evict_cached_block`, `block.touch`; has 2 explicit return paths. Allocate a new cache block. Returns: CacheBlock if available, None if out of memory. - Inputs: none - Return annotation: `Optional[CacheBlock]` - Calls: logger.warning, self.free_block_queue.popleft, self._maybe_evict_cached_block, block.touch - State reads: self._lock, self.free_block_queue.num_free_blocks, self.free_block_queue, self.free_block_queue.popleft, self.enable_caching, self._maybe_evict_cached_block, self.allocated_blocks, self.stats - State writes: self.stats.allocated_blocks, self.stats.free_blocks - Return expressions: None; block ## `vllm_mlx.paged_cache.PagedCacheManager.get_new_blocks` - Kind: method - Signature: `def get_new_blocks(self, num_blocks: int) -> List[CacheBlock]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/paged_cache.py#L573-L606 - Implementation: Method `PagedCacheManager.get_new_blocks` updates `self.stats.allocated_blocks`, `self.stats.free_blocks`; calls `ValueError`, `self.free_block_queue.popleft_n`, `self._maybe_evict_cached_block`, `block.touch`; can raise `ValueError`; returns `blocks`. Allocate multiple blocks at once (vLLM style). Args: num_blocks: Number of blocks to allocate Returns: List of allocated blocks Raises: ValueError: If not enough free blocks - Inputs: - `num_blocks` (int; required): Number of blocks to allocate - Return annotation: `List[CacheBlock]` - Calls: ValueError, self.free_block_queue.popleft_n, self._maybe_evict_cached_block, block.touch - State reads: self._lock, self.free_block_queue.num_free_blocks, self.free_block_queue, self.free_block_queue.popleft_n, self.enable_caching, self._maybe_evict_cached_block, self.allocated_blocks, self.stats - State writes: self.stats.allocated_blocks, self.stats.free_blocks - Raises directly: ValueError - Return expressions: blocks ## `vllm_mlx.paged_cache.PagedCacheManager._maybe_evict_cached_block` - Kind: method - Signature: `def _maybe_evict_cached_block(self, block: CacheBlock) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/paged_cache.py#L608-L634 - Implementation: Method `PagedCacheManager._maybe_evict_cached_block` updates `self.stats.evictions`; calls `self.cached_block_hash_to_block.pop`, `block.reset_hash`; has 2 explicit return paths. Evict a block from the hash cache if present. Args: block: Block to evict Returns: True if block was evicted from cache - Inputs: - `block` (CacheBlock; required): Block to evict - Return annotation: `bool` - Calls: self.cached_block_hash_to_block.pop, block.reset_hash - State reads: self.cached_block_hash_to_block.pop, self.cached_block_hash_to_block, self.hash_to_block, self.stats - State writes: self.stats.evictions - Return expressions: False; True ## `vllm_mlx.paged_cache.PagedCacheManager.free_block` - Kind: method - Signature: `def free_block(self, block_id: int) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/paged_cache.py#L636-L667 - Implementation: Method `PagedCacheManager.free_block` updates `self.stats.allocated_blocks`, `self.stats.free_blocks`, `self.stats.total_tokens_cached`; calls `logger.warning`, `self.free_block_queue.append`; has 2 explicit return paths. Free a cache block (decrements ref_count, frees if 0). Returns: True if block was freed, False if still referenced. - Inputs: - `block_id` (int; required): Required positional or keyword input. - Return annotation: `bool` - Calls: logger.warning, self.free_block_queue.append - State reads: self._lock, self.allocated_blocks, self.free_block_queue.append, self.free_block_queue, self.stats - State writes: self.stats.allocated_blocks, self.stats.free_blocks, self.stats.total_tokens_cached - Return expressions: False; True ## `vllm_mlx.paged_cache.PagedCacheManager.free_blocks` - Kind: method - Signature: `def free_blocks(self, blocks: Iterable[CacheBlock]) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/paged_cache.py#L669-L696 - Implementation: Method `PagedCacheManager.free_blocks` updates `self.stats.allocated_blocks`, `self.stats.free_blocks`, `self.stats.total_tokens_cached`; calls `list`, `to_free.append`, `self.free_block_queue.append_n`. Free multiple blocks (vLLM style). Blocks with ref_count=0 are added to the free queue. Args: blocks: Blocks to free (in eviction order) - Inputs: - `blocks` (Iterable[CacheBlock]; required): Blocks to free (in eviction order) - Return annotation: `None` - Calls: list, to_free.append, self.free_block_queue.append_n - State reads: self._lock, self.allocated_blocks, self.stats, self.free_block_queue.append_n, self.free_block_queue - State writes: self.stats.allocated_blocks, self.stats.free_blocks, self.stats.total_tokens_cached ## `vllm_mlx.paged_cache.PagedCacheManager.touch` - Kind: method - Signature: `def touch(self, blocks: Iterable[CacheBlock]) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/paged_cache.py#L698-L720 - Implementation: Method `PagedCacheManager.touch` updates `self.stats.free_blocks`, `self.stats.allocated_blocks`; calls `self.free_block_queue.remove`, `block.touch`. Touch blocks to prevent eviction (cache hit, vLLM style). Increments ref_count and removes from free queue if needed. Args: blocks: Blocks to touch - Inputs: - `blocks` (Iterable[CacheBlock]; required): Blocks to touch - Return annotation: `None` - Calls: self.free_block_queue.remove, block.touch - State reads: self._lock, self.free_block_queue.remove, self.free_block_queue, self.stats, self.allocated_blocks - State writes: self.stats.free_blocks, self.stats.allocated_blocks ## `vllm_mlx.paged_cache.PagedCacheManager.increment_ref` - Kind: method - Signature: `def increment_ref(self, block_id: int) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/paged_cache.py#L726-L739 - Implementation: Method `PagedCacheManager.increment_ref` updates `self.stats.shared_blocks`; calls `block.touch`; has 2 explicit return paths. Increment reference count for a block. - Inputs: - `block_id` (int; required): Required positional or keyword input. - Return annotation: `bool` - Calls: block.touch - State reads: self._lock, self.allocated_blocks, self.stats - State writes: self.stats.shared_blocks - Return expressions: False; True ## `vllm_mlx.paged_cache.PagedCacheManager.decrement_ref` - Kind: method - Signature: `def decrement_ref(self, block_id: int) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/paged_cache.py#L741-L743 - Implementation: Method `PagedCacheManager.decrement_ref` calls `self.free_block`; returns `self.free_block(block_id)`. Decrement reference count (alias for free_block). - Inputs: - `block_id` (int; required): Required positional or keyword input. - Return annotation: `bool` - Calls: self.free_block - State reads: self.free_block - Return expressions: self.free_block(block_id) ## `vllm_mlx.paged_cache.PagedCacheManager.get_cached_block` - Kind: method - Signature: `def get_cached_block(self, block_hash: BlockHash) -> Optional[CacheBlock]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/paged_cache.py#L749-L768 - Implementation: Method `PagedCacheManager.get_cached_block` updates `self.stats.cache_hits`, `self.stats.cache_misses`; calls `self.cached_block_hash_to_block.get_block`; has 2 explicit return paths. Get a cached block by its hash (vLLM style). Args: block_hash: Content hash of the block Returns: Cached block if found, None otherwise - Inputs: - `block_hash` (BlockHash; required): Content hash of the block - Return annotation: `Optional[CacheBlock]` - Calls: self.cached_block_hash_to_block.get_block - State reads: self.enable_caching, self._lock, self.cached_block_hash_to_block.get_block, self.cached_block_hash_to_block, self.stats - State writes: self.stats.cache_hits, self.stats.cache_misses - Return expressions: None; block ## `vllm_mlx.paged_cache.PagedCacheManager.cache_full_blocks` - Kind: method - Signature: `def cache_full_blocks(self, blocks: List[CacheBlock], token_ids: List[int], num_cached_blocks: int, num_full_blocks: int) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/paged_cache.py#L770-L824 - Implementation: Method `PagedCacheManager.cache_full_blocks` calls `range`, `compute_block_hash`, `len`, `self.cached_block_hash_to_block.insert`; returns `None`. Cache full blocks for prefix caching (vLLM style). Computes chain hashes and adds blocks to the cache. Args: blocks: All blocks for the request token_ids: All token IDs for the request num_cached_blocks: Number of blocks already cached num_full_blocks: Number of full blocks to cache - Inputs: - `blocks` (List[CacheBlock]; required): All blocks for the request - `token_ids` (List[int]; required): All token IDs for the request - `num_cached_blocks` (int; required): Number of blocks already cached - `num_full_blocks` (int; required): Number of full blocks to cache - Return annotation: `None` - Calls: range, compute_block_hash, len, self.cached_block_hash_to_block.insert, self.compute_block_hash - State reads: self.enable_caching, self._lock, self.block_size, self.cached_block_hash_to_block.insert, self.cached_block_hash_to_block, self.compute_block_hash, self.hash_to_block - Return expressions: None ## `vllm_mlx.paged_cache.PagedCacheManager.get_computed_blocks` - Kind: method - Signature: `def get_computed_blocks(self, token_ids: List[int]) -> Tuple[List[CacheBlock], int]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/paged_cache.py#L826-L868 - Implementation: Method `PagedCacheManager.get_computed_blocks` updates `self.stats.cache_misses`, `self.stats.cache_hits`; calls `len`, `range`, `compute_block_hash`, `self.cached_block_hash_to_block.get_block`; has 2 explicit return paths. Find cached blocks for a token prefix (vLLM style). Args: token_ids: Token IDs to look up Returns: Tuple of (cached_blocks, num_cached_tokens) - Inputs: - `token_ids` (List[int]; required): Token IDs to look up - Return annotation: `Tuple[List[CacheBlock], int]` - Calls: len, range, compute_block_hash, self.cached_block_hash_to_block.get_block, cached_blocks.append - State reads: self.enable_caching, self._lock, self.block_size, self.cached_block_hash_to_block.get_block, self.cached_block_hash_to_block, self.stats - State writes: self.stats.cache_misses, self.stats.cache_hits - Return expressions: ([], 0); (cached_blocks, num_cached_tokens) ## `vllm_mlx.paged_cache.PagedCacheManager.compute_block_hash` - Kind: method - Signature: `def compute_block_hash(tokens: List[int]) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/paged_cache.py#L875-L878 - Implementation: Method `PagedCacheManager.compute_block_hash` calls `b''.join`, `t.to_bytes`, `hashlib.sha256(token_bytes).hexdigest`, `hashlib.sha256`; returns `hashlib.sha256(token_bytes).hexdigest()[:16]`. Compute legacy string hash for a sequence of tokens. - Inputs: - `tokens` (List[int]; required): Required positional or keyword input. - Return annotation: `str` - Decorators: staticmethod - Calls: b''.join, t.to_bytes, hashlib.sha256(token_bytes).hexdigest, hashlib.sha256 - Return expressions: hashlib.sha256(token_bytes).hexdigest()[:16] ## `vllm_mlx.paged_cache.PagedCacheManager.find_cached_block` - Kind: method - Signature: `def find_cached_block(self, tokens: List[int]) -> Optional[CacheBlock]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/paged_cache.py#L880-L896 - Implementation: Method `PagedCacheManager.find_cached_block` updates `self.stats.cache_hits`, `self.stats.cache_misses`; calls `self.compute_block_hash`, `block.touch`; has 2 explicit return paths. Find a cached block matching the given tokens (legacy method). - Inputs: - `tokens` (List[int]; required): Required positional or keyword input. - Return annotation: `Optional[CacheBlock]` - Calls: self.compute_block_hash, block.touch - State reads: self._lock, self.compute_block_hash, self.hash_to_block, self.allocated_blocks, self.stats - State writes: self.stats.cache_hits, self.stats.cache_misses - Return expressions: block; None ## `vllm_mlx.paged_cache.PagedCacheManager.register_block_hash` - Kind: method - Signature: `def register_block_hash(self, block: CacheBlock, tokens: List[int]) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/paged_cache.py#L898-L903 - Implementation: Method `PagedCacheManager.register_block_hash` calls `self.compute_block_hash`. Register a block's hash for deduplication (legacy method). - Inputs: - `block` (CacheBlock; required): Required positional or keyword input. - `tokens` (List[int]; required): Required positional or keyword input. - Return annotation: `None` - Calls: self.compute_block_hash - State reads: self._lock, self.compute_block_hash, self.hash_to_block ## `vllm_mlx.paged_cache.PagedCacheManager.create_block_table` - Kind: method - Signature: `def create_block_table(self, request_id: str) -> BlockTable` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/paged_cache.py#L909-L914 - Implementation: Method `PagedCacheManager.create_block_table` calls `BlockTable`; returns `table`. Create a new block table for a request. - Inputs: - `request_id` (str; required): Required positional or keyword input. - Return annotation: `BlockTable` - Calls: BlockTable - State reads: self._lock, self.request_tables - Return expressions: table ## `vllm_mlx.paged_cache.PagedCacheManager.get_block_table` - Kind: method - Signature: `def get_block_table(self, request_id: str) -> Optional[BlockTable]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/paged_cache.py#L916-L919 - Implementation: Method `PagedCacheManager.get_block_table` calls `self.request_tables.get`; returns `self.request_tables.get(request_id)`. Get block table for a request. - Inputs: - `request_id` (str; required): Required positional or keyword input. - Return annotation: `Optional[BlockTable]` - Calls: self.request_tables.get - State reads: self._lock, self.request_tables.get, self.request_tables - Return expressions: self.request_tables.get(request_id) ## `vllm_mlx.paged_cache.PagedCacheManager.get_or_create_block_table` - Kind: method - Signature: `def get_or_create_block_table(self, request_id: str) -> BlockTable` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/paged_cache.py#L921-L926 - Implementation: Method `PagedCacheManager.get_or_create_block_table` calls `BlockTable`; returns `self.request_tables[request_id]`. Get or create block table for a request. - Inputs: - `request_id` (str; required): Required positional or keyword input. - Return annotation: `BlockTable` - Calls: BlockTable - State reads: self._lock, self.request_tables - Return expressions: self.request_tables[request_id] ## `vllm_mlx.paged_cache.PagedCacheManager.delete_block_table` - Kind: method - Signature: `def delete_block_table(self, request_id: str) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/paged_cache.py#L928-L934 - Implementation: Method `PagedCacheManager.delete_block_table` calls `self.request_tables.pop`, `self.free_block`. Delete block table and free associated blocks. - Inputs: - `request_id` (str; required): Required positional or keyword input. - Return annotation: `None` - Calls: self.request_tables.pop, self.free_block - State reads: self._lock, self.request_tables.pop, self.request_tables, self.free_block ## `vllm_mlx.paged_cache.PagedCacheManager.add_block_to_table` - Kind: method - Signature: `def add_block_to_table(self, table: BlockTable, block: CacheBlock, tokens_in_block: int) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/paged_cache.py#L936-L947 - Implementation: Method `PagedCacheManager.add_block_to_table` updates `self.stats.total_tokens_cached`; calls `table.block_ids.append`. Add a block to a block table. - Inputs: - `table` (BlockTable; required): Required positional or keyword input. - `block` (CacheBlock; required): Required positional or keyword input. - `tokens_in_block` (int; required): Required positional or keyword input. - Return annotation: `None` - Calls: table.block_ids.append - State reads: self._lock, self.stats - State writes: self.stats.total_tokens_cached ## `vllm_mlx.paged_cache.PagedCacheManager.find_shared_prefix` - Kind: method - Signature: `def find_shared_prefix(self, tokens: List[int]) -> Tuple[List[int], List[int]]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/paged_cache.py#L953-L974 - Implementation: Method `PagedCacheManager.find_shared_prefix` calls `tokens.copy`, `len`, `self.find_cached_block`, `shared_blocks.append`; returns `(shared_blocks, remaining_tokens)`. Find shared prefix blocks for a token sequence. - Inputs: - `tokens` (List[int]; required): Required positional or keyword input. - Return annotation: `Tuple[List[int], List[int]]` - Calls: tokens.copy, len, self.find_cached_block, shared_blocks.append - State reads: self._lock, self.block_size, self.find_cached_block - Return expressions: (shared_blocks, remaining_tokens) ## `vllm_mlx.paged_cache.PagedCacheManager.fork_block_table` - Kind: method - Signature: `def fork_block_table(self, source_table: BlockTable, new_request_id: str) -> BlockTable` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/paged_cache.py#L976-L997 - Implementation: Method `PagedCacheManager.fork_block_table` calls `source_table.copy`, `self.increment_ref`, `logger.debug`, `len`; returns `new_table`. Fork a block table for a new request (COW). - Inputs: - `source_table` (BlockTable; required): Required positional or keyword input. - `new_request_id` (str; required): Required positional or keyword input. - Return annotation: `BlockTable` - Calls: source_table.copy, self.increment_ref, logger.debug, len - State reads: self._lock, self.increment_ref, self.request_tables - Return expressions: new_table ## `vllm_mlx.paged_cache.PagedCacheManager.get_blocks_for_generation` - Kind: method - Signature: `def get_blocks_for_generation(self, table: BlockTable) -> Tuple[List[CacheBlock], bool]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/paged_cache.py#L999-L1029 - Implementation: Method `PagedCacheManager.get_blocks_for_generation` updates `self.stats.cow_copies`; calls `enumerate`, `self.allocated_blocks.get`, `block.is_shared`, `self._cow_copy_block`; returns `(blocks, was_copied)`. Get blocks for generation, applying COW if needed. - Inputs: - `table` (BlockTable; required): Required positional or keyword input. - Return annotation: `Tuple[List[CacheBlock], bool]` - Calls: enumerate, self.allocated_blocks.get, block.is_shared, self._cow_copy_block, blocks.append, block.touch - State reads: self._lock, self.allocated_blocks.get, self.allocated_blocks, self._cow_copy_block, self.stats - State writes: self.stats.cow_copies - Return expressions: (blocks, was_copied) ## `vllm_mlx.paged_cache.PagedCacheManager._cow_copy_block` - Kind: method - Signature: `def _cow_copy_block(self, source_block: CacheBlock) -> Optional[CacheBlock]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/paged_cache.py#L1031-L1046 - Implementation: Method `PagedCacheManager._cow_copy_block` updates `self.stats.shared_blocks`; calls `self.allocate_block`, `logger.debug`; has 2 explicit return paths. Create a copy of a block for COW. - Inputs: - `source_block` (CacheBlock; required): Required positional or keyword input. - Return annotation: `Optional[CacheBlock]` - Calls: self.allocate_block, logger.debug - State reads: self.allocate_block, self.stats - State writes: self.stats.shared_blocks - Return expressions: None; new_block ## `vllm_mlx.paged_cache.PagedCacheManager.allocate_blocks_for_tokens` - Kind: method - Signature: `def allocate_blocks_for_tokens(self, num_tokens: int) -> List[CacheBlock]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/paged_cache.py#L1052-L1055 - Implementation: Method `PagedCacheManager.allocate_blocks_for_tokens` calls `self.get_new_blocks`; returns `self.get_new_blocks(num_blocks_needed)`. Allocate enough blocks to hold num_tokens. - Inputs: - `num_tokens` (int; required): Required positional or keyword input. - Return annotation: `List[CacheBlock]` - Calls: self.get_new_blocks - State reads: self.block_size, self.get_new_blocks - Return expressions: self.get_new_blocks(num_blocks_needed) ## `vllm_mlx.paged_cache.PagedCacheManager.evict_lru_blocks` - Kind: method - Signature: `def evict_lru_blocks(self, num_blocks: int) -> int` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/paged_cache.py#L1061-L1085 - Implementation: Method `PagedCacheManager.evict_lru_blocks` calls `range`, `min`, `self.free_block_queue.popleft`, `self._maybe_evict_cached_block`; returns `evicted`. Evict least recently used blocks. With the doubly linked list, LRU blocks are already at the front of the free queue. We just need to pop from front. - Inputs: - `num_blocks` (int; required): Required positional or keyword input. - Return annotation: `int` - Calls: range, min, self.free_block_queue.popleft, self._maybe_evict_cached_block, self.free_block_queue.append, logger.info - State reads: self._lock, self.free_block_queue.num_free_blocks, self.free_block_queue, self.free_block_queue.popleft, self._maybe_evict_cached_block, self.free_block_queue.append - Return expressions: evicted ## `vllm_mlx.paged_cache.PagedCacheManager.handle_memory_pressure` - Kind: method - Signature: `def handle_memory_pressure(self, requested_blocks: int) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/paged_cache.py#L1087-L1096 - Implementation: Method `PagedCacheManager.handle_memory_pressure` calls `self.evict_lru_blocks`; has 2 explicit return paths. Handle memory pressure by evicting blocks. - Inputs: - `requested_blocks` (int; required): Required positional or keyword input. - Return annotation: `bool` - Calls: self.evict_lru_blocks - State reads: self._lock, self.free_block_queue.num_free_blocks, self.free_block_queue, self.evict_lru_blocks - Return expressions: True; self.free_block_queue.num_free_blocks >= requested_blocks ## `vllm_mlx.paged_cache.PagedCacheManager.free_blocks` - Kind: method - Signature: `def free_blocks(self) -> int` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/paged_cache.py#L1103-L1105 - Implementation: Method `PagedCacheManager.free_blocks` returns `self.free_block_queue.num_free_blocks`. Number of free blocks available. - Inputs: none - Return annotation: `int` - Decorators: property - State reads: self.free_block_queue.num_free_blocks, self.free_block_queue - Return expressions: self.free_block_queue.num_free_blocks ## `vllm_mlx.paged_cache.PagedCacheManager.usage` - Kind: method - Signature: `def usage(self) -> float` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/paged_cache.py#L1108-L1113 - Implementation: Method `PagedCacheManager.usage` has 2 explicit return paths. Cache usage ratio (0.0 to 1.0). - Inputs: none - Return annotation: `float` - Decorators: property - State reads: self.max_blocks, self.free_blocks - Return expressions: 0.0; 1.0 - self.free_blocks / total ## `vllm_mlx.paged_cache.PagedCacheManager.get_stats` - Kind: method - Signature: `def get_stats(self) -> CacheStats` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/paged_cache.py#L1115-L1122 - Implementation: Method `PagedCacheManager.get_stats` updates `self.stats.shared_blocks`, `self.stats.free_blocks`; calls `sum`, `self.allocated_blocks.values`; returns `self.stats`. Get current cache statistics. - Inputs: none - Return annotation: `CacheStats` - Calls: sum, self.allocated_blocks.values - State reads: self._lock, self.stats, self.allocated_blocks.values, self.allocated_blocks, self.free_block_queue.num_free_blocks, self.free_block_queue - State writes: self.stats.shared_blocks, self.stats.free_blocks - Return expressions: self.stats ## `vllm_mlx.paged_cache.PagedCacheManager.get_memory_usage` - Kind: method - Signature: `def get_memory_usage(self) -> Dict[str, Any]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/paged_cache.py#L1124-L1141 - Implementation: Method `PagedCacheManager.get_memory_usage` calls `self.get_stats`; returns `{'block_size': self.block_size, 'max_blocks': self.max_blocks, 'allocated_blocks': stats.allocated_blocks, 'free_blocks…`. Get memory usage information. - Inputs: none - Return annotation: `Dict[str, Any]` - Calls: self.get_stats - State reads: self._lock, self.get_stats, self.block_size, self.max_blocks - Return expressions: {'block_size': self.block_size, 'max_blocks': self.max_blocks, 'allocated_blocks': stats.allocated_blocks, 'free_blocks… ## `vllm_mlx.paged_cache.PagedCacheManager.reset_stats` - Kind: method - Signature: `def reset_stats(self) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/paged_cache.py#L1143-L1149 - Implementation: Method `PagedCacheManager.reset_stats` updates `self.stats.cache_hits`, `self.stats.cache_misses`, `self.stats.cow_copies`, `self.stats.evictions`. Reset statistics counters. - Inputs: none - Return annotation: `None` - State reads: self._lock, self.stats - State writes: self.stats.cache_hits, self.stats.cache_misses, self.stats.cow_copies, self.stats.evictions ## `vllm_mlx.paged_cache.PagedCacheManager.reset_prefix_cache` - Kind: method - Signature: `def reset_prefix_cache(self) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/paged_cache.py#L1151-L1171 - Implementation: Method `PagedCacheManager.reset_prefix_cache` updates `self.stats.evictions`, `self.stats.cache_hits`, `self.stats.cache_misses`; calls `logger.warning`, `self.cached_block_hash_to_block.clear`, `self.hash_to_block.clear`, `block.reset_hash`; has 2 explicit return paths. Reset the prefix cache. - Inputs: none - Return annotation: `bool` - Calls: logger.warning, self.cached_block_hash_to_block.clear, self.hash_to_block.clear, block.reset_hash, logger.info - State reads: self._lock, self.max_blocks, self.free_block_queue.num_free_blocks, self.free_block_queue, self.cached_block_hash_to_block.clear, self.cached_block_hash_to_block, self.hash_to_block.clear, self.hash_to_block, self.blocks, self.stats - State writes: self.stats.evictions, self.stats.cache_hits, self.stats.cache_misses - Return expressions: False; True ## `vllm_mlx.paged_cache.PagedCacheManager.clear` - Kind: method - Signature: `def clear(self) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/paged_cache.py#L1173-L1197 - Implementation: Method `PagedCacheManager.clear` updates `self.blocks`, `self.free_block_queue`, `self.null_block`, `self.null_block.is_null`; calls `CacheBlock`, `range`, `FreeKVCacheBlockQueue`, `self.cached_block_hash_to_block.clear`. Clear all cached data. - Inputs: none - Return annotation: `None` - Calls: CacheBlock, range, FreeKVCacheBlockQueue, self.cached_block_hash_to_block.clear, self.hash_to_block.clear, self.request_tables.clear, self.allocated_blocks.clear, self.free_block_queue.popleft, CacheStats, logger.info - State reads: self._lock, self.max_blocks, self.blocks, self.cached_block_hash_to_block.clear, self.cached_block_hash_to_block, self.hash_to_block.clear, self.hash_to_block, self.request_tables.clear, self.request_tables, self.allocated_blocks.clear, self.allocated_blocks, self.free_block_queue.popleft, self.free_block_queue, self.null_block, self.null_block.block_id - State writes: self.blocks, self.free_block_queue, self.null_block, self.null_block.is_null, self.null_block.ref_count, self.stats # Module `vllm_mlx.patches` Narrow runtime compatibility patches for supported model architectures. Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/patches/__init__.py#L1-L2 # Module `vllm_mlx.patches.gemma4_mllm` Runtime patch for mlx-vlm Gemma 4 Attention to trim oversized masks. mlx-vlm 0.5.0's stock Gemma 4 attention assumes the mask's last dim matches keys.shape[-2] exactly. vllm-mlx's BatchedEngine MLLM path (continuous batching) sometimes passes a mask sized for the max sequence in the batch while a specific layer's keys end up shorter — sliding-window layers cap keys at window=512, the mask is built once for the full prompt. Without a trim, scaled_dot_product_attention sees a shape mismatch. That mask trim is the only behavior this patch adds; everything else mirrors mlx-vlm 0.5.0 verbatim (signature, return shape, offset handling). The previous reason for this patch — BatchKVCache's in-place `+=` on `cache.offset` corrupting RoPE — is now handled upstream: mlx-vlm 0.5.0 line 223 does `offset = mx.array(cache.offset) if cache is not None else 0` which is a defensive copy. (Confirmed in review of PR #564.) Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/patches/gemma4_mllm.py#L1-L98 ## `vllm_mlx.patches.gemma4_mllm.patch_gemma4_attention_for_batching` - Kind: function - Signature: `def patch_gemma4_attention_for_batching() -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/patches/gemma4_mllm.py#L28-L98 - Implementation: Function `patch_gemma4_attention_for_batching` calls `logger.debug`, `getattr`, `logger.info`; has 2 explicit return paths. Patch Gemma 4 Attention.__call__ to trim oversized masks. Otherwise mirrors mlx-vlm 0.5.0 upstream verbatim. Returns True if applied, False if mlx-vlm is not installed or Gemma 4 unavailable. - Inputs: none - Return annotation: `bool` - Calls: logger.debug, getattr, logger.info - Return expressions: False; True ## `vllm_mlx.patches.gemma4_mllm.patch_gemma4_attention_for_batching._patched_call` - Kind: nested function - Signature: `def _patched_call(self, x: mx.array, mask: Optional[mx.array]=None, cache: Optional[Any]=None, shared_kv: Optional[tuple]=None, offset: Optional[Any]=None) -> Any` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/patches/gemma4_mllm.py#L45-L93 - Implementation: Nested Function `patch_gemma4_attention_for_batching._patched_call` calls `self.q_proj(x).reshape`, `self.q_proj`, `self.q_norm`, `self.k_proj(x).reshape`; returns `(self.o_proj(output), (keys, values), offset)`. Nested Function `patch_gemma4_attention_for_batching._patched_call` calls `self.q_proj(x).reshape`, `self.q_proj`, `self.q_norm`, `self.k_proj(x).reshape`; returns `(self.o_proj(output), (keys, values), offset)`. - Inputs: - `x` (mx.array; required): Required positional or keyword input. - `mask` (Optional[mx.array]; optional; default `None`): Optional positional or keyword input; defaults to `None`. - `cache` (Optional[Any]; optional; default `None`): Optional positional or keyword input; defaults to `None`. - `shared_kv` (Optional[tuple]; optional; default `None`): Optional positional or keyword input; defaults to `None`. - `offset` (Optional[Any]; optional; default `None`): Optional positional or keyword input; defaults to `None`. - Return annotation: `Any` - Calls: self.q_proj(x).reshape, self.q_proj, self.q_norm, self.k_proj(x).reshape, self.k_proj, self.v_proj(x).reshape, self.v_proj, mx.array, self.k_norm, keys.transpose, self.rope, self.v_norm, values.transpose, cache.update_and_fetch, queries.transpose, isinstance, scaled_dot_product_attention, output.transpose(0, 2, 1, 3).reshape, output.transpose, self.o_proj - State reads: self.q_proj, self.n_heads, self.head_dim, self.q_norm, self.k_proj, self.n_kv_heads, self.use_k_eq_v, self.v_proj, self.k_norm, self.rope, self.v_norm, self.scale, self.o_proj - Return expressions: (self.o_proj(output), (keys, values), offset) # Module `vllm_mlx.patches.glm4v_moe_mllm` Runtime patch for mlx-vlm's GLM-4.6V model to support BatchKVCache. GLM-4.6V (glm4v_moe) computes position_ids from cache[0].offset once at the start of GLM4VModel.__call__, then derives position_embeddings used by ALL decoder layers: position_ids = mx.arange(cache[0].offset, cache[0].offset + seq_len) position_embeddings = self.rotary_emb(h, position_ids) for layer in self.layers: h = layer(h, mask, cache[i], position_embeddings) For regular KVCache, cache.offset is a Python int, so mx.arange works fine. For BatchKVCache, cache.offset is an mx.array (per-batch-item offsets), and mx.arange does not support mx.array start/stop arguments, producing wrong position_ids that corrupt RoPE embeddings for ALL layers. This patch replaces GLM4VModel.__call__ with a version that converts cache[0].offset to int before computing position_ids. Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/patches/glm4v_moe_mllm.py#L1-L89 ## `vllm_mlx.patches.glm4v_moe_mllm.patch_glm4v_moe_for_batching` - Kind: function - Signature: `def patch_glm4v_moe_for_batching() -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/patches/glm4v_moe_mllm.py#L31-L89 - Implementation: Function `patch_glm4v_moe_for_batching` calls `logger.debug`, `getattr`, `logger.info`; has 2 explicit return paths. Monkey-patch GLM4VModel.__call__ to handle BatchKVCache offset. Returns True if patch was applied, False if mlx-vlm is not installed or GLM-4.6V module not available. - Inputs: none - Return annotation: `bool` - Calls: logger.debug, getattr, logger.info - Return expressions: False; True ## `vllm_mlx.patches.glm4v_moe_mllm.patch_glm4v_moe_for_batching._patched_call` - Kind: nested function - Signature: `def _patched_call(self, inputs: mx.array, inputs_embeds: Optional[mx.array]=None, cache: Optional[Any]=None, mask: Optional[mx.array]=None, position_ids: Optional[mx.array]=None) -> mx.array` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/patches/glm4v_moe_mllm.py#L50-L84 - Implementation: Nested Function `patch_glm4v_moe_for_batching._patched_call` calls `self.embed_tokens`, `inputs_embeds.astype`, `isinstance`, `int`; returns `self.norm(h)`. Nested Function `patch_glm4v_moe_for_batching._patched_call` calls `self.embed_tokens`, `inputs_embeds.astype`, `isinstance`, `int`; returns `self.norm(h)`. - Inputs: - `inputs` (mx.array; required): Required positional or keyword input. - `inputs_embeds` (Optional[mx.array]; optional; default `None`): Optional positional or keyword input; defaults to `None`. - `cache` (Optional[Any]; optional; default `None`): Optional positional or keyword input; defaults to `None`. - `mask` (Optional[mx.array]; optional; default `None`): Optional positional or keyword input; defaults to `None`. - `position_ids` (Optional[mx.array]; optional; default `None`): Optional positional or keyword input; defaults to `None`. - Return annotation: `mx.array` - Calls: self.embed_tokens, inputs_embeds.astype, isinstance, int, offset.max().item, offset.max, mx.arange, mx.expand_dims, mx.tile, self.rotary_emb, create_attention_mask, range, self.layers[self.start_idx + i], self.norm - State reads: self.embed_tokens, self.norm.weight.dtype, self.norm.weight, self.norm, self.rotary_emb, self.num_layers, self.layers, self.start_idx - Return expressions: self.norm(h) # Module `vllm_mlx.patches.qwen3_5_mllm` Runtime patch for mlx-vlm's Qwen3.5 attention to support BatchKVCache. Qwen 3.6 artifacts use the mlx-vlm Qwen3.5 language module in this stack. The attention patch therefore lives in the Qwen3.5 compatibility module while serving Qwen 3.6 27B/35B/122B artifacts. mlx-vlm's Qwen3_5Attention uses cache.offset directly for kv_seq_len computation and mask slicing. BatchKVCache stores offset as mx.array (per-batch-item), not int, causing: mask = mask[..., :kv_seq_len] ValueError: Slice indices must be integers or None. This patch replaces Qwen3_5Attention.__call__ with a version that converts cache.offset to int before using it for arithmetic/slicing, while leaving the actual cache.offset untouched so update_and_fetch still works correctly with per-batch offsets. Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/patches/qwen3_5_mllm.py#L1-L266 ## `vllm_mlx.patches.qwen3_5_mllm._cache_offset_to_int` - Kind: function - Signature: `def _cache_offset_to_int(cache) -> int` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/patches/qwen3_5_mllm.py#L33-L42 - Implementation: Function `_cache_offset_to_int` calls `isinstance`, `int`, `off.max().item`, `off.max`; has 4 explicit return paths. Extract cache offset as int, handling BatchKVCache mx.array offset. - Inputs: - `cache` (not annotated; required): Required positional or keyword input. - Return annotation: `int` - Calls: isinstance, int, off.max().item, off.max, off.item - Return expressions: 0; off; int(off.max().item()) if off.ndim > 0 else int(off.item()); int(off) ## `vllm_mlx.patches.qwen3_5_mllm._default_target_verify_linears` - Kind: function - Signature: `def _default_target_verify_linears(linears, x, target_verify: bool)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/patches/qwen3_5_mllm.py#L45-L46 - Implementation: Function `_default_target_verify_linears` calls `tuple`, `linear`; returns `tuple((linear(x) for linear in linears))`. Function `_default_target_verify_linears` calls `tuple`, `linear`; returns `tuple((linear(x) for linear in linears))`. - Inputs: - `linears` (not annotated; required): Required positional or keyword input. - `x` (not annotated; required): Required positional or keyword input. - `target_verify` (bool; required): Required positional or keyword input. - Return annotation: `not annotated` - Calls: tuple, linear - Return expressions: tuple((linear(x) for linear in linears)) ## `vllm_mlx.patches.qwen3_5_mllm._default_target_verify_left_padded_attention` - Kind: function - Signature: `def _default_target_verify_left_padded_attention(*args, **kwargs)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/patches/qwen3_5_mllm.py#L49-L50 - Implementation: Function `_default_target_verify_left_padded_attention` returns `None`. Function `_default_target_verify_left_padded_attention` returns `None`. - Inputs: - `*args` (not annotated; optional): Additional variadic positional inputs accepted by this callable. - `**kwargs` (not annotated; optional): Additional variadic keyword inputs accepted by this callable. - Return annotation: `not annotated` - Return expressions: None ## `vllm_mlx.patches.qwen3_5_mllm._normalize_position_inputs` - Kind: function - Signature: `def _normalize_position_inputs(position_ids: Optional[mx.array], position_embeddings: Optional[tuple[mx.array, mx.array]], length: int) -> tuple[Optional[mx.array], Optional[tuple[mx.array, mx.array]]]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/patches/qwen3_5_mllm.py#L53-L65 - Implementation: Function `_normalize_position_inputs` calls `logger.debug`; has 2 explicit return paths. Function `_normalize_position_inputs` calls `logger.debug`; has 2 explicit return paths. - Inputs: - `position_ids` (Optional[mx.array]; required): Required positional or keyword input. - `position_embeddings` (Optional[tuple[mx.array, mx.array]]; required): Required positional or keyword input. - `length` (int; required): Required positional or keyword input. - Return annotation: `tuple[Optional[mx.array], Optional[tuple[mx.array, mx.array]]]` - Calls: logger.debug - Return expressions: (position_ids, position_embeddings); (None, None) ## `vllm_mlx.patches.qwen3_5_mllm._position_ids_for_offset` - Kind: function - Signature: `def _position_ids_for_offset(offset: int, length: int) -> mx.array` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/patches/qwen3_5_mllm.py#L68-L71 - Implementation: Function `_position_ids_for_offset` calls `mx.arange`, `mx.expand_dims`, `mx.tile`; returns `mx.tile(position_ids, (3, 1, 1))`. Function `_position_ids_for_offset` calls `mx.arange`, `mx.expand_dims`, `mx.tile`; returns `mx.tile(position_ids, (3, 1, 1))`. - Inputs: - `offset` (int; required): Required positional or keyword input. - `length` (int; required): Required positional or keyword input. - Return annotation: `mx.array` - Calls: mx.arange, mx.expand_dims, mx.tile - Return expressions: mx.tile(position_ids, (3, 1, 1)) ## `vllm_mlx.patches.qwen3_5_mllm._kv_seq_len` - Kind: function - Signature: `def _kv_seq_len(keys: mx.array, cache: Optional[Any], offset: int) -> int` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/patches/qwen3_5_mllm.py#L74-L76 - Implementation: Function `_kv_seq_len` returns `length + offset + 1 if cache is not None else length`. Function `_kv_seq_len` returns `length + offset + 1 if cache is not None else length`. - Inputs: - `keys` (mx.array; required): Required positional or keyword input. - `cache` (Optional[Any]; required): Required positional or keyword input. - `offset` (int; required): Required positional or keyword input. - Return annotation: `int` - Return expressions: length + offset + 1 if cache is not None else length ## `vllm_mlx.patches.qwen3_5_mllm._apply_rotary` - Kind: function - Signature: `def _apply_rotary(attention, queries: mx.array, keys: mx.array, values: mx.array, position_ids: mx.array, position_embeddings: Optional[tuple[mx.array, mx.array]], apply_multimodal_rotary_pos_emb) -> tuple[mx.array, mx.array]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/patches/qwen3_5_mllm.py#L79-L101 - Implementation: Function `_apply_rotary` calls `apply_multimodal_rotary_pos_emb`, `hasattr`, `attention.rotary_emb.apply_rotary`, `attention.rotary_emb`; has 2 explicit return paths. Function `_apply_rotary` calls `apply_multimodal_rotary_pos_emb`, `hasattr`, `attention.rotary_emb.apply_rotary`, `attention.rotary_emb`; has 2 explicit return paths. - Inputs: - `attention` (not annotated; required): Required positional or keyword input. - `queries` (mx.array; required): Required positional or keyword input. - `keys` (mx.array; required): Required positional or keyword input. - `values` (mx.array; required): Required positional or keyword input. - `position_ids` (mx.array; required): Required positional or keyword input. - `position_embeddings` (Optional[tuple[mx.array, mx.array]]; required): Required positional or keyword input. - `apply_multimodal_rotary_pos_emb` (not annotated; required): Required positional or keyword input. - Return annotation: `tuple[mx.array, mx.array]` - Calls: apply_multimodal_rotary_pos_emb, hasattr, attention.rotary_emb.apply_rotary, attention.rotary_emb - Return expressions: apply_multimodal_rotary_pos_emb(queries, keys, cos, sin); attention.rotary_emb.apply_rotary(queries, keys, position_ids, unsqueeze_dim=1) ## `vllm_mlx.patches.qwen3_5_mllm._slice_attention_mask` - Kind: function - Signature: `def _slice_attention_mask(mask: Optional[mx.array], cache: Optional[Any], kv_seq_len: int, length: int) -> Optional[mx.array]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/patches/qwen3_5_mllm.py#L104-L116 - Implementation: Function `_slice_attention_mask` calls `isinstance`, `hasattr`, `int`, `kv_seq_len.max().item`; has 2 explicit return paths. Function `_slice_attention_mask` calls `isinstance`, `hasattr`, `int`, `kv_seq_len.max().item`; has 2 explicit return paths. - Inputs: - `mask` (Optional[mx.array]; required): Required positional or keyword input. - `cache` (Optional[Any]; required): Required positional or keyword input. - `kv_seq_len` (int; required): Required positional or keyword input. - `length` (int; required): Required positional or keyword input. - Return annotation: `Optional[mx.array]` - Calls: isinstance, hasattr, int, kv_seq_len.max().item, kv_seq_len.max - Return expressions: mask; mask[..., :int(kv_seq_len)] ## `vllm_mlx.patches.qwen3_5_mllm._maybe_target_verify_attention` - Kind: function - Signature: `def _maybe_target_verify_attention(queries: mx.array, keys: mx.array, values: mx.array, *, cache: Optional[Any], mask: Optional[mx.array], scale: float, target_verify: bool, length: int, left_padded_decode: bool, target_verify_left_padded_attention) -> Optional[mx.array]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/patches/qwen3_5_mllm.py#L119-L141 - Implementation: Function `_maybe_target_verify_attention` calls `target_verify_left_padded_attention`; has 2 explicit return paths. Function `_maybe_target_verify_attention` calls `target_verify_left_padded_attention`; has 2 explicit return paths. - Inputs: - `queries` (mx.array; required): Required positional or keyword input. - `keys` (mx.array; required): Required positional or keyword input. - `values` (mx.array; required): Required positional or keyword input. - `cache` (Optional[Any]; required): Required keyword-only input. - `mask` (Optional[mx.array]; required): Required keyword-only input. - `scale` (float; required): Required keyword-only input. - `target_verify` (bool; required): Required keyword-only input. - `length` (int; required): Required keyword-only input. - `left_padded_decode` (bool; required): Required keyword-only input. - `target_verify_left_padded_attention` (not annotated; required): Required keyword-only input. - Return annotation: `Optional[mx.array]` - Calls: target_verify_left_padded_attention - Return expressions: None; target_verify_left_padded_attention(queries, keys, values, cache=cache, scale=scale, mask=mask) ## `vllm_mlx.patches.qwen3_5_mllm.patch_qwen35_attention_for_batching` - Kind: function - Signature: `def patch_qwen35_attention_for_batching() -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/patches/qwen3_5_mllm.py#L144-L266 - Implementation: Function `patch_qwen35_attention_for_batching` calls `importlib.import_module`, `logger.debug`, `getattr`, `setattr`; has 2 explicit return paths. Monkey-patch Qwen3_5Attention.__call__ to handle BatchKVCache. Returns True if patch was applied, False if mlx-vlm is not installed or Qwen3.5 module not available. - Inputs: none - Return annotation: `bool` - Calls: importlib.import_module, logger.debug, getattr, setattr, logger.info - Return expressions: False; True ## `vllm_mlx.patches.qwen3_5_mllm.patch_qwen35_attention_for_batching._patched_call` - Kind: nested function - Signature: `def _patched_call(self, x: mx.array, mask: Optional[mx.array]=None, cache: Optional[Any]=None, position_ids: Optional[mx.array]=None, position_embeddings: Optional[tuple[mx.array, mx.array]]=None, target_verify: bool=False) -> mx.array` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/patches/qwen3_5_mllm.py#L174-L261 - Implementation: Nested Function `patch_qwen35_attention_for_batching._patched_call` calls `target_verify_linears`, `mx.split`, `q_proj_output.reshape`, `gate.reshape`; returns `self.o_proj(output * mx.sigmoid(gate))`. Nested Function `patch_qwen35_attention_for_batching._patched_call` calls `target_verify_linears`, `mx.split`, `q_proj_output.reshape`, `gate.reshape`; returns `self.o_proj(output * mx.sigmoid(gate))`. - Inputs: - `x` (mx.array; required): Required positional or keyword input. - `mask` (Optional[mx.array]; optional; default `None`): Optional positional or keyword input; defaults to `None`. - `cache` (Optional[Any]; optional; default `None`): Optional positional or keyword input; defaults to `None`. - `position_ids` (Optional[mx.array]; optional; default `None`): Optional positional or keyword input; defaults to `None`. - `position_embeddings` (Optional[tuple[mx.array, mx.array]]; optional; default `None`): Optional positional or keyword input; defaults to `None`. - `target_verify` (bool; optional; default `False`): Optional positional or keyword input; defaults to `False`. - Return annotation: `mx.array` - Calls: target_verify_linears, mx.split, q_proj_output.reshape, gate.reshape, self.q_norm(queries).transpose, self.q_norm, self.k_norm(keys.reshape(B, L, self.num_key_value_heads, -1)).transpose, self.k_norm, keys.reshape, values.reshape(B, L, self.num_key_value_heads, -1).transpose, values.reshape, _cache_offset_to_int, _normalize_position_inputs, _position_ids_for_offset, _kv_seq_len, _apply_rotary, _slice_attention_mask, cache.update_and_fetch, isinstance, _maybe_target_verify_attention, scaled_dot_product_attention, output.transpose(0, 2, 1, 3).reshape, output.transpose, self.o_proj, mx.sigmoid - State reads: self.q_proj, self.k_proj, self.v_proj, self.num_attention_heads, self.q_norm, self.k_norm, self.num_key_value_heads, self.scale, self.o_proj - Return expressions: self.o_proj(output * mx.sigmoid(gate)) # Module `vllm_mlx.patches.qwen3_5_mtp` Runtime MTP (Multi-Token Prediction) support for Qwen3.5 models. Qwen3.5 models may include a built-in MTP head that predicts token n+2 from hidden states + token n+1. MTP weights are added to the quantized MLX model via scripts/add_mtp_weights_qwen35.py. Since mlx_lm's qwen3_5.py does NOT define MTP module/methods, this module provides: - inject_mtp_support(): dynamically creates MTP module, loads weights, and monkey-patches the model class with return_hidden, mtp_forward, and make_mtp_cache - validate_mtp_support(): checks whether a loaded model has working MTP Supports both Dense (27B) and MoE (122B-A10B, 35B-A3B) architectures. The actual MTP scheduling logic lives in: - vllm_mlx/scheduler.py (_install_mtp, _mtp_step, _mtp_next) Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/patches/qwen3_5_mtp.py#L1-L512 ## `vllm_mlx.patches.qwen3_5_mtp._strip_mtp_key_prefix` - Kind: function - Signature: `def _strip_mtp_key_prefix(key: str) -> str | None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/patches/qwen3_5_mtp.py#L31-L36 - Implementation: Function `_strip_mtp_key_prefix` calls `key.startswith`, `key.removeprefix`; has 2 explicit return paths. Return an MTP-relative key for supported standalone shard layouts. - Inputs: - `key` (str; required): Required positional or keyword input. - Return annotation: `str | None` - Calls: key.startswith, key.removeprefix - Return expressions: key.removeprefix(prefix); None ## `vllm_mlx.patches.qwen3_5_mtp._resolve_qwen_mtp_hidden_state_mode` - Kind: function - Signature: `def _resolve_qwen_mtp_hidden_state_mode(config: dict) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/patches/qwen3_5_mtp.py#L52-L65 - Implementation: Function `_resolve_qwen_mtp_hidden_state_mode` calls `config.get`, `text_config.get`, `isinstance`, `logger.warning`; has 2 explicit return paths. Resolve the checkpoint's MTP hidden-state contract safely. - Inputs: - `config` (dict; required): Required positional or keyword input. - Return annotation: `str` - Calls: config.get, text_config.get, isinstance, logger.warning - Return expressions: 'post_norm'; mode ## `vllm_mlx.patches.qwen3_5_mtp._select_qwen_mtp_hidden_state` - Kind: function - Signature: `def _select_qwen_mtp_hidden_state(mode: str, hidden_states, normed)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/patches/qwen3_5_mtp.py#L68-L70 - Implementation: Function `_select_qwen_mtp_hidden_state` returns `hidden_states if mode == 'pre_norm' else normed`. Select the representation expected by the checkpoint's MTP head. - Inputs: - `mode` (str; required): Required positional or keyword input. - `hidden_states` (not annotated; required): Required positional or keyword input. - `normed` (not annotated; required): Required positional or keyword input. - Return annotation: `not annotated` - Return expressions: hidden_states if mode == 'pre_norm' else normed ## `vllm_mlx.patches.qwen3_5_mtp._is_qwen_mtp_rmsnorm_weight` - Kind: function - Signature: `def _is_qwen_mtp_rmsnorm_weight(key: str, weight) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/patches/qwen3_5_mtp.py#L73-L77 - Implementation: Function `_is_qwen_mtp_rmsnorm_weight` calls `any`, `key.endswith`; returns `weight.ndim == 1 and any((key.endswith(suffix) for suffix in _QWEN_MTP_RMSNORM_WEIGHT_SUFFIXES))`. Return True for MTP RMSNorm weights that use Qwen's offset convention. - Inputs: - `key` (str; required): Required positional or keyword input. - `weight` (not annotated; required): Required positional or keyword input. - Return annotation: `bool` - Calls: any, key.endswith - Return expressions: weight.ndim == 1 and any((key.endswith(suffix) for suffix in _QWEN_MTP_RMSNORM_WEIGHT_SUFFIXES)) ## `vllm_mlx.patches.qwen3_5_mtp._apply_qwen_mtp_rmsnorm_offset_fixups` - Kind: function - Signature: `def _apply_qwen_mtp_rmsnorm_offset_fixups(mtp_weights: dict) -> int` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/patches/qwen3_5_mtp.py#L80-L90 - Implementation: Function `_apply_qwen_mtp_rmsnorm_offset_fixups` calls `list`, `mtp_weights.items`, `_is_qwen_mtp_rmsnorm_weight`, `weight.mean().item`; returns `norm_fixup_count`. Apply Qwen raw-offset RMSNorm fixups without double-shifting MLX weights. - Inputs: - `mtp_weights` (dict; required): Required positional or keyword input. - Return annotation: `int` - Calls: list, mtp_weights.items, _is_qwen_mtp_rmsnorm_weight, weight.mean().item, weight.mean - Return expressions: norm_fixup_count ## `vllm_mlx.patches.qwen3_5_mtp._fixup_moe_mtp` - Kind: function - Signature: `def _fixup_moe_mtp(mtp, inner_model, loaded_keys: set, mx) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/patches/qwen3_5_mtp.py#L93-L157 - Implementation: Function `_fixup_moe_mtp` calls `reversed`, `logger.warning`, `getattr`, `mlx.utils.tree_flatten`; returns `None`. Fix missing weights in MoE MTP module. MoE MTP checkpoints (122B, 35B) only contain: fc, q_proj, o_proj, shared_expert.*, and per-expert weights. Missing: - k_proj, v_proj → zero out (attention becomes no-op) - gate, shared_expert_gate → copy from main model's last full-attn layer - norms → already at identity (weight=1.0), no action needed - Inputs: - `mtp` (not annotated; required): Required positional or keyword input. - `inner_model` (not annotated; required): Required positional or keyword input. - `loaded_keys` (set; required): Required positional or keyword input. - `mx` (not annotated; required): Required positional or keyword input. - Return annotation: `None` - Calls: reversed, logger.warning, getattr, mlx.utils.tree_flatten, src.parameters, dst.load_weights, mx.eval, dst.parameters, logger.info, hasattr, mx.zeros_like, proj.parameters - Return expressions: None ## `vllm_mlx.patches.qwen3_5_mtp.inject_mtp_support` - Kind: function - Signature: `def inject_mtp_support(model: Any, model_path, config: dict) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/patches/qwen3_5_mtp.py#L160-L447 - Implementation: Function `inject_mtp_support` calls `config.get`, `_resolve_qwen_mtp_hidden_state_mode`, `text_config.get`, `logger.info`; has 2 explicit return paths. Inject MTP module into a loaded Qwen3.5 model. mlx_lm's qwen3_5.py does not define MTP layers, so we: 1. Create MTP module matching the weight structure 2. Quantize it to match the base model 3. Load MTP weights from model-mtp.safetensors 4. Monkey-patch Model with return_hidden, mtp_forward, make_mtp_cache Args: model: A model loaded via mlx_lm (strict=False, MTP weights ignored) model_path: Path to model directory (contains model-mtp.safetensors) config: Parsed config.json dict Returns: True if MTP was successfully injected, False otherwise. - Inputs: - `model` (Any; required): A model loaded via mlx_lm (strict=False, MTP weights ignored) - `model_path` (not annotated; required): Path to model directory (contains model-mtp.safetensors) - `config` (dict; required): Parsed config.json dict - Return annotation: `bool` - Calls: config.get, _resolve_qwen_mtp_hidden_state_mode, text_config.get, logger.info, Path, mtp_file.exists, logger.warning, hasattr, isinstance, TextModelArgs.from_dict, getattr, _MTPModule, quant_config.get, mx.load, str, raw.items, _strip_mtp_key_prefix, set, sorted, raw_mtp.keys, key.endswith, key.replace, mx.dequantize, processed.update, processed.add, list, mtp_weights.keys, mtp_weights.pop, _apply_qwen_mtp_rmsnorm_offset_fixups, mtp.load_weights, mtp_weights.items, mx.eval, mtp.parameters, sum, k.endswith, any, _fixup_moe_mtp - Return expressions: False; True ## `vllm_mlx.patches.qwen3_5_mtp.inject_mtp_support._MTPModule` - Kind: nested class - Signature: `class _MTPModule(nn.Module)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/patches/qwen3_5_mtp.py#L239-L252 - Implementation: Nested Class `inject_mtp_support._MTPModule` derives from `nn.Module` and declares 1 direct member(s). Nested Class `inject_mtp_support._MTPModule` derives from `nn.Module` and declares 1 direct member(s). - Inputs: - `args` (not annotated; required): Required positional or keyword input. - `n_layers` (not annotated; required): Required positional or keyword input. - Constructs: `vllm_mlx.patches.qwen3_5_mtp.inject_mtp_support._MTPModule` ## `vllm_mlx.patches.qwen3_5_mtp.inject_mtp_support._MTPModule.__init__` - Kind: nested function - Signature: `def __init__(self, args, n_layers)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/patches/qwen3_5_mtp.py#L240-L252 - Implementation: Nested Function `inject_mtp_support._MTPModule.__init__` updates `self.pre_fc_norm_hidden`, `self.pre_fc_norm_embedding`, `self.fc`, `self.layers`; calls `super().__init__`, `super`, `nn.RMSNorm`, `nn.Linear`. Nested Function `inject_mtp_support._MTPModule.__init__` updates `self.pre_fc_norm_hidden`, `self.pre_fc_norm_embedding`, `self.fc`, `self.layers`; calls `super().__init__`, `super`, `nn.RMSNorm`, `nn.Linear`. - Inputs: - `args` (not annotated; required): Required positional or keyword input. - `n_layers` (not annotated; required): Required positional or keyword input. - Return annotation: `not annotated` - Calls: super().__init__, super, nn.RMSNorm, nn.Linear, DecoderLayer, range - State writes: self.pre_fc_norm_hidden, self.pre_fc_norm_embedding, self.fc, self.layers, self.norm ## `vllm_mlx.patches.qwen3_5_mtp.inject_mtp_support._Qwen3_5MTP` - Kind: nested class - Signature: `class _Qwen3_5MTP(original_class)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/patches/qwen3_5_mtp.py#L368-L438 - Implementation: Nested Class `inject_mtp_support._Qwen3_5MTP` derives from `original_class` and declares 3 direct member(s). Qwen3.5 with MTP support (injected at runtime). - Inputs: none - Constructs: `vllm_mlx.patches.qwen3_5_mtp.inject_mtp_support._Qwen3_5MTP` ## `vllm_mlx.patches.qwen3_5_mtp.inject_mtp_support._Qwen3_5MTP.__call__` - Kind: nested function - Signature: `def __call__(self, inputs, cache=None, return_hidden: bool=False, input_embeddings=None, **kwargs)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/patches/qwen3_5_mtp.py#L371-L408 - Implementation: Nested Function `inject_mtp_support._Qwen3_5MTP.__call__` calls `inner.embed_tokens`, `len`, `create_attention_mask`, `create_ssm_mask`; has 2 explicit return paths. Nested Function `inject_mtp_support._Qwen3_5MTP.__call__` calls `inner.embed_tokens`, `len`, `create_attention_mask`, `create_ssm_mask`; has 2 explicit return paths. - Inputs: - `inputs` (not annotated; required): Required positional or keyword input. - `cache` (not annotated; optional; default `None`): Optional positional or keyword input; defaults to `None`. - `return_hidden` (bool; optional; default `False`): Optional positional or keyword input; defaults to `False`. - `input_embeddings` (not annotated; optional; default `None`): Optional positional or keyword input; defaults to `None`. - `**kwargs` (not annotated; optional): Additional variadic keyword inputs accepted by this callable. - Return annotation: `not annotated` - Calls: inner.embed_tokens, len, create_attention_mask, create_ssm_mask, zip, layer, inner.norm, inner.embed_tokens.as_linear, self.lm_head, _select_qwen_mtp_hidden_state - State reads: self.model, self.args.tie_word_embeddings, self.args, self.lm_head - Return expressions: (out, _select_qwen_mtp_hidden_state(hidden_state_mode, hidden_states, normed)); out ## `vllm_mlx.patches.qwen3_5_mtp.inject_mtp_support._Qwen3_5MTP.mtp_forward` - Kind: nested function - Signature: `def mtp_forward(self, hidden_states, next_token_ids, cache=None, mtp_cache=None)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/patches/qwen3_5_mtp.py#L410-L432 - Implementation: Nested Function `inject_mtp_support._Qwen3_5MTP.mtp_forward` calls `self.model.embed_tokens`, `self.mtp.pre_fc_norm_embedding`, `self.mtp.pre_fc_norm_hidden`, `self.mtp.fc`; has 2 explicit return paths. Run MTP head: predict token n+2 from hidden states + token n+1. - Inputs: - `hidden_states` (not annotated; required): Required positional or keyword input. - `next_token_ids` (not annotated; required): Required positional or keyword input. - `cache` (not annotated; optional; default `None`): Optional positional or keyword input; defaults to `None`. - `mtp_cache` (not annotated; optional; default `None`): Optional positional or keyword input; defaults to `None`. - Return annotation: `not annotated` - Calls: self.model.embed_tokens, self.mtp.pre_fc_norm_embedding, self.mtp.pre_fc_norm_hidden, self.mtp.fc, mx.concatenate, create_attention_mask, layer, self.mtp.norm, self.model.embed_tokens.as_linear, self.lm_head - State reads: self.model.embed_tokens, self.model, self.mtp.pre_fc_norm_embedding, self.mtp, self.mtp.pre_fc_norm_hidden, self.mtp.fc, self.mtp.layers, self.mtp.norm, self.args.tie_word_embeddings, self.args, self.model.embed_tokens.as_linear, self.lm_head - Return expressions: self.model.embed_tokens.as_linear(x); self.lm_head(x) ## `vllm_mlx.patches.qwen3_5_mtp.inject_mtp_support._Qwen3_5MTP.make_mtp_cache` - Kind: nested function - Signature: `def make_mtp_cache(self)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/patches/qwen3_5_mtp.py#L434-L438 - Implementation: Nested Function `inject_mtp_support._Qwen3_5MTP.make_mtp_cache` calls `KVCache`; has 2 explicit return paths. Create KV cache for MTP layers. - Inputs: none - Return annotation: `not annotated` - Calls: KVCache - State reads: self.mtp, self.mtp.layers - Return expressions: None; [KVCache() for _ in self.mtp.layers] ## `vllm_mlx.patches.qwen3_5_mtp.validate_mtp_support` - Kind: function - Signature: `def validate_mtp_support(model: Any) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/patches/qwen3_5_mtp.py#L450-L512 - Implementation: Function `validate_mtp_support` calls `hasattr`, `getattr`, `logger.warning`, `inspect.signature`; has 2 explicit return paths. Validate that a loaded model has working MTP support. Checks: 1. model.mtp exists and is not None 2. model.mtp has layers with loaded weights 3. model has return_hidden support in __call__ 4. model has mtp_forward method 5. model has make_mtp_cache method Args: model: A model loaded via mlx_lm.load() Returns: True if MTP is fully functional, False otherwise. - Inputs: - `model` (Any; required): A model loaded via mlx_lm.load() - Return annotation: `bool` - Calls: hasattr, getattr, logger.warning, inspect.signature, type, callable, logger.info, len - Return expressions: False; True # Module `vllm_mlx.patches.qwen3_next_mtp` Runtime MTP (Multi-Token Prediction) support for Qwen3-Next models. Qwen3-Next models may include a built-in MTP head that predicts token n+2 from hidden states + token n+1. MTP weights are added to the quantized MLX model via scripts/add_mtp_weights.py. Since mlx_lm's qwen3_next.py does NOT define MTP module/methods, this module provides: - inject_mtp_support(): dynamically creates MTP module, loads weights, and monkey-patches the model class with return_hidden, mtp_forward, and make_mtp_cache - validate_mtp_support(): checks whether a loaded model has working MTP The actual MTP scheduling logic lives in: - vllm_mlx/scheduler.py (_install_mtp, _mtp_step, _mtp_next) Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/patches/qwen3_next_mtp.py#L1-L261 ## `vllm_mlx.patches.qwen3_next_mtp.inject_mtp_support` - Kind: function - Signature: `def inject_mtp_support(model: Any, model_path, config: dict) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/patches/qwen3_next_mtp.py#L27-L181 - Implementation: Function `inject_mtp_support` calls `config.get`, `logger.info`, `Path`, `mtp_file.exists`; has 2 explicit return paths. Inject MTP module into a loaded Qwen3-Next model. mlx_lm's qwen3_next.py does not define MTP layers, so we: 1. Create MTP module matching the weight structure 2. Quantize it to match the base model 3. Load MTP weights from model-mtp.safetensors 4. Monkey-patch Model with return_hidden, mtp_forward, make_mtp_cache Args: model: A model loaded via mlx_lm (strict=False, MTP weights ignored) model_path: Path to model directory (contains model-mtp.safetensors) config: Parsed config.json dict Returns: True if MTP was successfully injected, False otherwise. - Inputs: - `model` (Any; required): A model loaded via mlx_lm (strict=False, MTP weights ignored) - `model_path` (not annotated; required): Path to model directory (contains model-mtp.safetensors) - `config` (dict; required): Parsed config.json dict - Return annotation: `bool` - Calls: config.get, logger.info, Path, mtp_file.exists, logger.warning, _MTPModule, quant_config.get, nn.quantize, mx.load, str, k.removeprefix, raw.items, k.startswith, mtp.load_weights, list, mtp_weights.items, mx.eval, mtp.parameters, len - Return expressions: False; True ## `vllm_mlx.patches.qwen3_next_mtp.inject_mtp_support._MTPModule` - Kind: nested class - Signature: `class _MTPModule(nn.Module)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/patches/qwen3_next_mtp.py#L68-L83 - Implementation: Nested Class `inject_mtp_support._MTPModule` derives from `nn.Module` and declares 1 direct member(s). Nested Class `inject_mtp_support._MTPModule` derives from `nn.Module` and declares 1 direct member(s). - Inputs: - `args` (not annotated; required): Required positional or keyword input. - `n_layers` (not annotated; required): Required positional or keyword input. - Constructs: `vllm_mlx.patches.qwen3_next_mtp.inject_mtp_support._MTPModule` ## `vllm_mlx.patches.qwen3_next_mtp.inject_mtp_support._MTPModule.__init__` - Kind: nested function - Signature: `def __init__(self, args, n_layers)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/patches/qwen3_next_mtp.py#L69-L83 - Implementation: Nested Function `inject_mtp_support._MTPModule.__init__` updates `self.pre_fc_norm_hidden`, `self.pre_fc_norm_embedding`, `self.fc`, `self.layers`; calls `super().__init__`, `super`, `nn.RMSNorm`, `nn.Linear`. Nested Function `inject_mtp_support._MTPModule.__init__` updates `self.pre_fc_norm_hidden`, `self.pre_fc_norm_embedding`, `self.fc`, `self.layers`; calls `super().__init__`, `super`, `nn.RMSNorm`, `nn.Linear`. - Inputs: - `args` (not annotated; required): Required positional or keyword input. - `n_layers` (not annotated; required): Required positional or keyword input. - Return annotation: `not annotated` - Calls: super().__init__, super, nn.RMSNorm, nn.Linear, Qwen3NextDecoderLayer, range - State writes: self.pre_fc_norm_hidden, self.pre_fc_norm_embedding, self.fc, self.layers, self.norm ## `vllm_mlx.patches.qwen3_next_mtp.inject_mtp_support._mtp_quant_pred` - Kind: nested function - Signature: `def _mtp_quant_pred(path, module)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/patches/qwen3_next_mtp.py#L93-L103 - Implementation: Nested Function `inject_mtp_support._mtp_quant_pred` calls `isinstance`, `path.endswith`; has 3 explicit return paths. Nested Function `inject_mtp_support._mtp_quant_pred` calls `isinstance`, `path.endswith`; has 3 explicit return paths. - Inputs: - `path` (not annotated; required): Required positional or keyword input. - `module` (not annotated; required): Required positional or keyword input. - Return annotation: `not annotated` - Calls: isinstance, path.endswith - Return expressions: False; {'group_size': 64, 'bits': 8}; True ## `vllm_mlx.patches.qwen3_next_mtp.inject_mtp_support._Qwen3NextMTP` - Kind: nested class - Signature: `class _Qwen3NextMTP(original_class)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/patches/qwen3_next_mtp.py#L125-L177 - Implementation: Nested Class `inject_mtp_support._Qwen3NextMTP` derives from `original_class` and declares 3 direct member(s). Qwen3-Next with MTP support (injected at runtime). - Inputs: none - Constructs: `vllm_mlx.patches.qwen3_next_mtp.inject_mtp_support._Qwen3NextMTP` ## `vllm_mlx.patches.qwen3_next_mtp.inject_mtp_support._Qwen3NextMTP.__call__` - Kind: nested function - Signature: `def __call__(self, inputs, cache=None, return_hidden: bool=False)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/patches/qwen3_next_mtp.py#L128-L150 - Implementation: Nested Function `inject_mtp_support._Qwen3NextMTP.__call__` calls `inner.embed_tokens`, `len`, `create_attention_mask`, `create_ssm_mask`; has 2 explicit return paths. Nested Function `inject_mtp_support._Qwen3NextMTP.__call__` calls `inner.embed_tokens`, `len`, `create_attention_mask`, `create_ssm_mask`; has 2 explicit return paths. - Inputs: - `inputs` (not annotated; required): Required positional or keyword input. - `cache` (not annotated; optional; default `None`): Optional positional or keyword input; defaults to `None`. - `return_hidden` (bool; optional; default `False`): Optional positional or keyword input; defaults to `False`. - Return annotation: `not annotated` - Calls: inner.embed_tokens, len, create_attention_mask, create_ssm_mask, zip, layer, inner.norm, inner.embed_tokens.as_linear, self.lm_head - State reads: self.model, self.args.tie_word_embeddings, self.args, self.lm_head - Return expressions: (out, hidden_states); out ## `vllm_mlx.patches.qwen3_next_mtp.inject_mtp_support._Qwen3NextMTP.mtp_forward` - Kind: nested function - Signature: `def mtp_forward(self, hidden_states, next_token_ids, cache=None, mtp_cache=None)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/patches/qwen3_next_mtp.py#L152-L171 - Implementation: Nested Function `inject_mtp_support._Qwen3NextMTP.mtp_forward` calls `self.model.embed_tokens`, `self.mtp.pre_fc_norm_hidden`, `self.mtp.pre_fc_norm_embedding`, `self.mtp.fc`; has 2 explicit return paths. Run MTP head: predict token n+2 from hidden states + token n+1. - Inputs: - `hidden_states` (not annotated; required): Required positional or keyword input. - `next_token_ids` (not annotated; required): Required positional or keyword input. - `cache` (not annotated; optional; default `None`): Optional positional or keyword input; defaults to `None`. - `mtp_cache` (not annotated; optional; default `None`): Optional positional or keyword input; defaults to `None`. - Return annotation: `not annotated` - Calls: self.model.embed_tokens, self.mtp.pre_fc_norm_hidden, self.mtp.pre_fc_norm_embedding, self.mtp.fc, mx.concatenate, create_attention_mask, layer, self.mtp.norm, self.model.embed_tokens.as_linear, self.lm_head - State reads: self.model.embed_tokens, self.model, self.mtp.pre_fc_norm_hidden, self.mtp, self.mtp.pre_fc_norm_embedding, self.mtp.fc, self.mtp.layers, self.mtp.norm, self.args.tie_word_embeddings, self.args, self.model.embed_tokens.as_linear, self.lm_head - Return expressions: self.model.embed_tokens.as_linear(x); self.lm_head(x) ## `vllm_mlx.patches.qwen3_next_mtp.inject_mtp_support._Qwen3NextMTP.make_mtp_cache` - Kind: nested function - Signature: `def make_mtp_cache(self)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/patches/qwen3_next_mtp.py#L173-L177 - Implementation: Nested Function `inject_mtp_support._Qwen3NextMTP.make_mtp_cache` calls `KVCache`; has 2 explicit return paths. Create KV cache for MTP layers. - Inputs: none - Return annotation: `not annotated` - Calls: KVCache - State reads: self.mtp, self.mtp.layers - Return expressions: None; [KVCache() for _ in self.mtp.layers] ## `vllm_mlx.patches.qwen3_next_mtp.validate_mtp_support` - Kind: function - Signature: `def validate_mtp_support(model: Any) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/patches/qwen3_next_mtp.py#L184-L261 - Implementation: Function `validate_mtp_support` calls `getattr`, `logger.warning`, `logger.info`, `inspect.signature`; has 2 explicit return paths. Validate that a loaded model has working MTP support. Checks: 1. model.mtp exists and is not None (MTP module instantiated) 2. model.mtp has layers with loaded weights 3. model has return_hidden support in __call__ 4. model has mtp_forward method 5. model has make_mtp_cache method Args: model: A model loaded via mlx_lm.load() Returns: True if MTP is fully functional, False otherwise. - Inputs: - `model` (Any; required): A model loaded via mlx_lm.load() - Return annotation: `bool` - Calls: getattr, logger.warning, logger.info, inspect.signature, type, hasattr, callable, len - Return expressions: False; True # Module `vllm_mlx.plugin` vLLM Platform Plugin for MLX. This module provides the entry point for vLLM's platform plugin system, enabling automatic detection and activation of the MLX platform on Apple Silicon Macs. Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/plugin.py#L1-L155 ## `vllm_mlx.plugin.mlx_platform_plugin` - Kind: function - Signature: `def mlx_platform_plugin() -> str | None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/plugin.py#L17-L70 - Implementation: Function `mlx_platform_plugin` calls `logger.debug`, `platform.machine`, `mx.array`, `mx.sum`; has 2 explicit return paths. Platform plugin entry point for vLLM. This function is called by vLLM's platform detection system to determine if the MLX platform should be activated. Returns: str: Fully qualified class name of MLXPlatform if conditions are met None: If MLX platform should not be activated - Inputs: none - Return annotation: `str | None` - Calls: logger.debug, platform.machine, mx.array, mx.sum, mx.default_device, getattr, logger.warning, logger.info - Return expressions: None; 'vllm_mlx.vllm_platform.MLXPlatform' ## `vllm_mlx.plugin.is_mlx_available` - Kind: function - Signature: `def is_mlx_available() -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/plugin.py#L73-L80 - Implementation: Function `is_mlx_available` calls `mlx_platform_plugin`; returns `mlx_platform_plugin() is not None`. Check if MLX platform can be used. Returns: bool: True if MLX is available and working - Inputs: none - Return annotation: `bool` - Calls: mlx_platform_plugin - Return expressions: mlx_platform_plugin() is not None ## `vllm_mlx.plugin.get_mlx_device_info` - Kind: function - Signature: `def get_mlx_device_info() -> dict` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/plugin.py#L83-L155 - Implementation: Function `get_mlx_device_info` calls `is_mlx_available`, `subprocess.run`, `result.stdout.strip`, `int`; returns `info`. Get information about the MLX device. Returns: dict: Device information including chip name, memory, etc. - Inputs: none - Return annotation: `dict` - Calls: is_mlx_available, subprocess.run, result.stdout.strip, int, getattr - Return expressions: info # Module `vllm_mlx.prefix_cache` Prefix Cache Manager for vllm-mlx. Wraps mlx-lm's LRUPromptCache to provide prefix caching functionality, allowing reuse of computed KV cache for common prompt prefixes. This module provides two implementations: - PrefixCacheManager: Original trie-based LRU cache (for backward compatibility) - BlockAwarePrefixCache: Block-based cache with PagedCacheManager integration Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/prefix_cache.py#L1-L1039 ## `vllm_mlx.prefix_cache.CacheEntry` - Kind: class - Signature: `class CacheEntry` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/prefix_cache.py#L33-L37 - Implementation: Class `CacheEntry` declares 0 direct member(s). Entry in the prefix cache. - Inputs: - `prompt_cache` (List[Any]; required): Required constructor field. - `count` (int; required): Required constructor field. - Constructs: `vllm_mlx.prefix_cache.CacheEntry` - Decorators: dataclass ## `vllm_mlx.prefix_cache.PrefixCacheStats` - Kind: class - Signature: `class PrefixCacheStats` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/prefix_cache.py#L41-L66 - Implementation: Class `PrefixCacheStats` declares 2 direct member(s). Statistics for prefix cache performance. - Inputs: - `hits` (int; optional; default `0`): Optional constructor field; defaults to `0`. - `misses` (int; optional; default `0`): Optional constructor field; defaults to `0`. - `tokens_saved` (int; optional; default `0`): Optional constructor field; defaults to `0`. - `total_queries` (int; optional; default `0`): Optional constructor field; defaults to `0`. - `evictions` (int; optional; default `0`): Optional constructor field; defaults to `0`. - Constructs: `vllm_mlx.prefix_cache.PrefixCacheStats` - Decorators: dataclass ## `vllm_mlx.prefix_cache.PrefixCacheStats.hit_rate` - Kind: method - Signature: `def hit_rate(self) -> float` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/prefix_cache.py#L51-L55 - Implementation: Method `PrefixCacheStats.hit_rate` has 2 explicit return paths. Calculate cache hit rate. - Inputs: none - Return annotation: `float` - Decorators: property - State reads: self.total_queries, self.hits - Return expressions: 0.0; self.hits / self.total_queries ## `vllm_mlx.prefix_cache.PrefixCacheStats.to_dict` - Kind: method - Signature: `def to_dict(self) -> Dict[str, Any]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/prefix_cache.py#L57-L66 - Implementation: Method `PrefixCacheStats.to_dict` returns `{'hits': self.hits, 'misses': self.misses, 'hit_rate': self.hit_rate, 'tokens_saved': self.tokens_saved, 'total_queries…`. Convert stats to dictionary. - Inputs: none - Return annotation: `Dict[str, Any]` - State reads: self.hits, self.misses, self.hit_rate, self.tokens_saved, self.total_queries, self.evictions - Return expressions: {'hits': self.hits, 'misses': self.misses, 'hit_rate': self.hit_rate, 'tokens_saved': self.tokens_saved, 'total_queries… ## `vllm_mlx.prefix_cache.PrefixCacheManager` - Kind: class - Signature: `class PrefixCacheManager` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/prefix_cache.py#L69-L355 - Implementation: Class `PrefixCacheManager` declares 14 direct member(s). Manages prefix caching for vllm-mlx using a trie-based LRU cache. This implementation is inspired by mlx-lm's LRUPromptCache but adapted for vllm-mlx's batching architecture. The cache stores KV states keyed by token sequences, allowing: - Exact match: Full prompt found in cache - Shorter match: Partial prefix found, process remaining tokens - Longer match: Cached prefix longer than request, trim excess Example: cache_manager = PrefixCacheManager(model, max_entries=100) # Check for cached prefix cache, remaining_tokens = cache_manager.fetch_cache(tokens) if cache: # Use cached KV, only process remaining_tokens pass # After generation, store cache for reuse cache_manager.store_cache(full_tokens, prompt_cache) - Inputs: - `model` (Any; required): The MLX model (used for cache key identification) - `max_entries` (int; optional; default `100`): Maximum number of cached entries before LRU eviction - Constructs: `vllm_mlx.prefix_cache.PrefixCacheManager` ## `vllm_mlx.prefix_cache.PrefixCacheManager.__init__` - Kind: method - Signature: `def __init__(self, model: Any, max_entries: int=100)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/prefix_cache.py#L94-L115 - Implementation: Method `PrefixCacheManager.__init__` updates `self.model`, `self.model_key`, `self.max_size`, `self._cache`; calls `id`, `OrderedDict`, `PrefixCacheStats`. Initialize the prefix cache manager. Args: model: The MLX model (used for cache key identification) max_entries: Maximum number of cached entries before LRU eviction - Inputs: - `model` (Any; required): The MLX model (used for cache key identification) - `max_entries` (int; optional; default `100`): Maximum number of cached entries before LRU eviction - Return annotation: `not annotated` - Calls: id, OrderedDict, PrefixCacheStats - State writes: self.model, self.model_key, self.max_size, self._cache, self._lru, self.stats ## `vllm_mlx.prefix_cache.PrefixCacheManager._search` - Kind: method - Signature: `def _search(self, tokens: List[int]) -> Tuple[Optional[List[int]], Optional[List[int]], Optional[List[int]], int]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/prefix_cache.py#L117-L164 - Implementation: Method `PrefixCacheManager._search` calls `enumerate`, `list`, `path.append`, `stack.pop`; has 4 explicit return paths. Search for cached prefix matching tokens. Returns: Tuple of (exact, shorter, longer, common_prefix_len) - exact: Tokens if exact match found - shorter: Tokens of shorter cached prefix - longer: Tokens of longer cached prefix - common_prefix_len: Length of common prefix with longer match - Inputs: - `tokens` (List[int]; required): Required positional or keyword input. - Return annotation: `Tuple[Optional[List[int]], Optional[List[int]], Optional[List[int]], int]` - Calls: enumerate, list, path.append, stack.pop, len, node.items, stack.append - State reads: self.model_key, self._cache - Return expressions: (None, None, None, 0); (None, list(path), None, 0); (list(tokens), None, None, 0); (None, None, node_path, len(tokens)) ## `vllm_mlx.prefix_cache.PrefixCacheManager.fetch_cache` - Kind: method - Signature: `def fetch_cache(self, tokens: List[int]) -> Tuple[Optional[List[Any]], List[int]]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/prefix_cache.py#L166-L221 - Implementation: Method `PrefixCacheManager.fetch_cache` updates `self.stats.total_queries`, `self.stats.hits`, `self.stats.tokens_saved`, `self.stats.misses`; calls `tuple`, `self._search`, `self._get_cache_entry`, `len`; has 4 explicit return paths. Find cached prefix for the given tokens. Args: tokens: Input token sequence Returns: Tuple of (cache, remaining_tokens) - cache: Cached KV state if found, None otherwise - remaining_tokens: Tokens that still need processing - Inputs: - `tokens` (List[int]; required): Input token sequence - Return annotation: `Tuple[Optional[List[Any]], List[int]]` - Calls: tuple, self._search, self._get_cache_entry, len, self._touch_lru, self._can_trim_cache, self._trim_cache, copy.deepcopy - State reads: self.stats, self._search, self._get_cache_entry, self._touch_lru, self._can_trim_cache, self._trim_cache - State writes: self.stats.total_queries, self.stats.hits, self.stats.tokens_saved, self.stats.misses - Return expressions: (cache_entry.prompt_cache, []); (cache_entry.prompt_cache, remaining); (trimmed_cache, []); (None, tokens) ## `vllm_mlx.prefix_cache.PrefixCacheManager.store_cache` - Kind: method - Signature: `def store_cache(self, tokens: List[int], prompt_cache: List[Any]) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/prefix_cache.py#L223-L258 - Implementation: Method `PrefixCacheManager.store_cache` calls `tuple`, `self._lru.move_to_end`, `CacheEntry`, `len`; returns `None`. Store computed cache for future reuse. Args: tokens: Token sequence that was processed prompt_cache: The computed KV cache to store - Inputs: - `tokens` (List[int]; required): Token sequence that was processed - `prompt_cache` (List[Any]; required): The computed KV cache to store - Return annotation: `None` - Calls: tuple, self._lru.move_to_end, CacheEntry, len, self._evict_lru - State reads: self.model_key, self._cache, self._lru.move_to_end, self._lru, self.max_size, self._evict_lru - Return expressions: None ## `vllm_mlx.prefix_cache.PrefixCacheManager._get_cache_entry` - Kind: method - Signature: `def _get_cache_entry(self, tokens: List[int]) -> Optional[CacheEntry]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/prefix_cache.py#L260-L271 - Implementation: Method `PrefixCacheManager._get_cache_entry` calls `current.get`; has 2 explicit return paths. Get cache entry for given tokens. - Inputs: - `tokens` (List[int]; required): Required positional or keyword input. - Return annotation: `Optional[CacheEntry]` - Calls: current.get - State reads: self.model_key, self._cache - Return expressions: None; current.get('cache') ## `vllm_mlx.prefix_cache.PrefixCacheManager._touch_lru` - Kind: method - Signature: `def _touch_lru(self, tokens_tuple: tuple) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/prefix_cache.py#L273-L279 - Implementation: Method `PrefixCacheManager._touch_lru` calls `self._lru.move_to_end`. Move entry to most-recently-used position — O(1) with OrderedDict. - Inputs: - `tokens_tuple` (tuple; required): Required positional or keyword input. - Return annotation: `None` - Calls: self._lru.move_to_end - State reads: self.model_key, self._lru, self._lru.move_to_end ## `vllm_mlx.prefix_cache.PrefixCacheManager._evict_lru` - Kind: method - Signature: `def _evict_lru(self) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/prefix_cache.py#L281-L288 - Implementation: Method `PrefixCacheManager._evict_lru` updates `self.stats.evictions`; calls `self._lru.popitem`, `self._delete_cache`, `list`; returns `None`. Evict least recently used entry — O(1) popitem from OrderedDict. - Inputs: none - Return annotation: `None` - Calls: self._lru.popitem, self._delete_cache, list - State reads: self._lru, self._lru.popitem, self._delete_cache, self.stats - State writes: self.stats.evictions - Return expressions: None ## `vllm_mlx.prefix_cache.PrefixCacheManager._delete_cache` - Kind: method - Signature: `def _delete_cache(self, model_key: Any, tokens: List[int]) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/prefix_cache.py#L290-L314 - Implementation: Method `PrefixCacheManager._delete_cache` calls `path.append`, `range`, `len`; returns `None`. Delete cache entry and clean up empty trie branches. - Inputs: - `model_key` (Any; required): Required positional or keyword input. - `tokens` (List[int]; required): Required positional or keyword input. - Return annotation: `None` - Calls: path.append, range, len - State reads: self._cache - Return expressions: None ## `vllm_mlx.prefix_cache.PrefixCacheManager._can_trim_cache` - Kind: method - Signature: `def _can_trim_cache(self, prompt_cache: List[Any]) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/prefix_cache.py#L316-L330 - Implementation: Method `PrefixCacheManager._can_trim_cache` calls `hasattr`, `first_cache.is_trimmable`, `logger.debug`; has 3 explicit return paths. Check if cache can be trimmed. - Inputs: - `prompt_cache` (List[Any]; required): Required positional or keyword input. - Return annotation: `bool` - Calls: hasattr, first_cache.is_trimmable, logger.debug - Return expressions: False; trimmable; hasattr(first_cache, 'trim') ## `vllm_mlx.prefix_cache.PrefixCacheManager._trim_cache` - Kind: method - Signature: `def _trim_cache(self, prompt_cache: List[Any], num_tokens: int) -> List[Any]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/prefix_cache.py#L332-L337 - Implementation: Method `PrefixCacheManager._trim_cache` calls `hasattr`, `cache.trim`; returns `prompt_cache`. Trim cache by removing num_tokens from the end. - Inputs: - `prompt_cache` (List[Any]; required): Required positional or keyword input. - `num_tokens` (int; required): Required positional or keyword input. - Return annotation: `List[Any]` - Calls: hasattr, cache.trim - Return expressions: prompt_cache ## `vllm_mlx.prefix_cache.PrefixCacheManager.get_stats` - Kind: method - Signature: `def get_stats(self) -> Dict[str, Any]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/prefix_cache.py#L339-L341 - Implementation: Method `PrefixCacheManager.get_stats` calls `self.stats.to_dict`; returns `self.stats.to_dict()`. Get cache statistics. - Inputs: none - Return annotation: `Dict[str, Any]` - Calls: self.stats.to_dict - State reads: self.stats.to_dict, self.stats - Return expressions: self.stats.to_dict() ## `vllm_mlx.prefix_cache.PrefixCacheManager.reset_stats` - Kind: method - Signature: `def reset_stats(self) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/prefix_cache.py#L343-L345 - Implementation: Method `PrefixCacheManager.reset_stats` updates `self.stats`; calls `PrefixCacheStats`. Reset statistics. - Inputs: none - Return annotation: `None` - Calls: PrefixCacheStats - State writes: self.stats ## `vllm_mlx.prefix_cache.PrefixCacheManager.clear` - Kind: method - Signature: `def clear(self) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/prefix_cache.py#L347-L351 - Implementation: Method `PrefixCacheManager.clear` calls `self._cache.clear`, `self._lru.clear`, `self.reset_stats`. Clear all cached entries. - Inputs: none - Return annotation: `None` - Calls: self._cache.clear, self._lru.clear, self.reset_stats - State reads: self._cache.clear, self._cache, self._lru.clear, self._lru, self.reset_stats ## `vllm_mlx.prefix_cache.PrefixCacheManager.__len__` - Kind: method - Signature: `def __len__(self) -> int` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/prefix_cache.py#L353-L355 - Implementation: Method `PrefixCacheManager.__len__` calls `len`; returns `len(self._lru)`. Return number of cached entries. - Inputs: none - Return annotation: `int` - Calls: len - State reads: self._lru - Return expressions: len(self._lru) ## `vllm_mlx.prefix_cache.BlockCacheEntry` - Kind: class - Signature: `class BlockCacheEntry` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/prefix_cache.py#L364-L369 - Implementation: Class `BlockCacheEntry` declares 0 direct member(s). Entry mapping a token sequence to cache blocks. - Inputs: - `block_table` (BlockTable; required): Required constructor field. - `cache_data` (List[Any]; required): Required constructor field. - `last_access` (float; required): Required constructor field. - Constructs: `vllm_mlx.prefix_cache.BlockCacheEntry` - Decorators: dataclass ## `vllm_mlx.prefix_cache.BlockAwarePrefixCache` - Kind: class - Signature: `class BlockAwarePrefixCache` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/prefix_cache.py#L372-L1039 - Implementation: Class `BlockAwarePrefixCache` declares 17 direct member(s). Prefix cache that uses PagedCacheManager for block-based storage. Features: - Block-level prefix sharing (64 tokens per block) - Copy-on-Write for efficient forking - Hash-based deduplication across requests - Reference counting for memory efficiency This is the recommended cache for production use when memory efficiency for concurrent requests is important. Example: paged_manager = PagedCacheManager(block_size=64, max_blocks=1000) cache = BlockAwarePrefixCache(model, paged_manager) # Check for cached prefix block_table, remaining_tokens = cache.fetch_cache(request_id, tokens) # After generation, store cache cache.store_cache(request_id, tokens, kv_cache_data) # Clean up when request completes cache.release_cache(request_id) - Inputs: - `model` (Any; required): The MLX model (used for identification) - `paged_cache_manager` (PagedCacheManager; required): The PagedCacheManager instance for block management - Constructs: `vllm_mlx.prefix_cache.BlockAwarePrefixCache` ## `vllm_mlx.prefix_cache.BlockAwarePrefixCache.__init__` - Kind: method - Signature: `def __init__(self, model: Any, paged_cache_manager: PagedCacheManager)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/prefix_cache.py#L399-L426 - Implementation: Method `BlockAwarePrefixCache.__init__` updates `self.model`, `self.model_key`, `self.paged_cache`, `self.block_size`; calls `id`. Initialize block-aware prefix cache. Args: model: The MLX model (used for identification) paged_cache_manager: The PagedCacheManager instance for block management - Inputs: - `model` (Any; required): The MLX model (used for identification) - `paged_cache_manager` (PagedCacheManager; required): The PagedCacheManager instance for block management - Return annotation: `not annotated` - Calls: id - State writes: self.model, self.model_key, self.paged_cache, self.block_size, self._prefix_index, self._request_tables, self._hits, self._misses, self._tokens_saved ## `vllm_mlx.prefix_cache.BlockAwarePrefixCache.fetch_cache` - Kind: method - Signature: `def fetch_cache(self, request_id: str, tokens: List[int]) -> Tuple[Optional[BlockTable], List[int]]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/prefix_cache.py#L428-L502 - Implementation: Method `BlockAwarePrefixCache.fetch_cache` updates `self._hits`, `self._tokens_saved`, `self._misses`; calls `self.paged_cache.find_shared_prefix`, `self.paged_cache.create_block_table`, `self.paged_cache.increment_ref`, `self.paged_cache.allocated_blocks.get`; has 2 explicit return paths. Find cached prefix blocks for the given tokens. Args: request_id: Unique request identifier tokens: Input token sequence Returns: Tuple of (block_table, remaining_tokens) - block_table: BlockTable if prefix found, None otherwise - remaining_tokens: Tokens that need processing - Inputs: - `request_id` (str; required): Unique request identifier - `tokens` (List[int]; required): Input token sequence - Return annotation: `Tuple[Optional[BlockTable], List[int]]` - Calls: self.paged_cache.find_shared_prefix, self.paged_cache.create_block_table, self.paged_cache.increment_ref, self.paged_cache.allocated_blocks.get, block_table.block_ids.append, len, logger.debug, self._find_best_prefix_match - State reads: self.paged_cache.find_shared_prefix, self.paged_cache, self.paged_cache.create_block_table, self.paged_cache.increment_ref, self.paged_cache.allocated_blocks.get, self.paged_cache.allocated_blocks, self._find_best_prefix_match - State writes: self._hits, self._tokens_saved, self._misses - Return expressions: (None, tokens); (block_table, remaining) ## `vllm_mlx.prefix_cache.BlockAwarePrefixCache.store_cache` - Kind: method - Signature: `def store_cache(self, request_id: str, tokens: List[int], cache_data: List[Any]) -> Optional[BlockTable]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/prefix_cache.py#L504-L628 - Implementation: Method `BlockAwarePrefixCache.store_cache` calls `isinstance`, `len`, `self.paged_cache.get_block_table`, `self.paged_cache.create_block_table`; has 2 explicit return paths. Store computed cache for future reuse. This method stores actual tensor data (not references) when cache_data contains extracted states from mlx-lm's KVCache.state property. Args: request_id: Unique request identifier tokens: Token sequence that was processed cache_data: The computed KV cache to store. Can be: - List of KVCache objects (legacy, stores references) - List of dicts with 'state': (keys, values) tensors (new, stores slices) Returns: BlockTable for the stored cache, or None on failure - Inputs: - `request_id` (str; required): Unique request identifier - `tokens` (List[int]; required): Token sequence that was processed - `cache_data` (List[Any]; required): The computed KV cache to store. Can be: - List of KVCache objects (legacy, stores references) - List of dicts with 'state': (keys, values) tensors (new, stores slices) - Return annotation: `Optional[BlockTable]` - Calls: isinstance, len, self.paged_cache.get_block_table, self.paged_cache.create_block_table, range, min, self.paged_cache.find_cached_block, self.paged_cache.increment_ref, block_table.block_ids.append, self.paged_cache.allocate_block, self.paged_cache.handle_memory_pressure, logger.warning, self._extract_block_tensor_slice, logger.debug, self.paged_cache.register_block_hash, self._update_prefix_index, BlockCacheEntry, time.time, sum, self.paged_cache.allocated_blocks.get - State reads: self.paged_cache.get_block_table, self.paged_cache, self.paged_cache.create_block_table, self.block_size, self.paged_cache.find_cached_block, self.paged_cache.increment_ref, self.paged_cache.allocate_block, self.paged_cache.handle_memory_pressure, self._extract_block_tensor_slice, self.paged_cache.register_block_hash, self._update_prefix_index, self._request_tables, self.paged_cache.allocated_blocks.get, self.paged_cache.allocated_blocks - Return expressions: None; block_table ## `vllm_mlx.prefix_cache.BlockAwarePrefixCache._extract_block_tensor_slice` - Kind: method - Signature: `def _extract_block_tensor_slice(self, cache_data: List[Dict[str, Any]], start_idx: int, end_idx: int, total_tokens: int) -> Optional[List[Optional[Dict[str, Any]]]]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/prefix_cache.py#L630-L702 - Implementation: Method `BlockAwarePrefixCache._extract_block_tensor_slice` calls `block_slices.append`, `layer_state.get`, `self._cache_state_seq_axis`, `self._slice_concat_cache_state`; has 2 explicit return paths. Extract per-layer cache data for a single block. Args: cache_data: List of extracted layer states start_idx: Start token index in the sequence end_idx: End token index in the sequence total_tokens: Total number of tokens covered by cache_data Returns: Per-layer block cache state, or None on failure - Inputs: - `cache_data` (List[Dict[str, Any]]; required): List of extracted layer states - `start_idx` (int; required): Start token index in the sequence - `end_idx` (int; required): End token index in the sequence - `total_tokens` (int; required): Total number of tokens covered by cache_data - Return annotation: `Optional[List[Optional[Dict[str, Any]]]]` - Calls: block_slices.append, layer_state.get, self._cache_state_seq_axis, self._slice_concat_cache_state, any, logger.warning - State reads: self._cache_state_seq_axis, self._slice_concat_cache_state - Return expressions: None; block_slices if any((entry is not None for entry in block_slices)) else None ## `vllm_mlx.prefix_cache.BlockAwarePrefixCache._cache_state_seq_axis` - Kind: method - Signature: `def _cache_state_seq_axis(self, state: Any) -> Optional[int]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/prefix_cache.py#L704-L725 - Implementation: Method `BlockAwarePrefixCache._cache_state_seq_axis` calls `isinstance`, `len`, `hasattr`, `next`; has 3 explicit return paths. Return the sequence axis for cache states that support block concat. - Inputs: - `state` (Any; required): Required positional or keyword input. - Return annotation: `Optional[int]` - Calls: isinstance, len, hasattr, next, iter - Return expressions: None; 2; 1 ## `vllm_mlx.prefix_cache.BlockAwarePrefixCache._slice_concat_cache_state` - Kind: method - Signature: `def _slice_concat_cache_state(self, state: Tuple[Any, ...] | List[Any], start_idx: int, end_idx: int) -> Tuple[Any, ...] | List[Any]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/prefix_cache.py#L727-L751 - Implementation: Method `BlockAwarePrefixCache._slice_concat_cache_state` calls `self._cache_state_seq_axis`, `ValueError`, `min`, `_slice_tensor`; can raise `ValueError`; returns `tuple(sliced) if isinstance(state, tuple) else sliced`. Slice a sequence-backed cache state across the token axis. - Inputs: - `state` (Tuple[Any, ...] | List[Any]; required): Required positional or keyword input. - `start_idx` (int; required): Required positional or keyword input. - `end_idx` (int; required): Required positional or keyword input. - Return annotation: `Tuple[Any, ...] | List[Any]` - Calls: self._cache_state_seq_axis, ValueError, min, _slice_tensor, isinstance, tuple - State reads: self._cache_state_seq_axis - Raises directly: ValueError - Return expressions: tuple(sliced) if isinstance(state, tuple) else sliced ## `vllm_mlx.prefix_cache.BlockAwarePrefixCache._slice_concat_cache_state._slice_tensor` - Kind: nested function - Signature: `def _slice_tensor(tensor: Any) -> Any` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/prefix_cache.py#L745-L748 - Implementation: Nested Function `BlockAwarePrefixCache._slice_concat_cache_state._slice_tensor` calls `slice`, `len`, `tuple`; returns `tensor[tuple(slices)]`. Nested Function `BlockAwarePrefixCache._slice_concat_cache_state._slice_tensor` calls `slice`, `len`, `tuple`; returns `tensor[tuple(slices)]`. - Inputs: - `tensor` (Any; required): Required positional or keyword input. - Return annotation: `Any` - Calls: slice, len, tuple - Return expressions: tensor[tuple(slices)] ## `vllm_mlx.prefix_cache.BlockAwarePrefixCache._concat_cache_states` - Kind: method - Signature: `def _concat_cache_states(self, states: List[Tuple[Any, ...] | List[Any]], seq_axis: int) -> Optional[Tuple[Any, ...] | List[Any]]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/prefix_cache.py#L753-L768 - Implementation: Method `BlockAwarePrefixCache._concat_cache_states` calls `len`, `range`, `any`, `concatenated.append`; has 2 explicit return paths. Concatenate state fragments for a sequence-backed cache layer. - Inputs: - `states` (List[Tuple[Any, ...] | List[Any]]; required): Required positional or keyword input. - `seq_axis` (int; required): Required positional or keyword input. - Return annotation: `Optional[Tuple[Any, ...] | List[Any]]` - Calls: len, range, any, concatenated.append, mx.concatenate, isinstance, tuple - Return expressions: None; tuple(concatenated) if isinstance(states[0], tuple) else concatenated ## `vllm_mlx.prefix_cache.BlockAwarePrefixCache.get_cache_for_generation` - Kind: method - Signature: `def get_cache_for_generation(self, request_id: str) -> Tuple[Optional[List[Any]], bool]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/prefix_cache.py#L770-L799 - Implementation: Method `BlockAwarePrefixCache.get_cache_for_generation` calls `self._request_tables.get`, `self.paged_cache.get_blocks_for_generation`, `copy.deepcopy`, `time.time`; has 2 explicit return paths. Get cache data for generation, applying COW if needed. Args: request_id: Request identifier Returns: Tuple of (cache_data, was_copied) - Inputs: - `request_id` (str; required): Request identifier - Return annotation: `Tuple[Optional[List[Any]], bool]` - Calls: self._request_tables.get, self.paged_cache.get_blocks_for_generation, copy.deepcopy, time.time - State reads: self._request_tables.get, self._request_tables, self.paged_cache.get_blocks_for_generation, self.paged_cache - Return expressions: (None, False); (cache_data, was_copied) ## `vllm_mlx.prefix_cache.BlockAwarePrefixCache.release_cache` - Kind: method - Signature: `def release_cache(self, request_id: str) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/prefix_cache.py#L801-L811 - Implementation: Method `BlockAwarePrefixCache.release_cache` calls `self._request_tables.pop`, `self.paged_cache.delete_block_table`, `logger.debug`. Release cache blocks for a completed request. Args: request_id: Request identifier - Inputs: - `request_id` (str; required): Request identifier - Return annotation: `None` - Calls: self._request_tables.pop, self.paged_cache.delete_block_table, logger.debug - State reads: self._request_tables.pop, self._request_tables, self.paged_cache.delete_block_table, self.paged_cache ## `vllm_mlx.prefix_cache.BlockAwarePrefixCache.fork_cache` - Kind: method - Signature: `def fork_cache(self, source_request_id: str, new_request_id: str) -> Optional[BlockTable]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/prefix_cache.py#L813-L847 - Implementation: Method `BlockAwarePrefixCache.fork_cache` calls `self._request_tables.get`, `self.paged_cache.fork_block_table`, `BlockCacheEntry`, `time.time`; has 2 explicit return paths. Fork cache from one request to another (COW). Args: source_request_id: Source request ID new_request_id: New request ID Returns: Forked BlockTable, or None if source not found - Inputs: - `source_request_id` (str; required): Source request ID - `new_request_id` (str; required): New request ID - Return annotation: `Optional[BlockTable]` - Calls: self._request_tables.get, self.paged_cache.fork_block_table, BlockCacheEntry, time.time, logger.debug - State reads: self._request_tables.get, self._request_tables, self.paged_cache.fork_block_table, self.paged_cache - Return expressions: None; forked_table ## `vllm_mlx.prefix_cache.BlockAwarePrefixCache.reconstruct_cache` - Kind: method - Signature: `def reconstruct_cache(self, block_table: BlockTable) -> Optional[List[Any]]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/prefix_cache.py#L849-L967 - Implementation: Method `BlockAwarePrefixCache.reconstruct_cache` calls `logger.warning`, `self.paged_cache.allocated_blocks.get`, `logger.debug`, `all_block_data.append`; has 2 explicit return paths. Reconstruct cache objects from stored block tensor data. Sequence-backed caches are concatenated block-by-block. Recurrent caches such as ArraysCache are restored from the latest sequence boundary snapshot that was actually stored. Args: block_table: BlockTable containing block IDs to reconstruct from Returns: List of reconstructed KVCache objects (one per layer), or None if reconstruction fails - Inputs: - `block_table` (BlockTable; required): BlockTable containing block IDs to reconstruct from - Return annotation: `Optional[List[Any]]` - Calls: logger.warning, self.paged_cache.allocated_blocks.get, logger.debug, all_block_data.append, max, len, range, self._concat_cache_states, layer_meta.get, hasattr, _KVCache, self._cache_state_seq_axis, cache_cls.from_state, KVCache, reconstructed_caches.append, traceback.format_exc - State reads: self.paged_cache.allocated_blocks.get, self.paged_cache.allocated_blocks, self.paged_cache, self._concat_cache_states, self._cache_state_seq_axis - Return expressions: None; reconstructed_caches ## `vllm_mlx.prefix_cache.BlockAwarePrefixCache._find_best_prefix_match` - Kind: method - Signature: `def _find_best_prefix_match(self, tokens: List[int]) -> Optional[Tuple[List[int], List[int]]]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/prefix_cache.py#L969-L992 - Implementation: Method `BlockAwarePrefixCache._find_best_prefix_match` calls `range`, `len`, `self.paged_cache.compute_block_hash`; returns `best_match`. Find best matching prefix in the index. - Inputs: - `tokens` (List[int]; required): Required positional or keyword input. - Return annotation: `Optional[Tuple[List[int], List[int]]]` - Calls: range, len, self.paged_cache.compute_block_hash - State reads: self.block_size, self.paged_cache.compute_block_hash, self.paged_cache, self._prefix_index - Return expressions: best_match ## `vllm_mlx.prefix_cache.BlockAwarePrefixCache._update_prefix_index` - Kind: method - Signature: `def _update_prefix_index(self, tokens: List[int], block_ids: List[int]) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/prefix_cache.py#L994-L1005 - Implementation: Method `BlockAwarePrefixCache._update_prefix_index` calls `range`, `len`, `min`, `self.paged_cache.compute_block_hash`. Update prefix index with new token sequence. - Inputs: - `tokens` (List[int]; required): Required positional or keyword input. - `block_ids` (List[int]; required): Required positional or keyword input. - Return annotation: `None` - Calls: range, len, min, self.paged_cache.compute_block_hash - State reads: self.block_size, self.paged_cache.compute_block_hash, self.paged_cache, self._prefix_index ## `vllm_mlx.prefix_cache.BlockAwarePrefixCache.get_stats` - Kind: method - Signature: `def get_stats(self) -> Dict[str, Any]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/prefix_cache.py#L1007-L1021 - Implementation: Method `BlockAwarePrefixCache.get_stats` calls `self.paged_cache.get_memory_usage`, `len`; returns `{'hits': self._hits, 'misses': self._misses, 'hit_rate': self._hits / (self._hits + self._misses) if self._hits + self.…`. Get cache statistics. - Inputs: none - Return annotation: `Dict[str, Any]` - Calls: self.paged_cache.get_memory_usage, len - State reads: self.paged_cache.get_memory_usage, self.paged_cache, self._hits, self._misses, self._tokens_saved, self._request_tables - Return expressions: {'hits': self._hits, 'misses': self._misses, 'hit_rate': self._hits / (self._hits + self._misses) if self._hits + self.… ## `vllm_mlx.prefix_cache.BlockAwarePrefixCache.reset_stats` - Kind: method - Signature: `def reset_stats(self) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/prefix_cache.py#L1023-L1028 - Implementation: Method `BlockAwarePrefixCache.reset_stats` updates `self._hits`, `self._misses`, `self._tokens_saved`; calls `self.paged_cache.reset_stats`. Reset statistics. - Inputs: none - Return annotation: `None` - Calls: self.paged_cache.reset_stats - State reads: self.paged_cache.reset_stats, self.paged_cache - State writes: self._hits, self._misses, self._tokens_saved ## `vllm_mlx.prefix_cache.BlockAwarePrefixCache.clear` - Kind: method - Signature: `def clear(self) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/prefix_cache.py#L1030-L1035 - Implementation: Method `BlockAwarePrefixCache.clear` calls `self._request_tables.clear`, `self._prefix_index.clear`, `self.paged_cache.clear`, `self.reset_stats`. Clear all cached data. - Inputs: none - Return annotation: `None` - Calls: self._request_tables.clear, self._prefix_index.clear, self.paged_cache.clear, self.reset_stats - State reads: self._request_tables.clear, self._request_tables, self._prefix_index.clear, self._prefix_index, self.paged_cache.clear, self.paged_cache, self.reset_stats ## `vllm_mlx.prefix_cache.BlockAwarePrefixCache.__len__` - Kind: method - Signature: `def __len__(self) -> int` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/prefix_cache.py#L1037-L1039 - Implementation: Method `BlockAwarePrefixCache.__len__` calls `len`; returns `len(self._request_tables)`. Return number of active request entries. - Inputs: none - Return annotation: `int` - Calls: len - State reads: self._request_tables - Return expressions: len(self._request_tables) # Module `vllm_mlx.prompt_warmup` Prompt warm-up for vllm-mlx. At server startup, pre-populates the prefix cache by running one short generation per warm-up prompt. The first user request that shares a prefix with a warmed prompt sees cache-hit TTFT instead of cold prefill latency. File format (JSON): [ [{"role": "system", "content": "You are ..."}], [{"role": "system", "content": "..."}, {"role": "user", "content": "hi"}] ] Each entry is a list of chat messages — same shape as a ``/v1/chat/completions`` ``messages`` field. The warmer runs a ``max_tokens=1`` chat completion for each, which flows through the exact same path as a real request and writes the KV state to the prefix cache. Paths resolve from the current working directory. A single-message system prompt is sufficient if that is the shared prefix. Sizing note: prompts are warmed concurrently via ``asyncio.gather``, so N entries fire N concurrent prefills at startup. Each prefill allocates KV cache for its prompt length. For typical agent deployments 1–3 entries (one per active persona) cover the hot paths; a very large warm-prompts file on a memory-tight model can exhaust headroom at boot. Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/prompt_warmup.py#L1-L275 ## `vllm_mlx.prompt_warmup.load_warmup_file` - Kind: function - Signature: `def load_warmup_file(path: str) -> list[list[dict[str, Any]]]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/prompt_warmup.py#L41-L76 - Implementation: Function `load_warmup_file` calls `Path(path).expanduser`, `Path`, `p.exists`, `FileNotFoundError`; can raise `FileNotFoundError`, `ValueError`; returns `data`. Load and validate a warm-up prompts JSON file. Raises: FileNotFoundError: If the file does not exist. ValueError: If the file shape is invalid. - Inputs: - `path` (str; required): Required positional or keyword input. - Return annotation: `list[list[dict[str, Any]]]` - Calls: Path(path).expanduser, Path, p.exists, FileNotFoundError, json.loads, p.read_text, isinstance, ValueError, type, enumerate - Raises directly: FileNotFoundError, ValueError - Return expressions: data ## `vllm_mlx.prompt_warmup._ensure_user_terminator` - Kind: function - Signature: `def _ensure_user_terminator(messages: list[dict[str, Any]]) -> list[dict[str, Any]]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/prompt_warmup.py#L79-L91 - Implementation: Function `_ensure_user_terminator` calls `messages[-1].get`; has 2 explicit return paths. Ensure the message list ends with a user message. Some chat templates (Qwen3.6, DeepSeek-VL, a handful of others) require at least one user message or raise ``TemplateError: No user query found``. We prefer to cache just the system prefix, but when the template won't render without a user, append a minimal placeholder. The common prefix up to the start of user content still matches real requests, so the system tokens still get cached. - Inputs: - `messages` (list[dict[str, Any]]; required): Required positional or keyword input. - Return annotation: `list[dict[str, Any]]` - Calls: messages[-1].get - Return expressions: messages; [*messages, {'role': 'user', 'content': ' '}] ## `vllm_mlx.prompt_warmup._build_strict_prefix_string` - Kind: function - Signature: `def _build_strict_prefix_string(tokenizer: Any, messages: list[dict[str, Any]], enable_thinking: bool=True) -> str | None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/prompt_warmup.py#L94-L176 - Implementation: Function `_build_strict_prefix_string` calls `getattr`, `apply`, `_with_user`, `kwargs.pop`; has 2 explicit return paths. Build a STRING prefix that is a prefix of any real request's rendered chat template for the same system and empty chat history. Strategy: render the chat template twice with two DIFFERENT user contents and ``tokenize=False`` (matching what the server does). Truncate the first output at the position where the two strings diverge — that's where user content gets inserted. We return a STRING (not tokens) because the engine's request path also applies the template with ``tokenize=False`` and then lets the tokenizer encode the result. Going through the same pipeline guarantees the warm entry's tokens are a strict prefix of a real request's tokens. This enables warm-prompts on hybrid SSM+attention models where LCP matching is disabled (SSM state can't be trimmed) — they rely purely on strict PREFIX match. Returns None if rendering fails or the two probes don't diverge past a reasonable prefix length (unusual template). - Inputs: - `tokenizer` (Any; required): Required positional or keyword input. - `messages` (list[dict[str, Any]]; required): Required positional or keyword input. - `enable_thinking` (bool; optional; default `True`): Optional positional or keyword input; defaults to `True`. - Return annotation: `str | None` - Calls: getattr, apply, _with_user, kwargs.pop, isinstance, range, min, len - Return expressions: None; a[:boundary] ## `vllm_mlx.prompt_warmup._build_strict_prefix_string._with_user` - Kind: nested function - Signature: `def _with_user(user_content: str) -> list[dict[str, Any]]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/prompt_warmup.py#L121-L127 - Implementation: Nested Function `_build_strict_prefix_string._with_user` calls `dict`, `msgs[-1].get`; returns `msgs`. Nested Function `_build_strict_prefix_string._with_user` calls `dict`, `msgs[-1].get`; returns `msgs`. - Inputs: - `user_content` (str; required): Required positional or keyword input. - Return annotation: `list[dict[str, Any]]` - Calls: dict, msgs[-1].get - Return expressions: msgs ## `vllm_mlx.prompt_warmup.warm_prefix_cache` - Kind: function - Signature: `async def warm_prefix_cache(engine: Any, prompts: list[list[dict[str, Any]]], *, max_tokens: int=1) -> dict[str, Any]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/prompt_warmup.py#L179-L275 - Implementation: Function `warm_prefix_cache` calls `getattr`, `hasattr`, `time.perf_counter`, `asyncio.gather`; awaits asynchronous work; returns `{'count': completed, 'skipped': skipped, 'elapsed_ms': elapsed_ms, 'total_prompt_tokens': total_prompt_tokens, 'mode': …`. Run each prompt through the engine to populate the prefix cache. Prefers the strict-prefix path when the engine exposes a tokenizer: manually tokenize with ``add_generation_prompt=False`` and feed the raw token IDs to the engine's ``stream_generate`` (which accepts ``prompt: str | list[int]``). Real requests — which always use ``add_generation_prompt=True`` — will then find the warm entry as an exact strict prefix, independent of the engine's LCP matcher. This is the difference between warm-prompts helping dense models only and helping hybrid SSM+attention models too. Falls back to ``engine.stream_chat`` with a placeholder user message appended if no tokenizer is exposed — strict-prefix match won't apply there, so the feature is effectively LCP-only for that engine. Runs all prompts concurrently (``asyncio.gather``). Args: engine: The vllm-mlx engine (exposes ``stream_chat`` and optionally ``tokenizer`` + ``stream_generate``). prompts: List of message arrays. max_tokens: Tokens to generate per warm-up. 1 is enough. Returns: Dict with ``count``, ``skipped``, ``elapsed_ms``, ``total_prompt_tokens``, and ``mode`` (``"strict-prefix"`` or ``"chat-fallback"``) describing which path was used. - Inputs: - `engine` (Any; required): The vllm-mlx engine (exposes ``stream_chat`` and optionally ``tokenizer`` + ``stream_generate``). - `prompts` (list[list[dict[str, Any]]]; required): List of message arrays. - `max_tokens` (int; optional; default `1`): Tokens to generate per warm-up. 1 is enough. - Return annotation: `dict[str, Any]` - Calls: getattr, hasattr, time.perf_counter, asyncio.gather, runner, enumerate, sum - Return expressions: {'count': completed, 'skipped': skipped, 'elapsed_ms': elapsed_ms, 'total_prompt_tokens': total_prompt_tokens, 'mode': … ## `vllm_mlx.prompt_warmup.warm_prefix_cache._one_strict` - Kind: nested function - Signature: `async def _one_strict(idx: int, messages: list[dict[str, Any]]) -> tuple[int, int, str | None]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/prompt_warmup.py#L217-L235 - Implementation: Nested Function `warm_prefix_cache._one_strict` calls `_build_strict_prefix_string`, `_one_chat`, `engine.stream_generate`, `int`; awaits asynchronous work; has 3 explicit return paths. Nested Function `warm_prefix_cache._one_strict` calls `_build_strict_prefix_string`, `_one_chat`, `engine.stream_generate`, `int`; awaits asynchronous work; has 3 explicit return paths. - Inputs: - `idx` (int; required): Required positional or keyword input. - `messages` (list[dict[str, Any]]; required): Required positional or keyword input. - Return annotation: `tuple[int, int, str | None]` - Calls: _build_strict_prefix_string, _one_chat, engine.stream_generate, int, type, str, logger.warning - Return expressions: await _one_chat(idx, messages); (1, int(output.prompt_tokens or 0), None); (0, 0, 'no finished output') ## `vllm_mlx.prompt_warmup.warm_prefix_cache._one_chat` - Kind: nested function - Signature: `async def _one_chat(idx: int, messages: list[dict[str, Any]]) -> tuple[int, int, str | None]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/prompt_warmup.py#L237-L253 - Implementation: Nested Function `warm_prefix_cache._one_chat` calls `_ensure_user_terminator`, `engine.stream_chat`, `int`, `type`; has 3 explicit return paths. Nested Function `warm_prefix_cache._one_chat` calls `_ensure_user_terminator`, `engine.stream_chat`, `int`, `type`; has 3 explicit return paths. - Inputs: - `idx` (int; required): Required positional or keyword input. - `messages` (list[dict[str, Any]]; required): Required positional or keyword input. - Return annotation: `tuple[int, int, str | None]` - Calls: _ensure_user_terminator, engine.stream_chat, int, type, str, logger.warning - Return expressions: (1, int(output.prompt_tokens or 0), None); (0, 0, 'no finished output'); (0, 0, err) # Module `vllm_mlx.reasoning` Reasoning parser module for vllm-mlx. This module provides parsers for extracting reasoning/thinking content from model outputs. Supports models like Qwen3, DeepSeek-R1, etc. that use special tokens (e.g., ...) to separate reasoning from final responses. Usage: from vllm_mlx.reasoning import get_parser, list_parsers # Get a parser by name parser = get_parser("qwen3")() # Extract reasoning from complete output reasoning, content = parser.extract_reasoning(model_output) # For streaming parser.reset_state() for delta in stream: msg = parser.extract_reasoning_streaming(prev, curr, delta) if msg: # msg.reasoning and/or msg.content will be populated ... Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/reasoning/__init__.py#L1-L110 ## `vllm_mlx.reasoning.register_parser` - Kind: function - Signature: `def register_parser(name: str, parser_class: type[ReasoningParser]) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/reasoning/__init__.py#L34-L42 - Implementation: Function `register_parser` contains no state mutation, call, raise, return, await, or yield. Register a reasoning parser. Args: name: Name to register the parser under (e.g., "qwen3"). parser_class: The parser class to register. - Inputs: - `name` (str; required): Name to register the parser under (e.g., "qwen3"). - `parser_class` (type[ReasoningParser]; required): The parser class to register. - Return annotation: `None` ## `vllm_mlx.reasoning.get_parser` - Kind: function - Signature: `def get_parser(name: str) -> type[ReasoningParser]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/reasoning/__init__.py#L45-L63 - Implementation: Function `get_parser` calls `list`, `_REASONING_PARSERS.keys`, `KeyError`; can raise `KeyError`; returns `_REASONING_PARSERS[name]`. Get a reasoning parser class by name. Args: name: Name of the parser (e.g., "qwen3", "deepseek_r1"). Returns: The parser class (not an instance). Raises: KeyError: If parser name is not found. - Inputs: - `name` (str; required): Name of the parser (e.g., "qwen3", "deepseek_r1"). - Return annotation: `type[ReasoningParser]` - Calls: list, _REASONING_PARSERS.keys, KeyError - Raises directly: KeyError - Return expressions: _REASONING_PARSERS[name] ## `vllm_mlx.reasoning.list_parsers` - Kind: function - Signature: `def list_parsers() -> list[str]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/reasoning/__init__.py#L66-L73 - Implementation: Function `list_parsers` calls `list`, `_REASONING_PARSERS.keys`; returns `list(_REASONING_PARSERS.keys())`. List available parser names. Returns: List of registered parser names. - Inputs: none - Return annotation: `list[str]` - Calls: list, _REASONING_PARSERS.keys - Return expressions: list(_REASONING_PARSERS.keys()) ## `vllm_mlx.reasoning._register_builtin_parsers` - Kind: function - Signature: `def _register_builtin_parsers()` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/reasoning/__init__.py#L76-L94 - Implementation: Function `_register_builtin_parsers` calls `register_parser`. Register built-in parsers. - Inputs: none - Return annotation: `not annotated` - Calls: register_parser # Module `vllm_mlx.reasoning.base` Base classes for reasoning content extraction. This module provides the abstract base class for reasoning parsers that extract thinking/reasoning content from model outputs (e.g., ... tags). Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/reasoning/base.py#L1-L126 ## `vllm_mlx.reasoning.base.DeltaMessage` - Kind: class - Signature: `class DeltaMessage` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/reasoning/base.py#L15-L33 - Implementation: Class `DeltaMessage` declares 1 direct member(s). Delta message for streaming reasoning output. Contains either reasoning content, regular content, or both when transitioning from reasoning to content phase. Note: reasoning and content should typically not both be non-None except during the transition chunk. - Inputs: - `role` (str | None; optional; default `None`): Optional constructor field; defaults to `None`. - `content` (str | None; optional; default `None`): Optional constructor field; defaults to `None`. - `reasoning` (str | None; optional; default `None`): Optional constructor field; defaults to `None`. - Constructs: `vllm_mlx.reasoning.base.DeltaMessage` - Decorators: dataclass ## `vllm_mlx.reasoning.base.DeltaMessage.reasoning_content` - Kind: method - Signature: `def reasoning_content(self) -> str | None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/reasoning/base.py#L31-L33 - Implementation: Method `DeltaMessage.reasoning_content` returns `self.reasoning`. Deprecated: use reasoning instead. Maintained for backward compatibility. - Inputs: none - Return annotation: `str | None` - Decorators: property - State reads: self.reasoning - Return expressions: self.reasoning ## `vllm_mlx.reasoning.base.ReasoningParser` - Kind: class - Signature: `class ReasoningParser(ABC)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/reasoning/base.py#L36-L126 - Implementation: Class `ReasoningParser` derives from `ABC` and declares 5 direct member(s). Abstract base class for reasoning content extraction. Reasoning parsers extract thinking/reasoning content from model outputs, separating it from the final response content. This is useful for models like DeepSeek-R1, Qwen3, etc. that use special tokens to denote reasoning. Example: Input: "Let me solve this step by step...The answer is 42." Output: reasoning="Let me solve this step by step...", content="The answer is 42." - Inputs: - `tokenizer` (Any | None; optional; default `None`): Optional tokenizer for token-based parsing. For vllm-mlx, text-based parsing is sufficient, so this is optional. - Constructs: `vllm_mlx.reasoning.base.ReasoningParser` ## `vllm_mlx.reasoning.base.ReasoningParser.__init__` - Kind: method - Signature: `def __init__(self, tokenizer: Any | None=None)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/reasoning/base.py#L49-L57 - Implementation: Method `ReasoningParser.__init__` updates `self.tokenizer`. Initialize parser with optional tokenizer. Args: tokenizer: Optional tokenizer for token-based parsing. For vllm-mlx, text-based parsing is sufficient, so this is optional. - Inputs: - `tokenizer` (Any | None; optional; default `None`): Optional tokenizer for token-based parsing. For vllm-mlx, text-based parsing is sufficient, so this is optional. - Return annotation: `not annotated` - State writes: self.tokenizer ## `vllm_mlx.reasoning.base.ReasoningParser.extract_reasoning` - Kind: method - Signature: `def extract_reasoning(self, model_output: str) -> tuple[str | None, str | None]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/reasoning/base.py#L60-L74 - Implementation: Method `ReasoningParser.extract_reasoning` contains no state mutation, call, raise, return, await, or yield. Extract reasoning content from complete model output. Args: model_output: Complete text output from the model. Returns: Tuple of (reasoning_content, final_content). Either may be None if not present. - Inputs: - `model_output` (str; required): Complete text output from the model. - Return annotation: `tuple[str | None, str | None]` - Decorators: abstractmethod ## `vllm_mlx.reasoning.base.ReasoningParser.extract_reasoning_streaming` - Kind: method - Signature: `def extract_reasoning_streaming(self, previous_text: str, current_text: str, delta_text: str) -> DeltaMessage | None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/reasoning/base.py#L77-L100 - Implementation: Method `ReasoningParser.extract_reasoning_streaming` contains no state mutation, call, raise, return, await, or yield. Extract reasoning from streaming delta. Uses the "previous + delta = current" model where: - previous_text: All text accumulated before this delta - current_text: All text including this delta (previous + delta) - delta_text: Just the new text in this chunk Args: previous_text: Accumulated text before this delta. current_text: Accumulated text including this delta. delta_text: The new text in this streaming chunk. Returns: DeltaMessage with reasoning and/or content populated, or None if this delta should be skipped (e.g., special tokens). - Inputs: - `previous_text` (str; required): Accumulated text before this delta. - `current_text` (str; required): Accumulated text including this delta. - `delta_text` (str; required): The new text in this streaming chunk. - Return annotation: `DeltaMessage | None` - Decorators: abstractmethod ## `vllm_mlx.reasoning.base.ReasoningParser.reset_state` - Kind: method - Signature: `def reset_state(self)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/reasoning/base.py#L102-L110 - Implementation: Method `ReasoningParser.reset_state` contains no state mutation, call, raise, return, await, or yield. Reset any internal state for a new request. Called before starting to process a new streaming request. Override in subclasses if stateful parsing is needed. This is intentionally a default no-op implementation. - Inputs: none - Return annotation: `not annotated` ## `vllm_mlx.reasoning.base.ReasoningParser.finalize_stream` - Kind: method - Signature: `def finalize_stream(self) -> DeltaMessage | None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/reasoning/base.py#L112-L126 - Implementation: Method `ReasoningParser.finalize_stream` returns `None`. Finalize streaming state at end of stream. Called after the last delta is processed but before the stream closes. Parsers that buffer partial markers internally should flush any remaining text here. Default implementation is a no-op (returns None). Returns: DeltaMessage with any pending reasoning/content to emit, or None if nothing to flush. - Inputs: none - Return annotation: `DeltaMessage | None` - Return expressions: None # Module `vllm_mlx.reasoning.deepseek_r1_parser` Reasoning parser for DeepSeek-R1 models. DeepSeek-R1 uses ... tags for reasoning content. The model may sometimes start outputting reasoning without the explicit tag, so this parser is more lenient than Qwen3. Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/reasoning/deepseek_r1_parser.py#L1-L114 ## `vllm_mlx.reasoning.deepseek_r1_parser.DeepSeekR1ReasoningParser` - Kind: class - Signature: `class DeepSeekR1ReasoningParser(BaseThinkingReasoningParser)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/reasoning/deepseek_r1_parser.py#L14-L114 - Implementation: Class `DeepSeekR1ReasoningParser` derives from `BaseThinkingReasoningParser` and declares 4 direct member(s). Reasoning parser for DeepSeek-R1 model. DeepSeek-R1 uses ... tokens to denote reasoning text. This parser is more lenient than Qwen3: - The tag may not be explicitly generated (model assumes it) - If only is found, everything before it is reasoning Example: Input: "Step 1: analyze... Step 2: solve...The answer is 42." Output: reasoning="Step 1: analyze... Step 2: solve...", content="The answer is 42." Input: "reasoning contentfinal answer" # No opening tag Output: reasoning="reasoning content", content="final answer" - Inputs: none - Constructs: `vllm_mlx.reasoning.deepseek_r1_parser.DeepSeekR1ReasoningParser` ## `vllm_mlx.reasoning.deepseek_r1_parser.DeepSeekR1ReasoningParser.start_token` - Kind: method - Signature: `def start_token(self) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/reasoning/deepseek_r1_parser.py#L32-L35 - Implementation: Method `DeepSeekR1ReasoningParser.start_token` returns `''`. Return the marker that opens an explicit DeepSeek reasoning span. - Inputs: none - Return annotation: `str` - Decorators: property - Return expressions: '' ## `vllm_mlx.reasoning.deepseek_r1_parser.DeepSeekR1ReasoningParser.end_token` - Kind: method - Signature: `def end_token(self) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/reasoning/deepseek_r1_parser.py#L38-L41 - Implementation: Method `DeepSeekR1ReasoningParser.end_token` returns `''`. Return the marker that closes a DeepSeek reasoning span. - Inputs: none - Return annotation: `str` - Decorators: property - Return expressions: '' ## `vllm_mlx.reasoning.deepseek_r1_parser.DeepSeekR1ReasoningParser.extract_reasoning` - Kind: method - Signature: `def extract_reasoning(self, model_output: str) -> tuple[str | None, str | None]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/reasoning/deepseek_r1_parser.py#L43-L67 - Implementation: Method `DeepSeekR1ReasoningParser.extract_reasoning` calls `self._extract_complete_reasoning`, `super().extract_reasoning`, `super`; has 3 explicit return paths. Extract reasoning from DeepSeek-R1 output. More lenient than Qwen3 - handles cases where start tag is implicit. Args: model_output: Complete model output text. Returns: (reasoning, content) tuple. - Inputs: - `model_output` (str; required): Complete model output text. - Return annotation: `tuple[str | None, str | None]` - Calls: self._extract_complete_reasoning, super().extract_reasoning, super - State reads: self.end_token, self.start_token, self._extract_complete_reasoning - Return expressions: self._extract_complete_reasoning(model_output); (None, model_output); super().extract_reasoning(model_output) ## `vllm_mlx.reasoning.deepseek_r1_parser.DeepSeekR1ReasoningParser.extract_reasoning_streaming` - Kind: method - Signature: `def extract_reasoning_streaming(self, previous_text: str, current_text: str, delta_text: str) -> DeltaMessage | None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/reasoning/deepseek_r1_parser.py#L69-L114 - Implementation: Method `DeepSeekR1ReasoningParser.extract_reasoning_streaming` calls `super().extract_reasoning_streaming`, `super`, `delta_text.find`, `len`; has 2 explicit return paths. Extract reasoning from streaming delta. Handles DeepSeek-R1's pattern where may be implicit. Args: previous_text: Text accumulated before this delta. current_text: Text including this delta. delta_text: Just the new text. Returns: DeltaMessage with reasoning/content, or None to skip. - Inputs: - `previous_text` (str; required): Text accumulated before this delta. - `current_text` (str; required): Text including this delta. - `delta_text` (str; required): Just the new text. - Return annotation: `DeltaMessage | None` - Calls: super().extract_reasoning_streaming, super, delta_text.find, len, DeltaMessage - State reads: self.start_token, self.end_token - Return expressions: DeltaMessage(reasoning=reasoning_part if reasoning_part else None, content=content_part if content_part else None); result # Module `vllm_mlx.reasoning.gemma4_parser` Reasoning parser for Gemma 4 models. Gemma 4 uses a channel-based protocol for reasoning: <|channel>thought ...thinking content... ...response content... Where: <|channel> = token 100 (channel switch marker) = token 101 (end-of-channel marker) The channel names "thought" and "response" appear as text after the special tokens and should be stripped from the output. Some model variants may use <|channel>response instead of to transition from thinking to response mode. This parser handles both. When thinking is disabled or not triggered, output contains no tags. Degenerate cycling: On long prompts with tools, Gemma 4 may oscillate between thought and response channels many times, producing garbage reasoning before finally emitting valid content/tool_calls. The parser handles this by splitting at the LAST so all cycles go into reasoning_content and only the final response goes into content. Channel tokens are stripped from both sides. Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/reasoning/gemma4_parser.py#L1-L386 ## `vllm_mlx.reasoning.gemma4_parser._strip_channel_name` - Kind: function - Signature: `def _strip_channel_name(text: str, prefix: str) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/reasoning/gemma4_parser.py#L46-L50 - Implementation: Function `_strip_channel_name` calls `text.startswith`, `len`, `text.lstrip`; returns `text.lstrip('\n')`. Strip channel name and leading whitespace/newline from text start. - Inputs: - `text` (str; required): Required positional or keyword input. - `prefix` (str; required): Required positional or keyword input. - Return annotation: `str` - Calls: text.startswith, len, text.lstrip - Return expressions: text.lstrip('\n') ## `vllm_mlx.reasoning.gemma4_parser._strip_channel_tokens` - Kind: function - Signature: `def _strip_channel_tokens(text: str) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/reasoning/gemma4_parser.py#L53-L82 - Implementation: Function `_strip_channel_tokens` calls `text.replace`, `text.split`, `line.strip`, `cleaned.append`; returns `text.strip()`. Remove all channel special tokens and bare channel names from text. Handles degenerate model output with multiple thought/response cycles by stripping all protocol tokens, leaving only the actual text content. - Inputs: - `text` (str; required): Required positional or keyword input. - Return annotation: `str` - Calls: text.replace, text.split, line.strip, cleaned.append, '\n'.join, text.strip, text.startswith, len, text[len(name)].isalpha - Return expressions: text.strip() ## `vllm_mlx.reasoning.gemma4_parser.Gemma4ReasoningParser` - Kind: class - Signature: `class Gemma4ReasoningParser(BaseThinkingReasoningParser)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/reasoning/gemma4_parser.py#L85-L386 - Implementation: Class `Gemma4ReasoningParser` derives from `BaseThinkingReasoningParser` and declares 10 direct member(s). Reasoning parser for Gemma 4 models. Handles two transition formats: 1. <|channel>thought...response (standard: token 100 + 101) 2. <|channel>thought...<|channel>response (alternative: token 100 + 100) Channel names ("thought", "response") are stripped from output. Example: Input: "<|channel>thought\nLet me think...The answer is 42." Output: reasoning="Let me think...", content="The answer is 42." When no tags are present, the entire output is treated as content. Degenerate cycling (long prompts + tools): Uses rpartition to split at the LAST , so all intermediate thought/response cycles go into reasoning and only the final response goes into content. Streaming buffering: Partial markers at a delta boundary (e.g. "<|channel>" without a following "response" yet) are buffered internally so they don't leak into reasoning/content. The buffer is either consumed when the marker completes in a later delta, or flushed as reasoning via finalize_stream() when the stream ends. - Inputs: - `tokenizer` (not annotated; optional; default `None`): Optional positional or keyword input; defaults to `None`. - Constructs: `vllm_mlx.reasoning.gemma4_parser.Gemma4ReasoningParser` ## `vllm_mlx.reasoning.gemma4_parser.Gemma4ReasoningParser.start_token` - Kind: method - Signature: `def start_token(self) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/reasoning/gemma4_parser.py#L115-L118 - Implementation: Method `Gemma4ReasoningParser.start_token` returns `'<|channel>'`. Return Gemma's marker for entering the thought channel. - Inputs: none - Return annotation: `str` - Decorators: property - Return expressions: '<|channel>' ## `vllm_mlx.reasoning.gemma4_parser.Gemma4ReasoningParser.end_token` - Kind: method - Signature: `def end_token(self) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/reasoning/gemma4_parser.py#L121-L124 - Implementation: Method `Gemma4ReasoningParser.end_token` returns `''`. Return Gemma's marker for entering the response channel. - Inputs: none - Return annotation: `str` - Decorators: property - Return expressions: '' ## `vllm_mlx.reasoning.gemma4_parser.Gemma4ReasoningParser.__init__` - Kind: method - Signature: `def __init__(self, tokenizer=None)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/reasoning/gemma4_parser.py#L126-L133 - Implementation: Method `Gemma4ReasoningParser.__init__` updates `self._pending`, `self._content_seen`; calls `super().__init__`, `super`. Method `Gemma4ReasoningParser.__init__` updates `self._pending`, `self._content_seen`; calls `super().__init__`, `super`. - Inputs: - `tokenizer` (not annotated; optional; default `None`): Optional positional or keyword input; defaults to `None`. - Return annotation: `not annotated` - Calls: super().__init__, super - State writes: self._pending, self._content_seen ## `vllm_mlx.reasoning.gemma4_parser.Gemma4ReasoningParser.reset_state` - Kind: method - Signature: `def reset_state(self)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/reasoning/gemma4_parser.py#L135-L140 - Implementation: Method `Gemma4ReasoningParser.reset_state` updates `self._pending`, `self._content_seen`; calls `super().reset_state`, `super`. Reset base parsing state and buffered Gemma channel markers. - Inputs: none - Return annotation: `not annotated` - Calls: super().reset_state, super - State writes: self._pending, self._content_seen ## `vllm_mlx.reasoning.gemma4_parser.Gemma4ReasoningParser._trailing_partial_marker_len` - Kind: method - Signature: `def _trailing_partial_marker_len(self, text: str) -> int` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/reasoning/gemma4_parser.py#L142-L166 - Implementation: Method `Gemma4ReasoningParser._trailing_partial_marker_len` calls `range`, `min`, `len`, `text.endswith`; returns `max_len`. Return length of trailing substring of `text` that is a proper prefix of any transition marker (, <|channel>response, <|channel>). Only counts PROPER prefixes — if the marker is already complete in `text`, no buffering is needed. Returns 0 if no partial match. We must never buffer legitimate content. For <|channel>, only buffer when it appears AT THE END and is not followed by more text (i.e., `response` or `thought` hasn't arrived yet). - Inputs: - `text` (str; required): Required positional or keyword input. - Return annotation: `int` - Calls: range, min, len, text.endswith - State reads: self.end_token, self.start_token - Return expressions: max_len ## `vllm_mlx.reasoning.gemma4_parser.Gemma4ReasoningParser.finalize_stream` - Kind: method - Signature: `def finalize_stream(self) -> DeltaMessage | None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/reasoning/gemma4_parser.py#L168-L183 - Implementation: Method `Gemma4ReasoningParser.finalize_stream` updates `self._pending`; calls `DeltaMessage`; has 3 explicit return paths. Flush any buffered partial marker at the end of stream. If the stream ends while we have a partial marker buffered (e.g. model emitted "<|channel>" as its last token and got truncated by max_tokens), emit it as reasoning so the client doesn't lose the text. Content phase flushes as content. - Inputs: none - Return annotation: `DeltaMessage | None` - Calls: DeltaMessage - State reads: self._pending, self._phase - State writes: self._pending - Return expressions: None; DeltaMessage(content=pending); DeltaMessage(reasoning=pending) ## `vllm_mlx.reasoning.gemma4_parser.Gemma4ReasoningParser.extract_reasoning` - Kind: method - Signature: `def extract_reasoning(self, model_output: str) -> tuple[str | None, str | None]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/reasoning/gemma4_parser.py#L185-L233 - Implementation: Method `Gemma4ReasoningParser.extract_reasoning` calls `text.partition`, `after_start.rpartition`, `_strip_channel_tokens`, `text.count`; has 3 explicit return paths. Extract reasoning from complete output. Uses rpartition (LAST ) to handle degenerate cycling: all intermediate thought/response cycles go into reasoning, only the final response goes into content. Channel tokens are stripped from both sides. - Inputs: - `model_output` (str; required): Required positional or keyword input. - Return annotation: `tuple[str | None, str | None]` - Calls: text.partition, after_start.rpartition, _strip_channel_tokens, text.count, after_start.rfind, len, text.rpartition - State reads: self.start_token, self.end_token - Return expressions: (reasoning or None, content or None); (reasoning or None, None); (None, model_output) ## `vllm_mlx.reasoning.gemma4_parser.Gemma4ReasoningParser.extract_reasoning_streaming` - Kind: method - Signature: `def extract_reasoning_streaming(self, previous_text: str, current_text: str, delta_text: str) -> DeltaMessage | None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/reasoning/gemma4_parser.py#L235-L272 - Implementation: Method `Gemma4ReasoningParser.extract_reasoning_streaming` updates `self._pending`; calls `self._trailing_partial_marker_len`, `len`, `self._extract_from_safe_text`; has 2 explicit return paths. Extract reasoning from streaming delta. Handles: - No tags: treat as content (Gemma 4 doesn't inject tags in prompt) - <|channel>thought: enter reasoning mode, strip channel name - or <|channel>response: transition to content mode - Re-entry into thought from content (degenerate cycling): back to reasoning Partial markers at delta boundaries are buffered internally to prevent leaking them as reasoning/content. - Inputs: - `previous_text` (str; required): Required positional or keyword input. - `current_text` (str; required): Required positional or keyword input. - `delta_text` (str; required): Required positional or keyword input. - Return annotation: `DeltaMessage | None` - Calls: self._trailing_partial_marker_len, len, self._extract_from_safe_text - State reads: self._trailing_partial_marker_len, self._extract_from_safe_text - State writes: self._pending - Return expressions: None; self._extract_from_safe_text(safe_previous, safe_current, safe_delta) ## `vllm_mlx.reasoning.gemma4_parser.Gemma4ReasoningParser._strip_channel_tokens_from_delta` - Kind: method - Signature: `def _strip_channel_tokens_from_delta(msg: DeltaMessage | None) -> DeltaMessage | None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/reasoning/gemma4_parser.py#L275-L291 - Implementation: Method `Gemma4ReasoningParser._strip_channel_tokens_from_delta` calls `c.replace('', '').replace`, `c.replace`, `r.replace('', '').replace`, `r.replace`; has 3 explicit return paths. Strip channel special tokens from content and reasoning in a delta. - Inputs: - `msg` (DeltaMessage | None; required): Required positional or keyword input. - Return annotation: `DeltaMessage | None` - Decorators: staticmethod - Calls: c.replace('', '').replace, c.replace, r.replace('', '').replace, r.replace, DeltaMessage - Return expressions: None; msg; DeltaMessage(reasoning=r or None, content=c or None) ## `vllm_mlx.reasoning.gemma4_parser.Gemma4ReasoningParser._extract_from_safe_text` - Kind: method - Signature: `def _extract_from_safe_text(self, previous_text: str, current_text: str, delta_text: str) -> DeltaMessage | None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/reasoning/gemma4_parser.py#L293-L386 - Implementation: Method `Gemma4ReasoningParser._extract_from_safe_text` updates `self._phase`, `self._content_seen`; calls `DeltaMessage`, `current_text.find`, `len`, `after_marker.lstrip`; has 8 explicit return paths. Parse safe (non-buffered) text. Uses count-based detection for channel tokens so that multiple thought/response cycles (degenerate model behaviour) are handled correctly — each NEW <|channel> re-enters reasoning, each NEW transitions to content. - Inputs: - `previous_text` (str; required): Required positional or keyword input. - `current_text` (str; required): Required positional or keyword input. - `delta_text` (str; required): Required positional or keyword input. - Return annotation: `DeltaMessage | None` - Calls: DeltaMessage, current_text.find, len, after_marker.lstrip, self._strip_channel_tokens_from_delta, current_text.count, previous_text.count, delta_text.rfind, _strip_channel_name, after.lstrip, delta_text.lstrip, bool, current_text.split, after_ch.startswith, after_ch[len(_THOUGHT_PREFIX):].lstrip, previous_text.split, prev_after.startswith, prev_after[len(_THOUGHT_PREFIX):].lstrip - State reads: self.start_token, self.end_token, self._strip_channel_tokens_from_delta, self._phase, self._content_seen - State writes: self._phase, self._content_seen - Return expressions: DeltaMessage(content=delta_text); self._strip_channel_tokens_from_delta(DeltaMessage(content=after_marker)); None; DeltaMessage(content=after); self._strip_channel_tokens_from_delta(DeltaMessage(content=stripped)); self._strip_channel_tokens_from_delta(DeltaMessage(content=delta_text)); DeltaMessage(reasoning=r) if r else None; DeltaMessage(reasoning=delta_text) if delta_text else None # Module `vllm_mlx.reasoning.glm4_parser` Reasoning parser for GLM-4 models (GLM-4.5-Air, GLM-4.6V, GLM-4.7, etc.). GLM-4 uses ... tags for reasoning content, same as Qwen3. However, unlike Qwen3, GLM-4 does NOT inject in the prompt — the model decides autonomously whether to reason. This means: - Output without tags = normal response (no reasoning) - Output with tags = reasoning + content This is the opposite of Qwen3 where no tags = pure reasoning (because was injected in the prompt and the model hit max_tokens). GLM-4.6V also wraps responses in <|begin_of_box|>...<|end_of_box|> container tags which must be stripped before returning content. Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/reasoning/glm4_parser.py#L1-L113 ## `vllm_mlx.reasoning.glm4_parser.Glm4ReasoningParser` - Kind: class - Signature: `class Glm4ReasoningParser(BaseThinkingReasoningParser)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/reasoning/glm4_parser.py#L27-L113 - Implementation: Class `Glm4ReasoningParser` derives from `BaseThinkingReasoningParser` and declares 4 direct member(s). Reasoning parser for GLM-4 models. GLM-4 uses ... tokens to denote reasoning text. Unlike Qwen3, the template does NOT inject in the prompt, so output without tags is a normal response (not truncated reasoning). Supports three scenarios: 1. Both tags in output: reasoningcontent 2. Only closing tag (think in prompt): reasoningcontent 3. No tags: pure content (NOT reasoning) Example (with thinking): Input: "Let me analyze...The answer is 42." Output: reasoning="Let me analyze...", content="The answer is 42." Example (no thinking): Input: "The answer is 42." Output: reasoning=None, content="The answer is 42." - Inputs: none - Constructs: `vllm_mlx.reasoning.glm4_parser.Glm4ReasoningParser` ## `vllm_mlx.reasoning.glm4_parser.Glm4ReasoningParser.start_token` - Kind: method - Signature: `def start_token(self) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/reasoning/glm4_parser.py#L50-L53 - Implementation: Method `Glm4ReasoningParser.start_token` returns `''`. Return the marker that opens a GLM reasoning span. - Inputs: none - Return annotation: `str` - Decorators: property - Return expressions: '' ## `vllm_mlx.reasoning.glm4_parser.Glm4ReasoningParser.end_token` - Kind: method - Signature: `def end_token(self) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/reasoning/glm4_parser.py#L56-L59 - Implementation: Method `Glm4ReasoningParser.end_token` returns `''`. Return the marker that closes a GLM reasoning span. - Inputs: none - Return annotation: `str` - Decorators: property - Return expressions: '' ## `vllm_mlx.reasoning.glm4_parser.Glm4ReasoningParser.extract_reasoning` - Kind: method - Signature: `def extract_reasoning(self, model_output: str) -> tuple[str | None, str | None]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/reasoning/glm4_parser.py#L61-L68 - Implementation: Method `Glm4ReasoningParser.extract_reasoning` calls `model_output.replace(_BOX_START, '').replace`, `model_output.replace`, `super().extract_reasoning`, `super`; returns `super().extract_reasoning(cleaned)`. Strip GLM box markers and split complete reasoning from content. - Inputs: - `model_output` (str; required): Required positional or keyword input. - Return annotation: `tuple[str | None, str | None]` - Calls: model_output.replace(_BOX_START, '').replace, model_output.replace, super().extract_reasoning, super - Return expressions: super().extract_reasoning(cleaned) ## `vllm_mlx.reasoning.glm4_parser.Glm4ReasoningParser.extract_reasoning_streaming` - Kind: method - Signature: `def extract_reasoning_streaming(self, previous_text: str, current_text: str, delta_text: str) -> DeltaMessage | None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/reasoning/glm4_parser.py#L70-L113 - Implementation: Method `Glm4ReasoningParser.extract_reasoning_streaming` calls `delta_text.replace(_BOX_START, '').replace`, `delta_text.replace`, `super().extract_reasoning_streaming`, `super`; has 3 explicit return paths. Extract reasoning from streaming delta. Overrides base class pre_think behavior: when no tags have been seen, emit delta as content (not reasoning). GLM-4 doesn't inject in the prompt, so early tokens without tags are normal content. Once is seen, delegates to base class state machine. - Inputs: - `previous_text` (str; required): Required positional or keyword input. - `current_text` (str; required): Required positional or keyword input. - `delta_text` (str; required): Required positional or keyword input. - Return annotation: `DeltaMessage | None` - Calls: delta_text.replace(_BOX_START, '').replace, delta_text.replace, super().extract_reasoning_streaming, super, DeltaMessage - State reads: self.start_token, self.end_token, self._phase - Return expressions: None; super().extract_reasoning_streaming(previous_text, current_text, delta_text); DeltaMessage(content=delta_text) # Module `vllm_mlx.reasoning.gpt_oss_parser` Reasoning parser for GPT-OSS models using channel-based format. GPT-OSS models use a channel-based token format instead of ... tags: <|channel|>analysis<|message|>[reasoning]<|start|>assistant<|channel|>final<|message|>[content]<|return|> Some models also emit an extended format with a constrain token: <|channel|>final <|constrain|>JSON<|message|>[content]<|return|> This parser extracts reasoning from the 'analysis' channel and content from the 'final' channel, stripping all structural tokens from API responses. Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/reasoning/gpt_oss_parser.py#L1-L214 ## `vllm_mlx.reasoning.gpt_oss_parser._extract_channel` - Kind: function - Signature: `def _extract_channel(text: str, channel_name: str) -> str | None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/reasoning/gpt_oss_parser.py#L33-L55 - Implementation: Function `_extract_channel` calls `_CHANNEL_RE.finditer`, `m.group`, `m.end`, `_STRUCTURAL_TOKENS.search`; has 2 explicit return paths. Extract content from a named channel. Finds <|channel|>{name}...<|message|> (with optional constrain token) and extracts text up to the next structural token or end of string. Args: text: Full model output text. channel_name: Channel name to extract (e.g., "analysis", "final"). Returns: Extracted channel content, or None if channel not found. - Inputs: - `text` (str; required): Full model output text. - `channel_name` (str; required): Channel name to extract (e.g., "analysis", "final"). - Return annotation: `str | None` - Calls: _CHANNEL_RE.finditer, m.group, m.end, _STRUCTURAL_TOKENS.search, end_match.start, content.strip - Return expressions: content if content else None; None ## `vllm_mlx.reasoning.gpt_oss_parser.GptOssReasoningParser` - Kind: class - Signature: `class GptOssReasoningParser(ReasoningParser)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/reasoning/gpt_oss_parser.py#L58-L214 - Implementation: Class `GptOssReasoningParser` derives from `ReasoningParser` and declares 5 direct member(s). Reasoning parser for GPT-OSS models. GPT-OSS uses channel-based tokens: <|channel|>analysis<|message|>[reasoning] <|start|>assistant<|channel|>final<|message|>[content]<|return|> The 'analysis' channel maps to reasoning, 'final' to content. Also handles extended format with constrain token: <|channel|>final <|constrain|>JSON<|message|>[content]<|return|> - Inputs: none - Constructs: `vllm_mlx.reasoning.gpt_oss_parser.GptOssReasoningParser` ## `vllm_mlx.reasoning.gpt_oss_parser.GptOssReasoningParser.extract_reasoning` - Kind: method - Signature: `def extract_reasoning(self, model_output: str) -> tuple[str | None, str | None]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/reasoning/gpt_oss_parser.py#L72-L106 - Implementation: Method `GptOssReasoningParser.extract_reasoning` calls `_extract_channel`, `content.replace('<|return|>', '').strip`, `content.replace`, `_STRUCTURAL_TOKENS.sub('', content).strip`; has 3 explicit return paths. Extract reasoning and content from complete model output. Args: model_output: Complete text output from the model. Returns: (reasoning, content) tuple. Either may be None. - Inputs: - `model_output` (str; required): Complete text output from the model. - Return annotation: `tuple[str | None, str | None]` - Calls: _extract_channel, content.replace('<|return|>', '').strip, content.replace, _STRUCTURAL_TOKENS.sub('', content).strip, _STRUCTURAL_TOKENS.sub, _STRUCTURAL_TOKENS.sub('', reasoning).strip - Return expressions: (None, model_output if model_output else None); (None, model_output); (reasoning, content) ## `vllm_mlx.reasoning.gpt_oss_parser.GptOssReasoningParser.extract_reasoning_streaming` - Kind: method - Signature: `def extract_reasoning_streaming(self, previous_text: str, current_text: str, delta_text: str) -> DeltaMessage | None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/reasoning/gpt_oss_parser.py#L108-L161 - Implementation: Method `GptOssReasoningParser.extract_reasoning_streaming` calls `self._detect_phase`, `self._extract_content_after_marker_in_delta`, `self._strip_return`, `DeltaMessage`; has 5 explicit return paths. Extract reasoning from streaming delta. Uses stateless phase detection from current_text on each call. Args: previous_text: Accumulated text before this delta. current_text: Accumulated text including this delta. delta_text: Just the new text in this streaming chunk. Returns: DeltaMessage with reasoning and/or content, or None to skip. - Inputs: - `previous_text` (str; required): Accumulated text before this delta. - `current_text` (str; required): Accumulated text including this delta. - `delta_text` (str; required): Just the new text in this streaming chunk. - Return annotation: `DeltaMessage | None` - Calls: self._detect_phase, self._extract_content_after_marker_in_delta, self._strip_return, DeltaMessage, _STRUCTURAL_TOKENS.search, _STRUCTURAL_TOKENS.sub - State reads: self._detect_phase, self._extract_content_after_marker_in_delta, self._strip_return - Return expressions: DeltaMessage(reasoning=after_marker); DeltaMessage(content=after_marker); None; DeltaMessage(reasoning=cleaned); DeltaMessage(content=cleaned) ## `vllm_mlx.reasoning.gpt_oss_parser.GptOssReasoningParser._detect_phase` - Kind: method - Signature: `def _detect_phase(text: str) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/reasoning/gpt_oss_parser.py#L164-L187 - Implementation: Method `GptOssReasoningParser._detect_phase` calls `list`, `_CHANNEL_RE.finditer`, `last.group`, `last.end`; has 4 explicit return paths. Detect current streaming phase from accumulated text. Returns: "final" — final channel marker complete "analysis" — analysis marker complete, no structural token after "transition" — analysis present but structural token follows "init" — no channel marker yet - Inputs: - `text` (str; required): Required positional or keyword input. - Return annotation: `str` - Decorators: staticmethod - Calls: list, _CHANNEL_RE.finditer, last.group, last.end, _STRUCTURAL_TOKENS.search - Return expressions: 'init'; 'final'; 'transition'; 'analysis' ## `vllm_mlx.reasoning.gpt_oss_parser.GptOssReasoningParser._extract_content_after_marker_in_delta` - Kind: method - Signature: `def _extract_content_after_marker_in_delta(current_text: str, phase: str) -> str | None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/reasoning/gpt_oss_parser.py#L190-L209 - Implementation: Method `GptOssReasoningParser._extract_content_after_marker_in_delta` calls `list`, `_CHANNEL_RE.finditer`, `reversed`, `m.group`; has 2 explicit return paths. When phase changes, extract only the content after the phase marker that falls within the current accumulated text's tail. Args: current_text: Full accumulated text. phase: Current phase ("analysis" or "final"). Returns: Content after the marker, or None. - Inputs: - `current_text` (str; required): Full accumulated text. - `phase` (str; required): Current phase ("analysis" or "final"). - Return annotation: `str | None` - Decorators: staticmethod - Calls: list, _CHANNEL_RE.finditer, reversed, m.group, m.end - Return expressions: current_text[m.end():]; None ## `vllm_mlx.reasoning.gpt_oss_parser.GptOssReasoningParser._strip_return` - Kind: method - Signature: `def _strip_return(text: str) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/reasoning/gpt_oss_parser.py#L212-L214 - Implementation: Method `GptOssReasoningParser._strip_return` calls `text.replace`; returns `text.replace('<|return|>', '')`. Strip <|return|> from text. - Inputs: - `text` (str; required): Required positional or keyword input. - Return annotation: `str` - Decorators: staticmethod - Calls: text.replace - Return expressions: text.replace('<|return|>', '') # Module `vllm_mlx.reasoning.harmony_parser` Reasoning parser for GPT-OSS models using Harmony format. Harmony uses channels for reasoning vs final content: <|channel|>analysis <|message|>Let me think about this... <|end|> <|channel|>final <|message|>The answer is 42. <|return|> The analysis channel contains reasoning, and the final channel contains the user-facing response. Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/reasoning/harmony_parser.py#L1-L157 ## `vllm_mlx.reasoning.harmony_parser.HarmonyReasoningParser` - Kind: class - Signature: `class HarmonyReasoningParser(ReasoningParser)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/reasoning/harmony_parser.py#L35-L157 - Implementation: Class `HarmonyReasoningParser` derives from `ReasoningParser` and declares 4 direct member(s). Reasoning parser for GPT-OSS models using Harmony format. Extracts reasoning from the 'analysis' channel and content from the 'final' channel. Commentary channels (tool calls) are ignored since they are handled by the tool parser. Example: Input: "<|channel|>analysis<|message|>Thinking...<|end|> <|channel|>final<|message|>Result.<|return|>" Output: reasoning="Thinking...", content="Result." - Inputs: - `tokenizer` (not annotated; optional; default `None`): Optional positional or keyword input; defaults to `None`. - Constructs: `vllm_mlx.reasoning.harmony_parser.HarmonyReasoningParser` ## `vllm_mlx.reasoning.harmony_parser.HarmonyReasoningParser.__init__` - Kind: method - Signature: `def __init__(self, tokenizer=None)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/reasoning/harmony_parser.py#L49-L52 - Implementation: Method `HarmonyReasoningParser.__init__` updates `self._current_channel`, `self._in_message`; calls `super().__init__`, `super`. Method `HarmonyReasoningParser.__init__` updates `self._current_channel`, `self._in_message`; calls `super().__init__`, `super`. - Inputs: - `tokenizer` (not annotated; optional; default `None`): Optional positional or keyword input; defaults to `None`. - Return annotation: `not annotated` - Calls: super().__init__, super - State writes: self._current_channel, self._in_message ## `vllm_mlx.reasoning.harmony_parser.HarmonyReasoningParser.extract_reasoning` - Kind: method - Signature: `def extract_reasoning(self, model_output: str) -> tuple[str | None, str | None]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/reasoning/harmony_parser.py#L54-L78 - Implementation: Method `HarmonyReasoningParser.extract_reasoning` calls `_ANALYSIS_PATTERN.findall`, `'\n'.join`, `block.strip`, `_FINAL_PATTERN.search`; returns `(reasoning, content)`. Extract reasoning from complete Harmony output. Collects all analysis channel blocks as reasoning and the final channel block as content. Args: model_output: Complete model output text. Returns: (reasoning, content) tuple. Either may be None. - Inputs: - `model_output` (str; required): Complete model output text. - Return annotation: `tuple[str | None, str | None]` - Calls: _ANALYSIS_PATTERN.findall, '\n'.join, block.strip, _FINAL_PATTERN.search, final_match.group(1).strip, final_match.group - Return expressions: (reasoning, content) ## `vllm_mlx.reasoning.harmony_parser.HarmonyReasoningParser.extract_reasoning_streaming` - Kind: method - Signature: `def extract_reasoning_streaming(self, previous_text: str, current_text: str, delta_text: str) -> DeltaMessage | None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/reasoning/harmony_parser.py#L80-L152 - Implementation: Method `HarmonyReasoningParser.extract_reasoning_streaming` updates `self._current_channel`, `self._in_message`; calls `current_text.rfind`, `len`, `after.startswith`, `any`; has 3 explicit return paths. Extract reasoning from streaming Harmony output. Tracks the current channel and emits reasoning deltas for analysis channel content and content deltas for final channel. Args: previous_text: Accumulated text before this delta. current_text: Accumulated text including this delta. delta_text: The new text in this streaming chunk. Returns: DeltaMessage with reasoning and/or content, or None. - Inputs: - `previous_text` (str; required): Accumulated text before this delta. - `current_text` (str; required): Accumulated text including this delta. - `delta_text` (str; required): The new text in this streaming chunk. - Return annotation: `DeltaMessage | None` - Calls: current_text.rfind, len, after.startswith, any, delta_text.strip().startswith, delta_text.strip, delta_text.strip().endswith, DeltaMessage - State reads: self._current_channel, self._in_message - State writes: self._current_channel, self._in_message - Return expressions: None; DeltaMessage(reasoning=delta_text); DeltaMessage(content=delta_text) ## `vllm_mlx.reasoning.harmony_parser.HarmonyReasoningParser.reset_state` - Kind: method - Signature: `def reset_state(self)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/reasoning/harmony_parser.py#L154-L157 - Implementation: Method `HarmonyReasoningParser.reset_state` updates `self._current_channel`, `self._in_message`. Reset streaming state for a new request. - Inputs: none - Return annotation: `not annotated` - State writes: self._current_channel, self._in_message # Module `vllm_mlx.reasoning.mistral_parser` Reasoning parser for Mistral / Ministral reasoning models. Models such as Magistral and Ministral-3-*-Reasoning wrap their reasoning in [THINK]...[/THINK] delimiters (registered as special tokens in the tokenizer) and support a strict switch via 'enable_thinking=False' in chat template kwargs. This mirrors the Qwen3 parser, which uses ..., but with the Mistral bracket-token delimiters. Supports implicit reasoning mode where [THINK] is injected in the prompt and only [/THINK] appears in the output. Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/reasoning/mistral_parser.py#L1-L72 ## `vllm_mlx.reasoning.mistral_parser.MistralReasoningParser` - Kind: class - Signature: `class MistralReasoningParser(BaseThinkingReasoningParser)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/reasoning/mistral_parser.py#L19-L72 - Implementation: Class `MistralReasoningParser` derives from `BaseThinkingReasoningParser` and declares 3 direct member(s). Reasoning parser for Mistral/Ministral reasoning models. Uses [THINK]...[/THINK] tokens to denote reasoning text. Supports three scenarios: 1. Both tags in output: [THINK]reasoning[/THINK]content 2. Only closing tag (think in prompt): reasoning[/THINK]content 3. No tags: pure content Example (normal): Input: "[THINK]Let me analyze this...[/THINK]The answer is 42." Output: reasoning="Let me analyze this...", content="The answer is 42." Example (think in prompt): Input: "Let me analyze this...[/THINK]The answer is 42." Output: reasoning="Let me analyze this...", content="The answer is 42." - Inputs: none - Constructs: `vllm_mlx.reasoning.mistral_parser.MistralReasoningParser` ## `vllm_mlx.reasoning.mistral_parser.MistralReasoningParser.start_token` - Kind: method - Signature: `def start_token(self) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/reasoning/mistral_parser.py#L40-L43 - Implementation: Method `MistralReasoningParser.start_token` returns `'[THINK]'`. Return the Mistral reasoning opening marker. - Inputs: none - Return annotation: `str` - Decorators: property - Return expressions: '[THINK]' ## `vllm_mlx.reasoning.mistral_parser.MistralReasoningParser.end_token` - Kind: method - Signature: `def end_token(self) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/reasoning/mistral_parser.py#L46-L49 - Implementation: Method `MistralReasoningParser.end_token` returns `'[/THINK]'`. Return the Mistral reasoning closing marker. - Inputs: none - Return annotation: `str` - Decorators: property - Return expressions: '[/THINK]' ## `vllm_mlx.reasoning.mistral_parser.MistralReasoningParser.extract_reasoning` - Kind: method - Signature: `def extract_reasoning(self, model_output: str) -> tuple[str | None, str | None]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/reasoning/mistral_parser.py#L51-L72 - Implementation: Method `MistralReasoningParser.extract_reasoning` calls `super().extract_reasoning`, `super`; has 2 explicit return paths. Extract reasoning from Mistral/Ministral output. Handles both explicit [THINK]...[/THINK] tags and implicit mode where [THINK] was in the prompt (only [/THINK] in output). Args: model_output: Complete model output text. Returns: (reasoning, content) tuple. - Inputs: - `model_output` (str; required): Complete model output text. - Return annotation: `tuple[str | None, str | None]` - Calls: super().extract_reasoning, super - State reads: self.end_token - Return expressions: (None, model_output); super().extract_reasoning(model_output) # Module `vllm_mlx.reasoning.poolside_v1_parser` Reasoning parser for Poolside Laguna models. Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/reasoning/poolside_v1_parser.py#L1-L13 ## `vllm_mlx.reasoning.poolside_v1_parser.PoolsideV1ReasoningParser` - Kind: class - Signature: `class PoolsideV1ReasoningParser(Qwen3ReasoningParser)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/reasoning/poolside_v1_parser.py#L7-L13 - Implementation: Class `PoolsideV1ReasoningParser` derives from `Qwen3ReasoningParser` and declares 0 direct member(s). Parse Laguna's template-injected ```` reasoning boundary. vllm-mlx reasoning parsers receive only the generated assistant text, not prompt-history token IDs. Historical ```` markers therefore cannot affect this output-scoped parser as they can in token-aware vLLM serving. - Inputs: none - Constructs: `vllm_mlx.reasoning.poolside_v1_parser.PoolsideV1ReasoningParser` # Module `vllm_mlx.reasoning.qwen3_parser` Reasoning parser for Qwen3 models. Qwen3 uses ... tags for reasoning content and supports a strict switch via 'enable_thinking=False' in chat template kwargs. Supports implicit reasoning mode where is injected in the prompt by AI agents (e.g., OpenCode) and only appears in the output. Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/reasoning/qwen3_parser.py#L1-L68 ## `vllm_mlx.reasoning.qwen3_parser.Qwen3ReasoningParser` - Kind: class - Signature: `class Qwen3ReasoningParser(BaseThinkingReasoningParser)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/reasoning/qwen3_parser.py#L15-L68 - Implementation: Class `Qwen3ReasoningParser` derives from `BaseThinkingReasoningParser` and declares 3 direct member(s). Reasoning parser for Qwen3 models. Qwen3 uses ... tokens to denote reasoning text. Supports three scenarios: 1. Both tags in output: reasoningcontent 2. Only closing tag (think in prompt): reasoningcontent 3. No tags: pure content Example (normal): Input: "Let me analyze this...The answer is 42." Output: reasoning="Let me analyze this...", content="The answer is 42." Example (think in prompt): Input: "Let me analyze this...The answer is 42." Output: reasoning="Let me analyze this...", content="The answer is 42." - Inputs: none - Constructs: `vllm_mlx.reasoning.qwen3_parser.Qwen3ReasoningParser` ## `vllm_mlx.reasoning.qwen3_parser.Qwen3ReasoningParser.start_token` - Kind: method - Signature: `def start_token(self) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/reasoning/qwen3_parser.py#L36-L39 - Implementation: Method `Qwen3ReasoningParser.start_token` returns `''`. Return the Qwen3 reasoning opening marker. - Inputs: none - Return annotation: `str` - Decorators: property - Return expressions: '' ## `vllm_mlx.reasoning.qwen3_parser.Qwen3ReasoningParser.end_token` - Kind: method - Signature: `def end_token(self) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/reasoning/qwen3_parser.py#L42-L45 - Implementation: Method `Qwen3ReasoningParser.end_token` returns `''`. Return the Qwen3 reasoning closing marker. - Inputs: none - Return annotation: `str` - Decorators: property - Return expressions: '' ## `vllm_mlx.reasoning.qwen3_parser.Qwen3ReasoningParser.extract_reasoning` - Kind: method - Signature: `def extract_reasoning(self, model_output: str) -> tuple[str | None, str | None]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/reasoning/qwen3_parser.py#L47-L68 - Implementation: Method `Qwen3ReasoningParser.extract_reasoning` calls `super().extract_reasoning`, `super`; has 2 explicit return paths. Extract reasoning from Qwen3 output. Handles both explicit ... tags and implicit mode where was in the prompt (only in output). Args: model_output: Complete model output text. Returns: (reasoning, content) tuple. - Inputs: - `model_output` (str; required): Complete model output text. - Return annotation: `tuple[str | None, str | None]` - Calls: super().extract_reasoning, super - State reads: self.end_token - Return expressions: (None, model_output); super().extract_reasoning(model_output) # Module `vllm_mlx.reasoning.think_parser` Base parser for models using ... tags for reasoning. This module provides BaseThinkingReasoningParser, a concrete implementation for extracting reasoning content from models that use thinking tags. Supports three scenarios: 1. Both tags in output: reasoningcontent 2. Only closing tag (think injected in prompt): reasoningcontent 3. No tags: pure content Performance: The streaming parser uses a simple state machine to track the current phase (pre-think / thinking / content). Tag completion is detected against the accumulated text for correctness when `` / `` are split across delta boundaries, but phase tracking still avoids the old whole-output rescanning behavior. Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/reasoning/think_parser.py#L1-L462 ## `vllm_mlx.reasoning.think_parser.BaseThinkingReasoningParser` - Kind: class - Signature: `class BaseThinkingReasoningParser(ReasoningParser)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/reasoning/think_parser.py#L29-L462 - Implementation: Class `BaseThinkingReasoningParser` derives from `ReasoningParser` and declares 12 direct member(s). Base parser for models using ... style tags. This parser handles the common pattern where reasoning content is wrapped in special tags. Subclasses define the specific start and end tokens. Supports "implicit reasoning mode" where is injected in the prompt and only appears in the model output. This is common with AI agents like OpenCode that force models to reason by injecting thinking tags. The streaming parser uses a state machine with three phases: pre_think -> thinking -> content Transitions are tracked by parser state. Accumulated text is consulted only to detect when a start/end tag has completed across delta boundaries. - Inputs: - `tokenizer` (not annotated; optional; default `None`): Optional positional or keyword input; defaults to `None`. - Constructs: `vllm_mlx.reasoning.think_parser.BaseThinkingReasoningParser` ## `vllm_mlx.reasoning.think_parser.BaseThinkingReasoningParser.start_token` - Kind: method - Signature: `def start_token(self) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/reasoning/think_parser.py#L50-L51 - Implementation: Method `BaseThinkingReasoningParser.start_token` contains no state mutation, call, raise, return, await, or yield. The token/tag that starts reasoning content (e.g., ''). - Inputs: none - Return annotation: `str` - Decorators: property, abstractmethod ## `vllm_mlx.reasoning.think_parser.BaseThinkingReasoningParser.end_token` - Kind: method - Signature: `def end_token(self) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/reasoning/think_parser.py#L55-L56 - Implementation: Method `BaseThinkingReasoningParser.end_token` contains no state mutation, call, raise, return, await, or yield. The token/tag that ends reasoning content (e.g., ''). - Inputs: none - Return annotation: `str` - Decorators: property, abstractmethod ## `vllm_mlx.reasoning.think_parser.BaseThinkingReasoningParser.__init__` - Kind: method - Signature: `def __init__(self, tokenizer=None)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/reasoning/think_parser.py#L63-L71 - Implementation: Method `BaseThinkingReasoningParser.__init__` updates `self._phase`, `self._content_started`, `self._content_buffer`, `self._in_tool_call`; calls `super().__init__`, `super`. Method `BaseThinkingReasoningParser.__init__` updates `self._phase`, `self._content_started`, `self._content_buffer`, `self._in_tool_call`; calls `super().__init__`, `super`. - Inputs: - `tokenizer` (not annotated; optional; default `None`): Optional positional or keyword input; defaults to `None`. - Return annotation: `not annotated` - Calls: super().__init__, super - State writes: self._phase, self._content_started, self._content_buffer, self._in_tool_call, self._tool_call_buffer ## `vllm_mlx.reasoning.think_parser.BaseThinkingReasoningParser.reset_state` - Kind: method - Signature: `def reset_state(self)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/reasoning/think_parser.py#L73-L79 - Implementation: Method `BaseThinkingReasoningParser.reset_state` updates `self._phase`, `self._content_started`, `self._content_buffer`, `self._in_tool_call`. Reset state machine for a new streaming request. - Inputs: none - Return annotation: `not annotated` - State writes: self._phase, self._content_started, self._content_buffer, self._in_tool_call, self._tool_call_buffer ## `vllm_mlx.reasoning.think_parser.BaseThinkingReasoningParser.extract_reasoning` - Kind: method - Signature: `def extract_reasoning(self, model_output: str) -> tuple[str | None, str | None]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/reasoning/think_parser.py#L81-L110 - Implementation: Method `BaseThinkingReasoningParser.extract_reasoning` calls `self._extract_complete_reasoning`, `self._promote_tool_calls`, `text.partition`, `reasoning.strip`; has 3 explicit return paths. Extract reasoning from complete output. Handles three cases: 1. Both tags present: reasoningcontent 2. Only closing tag: reasoningcontent (think in prompt) 3. No tags: pure content Args: model_output: Complete model output text. Returns: (reasoning, content) tuple. Either may be None. - Inputs: - `model_output` (str; required): Complete model output text. - Return annotation: `tuple[str | None, str | None]` - Calls: self._extract_complete_reasoning, self._promote_tool_calls, text.partition, reasoning.strip - State reads: self.end_token, self._extract_complete_reasoning, self._promote_tool_calls, self.start_token - Return expressions: self._promote_tool_calls(reasoning, content); self._promote_tool_calls(reasoning, None); (None, model_output) ## `vllm_mlx.reasoning.think_parser.BaseThinkingReasoningParser.extract_reasoning_streaming` - Kind: method - Signature: `def extract_reasoning_streaming(self, previous_text: str, current_text: str, delta_text: str) -> DeltaMessage | None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/reasoning/think_parser.py#L112-L224 - Implementation: Method `BaseThinkingReasoningParser.extract_reasoning_streaming` updates `self._phase`, `self._in_tool_call`, `self._tool_call_buffer`; calls `delta_text.find`, `len`, `after.find`, `self._transition_to_content`; has 8 explicit return paths. Extract reasoning from a streaming delta using state-machine tracking. Instead of rescanning the full accumulated text on every token, this method tracks the current phase (pre_think / thinking / content) and only consults accumulated text to detect completed start/end tags that were split across delta boundaries. Handles three scenarios: 1. Explicit ... in model output 2. Implicit mode ( in prompt, only in output) 3. No tags at all (pure content after first token with no reasoning) Args: previous_text: Text accumulated before this delta. current_text: Text including this delta. delta_text: Just the new text in this chunk. Returns: DeltaMessage with reasoning and/or content, or None to skip. - Inputs: - `previous_text` (str; required): Text accumulated before this delta. - `current_text` (str; required): Text including this delta. - `delta_text` (str; required): Just the new text in this chunk. - Return annotation: `DeltaMessage | None` - Calls: delta_text.find, len, after.find, self._transition_to_content, DeltaMessage, self._thinking_tool_call, self._content_delta - State reads: self.start_token, self.end_token, self._phase, self._transition_to_content, self._TOOL_CALL_START, self._in_tool_call, self._thinking_tool_call, self._content_delta - State writes: self._phase, self._in_tool_call, self._tool_call_buffer - Return expressions: None; self._transition_to_content(reasoning, content); DeltaMessage(reasoning=before) if before else None; DeltaMessage(reasoning=after) if after else None; DeltaMessage(reasoning=delta_text); self._thinking_tool_call(previous_text, current_text, delta_text); DeltaMessage(reasoning=reasoning) if reasoning else None; self._content_delta(delta_text) ## `vllm_mlx.reasoning.think_parser.BaseThinkingReasoningParser._extract_complete_reasoning` - Kind: method - Signature: `def _extract_complete_reasoning(self, text: str) -> tuple[str | None, str | None]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/reasoning/think_parser.py#L226-L260 - Implementation: Method `BaseThinkingReasoningParser._extract_complete_reasoning` calls `remainder.lstrip`, `stripped.startswith`, `len`, `after_start.partition`; returns `(reasoning, content)`. Split complete output into leading reasoning spans and final content. - Inputs: - `text` (str; required): Required positional or keyword input. - Return annotation: `tuple[str | None, str | None]` - Calls: remainder.lstrip, stripped.startswith, len, after_start.partition, reasoning_parts.append, reasoning.strip, stripped.find, '\n'.join(reasoning_parts).strip, '\n'.join, remainder.strip - State reads: self.start_token, self.end_token - Return expressions: (reasoning, content) ## `vllm_mlx.reasoning.think_parser.BaseThinkingReasoningParser._transition_to_content` - Kind: method - Signature: `def _transition_to_content(self, reasoning: str | None, content: str | None) -> DeltaMessage | None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/reasoning/think_parser.py#L262-L276 - Implementation: Method `BaseThinkingReasoningParser._transition_to_content` calls `self._promote_tool_calls`, `self._content_delta`, `DeltaMessage`; has 2 explicit return paths. Return a delta while suppressing leading post-transition think blocks. - Inputs: - `reasoning` (str | None; required): Required positional or keyword input. - `content` (str | None; required): Required positional or keyword input. - Return annotation: `DeltaMessage | None` - Calls: self._promote_tool_calls, self._content_delta, DeltaMessage - State reads: self._promote_tool_calls, self._content_delta - Return expressions: None; DeltaMessage(reasoning=reasoning_text or None, content=final_content or None) ## `vllm_mlx.reasoning.think_parser.BaseThinkingReasoningParser._content_delta` - Kind: method - Signature: `def _content_delta(self, delta_text: str) -> DeltaMessage | None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/reasoning/think_parser.py#L278-L325 - Implementation: Method `BaseThinkingReasoningParser._content_delta` updates `self._content_buffer`, `self._content_started`; calls `DeltaMessage`, `self._content_buffer.lstrip`, `buffer.startswith`, `buffer[len(self.end_token):].lstrip`; has 4 explicit return paths. Emit content after consuming repeated leading think blocks. - Inputs: - `delta_text` (str; required): Required positional or keyword input. - Return annotation: `DeltaMessage | None` - Calls: DeltaMessage, self._content_buffer.lstrip, buffer.startswith, buffer[len(self.end_token):].lstrip, len, after_start.find, reasoning_parts.append, after_start[end_idx + len(self.end_token):].lstrip, self.start_token.startswith, self.end_token.startswith, ''.join - State reads: self._content_buffer, self._content_started, self._content_buffer.lstrip, self.end_token, self.start_token, self.start_token.startswith, self.end_token.startswith - State writes: self._content_buffer, self._content_started - Return expressions: None; DeltaMessage(content=delta_text) if delta_text else None; DeltaMessage(reasoning=''.join(reasoning_parts) or None, content=buffer); DeltaMessage(reasoning=''.join(reasoning_parts)) ## `vllm_mlx.reasoning.think_parser.BaseThinkingReasoningParser._thinking_tool_call` - Kind: method - Signature: `def _thinking_tool_call(self, previous_text: str, current_text: str, delta_text: str) -> DeltaMessage | None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/reasoning/think_parser.py#L327-L396 - Implementation: Method `BaseThinkingReasoningParser._thinking_tool_call` updates `self._tool_call_buffer`, `self._in_tool_call`, `self._phase`; calls `self._tool_call_buffer.find`, `len`, `logger.warning`, `remainder.find`; has 4 explicit return paths. Handle streaming while inside a during thinking phase. - Inputs: - `previous_text` (str; required): Required positional or keyword input. - `current_text` (str; required): Required positional or keyword input. - `delta_text` (str; required): Required positional or keyword input. - Return annotation: `DeltaMessage | None` - Calls: self._tool_call_buffer.find, len, logger.warning, remainder.find, remainder[:eidx].strip, self._content_delta, DeltaMessage, remainder[:tc_idx].strip, remainder.strip - State reads: self._TOOL_CALL_END, self.end_token, self._tool_call_buffer.find, self._tool_call_buffer, self._content_delta, self._TOOL_CALL_START - State writes: self._tool_call_buffer, self._in_tool_call, self._phase - Return expressions: DeltaMessage(content=final_content or None, reasoning=r_text or None); DeltaMessage(content=promoted, reasoning=reasoning); DeltaMessage(content=final_content or None); None ## `vllm_mlx.reasoning.think_parser.BaseThinkingReasoningParser.finalize_stream` - Kind: method - Signature: `def finalize_stream(self) -> DeltaMessage | None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/reasoning/think_parser.py#L398-L406 - Implementation: Method `BaseThinkingReasoningParser.finalize_stream` updates `self._tool_call_buffer`, `self._in_tool_call`; calls `logger.warning`, `DeltaMessage`; has 2 explicit return paths. Flush any buffered tool call text at end of stream. - Inputs: none - Return annotation: `DeltaMessage | None` - Calls: logger.warning, DeltaMessage - State reads: self._in_tool_call, self._tool_call_buffer - State writes: self._tool_call_buffer, self._in_tool_call - Return expressions: DeltaMessage(content=promoted); None ## `vllm_mlx.reasoning.think_parser.BaseThinkingReasoningParser._promote_tool_calls` - Kind: method - Signature: `def _promote_tool_calls(cls, reasoning: str | None, content: str | None) -> tuple[str | None, str | None]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/reasoning/think_parser.py#L409-L462 - Implementation: Method `BaseThinkingReasoningParser._promote_tool_calls` calls `cls._TOOL_CALL_CLOSED_RE.sub`, `cls._TOOL_CALL_UNCLOSED_RE.search`, `unclosed_match.group`, `unclosed_match.start`; has 2 explicit return paths. Method `BaseThinkingReasoningParser._promote_tool_calls` calls `cls._TOOL_CALL_CLOSED_RE.sub`, `cls._TOOL_CALL_UNCLOSED_RE.search`, `unclosed_match.group`, `unclosed_match.start`; has 2 explicit return paths. - Inputs: - `reasoning` (str | None; required): Required positional or keyword input. - `content` (str | None; required): Required positional or keyword input. - Return annotation: `tuple[str | None, str | None]` - Decorators: classmethod - Calls: cls._TOOL_CALL_CLOSED_RE.sub, cls._TOOL_CALL_UNCLOSED_RE.search, unclosed_match.group, unclosed_match.start, cleaned.strip, len, '\n'.join, result_content.strip, logger.warning - State reads: cls._TOOL_CALL_CLOSED_RE.sub, cls._TOOL_CALL_CLOSED_RE, cls._TOOL_CALL_UNCLOSED_RE.search, cls._TOOL_CALL_UNCLOSED_RE - Return expressions: (reasoning, content); (cleaned, result_content) ## `vllm_mlx.reasoning.think_parser.BaseThinkingReasoningParser._promote_tool_calls._collect_closed` - Kind: nested function - Signature: `def _collect_closed(match)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/reasoning/think_parser.py#L419-L421 - Implementation: Nested Function `BaseThinkingReasoningParser._promote_tool_calls._collect_closed` calls `closed.append`, `match.group`; returns `''`. Nested Function `BaseThinkingReasoningParser._promote_tool_calls._collect_closed` calls `closed.append`, `match.group`; returns `''`. - Inputs: - `match` (not annotated; required): Required positional or keyword input. - Return annotation: `not annotated` - Calls: closed.append, match.group - Return expressions: '' # Module `vllm_mlx.request` Request management for vllm-mlx continuous batching. This module provides Request and RequestStatus classes adapted from vLLM's request management system, simplified for MLX backend. Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/request.py#L1-L227 ## `vllm_mlx.request.RequestStatus` - Kind: class - Signature: `class RequestStatus(enum.IntEnum)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/request.py#L18-L48 - Implementation: Class `RequestStatus` derives from `enum.IntEnum` and declares 2 direct member(s). Status of a request in the scheduling system. - Inputs: none - Constructs: `vllm_mlx.request.RequestStatus` ## `vllm_mlx.request.RequestStatus.is_finished` - Kind: method - Signature: `def is_finished(status: 'RequestStatus') -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/request.py#L35-L37 - Implementation: Method `RequestStatus.is_finished` returns `status > RequestStatus.PREEMPTED`. Check if the status indicates a finished request. - Inputs: - `status` ('RequestStatus'; required): Required positional or keyword input. - Return annotation: `bool` - Decorators: staticmethod - Return expressions: status > RequestStatus.PREEMPTED ## `vllm_mlx.request.RequestStatus.get_finish_reason` - Kind: method - Signature: `def get_finish_reason(status: 'RequestStatus') -> Optional[str]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/request.py#L40-L48 - Implementation: Method `RequestStatus.get_finish_reason` has 4 explicit return paths. Get the finish reason string for a finished status. - Inputs: - `status` ('RequestStatus'; required): Required positional or keyword input. - Return annotation: `Optional[str]` - Decorators: staticmethod - Return expressions: 'stop'; 'length'; 'abort'; None ## `vllm_mlx.request.SamplingParams` - Kind: class - Signature: `class SamplingParams` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/request.py#L52-L73 - Implementation: Class `SamplingParams` declares 1 direct member(s). Sampling parameters for text generation. - Inputs: - `max_tokens` (int; optional; default `256`): Optional constructor field; defaults to `256`. - `temperature` (float; optional; default `0.7`): Optional constructor field; defaults to `0.7`. - `top_p` (float; optional; default `0.9`): Optional constructor field; defaults to `0.9`. - `top_k` (int; optional; default `0`): Optional constructor field; defaults to `0`. - `min_p` (float; optional; default `0.0`): Optional constructor field; defaults to `0.0`. - `presence_penalty` (float; optional; default `0.0`): Optional constructor field; defaults to `0.0`. - `repetition_penalty` (float; optional; default `1.0`): Optional constructor field; defaults to `1.0`. - `stop` (Optional[List[str]]; optional; default `None`): Optional constructor field; defaults to `None`. - `stop_token_ids` (Optional[List[int]]; optional; default `None`): Optional constructor field; defaults to `None`. - `logits_processors` (Optional[List[Callable]]; optional; default `None`): Optional constructor field; defaults to `None`. - Constructs: `vllm_mlx.request.SamplingParams` - Decorators: dataclass ## `vllm_mlx.request.SamplingParams.__post_init__` - Kind: method - Signature: `def __post_init__(self)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/request.py#L69-L73 - Implementation: Method `SamplingParams.__post_init__` updates `self.stop`, `self.stop_token_ids`. Method `SamplingParams.__post_init__` updates `self.stop`, `self.stop_token_ids`. - Inputs: none - Return annotation: `not annotated` - State reads: self.stop, self.stop_token_ids - State writes: self.stop, self.stop_token_ids ## `vllm_mlx.request.Request` - Kind: class - Signature: `class Request` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/request.py#L77-L192 - Implementation: Class `Request` declares 10 direct member(s). Represents a single inference request in the scheduling system. Adapted from vLLM's Request class with simplifications for MLX backend. Attributes: request_id: Unique identifier for this request prompt: The input prompt (string or token ids) prompt_token_ids: Tokenized prompt sampling_params: Parameters for generation arrival_time: When the request was received status: Current status of the request num_prompt_tokens: Number of tokens in the prompt num_computed_tokens: Number of tokens processed so far output_token_ids: Generated token ids output_text: Generated text (decoded) - Inputs: - `request_id` (str; required): Required constructor field. - `prompt` (Union[str, List[int]]; required): Required constructor field. - `sampling_params` (SamplingParams; required): Required constructor field. - `arrival_time` (float; optional; default `field(default_factory=time.time)`): Optional constructor field; defaults to `field(default_factory=time.time)`. - `priority` (int; optional; default `0`): Optional constructor field; defaults to `0`. - `prompt_token_ids` (Optional[List[int]]; optional; default `None`): Optional constructor field; defaults to `None`. - `num_prompt_tokens` (int; optional; default `0`): Optional constructor field; defaults to `0`. - `status` (RequestStatus; optional; default `RequestStatus.WAITING`): Optional constructor field; defaults to `RequestStatus.WAITING`. - `num_computed_tokens` (int; optional; default `0`): Optional constructor field; defaults to `0`. - `output_token_ids` (List[int]; optional; default `field(default_factory=list)`): Optional constructor field; defaults to `field(default_factory=list)`. - `output_text` (str; optional; default `''`): Optional constructor field; defaults to `''`. - `batch_uid` (Optional[int]; optional; default `None`): Optional constructor field; defaults to `None`. - `prompt_cache` (Optional[List[Any]]; optional; default `None`): Optional constructor field; defaults to `None`. - `cached_tokens` (int; optional; default `0`): Optional constructor field; defaults to `0`. - `remaining_tokens` (Optional[List[int]]; optional; default `None`): Optional constructor field; defaults to `None`. - `prefix_boundary` (int; optional; default `0`): Optional constructor field; defaults to `0`. - `block_table` (Optional['BlockTable']; optional; default `None`): Optional constructor field; defaults to `None`. - `shared_prefix_blocks` (int; optional; default `0`): Optional constructor field; defaults to `0`. - `images` (Optional[List[Any]]; optional; default `None`): Optional constructor field; defaults to `None`. - `videos` (Optional[List[Any]]; optional; default `None`): Optional constructor field; defaults to `None`. - `pixel_values` (Optional[Any]; optional; default `None`): Optional constructor field; defaults to `None`. - `image_grid_thw` (Optional[Any]; optional; default `None`): Optional constructor field; defaults to `None`. - `attention_mask` (Optional[Any]; optional; default `None`): Optional constructor field; defaults to `None`. - `multimodal_kwargs` (Optional[Dict[str, Any]]; optional; default `None`): Optional constructor field; defaults to `None`. - `is_multimodal` (bool; optional; default `False`): Optional constructor field; defaults to `False`. - `finish_reason` (Optional[str]; optional; default `None`): Optional constructor field; defaults to `None`. - `first_token_time` (Optional[float]; optional; default `None`): Optional constructor field; defaults to `None`. - `cache_hit_type` (Optional[str]; optional; default `None`): Optional constructor field; defaults to `None`. - Constructs: `vllm_mlx.request.Request` - Decorators: dataclass ## `vllm_mlx.request.Request.num_output_tokens` - Kind: method - Signature: `def num_output_tokens(self) -> int` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/request.py#L146-L148 - Implementation: Method `Request.num_output_tokens` calls `len`; returns `len(self.output_token_ids)`. Number of output tokens generated so far. - Inputs: none - Return annotation: `int` - Decorators: property - Calls: len - State reads: self.output_token_ids - Return expressions: len(self.output_token_ids) ## `vllm_mlx.request.Request.num_tokens` - Kind: method - Signature: `def num_tokens(self) -> int` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/request.py#L151-L153 - Implementation: Method `Request.num_tokens` returns `self.num_prompt_tokens + self.num_output_tokens`. Total number of tokens (prompt + output). - Inputs: none - Return annotation: `int` - Decorators: property - State reads: self.num_prompt_tokens, self.num_output_tokens - Return expressions: self.num_prompt_tokens + self.num_output_tokens ## `vllm_mlx.request.Request.max_tokens` - Kind: method - Signature: `def max_tokens(self) -> int` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/request.py#L156-L158 - Implementation: Method `Request.max_tokens` returns `self.sampling_params.max_tokens`. Maximum output tokens for this request. - Inputs: none - Return annotation: `int` - Decorators: property - State reads: self.sampling_params.max_tokens, self.sampling_params - Return expressions: self.sampling_params.max_tokens ## `vllm_mlx.request.Request.is_finished` - Kind: method - Signature: `def is_finished(self) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/request.py#L160-L162 - Implementation: Method `Request.is_finished` calls `RequestStatus.is_finished`; returns `RequestStatus.is_finished(self.status)`. Check if request has finished. - Inputs: none - Return annotation: `bool` - Calls: RequestStatus.is_finished - State reads: self.status - Return expressions: RequestStatus.is_finished(self.status) ## `vllm_mlx.request.Request.get_finish_reason` - Kind: method - Signature: `def get_finish_reason(self) -> Optional[str]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/request.py#L164-L168 - Implementation: Method `Request.get_finish_reason` calls `RequestStatus.get_finish_reason`; has 2 explicit return paths. Get the finish reason if finished. - Inputs: none - Return annotation: `Optional[str]` - Calls: RequestStatus.get_finish_reason - State reads: self.finish_reason, self.status - Return expressions: self.finish_reason; RequestStatus.get_finish_reason(self.status) ## `vllm_mlx.request.Request.append_output_token` - Kind: method - Signature: `def append_output_token(self, token_id: int) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/request.py#L170-L173 - Implementation: Method `Request.append_output_token` updates `self.num_computed_tokens`; calls `self.output_token_ids.append`. Append a generated token to the output. - Inputs: - `token_id` (int; required): Required positional or keyword input. - Return annotation: `None` - Calls: self.output_token_ids.append - State reads: self.output_token_ids.append, self.output_token_ids - State writes: self.num_computed_tokens ## `vllm_mlx.request.Request.set_finished` - Kind: method - Signature: `def set_finished(self, status: RequestStatus, reason: Optional[str]=None) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/request.py#L175-L178 - Implementation: Method `Request.set_finished` updates `self.status`, `self.finish_reason`; calls `RequestStatus.get_finish_reason`. Mark the request as finished. - Inputs: - `status` (RequestStatus; required): Required positional or keyword input. - `reason` (Optional[str]; optional; default `None`): Optional positional or keyword input; defaults to `None`. - Return annotation: `None` - Calls: RequestStatus.get_finish_reason - State writes: self.status, self.finish_reason ## `vllm_mlx.request.Request.__lt__` - Kind: method - Signature: `def __lt__(self, other: 'Request') -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/request.py#L180-L184 - Implementation: Method `Request.__lt__` has 2 explicit return paths. Compare requests for priority queue ordering. - Inputs: - `other` ('Request'; required): Required positional or keyword input. - Return annotation: `bool` - State reads: self.priority, self.arrival_time - Return expressions: self.priority < other.priority; self.arrival_time < other.arrival_time ## `vllm_mlx.request.Request.__hash__` - Kind: method - Signature: `def __hash__(self) -> int` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/request.py#L186-L187 - Implementation: Method `Request.__hash__` calls `hash`; returns `hash(self.request_id)`. Method `Request.__hash__` calls `hash`; returns `hash(self.request_id)`. - Inputs: none - Return annotation: `int` - Calls: hash - State reads: self.request_id - Return expressions: hash(self.request_id) ## `vllm_mlx.request.Request.__eq__` - Kind: method - Signature: `def __eq__(self, other: object) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/request.py#L189-L192 - Implementation: Method `Request.__eq__` calls `isinstance`; has 2 explicit return paths. Method `Request.__eq__` calls `isinstance`; has 2 explicit return paths. - Inputs: - `other` (object; required): Required positional or keyword input. - Return annotation: `bool` - Calls: isinstance - State reads: self.request_id - Return expressions: False; self.request_id == other.request_id ## `vllm_mlx.request.RequestOutput` - Kind: class - Signature: `class RequestOutput` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/request.py#L196-L227 - Implementation: Class `RequestOutput` declares 1 direct member(s). Output for a single request after a generation step. This is returned by the engine to communicate results back to the API layer. - Inputs: - `request_id` (str; required): Required constructor field. - `new_token_ids` (List[int]; optional; default `field(default_factory=list)`): Optional constructor field; defaults to `field(default_factory=list)`. - `new_text` (str; optional; default `''`): Optional constructor field; defaults to `''`. - `output_token_ids` (List[int]; optional; default `field(default_factory=list)`): Optional constructor field; defaults to `field(default_factory=list)`. - `output_text` (str; optional; default `''`): Optional constructor field; defaults to `''`. - `finished` (bool; optional; default `False`): Optional constructor field; defaults to `False`. - `finish_reason` (Optional[str]; optional; default `None`): Optional constructor field; defaults to `None`. - `prompt_tokens` (int; optional; default `0`): Optional constructor field; defaults to `0`. - `completion_tokens` (int; optional; default `0`): Optional constructor field; defaults to `0`. - `mtp_drafts` (int; optional; default `0`): Optional constructor field; defaults to `0`. - `mtp_accepted` (int; optional; default `0`): Optional constructor field; defaults to `0`. - Constructs: `vllm_mlx.request.RequestOutput` - Decorators: dataclass ## `vllm_mlx.request.RequestOutput.usage` - Kind: method - Signature: `def usage(self) -> Dict[str, int]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/request.py#L221-L227 - Implementation: Method `RequestOutput.usage` returns `{'prompt_tokens': self.prompt_tokens, 'completion_tokens': self.completion_tokens, 'total_tokens': self.prompt_tokens +…`. Return usage statistics compatible with OpenAI API. - Inputs: none - Return annotation: `Dict[str, int]` - Decorators: property - State reads: self.prompt_tokens, self.completion_tokens - Return expressions: {'prompt_tokens': self.prompt_tokens, 'completion_tokens': self.completion_tokens, 'total_tokens': self.prompt_tokens +… # Module `vllm_mlx.rerank` Reranker engine for cross-encoder models. Provides a dedicated RerankEngine with adapter-based scoring for the OpenAI/Jina-compatible /v1/rerank endpoint. Cross-encoder models use AutoModelForSequenceClassification-style loading, not mlx_lm.load. Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/rerank.py#L1-L398 ## `vllm_mlx.rerank.RerankAdapter` - Kind: class - Signature: `class RerankAdapter(ABC)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/rerank.py#L29-L80 - Implementation: Class `RerankAdapter` derives from `ABC` and declares 3 direct member(s). Per-family adapter for reranker models. Different cross-encoder families use different tokenization patterns, score extraction logic, and normalization functions. This contract isolates those differences so RerankEngine stays family-agnostic. - Inputs: none - Constructs: `vllm_mlx.rerank.RerankAdapter` ## `vllm_mlx.rerank.RerankAdapter.tokenize_pair` - Kind: method - Signature: `def tokenize_pair(self, tokenizer, query: str, document: str, max_length: int) -> dict` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/rerank.py#L39-L54 - Implementation: Method `RerankAdapter.tokenize_pair` contains no state mutation, call, raise, return, await, or yield. Tokenize a (query, document) pair for the cross-encoder. Args: tokenizer: The HuggingFace tokenizer instance. query: The query string. document: The document string. max_length: Truncation length (from the model's context window). Returns: Dict with 'input_ids' and 'attention_mask' as numpy arrays. - Inputs: - `tokenizer` (not annotated; required): The HuggingFace tokenizer instance. - `query` (str; required): The query string. - `document` (str; required): The document string. - `max_length` (int; required): Truncation length (from the model's context window). - Return annotation: `dict` - Decorators: abstractmethod ## `vllm_mlx.rerank.RerankAdapter.extract_score` - Kind: method - Signature: `def extract_score(self, logits) -> float` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/rerank.py#L57-L67 - Implementation: Method `RerankAdapter.extract_score` contains no state mutation, call, raise, return, await, or yield. Extract a raw relevance score from model output logits. Args: logits: Model output logits (list or array), shape varies by model. Returns: A single float raw score. - Inputs: - `logits` (not annotated; required): Model output logits (list or array), shape varies by model. - Return annotation: `float` - Decorators: abstractmethod ## `vllm_mlx.rerank.RerankAdapter.normalize` - Kind: method - Signature: `def normalize(self, raw_score: float) -> float` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/rerank.py#L70-L80 - Implementation: Method `RerankAdapter.normalize` contains no state mutation, call, raise, return, await, or yield. Normalize a raw score to [0, 1] range. Args: raw_score: The raw score from extract_score(). Returns: Normalized relevance score in [0, 1]. - Inputs: - `raw_score` (float; required): The raw score from extract_score(). - Return annotation: `float` - Decorators: abstractmethod ## `vllm_mlx.rerank.SigmoidAdapter` - Kind: class - Signature: `class SigmoidAdapter(RerankAdapter)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/rerank.py#L83-L111 - Implementation: Class `SigmoidAdapter` derives from `RerankAdapter` and declares 3 direct member(s). Default adapter for single-logit sigmoid rerankers. Works with Jina Reranker v2, BGE Reranker v2, and MS-MARCO MiniLM families. These models output a single relevance logit at position 0, normalized via sigmoid. - Inputs: none - Constructs: `vllm_mlx.rerank.SigmoidAdapter` ## `vllm_mlx.rerank.SigmoidAdapter.tokenize_pair` - Kind: method - Signature: `def tokenize_pair(self, tokenizer, query: str, document: str, max_length: int) -> dict` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/rerank.py#L92-L103 - Implementation: Method `SigmoidAdapter.tokenize_pair` calls `tokenizer`; returns `tokenizer(query, document, padding=True, truncation=True, max_length=max_length, return_tensors='np')`. Tokenize as a sentence pair (query, document). - Inputs: - `tokenizer` (not annotated; required): Required positional or keyword input. - `query` (str; required): Required positional or keyword input. - `document` (str; required): Required positional or keyword input. - `max_length` (int; required): Required positional or keyword input. - Return annotation: `dict` - Calls: tokenizer - Return expressions: tokenizer(query, document, padding=True, truncation=True, max_length=max_length, return_tensors='np') ## `vllm_mlx.rerank.SigmoidAdapter.extract_score` - Kind: method - Signature: `def extract_score(self, logits) -> float` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/rerank.py#L105-L107 - Implementation: Method `SigmoidAdapter.extract_score` calls `float`; returns `float(logits[0])`. Extract the first logit as the relevance score. - Inputs: - `logits` (not annotated; required): Required positional or keyword input. - Return annotation: `float` - Calls: float - Return expressions: float(logits[0]) ## `vllm_mlx.rerank.SigmoidAdapter.normalize` - Kind: method - Signature: `def normalize(self, raw_score: float) -> float` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/rerank.py#L109-L111 - Implementation: Method `SigmoidAdapter.normalize` calls `math.exp`; returns `1.0 / (1.0 + math.exp(-raw_score))`. Apply sigmoid normalization. - Inputs: - `raw_score` (float; required): Required positional or keyword input. - Return annotation: `float` - Calls: math.exp - Return expressions: 1.0 / (1.0 + math.exp(-raw_score)) ## `vllm_mlx.rerank.get_adapter` - Kind: function - Signature: `def get_adapter(model_name: str) -> RerankAdapter` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/rerank.py#L124-L133 - Implementation: Function `get_adapter` calls `_ADAPTER_REGISTRY['default']`; returns `_ADAPTER_REGISTRY['default']()`. Return the appropriate adapter for a model. Falls back to SigmoidAdapter (works for Jina, BGE, MS-MARCO families). Extend _ADAPTER_REGISTRY for families that need different scoring. - Inputs: - `model_name` (str; required): Required positional or keyword input. - Return annotation: `RerankAdapter` - Calls: _ADAPTER_REGISTRY['default'] - Return expressions: _ADAPTER_REGISTRY['default']() ## `vllm_mlx.rerank.RerankEngine` - Kind: class - Signature: `class RerankEngine` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/rerank.py#L136-L338 - Implementation: Class `RerankEngine` declares 6 direct member(s). Reranker engine for cross-encoder sequence classification models. Loads cross-encoder models via transformers + MLX (safetensors weights). Scores (query, document) pairs using the adapter contract for family-specific tokenization, score extraction, and normalization. Supports token-budget batching to avoid OOM on large document lists. - Inputs: - `model_name` (str; required): Required positional or keyword input. - `token_budget` (int; optional; default `4096`): Optional positional or keyword input; defaults to `4096`. - `max_concurrency` (int; optional; default `1`): Optional positional or keyword input; defaults to `1`. - Constructs: `vllm_mlx.rerank.RerankEngine` ## `vllm_mlx.rerank.RerankEngine.__init__` - Kind: method - Signature: `def __init__(self, model_name: str, token_budget: int=4096, max_concurrency: int=1)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/rerank.py#L147-L159 - Implementation: Method `RerankEngine.__init__` updates `self.model_name`, `self.token_budget`, `self.max_concurrency`, `self._semaphore`; calls `asyncio.Semaphore`. Method `RerankEngine.__init__` updates `self.model_name`, `self.token_budget`, `self.max_concurrency`, `self._semaphore`; calls `asyncio.Semaphore`. - Inputs: - `model_name` (str; required): Required positional or keyword input. - `token_budget` (int; optional; default `4096`): Optional positional or keyword input; defaults to `4096`. - `max_concurrency` (int; optional; default `1`): Optional positional or keyword input; defaults to `1`. - Return annotation: `not annotated` - Calls: asyncio.Semaphore - State writes: self.model_name, self.token_budget, self.max_concurrency, self._semaphore, self._model, self._tokenizer, self._adapter ## `vllm_mlx.rerank.RerankEngine.is_loaded` - Kind: method - Signature: `def is_loaded(self) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/rerank.py#L162-L165 - Implementation: Method `RerankEngine.is_loaded` returns `self._model is not None`. Return whether the reranking model has been loaded. - Inputs: none - Return annotation: `bool` - Decorators: property - State reads: self._model - Return expressions: self._model is not None ## `vllm_mlx.rerank.RerankEngine.load` - Kind: method - Signature: `def load(self) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/rerank.py#L167-L184 - Implementation: Method `RerankEngine.load` updates `self._tokenizer`, `self._model`, `self._adapter`; calls `logger.info`, `time.perf_counter`, `AutoTokenizer.from_pretrained`, `self._load_mlx_model`. Load the cross-encoder model and tokenizer. Uses transformers AutoTokenizer and loads MLX weights from safetensors via the model's from_pretrained or equivalent MLX loading path. - Inputs: none - Return annotation: `None` - Calls: logger.info, time.perf_counter, AutoTokenizer.from_pretrained, self._load_mlx_model, get_adapter - State reads: self.model_name, self._load_mlx_model - State writes: self._tokenizer, self._model, self._adapter ## `vllm_mlx.rerank.RerankEngine._load_mlx_model` - Kind: method - Signature: `def _load_mlx_model(model_name: str)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/rerank.py#L187-L230 - Implementation: Method `RerankEngine._load_mlx_model` calls `snapshot_download`, `os.path.join`, `open`, `json.load`; can raise `FileNotFoundError`; returns `model`. Load an MLX cross-encoder model from HuggingFace Hub. Attempts mlx-community weights first (safetensors), then falls back to transformers AutoModelForSequenceClassification with MLX conversion. - Inputs: - `model_name` (str; required): Required positional or keyword input. - Return annotation: `not annotated` - Decorators: staticmethod - Calls: snapshot_download, os.path.join, open, json.load, glob.glob, FileNotFoundError, safe_open, f.keys, mx.array, f.get_tensor, config.get, _build_classifier_model, mx.eval, model.parameters, logger.error - Raises directly: FileNotFoundError - Return expressions: model ## `vllm_mlx.rerank.RerankEngine._ensure_loaded` - Kind: method - Signature: `def _ensure_loaded(self) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/rerank.py#L232-L234 - Implementation: Method `RerankEngine._ensure_loaded` calls `self.load`. Method `RerankEngine._ensure_loaded` calls `self.load`. - Inputs: none - Return annotation: `None` - Calls: self.load - State reads: self.is_loaded, self.load ## `vllm_mlx.rerank.RerankEngine.score_pairs` - Kind: method - Signature: `def score_pairs(self, query: str, documents: list[str]) -> tuple[list[float], int]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/rerank.py#L236-L338 - Implementation: Method `RerankEngine.score_pairs` calls `self._ensure_loaded`, `resolve_max_length`, `getattr`, `self._adapter.tokenize_pair`; returns `([score for _, score in all_scores], total_tokens)`. Score each (query, document) pair and return normalized relevance scores. Pairs are batched by token budget to control memory usage. Each batch is tokenized together and scored in a single forward pass. Returns (scores, total_tokens) where total_tokens reflects the actual tokenization used for scoring (consistent with adapter). Args: query: The query string. documents: List of document strings. Returns: List of normalized relevance scores, one per document, in the same order as the input documents. - Inputs: - `query` (str; required): The query string. - `documents` (list[str]; required): List of document strings. - Return annotation: `tuple[list[float], int]` - Calls: self._ensure_loaded, resolve_max_length, getattr, self._adapter.tokenize_pair, pair_encodings.append, hasattr, len, pair_token_counts.append, enumerate, zip, batches.append, current_batch.append, mx.array, max, raw_ids.tolist, list, raw_mask.tolist, padded_ids.append, padded_mask.append, self._model, output.logits.tolist, self._adapter.extract_score, self._adapter.normalize, all_scores.append, all_scores.sort, sum - State reads: self._ensure_loaded, self._model, self._tokenizer, self._adapter.tokenize_pair, self._adapter, self.token_budget, self._adapter.extract_score, self._adapter.normalize - Return expressions: ([score for _, score in all_scores], total_tokens) ## `vllm_mlx.rerank._build_classifier_model` - Kind: function - Signature: `def _build_classifier_model(model_type, config, weights, num_labels)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/rerank.py#L341-L349 - Implementation: Function `_build_classifier_model` calls `_MLXClassifierWrapper`; returns `_MLXClassifierWrapper(config, weights, num_labels)`. Build an MLX sequence classification model from config and weights. This is a thin wrapper that constructs the appropriate encoder architecture with a classification head on top. - Inputs: - `model_type` (not annotated; required): Required positional or keyword input. - `config` (not annotated; required): Required positional or keyword input. - `weights` (not annotated; required): Required positional or keyword input. - `num_labels` (not annotated; required): Required positional or keyword input. - Return annotation: `not annotated` - Calls: _MLXClassifierWrapper - Return expressions: _MLXClassifierWrapper(config, weights, num_labels) ## `vllm_mlx.rerank._MLXClassifierWrapper` - Kind: class - Signature: `class _MLXClassifierWrapper` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/rerank.py#L352-L391 - Implementation: Class `_MLXClassifierWrapper` declares 3 direct member(s). Minimal MLX wrapper for sequence classification models. Wraps loaded safetensors weights into a callable that returns logits for (input_ids, attention_mask) pairs. Supports BERT-family and XLM-RoBERTa-family architectures commonly used as cross-encoders. - Inputs: - `config` (dict; required): Required positional or keyword input. - `weights` (dict; required): Required positional or keyword input. - `num_labels` (int; required): Required positional or keyword input. - Constructs: `vllm_mlx.rerank._MLXClassifierWrapper` ## `vllm_mlx.rerank._MLXClassifierWrapper.__init__` - Kind: method - Signature: `def __init__(self, config: dict, weights: dict, num_labels: int)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/rerank.py#L361-L365 - Implementation: Method `_MLXClassifierWrapper.__init__` updates `self.config`, `self.weights`, `self.num_labels`, `self._params`; calls `list`, `weights.values`. Method `_MLXClassifierWrapper.__init__` updates `self.config`, `self.weights`, `self.num_labels`, `self._params`; calls `list`, `weights.values`. - Inputs: - `config` (dict; required): Required positional or keyword input. - `weights` (dict; required): Required positional or keyword input. - `num_labels` (int; required): Required positional or keyword input. - Return annotation: `not annotated` - Calls: list, weights.values - State writes: self.config, self.weights, self.num_labels, self._params ## `vllm_mlx.rerank._MLXClassifierWrapper.parameters` - Kind: method - Signature: `def parameters(self)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/rerank.py#L367-L369 - Implementation: Method `_MLXClassifierWrapper.parameters` returns `self._params`. Return model parameters for mx.eval. - Inputs: none - Return annotation: `not annotated` - State reads: self._params - Return expressions: self._params ## `vllm_mlx.rerank._MLXClassifierWrapper.__call__` - Kind: method - Signature: `def __call__(self, input_ids: mx.array, attention_mask: mx.array=None)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/rerank.py#L371-L391 - Implementation: Method `_MLXClassifierWrapper.__call__` calls `classifier_forward`, `_ClassifierOutput`; returns `_ClassifierOutput(logits=logits)`. Forward pass through the classifier. For encoder-only cross-encoders, this runs the full transformer encoder and classification head. The exact layer wiring depends on the model architecture. This initial implementation uses a weight-lookup forward pass that works for standard BERT/XLM-RoBERTa classifiers. For models with non-standard architectures, register a custom adapter via _ADAPTER_REGISTRY. - Inputs: - `input_ids` (mx.array; required): Required positional or keyword input. - `attention_mask` (mx.array; optional; default `None`): Optional positional or keyword input; defaults to `None`. - Return annotation: `not annotated` - Calls: classifier_forward, _ClassifierOutput - State reads: self.weights, self.config - Return expressions: _ClassifierOutput(logits=logits) ## `vllm_mlx.rerank._ClassifierOutput` - Kind: class - Signature: `class _ClassifierOutput` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/rerank.py#L394-L398 - Implementation: Class `_ClassifierOutput` declares 1 direct member(s). Simple container for classifier output logits. - Inputs: - `logits` (mx.array; required): Required positional or keyword input. - Constructs: `vllm_mlx.rerank._ClassifierOutput` ## `vllm_mlx.rerank._ClassifierOutput.__init__` - Kind: method - Signature: `def __init__(self, logits: mx.array)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/rerank.py#L397-L398 - Implementation: Method `_ClassifierOutput.__init__` updates `self.logits`. Method `_ClassifierOutput.__init__` updates `self.logits`. - Inputs: - `logits` (mx.array; required): Required positional or keyword input. - Return annotation: `not annotated` - State writes: self.logits # Module `vllm_mlx.rerank_forward` MLX forward pass for BERT-family sequence classification models. Implements a from-weights forward pass for cross-encoder rerankers that use the standard BERT/XLM-RoBERTa architecture with a classification head. This avoids pulling in the full transformers modeling stack at inference time — only the tokenizer is needed from transformers. Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/rerank_forward.py#L1-L265 ## `vllm_mlx.rerank_forward.classifier_forward` - Kind: function - Signature: `def classifier_forward(input_ids: mx.array, attention_mask: mx.array, weights: dict[str, mx.array], config: dict) -> mx.array` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/rerank_forward.py#L16-L85 - Implementation: Function `classifier_forward` calls `config.get`, `_detect_prefix`, `_position_ids_for_config`, `mx.zeros_like`; returns `logits`. Run a BERT-family classifier forward pass on MLX. Args: input_ids: (batch, seq_len) token IDs. attention_mask: (batch, seq_len) attention mask (1=attend, 0=pad). weights: Dict mapping weight name -> mx.array. config: Model config dict (from config.json). Returns: logits: (batch, num_labels) classification logits. - Inputs: - `input_ids` (mx.array; required): (batch, seq_len) token IDs. - `attention_mask` (mx.array; required): (batch, seq_len) attention mask (1=attend, 0=pad). - `weights` (dict[str, mx.array]; required): Dict mapping weight name -> mx.array. - `config` (dict; required): Model config dict (from config.json). - Return annotation: `mx.array` - Calls: config.get, _detect_prefix, _position_ids_for_config, mx.zeros_like, _layer_norm, attention_mask[:, None, None, :].astype, range, _encoder_layer, weights.get, mx.tanh, _classification_head_forward - Return expressions: logits ## `vllm_mlx.rerank_forward._position_ids_for_config` - Kind: function - Signature: `def _position_ids_for_config(config: dict, input_ids: mx.array, attention_mask: mx.array | None) -> mx.array` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/rerank_forward.py#L88-L105 - Implementation: Function `_position_ids_for_config` calls `str(config.get('model_type', '')).lower`, `str`, `config.get`, `mx.arange`; has 3 explicit return paths. Build BERT or RoBERTa-family absolute position IDs. - Inputs: - `config` (dict; required): Required positional or keyword input. - `input_ids` (mx.array; required): Required positional or keyword input. - `attention_mask` (mx.array | None; required): Required positional or keyword input. - Return annotation: `mx.array` - Calls: str(config.get('model_type', '')).lower, str, config.get, mx.arange, int, attention_mask.astype, mx.cumsum, positions.astype - Return expressions: mx.arange(seq_len)[None, :]; mx.arange(padding_idx + 1, seq_len + padding_idx + 1)[None, :]; positions.astype(mx.int32) ## `vllm_mlx.rerank_forward._classification_head_forward` - Kind: function - Signature: `def _classification_head_forward(pooled: mx.array, weights: dict[str, mx.array]) -> mx.array` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/rerank_forward.py#L108-L122 - Implementation: Function `_classification_head_forward` calls `mx.tanh`; has 2 explicit return paths. Run BERT flat or XLM-RoBERTa two-layer sequence-classification head. - Inputs: - `pooled` (mx.array; required): Required positional or keyword input. - `weights` (dict[str, mx.array]; required): Required positional or keyword input. - Return annotation: `mx.array` - Calls: mx.tanh - Return expressions: hidden @ weights['classifier.out_proj.weight'].T + weights['classifier.out_proj.bias']; pooled @ weights['classifier.weight'].T + weights['classifier.bias'] ## `vllm_mlx.rerank_forward._detect_prefix` - Kind: function - Signature: `def _detect_prefix(weights: dict) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/rerank_forward.py#L125-L135 - Implementation: Function `_detect_prefix` calls `key.startswith`; has 3 explicit return paths. Detect the model weight prefix (bert, roberta, xlm-roberta). - Inputs: - `weights` (dict; required): Required positional or keyword input. - Return annotation: `str` - Calls: key.startswith - Return expressions: 'bert'; 'roberta'; 'xlm-roberta' ## `vllm_mlx.rerank_forward._layer_norm` - Kind: function - Signature: `def _layer_norm(x: mx.array, weight: mx.array, bias: mx.array, eps: float) -> mx.array` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/rerank_forward.py#L138-L142 - Implementation: Function `_layer_norm` calls `mx.mean`, `mx.var`, `mx.sqrt`; returns `weight * (x - mean) / mx.sqrt(var + eps) + bias`. Apply layer normalization. - Inputs: - `x` (mx.array; required): Required positional or keyword input. - `weight` (mx.array; required): Required positional or keyword input. - `bias` (mx.array; required): Required positional or keyword input. - `eps` (float; required): Required positional or keyword input. - Return annotation: `mx.array` - Calls: mx.mean, mx.var, mx.sqrt - Return expressions: weight * (x - mean) / mx.sqrt(var + eps) + bias ## `vllm_mlx.rerank_forward._encoder_layer` - Kind: function - Signature: `def _encoder_layer(hidden: mx.array, ext_mask: mx.array | None, weights: dict, prefix: str, num_heads: int, head_dim: int, eps: float, config: dict) -> mx.array` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/rerank_forward.py#L145-L217 - Implementation: Function `_encoder_layer` calls `(hidden @ q_w.T + q_b).reshape(batch_size, seq_len, num_heads, head_dim).transpose`, `(hidden @ q_w.T + q_b).reshape`, `(hidden @ k_w.T + k_b).reshape(batch_size, seq_len, num_heads, head_dim).transpose`, `(hidden @ k_w.T + k_b).reshape`; returns `hidden`. Run one BERT encoder layer (self-attention + FFN). - Inputs: - `hidden` (mx.array; required): Required positional or keyword input. - `ext_mask` (mx.array | None; required): Required positional or keyword input. - `weights` (dict; required): Required positional or keyword input. - `prefix` (str; required): Required positional or keyword input. - `num_heads` (int; required): Required positional or keyword input. - `head_dim` (int; required): Required positional or keyword input. - `eps` (float; required): Required positional or keyword input. - `config` (dict; required): Required positional or keyword input. - Return annotation: `mx.array` - Calls: (hidden @ q_w.T + q_b).reshape(batch_size, seq_len, num_heads, head_dim).transpose, (hidden @ q_w.T + q_b).reshape, (hidden @ k_w.T + k_b).reshape(batch_size, seq_len, num_heads, head_dim).transpose, (hidden @ k_w.T + k_b).reshape, (hidden @ v_w.T + v_b).reshape(batch_size, seq_len, num_heads, head_dim).transpose, (hidden @ v_w.T + v_b).reshape, k.transpose, mx.softmax, (attn_probs @ v).transpose(0, 2, 1, 3).reshape, (attn_probs @ v).transpose, _layer_norm, _apply_hidden_activation - Return expressions: hidden ## `vllm_mlx.rerank_forward._gelu` - Kind: function - Signature: `def _gelu(x: mx.array) -> mx.array` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/rerank_forward.py#L220-L222 - Implementation: Function `_gelu` calls `nn.gelu`; returns `nn.gelu(x)`. GELU activation (exact form). - Inputs: - `x` (mx.array; required): Required positional or keyword input. - Return annotation: `mx.array` - Calls: nn.gelu - Return expressions: nn.gelu(x) ## `vllm_mlx.rerank_forward._gelu_new` - Kind: function - Signature: `def _gelu_new(x: mx.array) -> mx.array` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/rerank_forward.py#L225-L227 - Implementation: Function `_gelu_new` calls `mx.tanh`; returns `0.5 * x * (1.0 + mx.tanh(0.7978845608028654 * (x + 0.044715 * x ** 3)))`. BERT GELU approximation used by transformers gelu_new. - Inputs: - `x` (mx.array; required): Required positional or keyword input. - Return annotation: `mx.array` - Calls: mx.tanh - Return expressions: 0.5 * x * (1.0 + mx.tanh(0.7978845608028654 * (x + 0.044715 * x ** 3))) ## `vllm_mlx.rerank_forward._relu` - Kind: function - Signature: `def _relu(x: mx.array) -> mx.array` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/rerank_forward.py#L230-L232 - Implementation: Function `_relu` calls `mx.maximum`; returns `mx.maximum(x, 0)`. ReLU activation. - Inputs: - `x` (mx.array; required): Required positional or keyword input. - Return annotation: `mx.array` - Calls: mx.maximum - Return expressions: mx.maximum(x, 0) ## `vllm_mlx.rerank_forward._silu` - Kind: function - Signature: `def _silu(x: mx.array) -> mx.array` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/rerank_forward.py#L235-L237 - Implementation: Function `_silu` calls `mx.sigmoid`; returns `x * mx.sigmoid(x)`. SiLU/swish activation. - Inputs: - `x` (mx.array; required): Required positional or keyword input. - Return annotation: `mx.array` - Calls: mx.sigmoid - Return expressions: x * mx.sigmoid(x) ## `vllm_mlx.rerank_forward._apply_hidden_activation` - Kind: function - Signature: `def _apply_hidden_activation(x: mx.array, config: dict) -> mx.array` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/rerank_forward.py#L240-L265 - Implementation: Function `_apply_hidden_activation` calls `config.get`, `isinstance`, `hidden_act.get`, `str(hidden_act).lower`; can raise `ValueError`; has 4 explicit return paths. Apply the configured encoder hidden activation. The MLX reranker forward pass targets standard BERT/XLM-RoBERTa-style sequence classifiers. Configs that request an activation outside that supported contract fail explicitly instead of silently using GELU. - Inputs: - `x` (mx.array; required): Required positional or keyword input. - `config` (dict; required): Required positional or keyword input. - Return annotation: `mx.array` - Calls: config.get, isinstance, hidden_act.get, str(hidden_act).lower, str, _gelu, _gelu_new, _relu, _silu, ValueError - Raises directly: ValueError - Return expressions: _gelu(x); _gelu_new(x); _relu(x); _silu(x) # Module `vllm_mlx.scheduler` Scheduler for vllm-mlx continuous batching. This module provides a Scheduler class that manages request scheduling using mlx-lm's BatchGenerator for efficient continuous batching. The scheduler follows vLLM's design with: - Waiting queue for pending requests - Running set for active requests - Continuous batching via BatchGenerator Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/scheduler.py#L1-L3518 ## `vllm_mlx.scheduler._normalize_logits_processors` - Kind: function - Signature: `def _normalize_logits_processors(logits_processors)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/scheduler.py#L46-L50 - Implementation: Function `_normalize_logits_processors` has 2 explicit return paths. Normalize empty per-sequence processor slots to lists. - Inputs: - `logits_processors` (not annotated; required): Required positional or keyword input. - Return annotation: `not annotated` - Return expressions: None; [processors or [] for processors in logits_processors] ## `vllm_mlx.scheduler._sanitize_batch_generator_logits_processors` - Kind: function - Signature: `def _sanitize_batch_generator_logits_processors(batch_generator) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/scheduler.py#L53-L65 - Implementation: Function `_sanitize_batch_generator_logits_processors` calls `getattr`, `hasattr`, `_normalize_logits_processors`, `isinstance`. Sanitize stale BatchGenerator processor state before decode. - Inputs: - `batch_generator` (not annotated; required): Required positional or keyword input. - Return annotation: `None` - Calls: getattr, hasattr, _normalize_logits_processors, isinstance ## `vllm_mlx.scheduler.SchedulingPolicy` - Kind: class - Signature: `class SchedulingPolicy(Enum)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/scheduler.py#L68-L72 - Implementation: Class `SchedulingPolicy` derives from `Enum` and declares 0 direct member(s). Scheduling policy for request ordering. - Inputs: none - Constructs: `vllm_mlx.scheduler.SchedulingPolicy` ## `vllm_mlx.scheduler.SchedulerConfig` - Kind: class - Signature: `class SchedulerConfig` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/scheduler.py#L76-L140 - Implementation: Class `SchedulerConfig` declares 1 direct member(s). Configuration for the scheduler. - Inputs: - `max_num_seqs` (int; optional; default `256`): Optional constructor field; defaults to `256`. - `max_num_batched_tokens` (int; optional; default `8192`): Optional constructor field; defaults to `8192`. - `policy` (SchedulingPolicy; optional; default `SchedulingPolicy.FCFS`): Optional constructor field; defaults to `SchedulingPolicy.FCFS`. - `prefill_batch_size` (int; optional; default `8`): Optional constructor field; defaults to `8`. - `completion_batch_size` (int; optional; default `32`): Optional constructor field; defaults to `32`. - `prefill_step_size` (int; optional; default `2048`): Optional constructor field; defaults to `2048`. - `mllm_prefill_step_size` (Optional[int]; optional; default `None`): Optional constructor field; defaults to `None`. - `enable_prefix_cache` (bool; optional; default `True`): Optional constructor field; defaults to `True`. - `prefix_cache_size` (int; optional; default `100`): Optional constructor field; defaults to `100`. - `use_memory_aware_cache` (bool; optional; default `True`): Optional constructor field; defaults to `True`. - `cache_memory_mb` (Optional[int]; optional; default `None`): Optional constructor field; defaults to `None`. - `cache_memory_percent` (float; optional; default `0.2`): Optional constructor field; defaults to `0.2`. - `kv_cache_quantization` (bool; optional; default `False`): Optional constructor field; defaults to `False`. - `kv_cache_quantization_bits` (int; optional; default `8`): Optional constructor field; defaults to `8`. - `kv_cache_quantization_group_size` (int; optional; default `64`): Optional constructor field; defaults to `64`. - `kv_cache_min_quantize_tokens` (int; optional; default `256`): Optional constructor field; defaults to `256`. - `use_paged_cache` (bool; optional; default `False`): Optional constructor field; defaults to `False`. - `paged_cache_block_size` (int; optional; default `64`): Optional constructor field; defaults to `64`. - `max_cache_blocks` (int; optional; default `1000`): Optional constructor field; defaults to `1000`. - `chunked_prefill_tokens` (int; optional; default `0`): Optional constructor field; defaults to `0`. - `mid_prefill_save_interval` (int; optional; default `8192`): Optional constructor field; defaults to `8192`. - `ssd_cache_dir` (Optional[str]; optional; default `None`): Optional constructor field; defaults to `None`. - `ssd_cache_max_gb` (float; optional; default `10.0`): Optional constructor field; defaults to `10.0`. - `max_kv_size` (int; optional; default `0`): Optional constructor field; defaults to `0`. - `enable_mtp` (bool; optional; default `False`): Optional constructor field; defaults to `False`. - `mtp_num_draft_tokens` (int; optional; default `1`): Optional constructor field; defaults to `1`. - `mtp_optimistic` (bool; optional; default `False`): Optional constructor field; defaults to `False`. - Constructs: `vllm_mlx.scheduler.SchedulerConfig` - Decorators: dataclass ## `vllm_mlx.scheduler.SchedulerConfig.__post_init__` - Kind: method - Signature: `def __post_init__(self) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/scheduler.py#L138-L140 - Implementation: Method `SchedulerConfig.__post_init__` calls `ValueError`; can raise `ValueError`. Method `SchedulerConfig.__post_init__` calls `ValueError`; can raise `ValueError`. - Inputs: none - Return annotation: `None` - Calls: ValueError - State reads: self.mllm_prefill_step_size - Raises directly: ValueError ## `vllm_mlx.scheduler.SchedulerOutput` - Kind: class - Signature: `class SchedulerOutput` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/scheduler.py#L144-L160 - Implementation: Class `SchedulerOutput` declares 0 direct member(s). Output from a scheduling step. Contains information about what was scheduled and results. - Inputs: - `scheduled_request_ids` (List[str]; optional; default `field(default_factory=list)`): Optional constructor field; defaults to `field(default_factory=list)`. - `num_scheduled_tokens` (int; optional; default `0`): Optional constructor field; defaults to `0`. - `finished_request_ids` (Set[str]; optional; default `field(default_factory=set)`): Optional constructor field; defaults to `field(default_factory=set)`. - `outputs` (List[RequestOutput]; optional; default `field(default_factory=list)`): Optional constructor field; defaults to `field(default_factory=list)`. - `has_work` (bool; optional; default `False`): Optional constructor field; defaults to `False`. - Constructs: `vllm_mlx.scheduler.SchedulerOutput` - Decorators: dataclass ## `vllm_mlx.scheduler._install_prompt_cache_save` - Kind: function - Signature: `def _install_prompt_cache_save(batch_gen: 'BatchGenerator', prompt_cache_save) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/scheduler.py#L163-L187 - Implementation: Function `_install_prompt_cache_save` contains no state mutation, call, raise, return, await, or yield. Monkey-patch ``_process_prompts`` to capture prompt-only cache state. Can be installed independently of chunked prefill. If chunked prefill is also installed, *it* takes over ``_process_prompts`` and invokes the callback itself, so call this **before** ``_install_chunked_prefill``. - Inputs: - `batch_gen` ('BatchGenerator'; required): Required positional or keyword input. - `prompt_cache_save` (not annotated; required): Required positional or keyword input. - Return annotation: `None` ## `vllm_mlx.scheduler._install_prompt_cache_save._patched_process_prompts` - Kind: nested function - Signature: `def _patched_process_prompts(prompts, _self=batch_gen)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/scheduler.py#L177-L185 - Implementation: Nested Function `_install_prompt_cache_save._patched_process_prompts` calls `_orig_process_prompts`, `enumerate`, `prompt_cache_save`, `batch.extract_cache`; returns `batch`. Nested Function `_install_prompt_cache_save._patched_process_prompts` calls `_orig_process_prompts`, `enumerate`, `prompt_cache_save`, `batch.extract_cache`; returns `batch`. - Inputs: - `prompts` (not annotated; required): Required positional or keyword input. - `_self` (not annotated; optional; default `batch_gen`): Optional positional or keyword input; defaults to `batch_gen`. - Return annotation: `not annotated` - Calls: _orig_process_prompts, enumerate, prompt_cache_save, batch.extract_cache - Return expressions: batch ## `vllm_mlx.scheduler._install_chunked_prefill` - Kind: function - Signature: `def _install_chunked_prefill(batch_gen: 'BatchGenerator', budget: int, mid_prefill_save=None, prompt_cache_save=None, pending_abort_ids: Optional[Set[str]]=None, uid_to_request_id: Optional[Dict[int, str]]=None, requests: Optional[Dict[str, Any]]=None) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/scheduler.py#L190-L697 - Implementation: Function `_install_chunked_prefill` calls `logger.info`. Monkey-patch a BatchGenerator instance so that large prefills are broken into chunks of at most *budget* tokens each. Between chunks the generation loop gets a chance to produce one token for every active request, preventing starvation during long prefills. Args: batch_gen: The BatchGenerator to patch. budget: Max tokens per prefill chunk. mid_prefill_save: Optional callback(uid, processed, prompt_cache) called after each chunk to save intermediate KV cache state. - Inputs: - `batch_gen` ('BatchGenerator'; required): The BatchGenerator to patch. - `budget` (int; required): Max tokens per prefill chunk. - `mid_prefill_save` (not annotated; optional; default `None`): Optional callback(uid, processed, prompt_cache) called after each chunk to save intermediate KV cache state. - `prompt_cache_save` (not annotated; optional; default `None`): Optional positional or keyword input; defaults to `None`. - `pending_abort_ids` (Optional[Set[str]]; optional; default `None`): Optional positional or keyword input; defaults to `None`. - `uid_to_request_id` (Optional[Dict[int, str]]; optional; default `None`): Optional positional or keyword input; defaults to `None`. - `requests` (Optional[Dict[str, Any]]; optional; default `None`): Optional positional or keyword input; defaults to `None`. - Return annotation: `None` - Calls: logger.info ## `vllm_mlx.scheduler._install_chunked_prefill._lazy_extract_cache` - Kind: nested function - Signature: `def _lazy_extract_cache(cache, idx)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/scheduler.py#L225-L226 - Implementation: Nested Function `_install_chunked_prefill._lazy_extract_cache` calls `c.extract`; returns `(c.extract(idx) for c in cache)`. Nested Function `_install_chunked_prefill._lazy_extract_cache` calls `c.extract`; returns `(c.extract(idx) for c in cache)`. - Inputs: - `cache` (not annotated; required): Required positional or keyword input. - `idx` (not annotated; required): Required positional or keyword input. - Return annotation: `not annotated` - Calls: c.extract - Return expressions: (c.extract(idx) for c in cache) ## `vllm_mlx.scheduler._install_chunked_prefill._batch_cls` - Kind: nested class - Signature: `class _batch_cls` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/scheduler.py#L233-L273 - Implementation: Nested Class `_install_chunked_prefill._batch_cls` declares 4 direct member(s). Nested Class `_install_chunked_prefill._batch_cls` declares 4 direct member(s). - Inputs: - `uids` (List[int]; required): Required constructor field. - `y` (Any; required): Required constructor field. - `logprobs` (List[Any]; required): Required constructor field. - `max_tokens` (List[int]; required): Required constructor field. - `num_tokens` (List[int]; required): Required constructor field. - `cache` (List[Any]; required): Required constructor field. - `samplers` (List[Any]; required): Required constructor field. - `logits_processors` (List[Any]; required): Required constructor field. - `tokens` (List[Any]; required): Required constructor field. - Constructs: `vllm_mlx.scheduler._install_chunked_prefill._batch_cls` - Decorators: dataclass ## `vllm_mlx.scheduler._install_chunked_prefill._batch_cls.__len__` - Kind: nested function - Signature: `def __len__(self)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/scheduler.py#L244-L245 - Implementation: Nested Function `_install_chunked_prefill._batch_cls.__len__` calls `len`; returns `len(self.uids)`. Nested Function `_install_chunked_prefill._batch_cls.__len__` calls `len`; returns `len(self.uids)`. - Inputs: none - Return annotation: `not annotated` - Calls: len - State reads: self.uids - Return expressions: len(self.uids) ## `vllm_mlx.scheduler._install_chunked_prefill._batch_cls.filter` - Kind: nested function - Signature: `def filter(self, keep_idx: List[int])` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/scheduler.py#L247-L258 - Implementation: Nested Function `_install_chunked_prefill._batch_cls.filter` updates `self.uids`, `self.logprobs`, `self.max_tokens`, `self.num_tokens`; calls `mx.array`, `c.filter`. Nested Function `_install_chunked_prefill._batch_cls.filter` updates `self.uids`, `self.logprobs`, `self.max_tokens`, `self.num_tokens`; calls `mx.array`, `c.filter`. - Inputs: - `keep_idx` (List[int]; required): Required positional or keyword input. - Return annotation: `not annotated` - Calls: mx.array, c.filter - State reads: self.uids, self.logprobs, self.max_tokens, self.num_tokens, self.samplers, self.logits_processors, self.tokens, self.y, self.cache - State writes: self.uids, self.logprobs, self.max_tokens, self.num_tokens, self.samplers, self.logits_processors, self.tokens, self.y ## `vllm_mlx.scheduler._install_chunked_prefill._batch_cls.extend` - Kind: nested function - Signature: `def extend(self, other)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/scheduler.py#L260-L270 - Implementation: Nested Function `_install_chunked_prefill._batch_cls.extend` updates `self.y`; calls `self.uids.extend`, `mx.concatenate`, `self.logprobs.extend`, `self.num_tokens.extend`. Nested Function `_install_chunked_prefill._batch_cls.extend` updates `self.y`; calls `self.uids.extend`, `mx.concatenate`, `self.logprobs.extend`, `self.num_tokens.extend`. - Inputs: - `other` (not annotated; required): Required positional or keyword input. - Return annotation: `not annotated` - Calls: self.uids.extend, mx.concatenate, self.logprobs.extend, self.num_tokens.extend, self.max_tokens.extend, self.samplers.extend, self.logits_processors.extend, self.tokens.extend, zip, c.extend - State reads: self.uids.extend, self.uids, self.y, self.logprobs.extend, self.logprobs, self.num_tokens.extend, self.num_tokens, self.max_tokens.extend, self.max_tokens, self.samplers.extend, self.samplers, self.logits_processors.extend, self.logits_processors, self.tokens.extend, self.tokens, self.cache - State writes: self.y ## `vllm_mlx.scheduler._install_chunked_prefill._batch_cls.extract_cache` - Kind: nested function - Signature: `def extract_cache(self, idx)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/scheduler.py#L272-L273 - Implementation: Nested Function `_install_chunked_prefill._batch_cls.extract_cache` calls `c.extract`; returns `[c.extract(idx) for c in self.cache]`. Nested Function `_install_chunked_prefill._batch_cls.extract_cache` calls `c.extract`; returns `[c.extract(idx) for c in self.cache]`. - Inputs: - `idx` (not annotated; required): Required positional or keyword input. - Return annotation: `not annotated` - Calls: c.extract - State reads: self.cache - Return expressions: [c.extract(idx) for c in self.cache] ## `vllm_mlx.scheduler._install_chunked_prefill._patched_process_prompts` - Kind: nested function - Signature: `def _patched_process_prompts(prompts, _self=batch_gen)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/scheduler.py#L291-L299 - Implementation: Nested Function `_install_chunked_prefill._patched_process_prompts` calls `_orig_process_prompts`, `enumerate`, `prompt_cache_save`, `batch.extract_cache`; returns `batch`. Nested Function `_install_chunked_prefill._patched_process_prompts` calls `_orig_process_prompts`, `enumerate`, `prompt_cache_save`, `batch.extract_cache`; returns `batch`. - Inputs: - `prompts` (not annotated; required): Required positional or keyword input. - `_self` (not annotated; optional; default `batch_gen`): Optional positional or keyword input; defaults to `batch_gen`. - Return annotation: `not annotated` - Calls: _orig_process_prompts, enumerate, prompt_cache_save, batch.extract_cache - Return expressions: batch ## `vllm_mlx.scheduler._install_chunked_prefill._generation_step` - Kind: nested function - Signature: `def _generation_step(self=batch_gen)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/scheduler.py#L303-L360 - Implementation: Nested Function `_install_chunked_prefill._generation_step` updates `self._stats.generation_time`, `self.active_batch`, `self._stats.generation_tokens`; calls `len`, `_time.perf_counter`, `enumerate`, `mx.concatenate`; has 2 explicit return paths. Run one generation step on the active batch. Returns responses. - Inputs: none - Return annotation: `not annotated` - Calls: len, _time.perf_counter, enumerate, mx.concatenate, self._step, mx.async_eval, y.tolist, zip, end_idx.append, keep_idx.append, batch.extract_cache, responses.append, self.Response, batch.filter - State reads: self.active_batch, self._step, self._stats, self.stop_tokens, self.Response - State writes: self._stats.generation_time, self.active_batch, self._stats.generation_tokens - Return expressions: []; responses ## `vllm_mlx.scheduler._install_chunked_prefill._chunked_next` - Kind: nested function - Signature: `def _chunked_next(self=batch_gen)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/scheduler.py#L362-L678 - Implementation: Nested Function `_install_chunked_prefill._chunked_next` updates `self._partial`, `self.active_batch`, `self._stats.prompt_time`, `self._stats.generation_time`; calls `uid_to_request_id.get`, `logger.info`, `mx.clear_cache`, `self._generation_step`; returns `self._generation_step()`. Replacement for _next() that chunks large prefills. Only intercepts when: 1. A partial prefill is in progress (_partial is not None) 2. The next prompt batch exceeds the budget Everything else delegates to the original _next(). - Inputs: none - Return annotation: `not annotated` - Calls: uid_to_request_id.get, logger.info, mx.clear_cache, self._generation_step, _time.perf_counter, max, int, partial.get, min, self.model, mx.contiguous, mx.eval, self.prompt_progress_callback, len, mid_prefill_save, c.finalize, self.prompt_checkpoint_callback, _lazy_extract_cache, enumerate, self._step, mx.async_eval, _batch_cls, list, prompt_cache_save, new_batch.extract_cache, self.active_batch.extend, sum, requests.get, getattr, zip, mx.array, all, c[0].empty, _left_pad_prompts, _make_cache, _right_pad_prompts, _merge_caches, c.prepare, self._process_prompts - State reads: self._partial, self._generation_step, self.model, self.prompt_progress_callback, self.prompt_checkpoint_callback, self._step, self.active_batch, self.active_batch.extend, self._stats, self.completion_batch_size, self.prefill_batch_size, self.unprocessed_prompts, self.active_batch.y, self.active_batch.logprobs, self.max_kv_size, self._process_prompts - State writes: self._partial, self.active_batch, self._stats.prompt_time, self._stats.generation_time, self._stats.prompt_tokens, self.unprocessed_prompts - Return expressions: self._generation_step() ## `vllm_mlx.scheduler._install_chunked_prefill._patched_remove` - Kind: nested function - Signature: `def _patched_remove(uids_to_remove, _self=batch_gen)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/scheduler.py#L680-L691 - Implementation: Nested Function `_install_chunked_prefill._patched_remove` calls `set`, `logger.info`, `mx.clear_cache`, `_orig_remove`. Clear partial state if aborted request is being prefilled. - Inputs: - `uids_to_remove` (not annotated; required): Required positional or keyword input. - `_self` (not annotated; optional; default `batch_gen`): Optional positional or keyword input; defaults to `batch_gen`. - Return annotation: `not annotated` - Calls: set, logger.info, mx.clear_cache, _orig_remove ## `vllm_mlx.scheduler._MTPStatsState` - Kind: class - Signature: `class _MTPStatsState` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/scheduler.py#L701-L719 - Implementation: Class `_MTPStatsState` declares 0 direct member(s). Cumulative native-MTP counters shared across generator instances. - Inputs: - `counters` (Dict[str, int]; optional; default `field(default_factory=lambda: {'attempted': 0, 'accepted': 0, 'rejected': 0, 'errors': 0})`): Optional constructor field; defaults to `field(default_factory=lambda: {'attempted': 0, 'accepted': 0, 'rejected': 0, 'errors': 0})`. - `bypass_counts` (Dict[str, int]; optional; default `field(default_factory=lambda: {'prefill': 0, 'no_active_batch': 0, 'cache_mismatch': 0})`): Optional constructor field; defaults to `field(default_factory=lambda: {'prefill': 0, 'no_active_batch': 0, 'cache_mismatch': 0})`. - `lock` (Any; optional; default `field(default_factory=Lock)`): Optional constructor field; defaults to `field(default_factory=Lock)`. - Constructs: `vllm_mlx.scheduler._MTPStatsState` - Decorators: dataclass ## `vllm_mlx.scheduler._configure_chunked_prefill` - Kind: function - Signature: `def _configure_chunked_prefill(scheduler: 'Scheduler', batch_gen: 'BatchGenerator', budget: int, prompt_cache_save) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/scheduler.py#L722-L777 - Implementation: Function `_configure_chunked_prefill` calls `hasattr`, `scheduler._make_mid_prefill_save_callback`, `logger.info`, `_install_chunked_prefill`; returns `None`. Enable the matching legacy or native mlx-lm chunked-prefill API. - Inputs: - `scheduler` ('Scheduler'; required): Required positional or keyword input. - `batch_gen` ('BatchGenerator'; required): Required positional or keyword input. - `budget` (int; required): Required positional or keyword input. - `prompt_cache_save` (not annotated; required): Required positional or keyword input. - Return annotation: `None` - Calls: hasattr, scheduler._make_mid_prefill_save_callback, logger.info, _install_chunked_prefill, all, logger.warning - Return expressions: None ## `vllm_mlx.scheduler._install_mtp` - Kind: function - Signature: `def _install_mtp(batch_gen: 'BatchGenerator', model: Any, num_draft_tokens: int=1, optimistic: bool=False, stats_state: Optional['_MTPStatsState']=None) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/scheduler.py#L780-L1262 - Implementation: Function `_install_mtp` calls `make_sampler`, `_MTPStatsState`, `logger.warning`, `logger.info`. Monkey-patch a BatchGenerator to use MTP (Multi-Token Prediction) with always-advance strategy for hybrid MambaCache + KVCache. Flow per generation step: 1. Use skip_state logits/hidden OR run model forward -> sample primary 2. MTP head drafts one token after primary 3. Verify [primary, draft] in one model call (always advances cache) 4. Accept: skip_state from pos 1, defer draft for next step emission Reject: trim KVCache by 1, skip_state from pos 0 (no cold start) 5. Draft is emitted in the NEXT generation step after primary - Inputs: - `batch_gen` ('BatchGenerator'; required): Required positional or keyword input. - `model` (Any; required): Required positional or keyword input. - `num_draft_tokens` (int; optional; default `1`): Optional positional or keyword input; defaults to `1`. - `optimistic` (bool; optional; default `False`): Optional positional or keyword input; defaults to `False`. - `stats_state` (Optional['_MTPStatsState']; optional; default `None`): Optional positional or keyword input; defaults to `None`. - Return annotation: `None` - Calls: make_sampler, _MTPStatsState, logger.warning, logger.info ## `vllm_mlx.scheduler._install_mtp._get_mtp_stats` - Kind: nested function - Signature: `def _get_mtp_stats() -> Dict[str, Any]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/scheduler.py#L823-L845 - Implementation: Nested Function `_install_mtp._get_mtp_stats` calls `dict`; returns `{'enabled': True, 'requested_draft_tokens': num_draft_tokens, 'effective_draft_tokens': 1, 'mode': 'always_advance_opti…`. Nested Function `_install_mtp._get_mtp_stats` calls `dict`; returns `{'enabled': True, 'requested_draft_tokens': num_draft_tokens, 'effective_draft_tokens': 1, 'mode': 'always_advance_opti…`. - Inputs: none - Return annotation: `Dict[str, Any]` - Calls: dict - Return expressions: {'enabled': True, 'requested_draft_tokens': num_draft_tokens, 'effective_draft_tokens': 1, 'mode': 'always_advance_opti… ## `vllm_mlx.scheduler._install_mtp._mtp_bypass_reasons` - Kind: nested function - Signature: `def _mtp_bypass_reasons(input_tokens, prompt_cache)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/scheduler.py#L849-L857 - Implementation: Nested Function `_install_mtp._mtp_bypass_reasons` calls `reasons.append`; returns `reasons`. Nested Function `_install_mtp._mtp_bypass_reasons` calls `reasons.append`; returns `reasons`. - Inputs: - `input_tokens` (not annotated; required): Required positional or keyword input. - `prompt_cache` (not annotated; required): Required positional or keyword input. - Return annotation: `not annotated` - Calls: reasons.append - Return expressions: reasons ## `vllm_mlx.scheduler._install_mtp._record_mtp_bypass` - Kind: nested function - Signature: `def _record_mtp_bypass(reasons) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/scheduler.py#L859-L862 - Implementation: Nested Function `_install_mtp._record_mtp_bypass` contains no state mutation, call, raise, return, await, or yield. Nested Function `_install_mtp._record_mtp_bypass` contains no state mutation, call, raise, return, await, or yield. - Inputs: - `reasons` (not annotated; required): Required positional or keyword input. - Return annotation: `None` ## `vllm_mlx.scheduler._install_mtp._mtp_step` - Kind: nested function - Signature: `def _mtp_step(input_tokens, prompt_cache, samplers, logits_processors, tokens)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/scheduler.py#L864-L1138 - Implementation: Nested Function `_install_mtp._mtp_step` calls `_mtp_bypass_reasons`, `_record_mtp_bypass`, `_orig_step`, `model`; has 2 explicit return paths. Extended _step with MTP always-advance strategy. Every step (after skip): 1. Use skip_state logits/hidden OR run model forward 2. Sample primary token P 3. MTP head drafts token D 4. Verify [P, D] in one model call (always advances cache) 5. Accept: skip_state from position 1 (after D), defer D Reject: trim KVCache by 1, skip_state from position 0 (after P) No snapshot/restore — eliminates cold starts after rejection. MambaCache layers accept minor pollution on reject (exponential decay). During prefill (multi-token input), MTP is skipped entirely. - Inputs: - `input_tokens` (not annotated; required): Required positional or keyword input. - `prompt_cache` (not annotated; required): Required positional or keyword input. - `samplers` (not annotated; required): Required positional or keyword input. - `logits_processors` (not annotated; required): Required positional or keyword input. - `tokens` (not annotated; required): Required positional or keyword input. - Return annotation: `not annotated` - Calls: _mtp_bypass_reasons, _record_mtp_bypass, _orig_step, model, isinstance, _normalize_logits_processors, any, logger.debug, sum, len, range, processor, processed_logits.append, mx.concatenate, mx.logsumexp, sample_sampler, all_samples.append, batch_gen.sampler, list, model.mtp_forward, _draft_sampler, enumerate, hasattr, _c.is_trimmable, s.copy, mx.async_eval, mx.argmax, mx.eval, verify_pred.tolist, draft_tokens.tolist, c.is_trimmable, c.trim, _rnn_snapshots.items, _deferred_drafts.pop - Return expressions: _orig_step(input_tokens, prompt_cache, samplers, logits_processors, tokens); (primary_tokens, list(logprobs)) ## `vllm_mlx.scheduler._install_mtp._mtp_next` - Kind: nested function - Signature: `def _mtp_next(self=batch_gen)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/scheduler.py#L1147-L1247 - Implementation: Nested Function `_install_mtp._mtp_next` updates `self.active_batch`; calls `_deferred_drafts.clear`, `_deferred_drafts.pop`, `self._inner_next`, `set`; has 2 explicit return paths. Wrapper around _next that emits deferred MTP draft tokens. After each primary token, if the previous step's MTP draft was accepted, it is emitted as an additional response. - Inputs: none - Return annotation: `not annotated` - Calls: _deferred_drafts.clear, _deferred_drafts.pop, self._inner_next, set, augmented.append, prev_deferred.pop, draft_info['token_array'].item, self.Response, draft_end_uids.add, enumerate, mx.concatenate, mx.array, batch.extract_cache, self.active_batch.filter - State reads: self.active_batch, self.active_batch.uids, self._inner_next, self.stop_tokens, self.Response, self.active_batch.filter - State writes: self.active_batch - Return expressions: responses; augmented ## `vllm_mlx.scheduler._mtp_status_snapshot` - Kind: function - Signature: `def _mtp_status_snapshot(batch_generator) -> Dict[str, Any]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/scheduler.py#L1265-L1269 - Implementation: Function `_mtp_status_snapshot` calls `getattr`, `callable`, `get_mtp_stats`; has 2 explicit return paths. Function `_mtp_status_snapshot` calls `getattr`, `callable`, `get_mtp_stats`; has 2 explicit return paths. - Inputs: - `batch_generator` (not annotated; required): Required positional or keyword input. - Return annotation: `Dict[str, Any]` - Calls: getattr, callable, get_mtp_stats - Return expressions: {'mtp': get_mtp_stats()}; {} ## `vllm_mlx.scheduler.Scheduler` - Kind: class - Signature: `class Scheduler` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/scheduler.py#L1272-L3518 - Implementation: Class `Scheduler` declares 52 direct member(s). Scheduler for continuous batching using mlx-lm BatchGenerator. This scheduler manages the lifecycle of requests: 1. Requests arrive and are added to the waiting queue 2. Scheduler moves requests from waiting to running (via BatchGenerator) 3. BatchGenerator processes all running requests together 4. Finished requests are removed and outputs returned The key insight is that mlx-lm's BatchGenerator already implements continuous batching at the token level, so we use it as the backend. - Inputs: - `model` (Any; required): The MLX model - `tokenizer` (Any; required): The tokenizer - `config` (Optional[SchedulerConfig]; optional; default `None`): Scheduler configuration - Constructs: `vllm_mlx.scheduler.Scheduler` ## `vllm_mlx.scheduler.Scheduler.__init__` - Kind: method - Signature: `def __init__(self, model: Any, tokenizer: Any, config: Optional[SchedulerConfig]=None)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/scheduler.py#L1286-L1402 - Implementation: Method `Scheduler.__init__` updates `self.model`, `self.tokenizer`, `self.config`, `self._actual_tokenizer`; calls `SchedulerConfig`, `self._get_actual_tokenizer`, `deque`, `set`. Initialize the scheduler. Args: model: The MLX model tokenizer: The tokenizer config: Scheduler configuration - Inputs: - `model` (Any; required): The MLX model - `tokenizer` (Any; required): The tokenizer - `config` (Optional[SchedulerConfig]; optional; default `None`): Scheduler configuration - Return annotation: `not annotated` - Calls: SchedulerConfig, self._get_actual_tokenizer, deque, set, PagedCacheManager, BlockAwarePrefixCache, logger.info, MemoryCacheConfig, MemoryAwarePrefixCache, SSDCacheConfig, SSDCacheTier, self._ssd_tier.start_writer, self._ssd_tier.reconcile, self.memory_aware_cache.set_ssd_tier, PrefixCacheManager, _MTPStatsState - State reads: self._get_actual_tokenizer, self.config.enable_prefix_cache, self.config, self.config.use_paged_cache, self.config.paged_cache_block_size, self.config.max_cache_blocks, self.paged_cache_manager, self.config.use_memory_aware_cache, self.config.cache_memory_mb, self.config.cache_memory_percent, self.config.kv_cache_quantization, self.config.kv_cache_quantization_bits, self.config.kv_cache_quantization_group_size, self.config.kv_cache_min_quantize_tokens, self.memory_aware_cache.memory_limit_mb, self.memory_aware_cache, self.config.ssd_cache_dir, self.config.ssd_cache_max_gb, self._ssd_tier.start_writer, self._ssd_tier, self._ssd_tier.reconcile, self.memory_aware_cache.set_ssd_tier, self.config.prefix_cache_size - State writes: self.model, self.tokenizer, self.config, self._actual_tokenizer, self._detokenizer_pool, self.waiting, self.running, self.requests, self.finished_req_ids, self.request_id_to_uid, self.uid_to_request_id, self.batch_generator, self._current_sampler_params, self.prefix_cache, self.memory_aware_cache, self.paged_cache_manager, self.block_aware_cache, self._ssd_tier, self._pending_abort_ids, self.num_requests_processed, self.total_prompt_tokens, self.total_completion_tokens, self._mtp_stats_state, self._step_count, self._clear_cache_interval, self._memory_log_interval ## `vllm_mlx.scheduler.Scheduler._get_actual_tokenizer` - Kind: method - Signature: `def _get_actual_tokenizer(self, tokenizer: Any) -> Any` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/scheduler.py#L1404-L1418 - Implementation: Method `Scheduler._get_actual_tokenizer` calls `hasattr`, `callable`; has 2 explicit return paths. Get the actual tokenizer from a processor or tokenizer. MLLM models use processors (e.g., Qwen3VLProcessor) which wrap the tokenizer. This method extracts the actual tokenizer. - Inputs: - `tokenizer` (Any; required): Required positional or keyword input. - Return annotation: `Any` - Calls: hasattr, callable - Return expressions: tokenizer; tokenizer.tokenizer ## `vllm_mlx.scheduler.Scheduler._decode_tokens` - Kind: method - Signature: `def _decode_tokens(self, token_ids: List[int]) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/scheduler.py#L1420-L1424 - Implementation: Method `Scheduler._decode_tokens` calls `self._actual_tokenizer.decode`; returns `self._actual_tokenizer.decode(token_ids)`. Decode token IDs to text, handling both tokenizers and processors. - Inputs: - `token_ids` (List[int]; required): Required positional or keyword input. - Return annotation: `str` - Calls: self._actual_tokenizer.decode - State reads: self._actual_tokenizer.decode, self._actual_tokenizer - Return expressions: self._actual_tokenizer.decode(token_ids) ## `vllm_mlx.scheduler.Scheduler._get_detokenizer` - Kind: method - Signature: `def _get_detokenizer(self, request_id: str) -> Any` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/scheduler.py#L1426-L1431 - Implementation: Method `Scheduler._get_detokenizer` calls `NaiveStreamingDetokenizer`; returns `self._detokenizer_pool[request_id]`. Get or create a streaming detokenizer for a request. - Inputs: - `request_id` (str; required): Required positional or keyword input. - Return annotation: `Any` - Calls: NaiveStreamingDetokenizer - State reads: self._detokenizer_pool, self._actual_tokenizer - Return expressions: self._detokenizer_pool[request_id] ## `vllm_mlx.scheduler.Scheduler._cleanup_detokenizer` - Kind: method - Signature: `def _cleanup_detokenizer(self, request_id: str) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/scheduler.py#L1433-L1435 - Implementation: Method `Scheduler._cleanup_detokenizer` calls `self._detokenizer_pool.pop`. Remove the streaming detokenizer for a finished request. - Inputs: - `request_id` (str; required): Required positional or keyword input. - Return annotation: `None` - Calls: self._detokenizer_pool.pop - State reads: self._detokenizer_pool.pop, self._detokenizer_pool ## `vllm_mlx.scheduler.Scheduler._get_stop_tokens` - Kind: method - Signature: `def _get_stop_tokens(self) -> Set[int]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/scheduler.py#L1437-L1455 - Implementation: Method `Scheduler._get_stop_tokens` calls `set`, `hasattr`, `isinstance`, `stop_tokens.update`; returns `stop_tokens`. Get stop token IDs from tokenizer or processor. - Inputs: none - Return annotation: `Set[int]` - Calls: set, hasattr, isinstance, stop_tokens.update, stop_tokens.add - State reads: self.tokenizer, self._actual_tokenizer - Return expressions: stop_tokens ## `vllm_mlx.scheduler.Scheduler._create_batch_generator` - Kind: method - Signature: `def _create_batch_generator(self, sampling_params: SamplingParams) -> BatchGenerator` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/scheduler.py#L1457-L1539 - Implementation: Method `Scheduler._create_batch_generator` calls `make_sampler`, `self._get_stop_tokens`, `stop_tokens.update`, `BatchGenerator`; returns `bg`. Create a BatchGenerator with the given sampling parameters. - Inputs: - `sampling_params` (SamplingParams; required): Required positional or keyword input. - Return annotation: `BatchGenerator` - Calls: make_sampler, self._get_stop_tokens, stop_tokens.update, BatchGenerator, self._make_prompt_cache_save_callback, _configure_chunked_prefill, hasattr, _install_prompt_cache_save, _install_mtp, logger.warning - State reads: self._get_stop_tokens, self.model, self.config.prefill_batch_size, self.config, self.config.completion_batch_size, self.config.prefill_step_size, self.config.chunked_prefill_tokens, self.memory_aware_cache, self._make_prompt_cache_save_callback, self.config.enable_mtp, self.model.mtp, self.config.mtp_num_draft_tokens, self.config.mtp_optimistic, self._mtp_stats_state - Return expressions: bg ## `vllm_mlx.scheduler.Scheduler._create_batch_generator._prefill_progress` - Kind: nested function - Signature: `def _prefill_progress(progress_list)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/scheduler.py#L1472-L1479 - Implementation: Nested Function `Scheduler._create_batch_generator._prefill_progress` calls `self.uid_to_request_id.get`, `logger.info`, `isinstance`. Log prefill progress for each uid chunk. - Inputs: - `progress_list` (not annotated; required): Required positional or keyword input. - Return annotation: `not annotated` - Calls: self.uid_to_request_id.get, logger.info, isinstance - State reads: self.uid_to_request_id.get, self.uid_to_request_id ## `vllm_mlx.scheduler.Scheduler._make_prompt_cache_save_callback` - Kind: method - Signature: `def _make_prompt_cache_save_callback(self)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/scheduler.py#L1541-L1585 - Implementation: Method `Scheduler._make_prompt_cache_save_callback` returns `_prompt_cache_save`. Create a callback that stores prompt-only KV/Mamba cache. Called from ``_generation_step`` right before the first output token is fed into the model. At that point ``num_tokens == 0`` and the batch cache contains the exact prompt-only state (correct for both KVCache and MambaCache/ArraysCache layers). The cache is stored with key = prompt_token_ids so that a future request with the identical prompt gets an exact hit. - Inputs: none - Return annotation: `not annotated` - Return expressions: _prompt_cache_save ## `vllm_mlx.scheduler.Scheduler._make_prompt_cache_save_callback._prompt_cache_save` - Kind: nested function - Signature: `def _prompt_cache_save(uid, extracted_cache)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/scheduler.py#L1554-L1583 - Implementation: Nested Function `Scheduler._make_prompt_cache_save_callback._prompt_cache_save` calls `self.uid_to_request_id.get`, `self.requests.get`, `list`, `_trim_cache_offset`; returns `None`. Nested Function `Scheduler._make_prompt_cache_save_callback._prompt_cache_save` calls `self.uid_to_request_id.get`, `self.requests.get`, `list`, `_trim_cache_offset`; returns `None`. - Inputs: - `uid` (not annotated; required): Required positional or keyword input. - `extracted_cache` (not annotated; required): Required positional or keyword input. - Return annotation: `not annotated` - Calls: self.uid_to_request_id.get, self.requests.get, list, _trim_cache_offset, _time.monotonic, self.memory_aware_cache.store, logger.info, len - State reads: self.uid_to_request_id.get, self.uid_to_request_id, self.requests.get, self.requests, self.memory_aware_cache.store, self.memory_aware_cache - Return expressions: None ## `vllm_mlx.scheduler.Scheduler._make_mid_prefill_save_callback` - Kind: method - Signature: `def _make_mid_prefill_save_callback(self, save_interval: int)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/scheduler.py#L1587-L1655 - Implementation: Method `Scheduler._make_mid_prefill_save_callback` returns `_mid_prefill_save`. Create a callback for saving intermediate KV cache during chunked prefill. The callback is called after each chunk with (uid, processed_tokens, prompt_cache). It extracts the cache state (immutable MLX array snapshots), reconstructs KVCache objects, and stores them in the memory-aware prefix cache so that a subsequent request with the same prompt prefix can skip the already-computed tokens. - Inputs: - `save_interval` (int; required): Required positional or keyword input. - Return annotation: `not annotated` - Return expressions: _mid_prefill_save ## `vllm_mlx.scheduler.Scheduler._make_mid_prefill_save_callback._mid_prefill_save` - Kind: nested function - Signature: `def _mid_prefill_save(uid, processed_tokens, prompt_cache)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/scheduler.py#L1598-L1653 - Implementation: Nested Function `Scheduler._make_mid_prefill_save_callback._mid_prefill_save` calls `self.uid_to_request_id.get`, `self.requests.get`, `getattr`, `self._extract_cache_states`; returns `None`. Nested Function `Scheduler._make_mid_prefill_save_callback._mid_prefill_save` calls `self.uid_to_request_id.get`, `self.requests.get`, `getattr`, `self._extract_cache_states`; returns `None`. - Inputs: - `uid` (not annotated; required): Required positional or keyword input. - `processed_tokens` (not annotated; required): Required positional or keyword input. - `prompt_cache` (not annotated; required): Required positional or keyword input. - Return annotation: `not annotated` - Calls: self.uid_to_request_id.get, self.requests.get, getattr, self._extract_cache_states, self._reconstruct_cache_from_states, list, self.memory_aware_cache.remove, _time.monotonic, self.memory_aware_cache.store, tuple, logger.info, len, logger.debug - State reads: self.uid_to_request_id.get, self.uid_to_request_id, self.requests.get, self.requests, self._extract_cache_states, self._reconstruct_cache_from_states, self.memory_aware_cache.remove, self.memory_aware_cache, self.memory_aware_cache.store - Return expressions: None ## `vllm_mlx.scheduler.Scheduler._close_batch_generator` - Kind: method - Signature: `def _close_batch_generator(self) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/scheduler.py#L1657-L1665 - Implementation: Method `Scheduler._close_batch_generator` updates `self.batch_generator`; calls `hasattr`, `self.batch_generator.close`, `logger.debug`. Properly close BatchGenerator to restore wired_limit. - Inputs: none - Return annotation: `None` - Calls: hasattr, self.batch_generator.close, logger.debug - State reads: self.batch_generator, self.batch_generator.close - State writes: self.batch_generator ## `vllm_mlx.scheduler.Scheduler._ensure_batch_generator` - Kind: method - Signature: `def _ensure_batch_generator(self, sampling_params: SamplingParams) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/scheduler.py#L1667-L1709 - Implementation: Method `Scheduler._ensure_batch_generator` updates `self.batch_generator`, `self._current_sampler_params`; calls `logger.warning`, `len`, `hasattr`, `logger.info`; returns `None`. Ensure BatchGenerator exists with compatible settings. - Inputs: - `sampling_params` (SamplingParams; required): Required positional or keyword input. - Return annotation: `None` - Calls: logger.warning, len, hasattr, logger.info, self._close_batch_generator, self._create_batch_generator - State reads: self.batch_generator, self._current_sampler_params, self.running, self.memory_aware_cache, self.memory_aware_cache._entries, self.prefix_cache, self._close_batch_generator, self._create_batch_generator - State writes: self.batch_generator, self._current_sampler_params - Return expressions: None ## `vllm_mlx.scheduler.Scheduler._validate_cache` - Kind: method - Signature: `def _validate_cache(self, cache: Any) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/scheduler.py#L1711-L1769 - Implementation: Method `Scheduler._validate_cache` calls `isinstance`, `len`, `hasattr`, `logger.debug`; has 2 explicit return paths. Validate that a cache object is usable. Checks for None references AND shape compatibility. Restored cache entries must have batch_size == 1 (single sequence) so they can be merged into the running batch by _merge_caches. A shape mismatch here (e.g. batch=2 from a stale entry) would cause a concatenation crash inside _merge_caches. Args: cache: The cache object to validate Returns: True if cache is valid and usable - Inputs: - `cache` (Any; required): The cache object to validate - Return annotation: `bool` - Calls: isinstance, len, hasattr, logger.debug - Return expressions: False; True ## `vllm_mlx.scheduler.Scheduler._extract_cache_states` - Kind: method - Signature: `def _extract_cache_states(self, raw_cache: List[Any]) -> List[Dict[str, Any]]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/scheduler.py#L1771-L1806 - Implementation: Method `Scheduler._extract_cache_states` calls `hasattr`, `extracted.append`, `type`, `logger.debug`; has 2 explicit return paths. Extract actual tensor state from each layer cache. This extracts the real KV data using mlx-lm's cache.state property, allowing the data to be stored and reconstructed later even after the BatchGenerator is recreated. Args: raw_cache: List of KVCache objects from mlx-lm Returns: List of dicts with {state: (keys, values), meta_state: (offset,), class_name: str} - Inputs: - `raw_cache` (List[Any]; required): List of KVCache objects from mlx-lm - Return annotation: `List[Dict[str, Any]]` - Calls: hasattr, extracted.append, type, logger.debug, len - Return expressions: []; extracted if len(extracted) == len(raw_cache) else [] ## `vllm_mlx.scheduler.Scheduler._reconstruct_cache_from_states` - Kind: method - Signature: `def _reconstruct_cache_from_states(self, extracted_states: List[Dict[str, Any]]) -> Optional[List[Any]]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/scheduler.py#L1808-L1872 - Implementation: Method `Scheduler._reconstruct_cache_from_states` calls `layer_state.get`, `hasattr`, `_KVCache`, `cache_cls.from_state`; has 2 explicit return paths. Reconstruct cache objects from extracted cache states. This is the inverse of _extract_cache_states(). Uses mlx-lm's _BaseCache.from_state() to reconstruct any cache type (KVCache, MambaCache, etc.) from its state/meta_state. Args: extracted_states: List of dicts from _extract_cache_states() Returns: List of cache objects, or None if reconstruction fails - Inputs: - `extracted_states` (List[Dict[str, Any]]; required): List of dicts from _extract_cache_states() - Return annotation: `Optional[List[Any]]` - Calls: layer_state.get, hasattr, _KVCache, cache_cls.from_state, len, KVCache, int, caches.append, logger.info - Return expressions: None; caches ## `vllm_mlx.scheduler.Scheduler.add_request` - Kind: method - Signature: `def add_request(self, request: Request) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/scheduler.py#L1874-L1997 - Implementation: Method `Scheduler.add_request` calls `ValueError`, `isinstance`, `hasattr`, `self.tokenizer.encode`; can raise `ValueError`, `AttributeError`. Add a new request to the scheduler. Args: request: The request to add - Inputs: - `request` (Request; required): The request to add - Return annotation: `None` - Calls: ValueError, isinstance, hasattr, self.tokenizer.encode, self.tokenizer.tokenizer.encode, AttributeError, type, list, len, self.block_aware_cache.fetch_cache, self.block_aware_cache.reconstruct_cache, logger.debug, _time.monotonic, self.memory_aware_cache.fetch, logger.info, self.memory_aware_cache.check_ssd, self.prefix_cache.fetch_cache, self.waiting.append - State reads: self.requests, self.tokenizer, self.tokenizer.encode, self.tokenizer.tokenizer, self.tokenizer.tokenizer.encode, self.block_aware_cache, self.block_aware_cache.fetch_cache, self.block_aware_cache.reconstruct_cache, self.memory_aware_cache, self.memory_aware_cache.fetch, self.memory_aware_cache._last_match_type, self.memory_aware_cache._entries, self._ssd_tier, self.memory_aware_cache.check_ssd, self.prefix_cache, self.prefix_cache.fetch_cache, self.waiting.append, self.waiting - Raises directly: ValueError, AttributeError ## `vllm_mlx.scheduler.Scheduler.abort_request` - Kind: method - Signature: `def abort_request(self, request_id: str) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/scheduler.py#L1999-L2014 - Implementation: Method `Scheduler.abort_request` calls `self._pending_abort_ids.add`, `logger.info`; returns `True`. Queue request for abort. Thread-safe, called from any thread. The actual abort is deferred to the executor thread (inside step()) to avoid race conditions with in-flight Metal GPU operations. Args: request_id: The request ID to abort Returns: True (abort is always enqueued) - Inputs: - `request_id` (str; required): The request ID to abort - Return annotation: `bool` - Calls: self._pending_abort_ids.add, logger.info - State reads: self._pending_abort_ids.add, self._pending_abort_ids - Return expressions: True ## `vllm_mlx.scheduler.Scheduler._process_pending_aborts` - Kind: method - Signature: `def _process_pending_aborts(self) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/scheduler.py#L2016-L2020 - Implementation: Method `Scheduler._process_pending_aborts` calls `self._pending_abort_ids.pop`, `self._do_abort_request`. Drain and process pending abort requests. Called from executor thread. - Inputs: none - Return annotation: `None` - Calls: self._pending_abort_ids.pop, self._do_abort_request - State reads: self._pending_abort_ids, self._pending_abort_ids.pop, self._do_abort_request ## `vllm_mlx.scheduler.Scheduler._do_abort_request` - Kind: method - Signature: `def _do_abort_request(self, request_id: str) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/scheduler.py#L2022-L2087 - Implementation: Method `Scheduler._do_abort_request` updates `self.total_completion_tokens`, `self.total_prompt_tokens`; calls `self.requests.get`, `self.waiting.remove`, `self.batch_generator.remove`, `request.set_finished`; returns `True`. Actually abort a request. Must be called from the executor thread. Handles the case where the request was already removed from self.requests by _cleanup_request() but still lives in the BatchGenerator (e.g. in _partial or active_batch). Args: request_id: The request ID to abort Returns: True if any cleanup was performed, False otherwise - Inputs: - `request_id` (str; required): The request ID to abort - Return annotation: `bool` - Calls: self.requests.get, self.waiting.remove, self.batch_generator.remove, request.set_finished, self.finished_req_ids.add, self._cleanup_detokenizer, mx.clear_cache, logger.info, len - State reads: self.requests.get, self.requests, self.waiting.remove, self.waiting, self.request_id_to_uid, self.batch_generator, self.batch_generator.remove, self.uid_to_request_id, self.running, self.finished_req_ids.add, self.finished_req_ids, self._cleanup_detokenizer - State writes: self.total_completion_tokens, self.total_prompt_tokens - Return expressions: True ## `vllm_mlx.scheduler.Scheduler.has_requests` - Kind: method - Signature: `def has_requests(self) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/scheduler.py#L2089-L2091 - Implementation: Method `Scheduler.has_requests` calls `bool`; returns `bool(self.waiting or self.running)`. Check if there are any pending or running requests. - Inputs: none - Return annotation: `bool` - Calls: bool - State reads: self.waiting, self.running - Return expressions: bool(self.waiting or self.running) ## `vllm_mlx.scheduler.Scheduler.get_num_waiting` - Kind: method - Signature: `def get_num_waiting(self) -> int` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/scheduler.py#L2093-L2095 - Implementation: Method `Scheduler.get_num_waiting` calls `len`; returns `len(self.waiting)`. Get number of waiting requests. - Inputs: none - Return annotation: `int` - Calls: len - State reads: self.waiting - Return expressions: len(self.waiting) ## `vllm_mlx.scheduler.Scheduler.get_num_running` - Kind: method - Signature: `def get_num_running(self) -> int` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/scheduler.py#L2097-L2099 - Implementation: Method `Scheduler.get_num_running` calls `len`; returns `len(self.running)`. Get number of running requests. - Inputs: none - Return annotation: `int` - Calls: len - State reads: self.running - Return expressions: len(self.running) ## `vllm_mlx.scheduler.Scheduler._schedule_waiting` - Kind: method - Signature: `def _schedule_waiting(self) -> List[Request]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/scheduler.py#L2101-L2276 - Implementation: Method `Scheduler._schedule_waiting` updates `self.total_prompt_tokens`; calls `self._try_promote_ssd_pending`, `len`, `self.waiting.popleft`, `self._ensure_batch_generator`; returns `scheduled`. Move requests from waiting queue to running. Returns: List of requests that were scheduled - Inputs: none - Return annotation: `List[Request]` - Calls: self._try_promote_ssd_pending, len, self.waiting.popleft, self._ensure_batch_generator, self.waiting.appendleft, getattr, logger.debug, make_prompt_cache, self._validate_cache, combined_lp.extend, make_logits_processors, logger.info, self.batch_generator.insert, logger.warning, scheduled.append - State reads: self._ssd_tier, self._try_promote_ssd_pending, self.waiting, self.running, self.config.max_num_seqs, self.config, self.waiting.popleft, self._ensure_batch_generator, self.batch_generator, self.waiting.appendleft, self.config.max_kv_size, self.model, self._validate_cache, self.batch_generator.insert, self.request_id_to_uid, self.uid_to_request_id - State writes: self.total_prompt_tokens - Return expressions: scheduled ## `vllm_mlx.scheduler.Scheduler._copy_cache_state` - Kind: method - Signature: `def _copy_cache_state(value: Any) -> Any` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/scheduler.py#L2279-L2295 - Implementation: Method `Scheduler._copy_cache_state` calls `isinstance`, `Scheduler._copy_cache_state`, `type(value)`, `type`; has 3 explicit return paths. Deep-copy a cache ``state`` payload. Sharing the arrays is not safe: RotatingKVCache writes into its ring buffer and PoolingCache writes into its remainder buffer, both in place, so a snapshot that aliases them would be rewritten by the very generation it is supposed to predate. ``x + 0`` forces a fresh array while staying on the GPU. - Inputs: - `value` (Any; required): Required positional or keyword input. - Return annotation: `Any` - Decorators: staticmethod - Calls: isinstance, Scheduler._copy_cache_state, type(value), type - Return expressions: value + 0; type(value)(copied) if isinstance(value, tuple) else copied; value ## `vllm_mlx.scheduler.Scheduler._prompt_output_entry_is_useless` - Kind: method - Signature: `def _prompt_output_entry_is_useless(cache: Any) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/scheduler.py#L2303-L2317 - Implementation: Method `Scheduler._prompt_output_entry_is_useless` calls `can_trim_prompt_cache`; has 2 explicit return paths. Would a prompt+output entry built from this cache ever be reusable? Only via a trim: any later query is shorter than a prompt+output key, so the generated tail has to come off first. When the cache cannot be trimmed the entry is dead weight — and far from free, since each one holds a full-length KV copy and Metal runs out of buffers long before the byte budget is reached. - Inputs: - `cache` (Any; required): Required positional or keyword input. - Return annotation: `bool` - Decorators: staticmethod - Calls: can_trim_prompt_cache - Return expressions: not can_trim_prompt_cache(cache); False ## `vllm_mlx.scheduler.Scheduler._extract_cache_for_uid` - Kind: method - Signature: `def _extract_cache_for_uid(self, uid: int) -> Any` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/scheduler.py#L2319-L2336 - Implementation: Method `Scheduler._extract_cache_for_uid` calls `getattr`, `extract`, `uids.index`, `logger.debug`; has 2 explicit return paths. Pull one sequence's cache out of the live BatchGenerator batch. - Inputs: - `uid` (int; required): Required positional or keyword input. - Return annotation: `Any` - Calls: getattr, extract, uids.index, logger.debug - State reads: self.batch_generator - Return expressions: None; extract(uids.index(uid)) ## `vllm_mlx.scheduler.Scheduler._make_snapshot_destination` - Kind: method - Signature: `def _make_snapshot_destination(self, live_cache: Any) -> Any` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/scheduler.py#L2338-L2380 - Implementation: Method `Scheduler._make_snapshot_destination` calls `_mirror`, `logger.warning`; has 2 explicit return paths. Build a destination cache with the same topology as the live one. ``make_prompt_cache(model)`` is not a safe source for this. A plain ``KVCache`` destination cannot take a ``RotatingKVCache``'s state or meta_state; the assignment raises, the broad handler below logs a warning, and the snapshot is silently never stored — on exactly the sliding-window configurations this feature exists for. Deriving it from ``config.max_kv_size`` instead is also wrong, which I only found by measuring: ``_create_batch_generator`` does not pass ``max_kv_size`` to ``BatchGenerator``, so with ``max_kv_size=512`` configured the live layers were still plain ``KVCache`` and a config-derived destination mismatched in the opposite direction. So mirror the live objects themselves. A shallow copy keeps the class and every scalar attribute (``max_size``, ``keep``, ``step``, ``_idx``) and the caller overwrites the arrays, which is the only part that must not be shared. - Inputs: - `live_cache` (Any; required): Required positional or keyword input. - Return annotation: `Any` - Calls: _mirror, logger.warning - Return expressions: [_mirror(layer) for layer in live_cache]; None ## `vllm_mlx.scheduler.Scheduler._make_snapshot_destination._mirror` - Kind: nested function - Signature: `def _mirror(layer: Any) -> Any` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/scheduler.py#L2360-L2370 - Implementation: Nested Function `Scheduler._make_snapshot_destination._mirror` calls `getattr`, `_mirror`, `copy.copy`, `type(children)`; has 2 explicit return paths. Nested Function `Scheduler._make_snapshot_destination._mirror` calls `getattr`, `_mirror`, `copy.copy`, `type(children)`; has 2 explicit return paths. - Inputs: - `layer` (Any; required): Required positional or keyword input. - Return annotation: `Any` - Calls: getattr, _mirror, copy.copy, type(children), type - Return expressions: container; copy.copy(layer) ## `vllm_mlx.scheduler.Scheduler._cache_coverage` - Kind: method - Signature: `def _cache_coverage(cache: Any) -> int | None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/scheduler.py#L2383-L2409 - Implementation: Method `Scheduler._cache_coverage` calls `_offset_of`; has 2 explicit return paths. How many tokens the live cache actually holds. Containers have to be descended into: ``CacheList`` carries no ``offset`` of its own, so reading the attribute off the layer returns None and the caller silently falls back to a prompt-only key — the misalignment this is here to prevent, on exactly the architectures (DeepSeek-V4) that group several caches per layer. - Inputs: - `cache` (Any; required): Required positional or keyword input. - Return annotation: `int | None` - Decorators: staticmethod - Calls: _offset_of - Return expressions: found; None ## `vllm_mlx.scheduler.Scheduler._cache_coverage._offset_of` - Kind: nested function - Signature: `def _offset_of(layer: Any) -> int | None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/scheduler.py#L2393-L2403 - Implementation: Nested Function `Scheduler._cache_coverage._offset_of` calls `getattr`, `isinstance`, `_offset_of`; has 3 explicit return paths. Nested Function `Scheduler._cache_coverage._offset_of` calls `getattr`, `isinstance`, `_offset_of`; has 3 explicit return paths. - Inputs: - `layer` (Any; required): Required positional or keyword input. - Return annotation: `int | None` - Calls: getattr, isinstance, _offset_of - Return expressions: offset; found; None ## `vllm_mlx.scheduler.Scheduler._cache_key_for_snapshot` - Kind: method - Signature: `def _cache_key_for_snapshot(self, request: Any, response: Any, raw_cache: Any) -> list[int] | None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/scheduler.py#L2411-L2468 - Implementation: Method `Scheduler._cache_key_for_snapshot` calls `self._cache_coverage`, `list`, `logger.debug`, `', '.join`; has 3 explicit return paths. Key the entry by the tokens the cache covers, not by the prompt. The snapshot is taken while processing the response that carries the first generated token, and by then the batch has already fed that token through the cache: measured ``prompt_len=5, cache_offset=6``. Storing that under ``prompt_token_ids`` leaves every warm reuse one token ahead of its key. Trimming the overshoot off is not available here — these are precisely the caches that cannot be trimmed — so the key is extended instead. The extra token is the first token of the reply, which the next turn's prompt also contains, so the entry still matches by strict prefix. Returns None rather than storing a misaligned entry. - Inputs: - `request` (Any; required): Required positional or keyword input. - `response` (Any; required): Required positional or keyword input. - `raw_cache` (Any; required): Required positional or keyword input. - Return annotation: `list[int] | None` - Calls: self._cache_coverage, list, logger.debug, ', '.join, sorted, type, len, getattr, int - State reads: self._cache_coverage - Return expressions: None; prompt_ids; prompt_ids + generated[:overshoot] ## `vllm_mlx.scheduler.Scheduler._store_prompt_only_cache` - Kind: method - Signature: `def _store_prompt_only_cache(self, request: Any, response: Any) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/scheduler.py#L2470-L2581 - Implementation: Method `Scheduler._store_prompt_only_cache` calls `getattr`, `len`, `callable`, `raw_cache`; returns `None`. Store the post-prefill cache under the prompt tokens alone. Called once per request, at the point where the cache covers exactly the prompt. Entries keyed this way are reusable without any trimming, which is what models with sliding-window or pooled KV need. - Inputs: - `request` (Any; required): Required positional or keyword input. - `response` (Any; required): Required positional or keyword input. - Return annotation: `None` - Calls: getattr, len, callable, raw_cache, self._extract_cache_for_uid, self._prompt_output_entry_is_useless, self._cache_key_for_snapshot, _t.monotonic, self._make_snapshot_destination, zip, self._copy_cache_state, states.append, mx.eval, logger.debug, self.memory_aware_cache.store, logger.info, logger.warning - State reads: self.memory_aware_cache, self.SNAPSHOT_REFRESH_TOKENS, self._extract_cache_for_uid, self._prompt_output_entry_is_useless, self._cache_key_for_snapshot, self._make_snapshot_destination, self._copy_cache_state, self.memory_aware_cache.store, self.memory_aware_cache._entries - Return expressions: None ## `vllm_mlx.scheduler.Scheduler._process_batch_responses` - Kind: method - Signature: `def _process_batch_responses(self, responses: List[Any]) -> Tuple[List[RequestOutput], Set[str]]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/scheduler.py#L2583-L2710 - Implementation: Method `Scheduler._process_batch_responses` updates `self.total_completion_tokens`, `self.num_requests_processed`; calls `set`, `self.uid_to_request_id.get`, `self.running.get`, `self._store_prompt_only_cache`; returns `(outputs, finished_ids)`. Process responses from BatchGenerator. Args: responses: List of BatchGenerator.Response objects Returns: Tuple of (outputs, finished_request_ids) - Inputs: - `responses` (List[Any]; required): List of BatchGenerator.Response objects - Return annotation: `Tuple[List[RequestOutput], Set[str]]` - Calls: set, self.uid_to_request_id.get, self.running.get, self._store_prompt_only_cache, request.append_output_token, _time.time, self._get_detokenizer, detok.add_token, RequestOutput, request.set_finished, finished_ids.add, self._detokenizer_pool.get, detok.finalize, self._decode_tokens, self._cleanup_detokenizer, hasattr, callable, response.prompt_cache, self._prompt_output_entry_is_useless, self._extract_cache_states, logger.debug, len, outputs.append - State reads: self.uid_to_request_id.get, self.uid_to_request_id, self.running.get, self.running, self._store_prompt_only_cache, self._get_detokenizer, self._detokenizer_pool.get, self._detokenizer_pool, self._decode_tokens, self._cleanup_detokenizer, self._prompt_output_entry_is_useless, self.block_aware_cache, self._extract_cache_states - State writes: self.total_completion_tokens, self.num_requests_processed - Return expressions: (outputs, finished_ids) ## `vllm_mlx.scheduler.Scheduler._cleanup_finished` - Kind: method - Signature: `def _cleanup_finished(self, finished_ids: Set[str]) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/scheduler.py#L2712-L2865 - Implementation: Method `Scheduler._cleanup_finished` calls `self.running.get`, `hasattr`, `list`, `self.block_aware_cache.store_cache`. Clean up finished requests and store caches for reuse. - Inputs: - `finished_ids` (Set[str]; required): Required positional or keyword input. - Return annotation: `None` - Calls: self.running.get, hasattr, list, self.block_aware_cache.store_cache, logger.debug, len, _time.monotonic, self.memory_aware_cache.store, logger.info, self.prefix_cache.store_cache, isinstance, mx.eval, callable, self.finished_req_ids.add, mx.clear_cache - State reads: self.running.get, self.running, self.block_aware_cache, self.block_aware_cache.store_cache, self.memory_aware_cache, self.memory_aware_cache.store, self.memory_aware_cache._entries, self.memory_aware_cache._current_memory, self.prefix_cache, self.prefix_cache.store_cache, self.request_id_to_uid, self.uid_to_request_id, self.finished_req_ids.add, self.finished_req_ids ## `vllm_mlx.scheduler.Scheduler._is_cache_corruption_error` - Kind: method - Signature: `def _is_cache_corruption_error(self, error: Exception) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/scheduler.py#L2867-L2870 - Implementation: Method `Scheduler._is_cache_corruption_error` calls `str`, `any`; returns `any((pattern in error_str for pattern in CACHE_CORRUPTION_PATTERNS))`. Check if an error indicates cache corruption. - Inputs: - `error` (Exception; required): Required positional or keyword input. - Return annotation: `bool` - Calls: str, any - Return expressions: any((pattern in error_str for pattern in CACHE_CORRUPTION_PATTERNS)) ## `vllm_mlx.scheduler.Scheduler._is_stream_thread_error` - Kind: method - Signature: `def _is_stream_thread_error(self, error: Exception) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/scheduler.py#L2872-L2875 - Implementation: Method `Scheduler._is_stream_thread_error` calls `str`; returns `'no Stream(' in error_str or 'no Stream(gpu' in error_str`. Check if an error indicates MLX stream/thread ownership mismatch. - Inputs: - `error` (Exception; required): Required positional or keyword input. - Return annotation: `bool` - Calls: str - Return expressions: 'no Stream(' in error_str or 'no Stream(gpu' in error_str ## `vllm_mlx.scheduler.Scheduler._recover_from_cache_error` - Kind: method - Signature: `def _recover_from_cache_error(self) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/scheduler.py#L2877-L2895 - Implementation: Method `Scheduler._recover_from_cache_error` updates `self._current_sampler_params`; calls `self._close_batch_generator`, `self.block_aware_cache.clear`, `self.memory_aware_cache.clear`, `self.prefix_cache.clear`. Recover from cache corruption error. - Inputs: none - Return annotation: `None` - Calls: self._close_batch_generator, self.block_aware_cache.clear, self.memory_aware_cache.clear, self.prefix_cache.clear, self.request_id_to_uid.clear, self.uid_to_request_id.clear, logger.info - State reads: self._close_batch_generator, self.block_aware_cache, self.block_aware_cache.clear, self.memory_aware_cache, self.memory_aware_cache.clear, self.prefix_cache, self.prefix_cache.clear, self.request_id_to_uid.clear, self.request_id_to_uid, self.uid_to_request_id.clear, self.uid_to_request_id - State writes: self._current_sampler_params ## `vllm_mlx.scheduler.Scheduler._recover_from_generation_error` - Kind: method - Signature: `def _recover_from_generation_error(self) -> Set[str]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/scheduler.py#L2897-L2933 - Implementation: Method `Scheduler._recover_from_generation_error` updates `self._current_sampler_params`; calls `self._close_batch_generator`, `set`, `list`, `self.running.get`; returns `aborted_ids`. Recover from fatal generation error (OOM, Metal crash). Aborts all running requests and resets batch state. Unlike cache corruption recovery, does NOT reschedule — the request that OOMed would just OOM again. Returns: Set of aborted request IDs. - Inputs: none - Return annotation: `Set[str]` - Calls: self._close_batch_generator, set, list, self.running.get, request.set_finished, aborted_ids.add, self.finished_req_ids.add, self.running.clear, self._detokenizer_pool.clear, self.request_id_to_uid.clear, self.uid_to_request_id.clear, mx.clear_cache, logger.warning, len - State reads: self._close_batch_generator, self.running, self.running.get, self.finished_req_ids.add, self.finished_req_ids, self.running.clear, self._detokenizer_pool.clear, self._detokenizer_pool, self.request_id_to_uid.clear, self.request_id_to_uid, self.uid_to_request_id.clear, self.uid_to_request_id - State writes: self._current_sampler_params - Return expressions: aborted_ids ## `vllm_mlx.scheduler.Scheduler._reschedule_running_requests` - Kind: method - Signature: `def _reschedule_running_requests(self) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/scheduler.py#L2935-L2951 - Implementation: Method `Scheduler._reschedule_running_requests` calls `len`, `list`, `self.running.items`, `self.waiting.appendleft`. Move running requests back to waiting queue for retry. - Inputs: none - Return annotation: `None` - Calls: len, list, self.running.items, self.waiting.appendleft, logger.info - State reads: self.running, self.running.items, self.waiting.appendleft, self.waiting ## `vllm_mlx.scheduler.Scheduler.step` - Kind: method - Signature: `def step(self, max_retries: int=1) -> SchedulerOutput` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/scheduler.py#L2953-L3089 - Implementation: Method `Scheduler.step` updates `self.finished_req_ids`, `self._step_count`; calls `SchedulerOutput`, `self._process_pending_aborts`, `range`, `self._schedule_waiting`; returns `output`. Execute one scheduling step with automatic error recovery. This method: 1. Schedules waiting requests into the batch 2. Runs one generation step via BatchGenerator 3. Processes outputs and handles finished requests 4. Automatically recovers from cache corruption errors Args: max_retries: Number of times to retry on cache errors (default 1) Returns: SchedulerOutput with results of this step - Inputs: - `max_retries` (int; optional; default `1`): Number of times to retry on cache errors (default 1) - Return annotation: `SchedulerOutput` - Calls: SchedulerOutput, self._process_pending_aborts, range, self._schedule_waiting, sum, _sanitize_batch_generator_logits_processors, self.batch_generator.next, isinstance, self._process_batch_responses, self._cleanup_finished, self._is_cache_corruption_error, logger.warning, self._recover_from_cache_error, self._reschedule_running_requests, logger.error, self._is_stream_thread_error, traceback.format_exc, self._recover_from_generation_error, output.outputs.append, RequestOutput, set, len, max, hasattr, mx.eval, mx.clear_cache, mx.metal.is_available, mx.get_active_memory, mx.get_peak_memory, mx.get_cache_memory, logger.info - State reads: self._process_pending_aborts, self._schedule_waiting, self.batch_generator, self.running, self.batch_generator.next, self._process_batch_responses, self._cleanup_finished, self._is_cache_corruption_error, self._recover_from_cache_error, self._reschedule_running_requests, self._is_stream_thread_error, self._recover_from_generation_error, self.finished_req_ids, self._clear_cache_interval, self._step_count, self.batch_generator.active_batch, self.batch_generator.active_batch.tokens, self._memory_log_interval, self.waiting - State writes: self.finished_req_ids, self._step_count - Return expressions: output ## `vllm_mlx.scheduler.Scheduler.get_request` - Kind: method - Signature: `def get_request(self, request_id: str) -> Optional[Request]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/scheduler.py#L3091-L3093 - Implementation: Method `Scheduler.get_request` calls `self.requests.get`; returns `self.requests.get(request_id)`. Get a request by ID. - Inputs: - `request_id` (str; required): Required positional or keyword input. - Return annotation: `Optional[Request]` - Calls: self.requests.get - State reads: self.requests.get, self.requests - Return expressions: self.requests.get(request_id) ## `vllm_mlx.scheduler.Scheduler.remove_finished_request` - Kind: method - Signature: `def remove_finished_request(self, request_id: str) -> Optional[Request]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/scheduler.py#L3095-L3097 - Implementation: Method `Scheduler.remove_finished_request` calls `self.requests.pop`; returns `self.requests.pop(request_id, None)`. Remove a finished request from tracking. - Inputs: - `request_id` (str; required): Required positional or keyword input. - Return annotation: `Optional[Request]` - Calls: self.requests.pop - State reads: self.requests.pop, self.requests - Return expressions: self.requests.pop(request_id, None) ## `vllm_mlx.scheduler.Scheduler.get_running_requests_info` - Kind: method - Signature: `def get_running_requests_info(self) -> List[Dict[str, Any]]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/scheduler.py#L3099-L3165 - Implementation: Method `Scheduler.get_running_requests_info` calls `_time.time`, `result.append`, `round`, `self.running.values`; returns `result`. Per-request details for status endpoint. - Inputs: none - Return annotation: `List[Dict[str, Any]]` - Calls: _time.time, result.append, round, self.running.values, min - State reads: self.waiting, self.running.values, self.running - Return expressions: result ## `vllm_mlx.scheduler.Scheduler.get_stats` - Kind: method - Signature: `def get_stats(self) -> Dict[str, Any]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/scheduler.py#L3167-L3193 - Implementation: Method `Scheduler.get_stats` calls `len`, `stats.update`, `_mtp_status_snapshot`, `mx.metal.is_available`; returns `stats`. Get scheduler statistics. - Inputs: none - Return annotation: `Dict[str, Any]` - Calls: len, stats.update, _mtp_status_snapshot, mx.metal.is_available, round, mx.get_active_memory, mx.get_peak_memory, mx.get_cache_memory, self.block_aware_cache.get_stats, self.memory_aware_cache.get_stats, self.prefix_cache.get_stats - State reads: self.waiting, self.running, self.num_requests_processed, self.total_prompt_tokens, self.total_completion_tokens, self.batch_generator, self.block_aware_cache, self.block_aware_cache.get_stats, self.memory_aware_cache, self.memory_aware_cache.get_stats, self.prefix_cache, self.prefix_cache.get_stats - Return expressions: stats ## `vllm_mlx.scheduler.Scheduler.get_cache_stats` - Kind: method - Signature: `def get_cache_stats(self) -> Optional[Dict[str, Any]]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/scheduler.py#L3195-L3203 - Implementation: Method `Scheduler.get_cache_stats` calls `self.block_aware_cache.get_stats`, `self.memory_aware_cache.get_stats`, `self.prefix_cache.get_stats`; has 4 explicit return paths. Get cache statistics. - Inputs: none - Return annotation: `Optional[Dict[str, Any]]` - Calls: self.block_aware_cache.get_stats, self.memory_aware_cache.get_stats, self.prefix_cache.get_stats - State reads: self.block_aware_cache, self.block_aware_cache.get_stats, self.memory_aware_cache, self.memory_aware_cache.get_stats, self.prefix_cache, self.prefix_cache.get_stats - Return expressions: self.block_aware_cache.get_stats(); self.memory_aware_cache.get_stats(); self.prefix_cache.get_stats(); None ## `vllm_mlx.scheduler.Scheduler.clear_runtime_caches` - Kind: method - Signature: `def clear_runtime_caches(self) -> Dict[str, bool]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/scheduler.py#L3205-L3221 - Implementation: Method `Scheduler.clear_runtime_caches` calls `self.block_aware_cache.clear`, `self.memory_aware_cache.clear`, `self.prefix_cache.clear`; returns `cleared`. Clear prefix-cache state without resetting scheduler/request state. - Inputs: none - Return annotation: `Dict[str, bool]` - Calls: self.block_aware_cache.clear, self.memory_aware_cache.clear, self.prefix_cache.clear - State reads: self.block_aware_cache, self.block_aware_cache.clear, self.memory_aware_cache, self.memory_aware_cache.clear, self.prefix_cache, self.prefix_cache.clear - Return expressions: cleared ## `vllm_mlx.scheduler.Scheduler.reset` - Kind: method - Signature: `def reset(self) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/scheduler.py#L3223-L3246 - Implementation: Method `Scheduler.reset` updates `self._current_sampler_params`; calls `self._pending_abort_ids.clear`, `list`, `self.requests.keys`, `self._do_abort_request`. Reset the scheduler state. - Inputs: none - Return annotation: `None` - Calls: self._pending_abort_ids.clear, list, self.requests.keys, self._do_abort_request, self.waiting.clear, self.running.clear, self.requests.clear, self.finished_req_ids.clear, self.request_id_to_uid.clear, self.uid_to_request_id.clear, self._detokenizer_pool.clear, self._close_batch_generator, self.clear_runtime_caches, self.close_ssd_tier - State reads: self._pending_abort_ids.clear, self._pending_abort_ids, self.requests.keys, self.requests, self._do_abort_request, self.waiting.clear, self.waiting, self.running.clear, self.running, self.requests.clear, self.finished_req_ids.clear, self.finished_req_ids, self.request_id_to_uid.clear, self.request_id_to_uid, self.uid_to_request_id.clear, self.uid_to_request_id, self._detokenizer_pool.clear, self._detokenizer_pool, self._close_batch_generator, self.clear_runtime_caches, self.close_ssd_tier - State writes: self._current_sampler_params ## `vllm_mlx.scheduler.Scheduler.deep_reset` - Kind: method - Signature: `def deep_reset(self) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/scheduler.py#L3248-L3276 - Implementation: Method `Scheduler.deep_reset` updates `self.model.cache`; calls `self.reset`, `hasattr`, `gc.collect`, `logger.info`. Deep reset that clears ALL cache state including model-level caches. This is more aggressive than reset() and should be used when switching engines or recovering from errors. - Inputs: none - Return annotation: `None` - Calls: self.reset, hasattr, gc.collect, logger.info - State reads: self.reset, self.model, self.model.layers - State writes: self.model.cache ## `vllm_mlx.scheduler.Scheduler.save_cache_to_disk` - Kind: method - Signature: `def save_cache_to_disk(self, cache_dir: str) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/scheduler.py#L3282-L3287 - Implementation: Method `Scheduler.save_cache_to_disk` calls `self.memory_aware_cache.save_to_disk`, `logger.info`; has 2 explicit return paths. Save prefix cache to disk for persistence across restarts. - Inputs: - `cache_dir` (str; required): Required positional or keyword input. - Return annotation: `bool` - Calls: self.memory_aware_cache.save_to_disk, logger.info - State reads: self.memory_aware_cache, self.memory_aware_cache.save_to_disk - Return expressions: self.memory_aware_cache.save_to_disk(cache_dir); False ## `vllm_mlx.scheduler.Scheduler.load_cache_from_disk` - Kind: method - Signature: `def load_cache_from_disk(self, cache_dir: str) -> int` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/scheduler.py#L3289-L3294 - Implementation: Method `Scheduler.load_cache_from_disk` calls `self.memory_aware_cache.load_from_disk`, `logger.info`; has 2 explicit return paths. Load prefix cache from disk. Returns number of entries loaded. - Inputs: - `cache_dir` (str; required): Required positional or keyword input. - Return annotation: `int` - Calls: self.memory_aware_cache.load_from_disk, logger.info - State reads: self.memory_aware_cache, self.memory_aware_cache.load_from_disk - Return expressions: self.memory_aware_cache.load_from_disk(cache_dir); 0 ## `vllm_mlx.scheduler.Scheduler.clear_prefix_cache` - Kind: method - Signature: `def clear_prefix_cache(self) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/scheduler.py#L3296-L3306 - Implementation: Method `Scheduler.clear_prefix_cache` calls `hasattr`, `self.memory_aware_cache.clear`, `logger.info`, `self.prefix_cache.clear`; returns `None`. Clear the in-memory prefix cache (keeps disk cache untouched). - Inputs: none - Return annotation: `None` - Calls: hasattr, self.memory_aware_cache.clear, logger.info, self.prefix_cache.clear - State reads: self.memory_aware_cache, self.memory_aware_cache.clear, self.prefix_cache, self.prefix_cache.clear - Return expressions: None ## `vllm_mlx.scheduler.Scheduler.close_ssd_tier` - Kind: method - Signature: `def close_ssd_tier(self) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/scheduler.py#L3308-L3313 - Implementation: Method `Scheduler.close_ssd_tier` updates `self._ssd_tier`; calls `self._ssd_tier.close`, `logger.info`. Shut down the SSD cache tier if present. - Inputs: none - Return annotation: `None` - Calls: self._ssd_tier.close, logger.info - State reads: self._ssd_tier, self._ssd_tier.close - State writes: self._ssd_tier ## `vllm_mlx.scheduler.Scheduler._try_promote_ssd_pending` - Kind: method - Signature: `def _try_promote_ssd_pending(self) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/scheduler.py#L3315-L3395 - Implementation: Method `Scheduler._try_promote_ssd_pending` updates `self._ssd_tier._stats.promotion_failures`, `self._ssd_tier._stats.ssd_hits`; calls `getattr`, `self.memory_aware_cache.try_reserve_memory`, `logger.info`, `tuple`. Attempt synchronous SSD promotion for waiting requests tagged ssd_pending. Called from _schedule_waiting() before requests are moved to running. Reads SSD entries synchronously (disk I/O stays out of fetch() per spec). - Inputs: none - Return annotation: `None` - Calls: getattr, self.memory_aware_cache.try_reserve_memory, logger.info, tuple, self._ssd_tier._read_entry, self.memory_aware_cache.release_reserved_memory, logger.exception, self._reconstruct_ssd_layers, self.memory_aware_cache.store, list, self._ssd_tier._index.touch, len - State reads: self.waiting, self.memory_aware_cache, self.memory_aware_cache.try_reserve_memory, self._ssd_tier._stats, self._ssd_tier, self._ssd_tier._read_entry, self.memory_aware_cache.release_reserved_memory, self._reconstruct_ssd_layers, self.memory_aware_cache.store, self._ssd_tier._index.touch, self._ssd_tier._index - State writes: self._ssd_tier._stats.promotion_failures, self._ssd_tier._stats.ssd_hits ## `vllm_mlx.scheduler.Scheduler.promote_from_ssd` - Kind: method - Signature: `async def promote_from_ssd(self, request) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/scheduler.py#L3397-L3460 - Implementation: Method `Scheduler.promote_from_ssd` calls `getattr`, `candidate.get`, `len`, `tuple`; awaits asynchronous work; has 2 explicit return paths. Promote a cold-tier cache entry for a request (async version). Alternative to _try_promote_ssd_pending() for callers with an async event loop. Uses asyncio.to_thread for non-blocking disk I/O. Returns True if promotion succeeded and request was updated. - Inputs: - `request` (not annotated; required): Required positional or keyword input. - Return annotation: `bool` - Calls: getattr, candidate.get, len, tuple, self._ssd_tier.async_promote, release_budget, self._reconstruct_ssd_layers, self.memory_aware_cache.store, list, logger.info - State reads: self._ssd_tier, self._ssd_tier.async_promote, self._reconstruct_ssd_layers, self.memory_aware_cache.store, self.memory_aware_cache - Return expressions: False; True ## `vllm_mlx.scheduler.Scheduler.promote_from_ssd.reserve_budget` - Kind: nested function - Signature: `def reserve_budget(nbytes: int) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/scheduler.py#L3412-L3416 - Implementation: Nested Function `Scheduler.promote_from_ssd.reserve_budget` calls `self.memory_aware_cache.try_reserve_memory`; has 2 explicit return paths. Tentatively reserve RAM budget for promotion. - Inputs: - `nbytes` (int; required): Required positional or keyword input. - Return annotation: `bool` - Calls: self.memory_aware_cache.try_reserve_memory - State reads: self.memory_aware_cache, self.memory_aware_cache.try_reserve_memory - Return expressions: False; self.memory_aware_cache.try_reserve_memory(nbytes) ## `vllm_mlx.scheduler.Scheduler.promote_from_ssd.release_budget` - Kind: nested function - Signature: `def release_budget(nbytes: int) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/scheduler.py#L3418-L3421 - Implementation: Nested Function `Scheduler.promote_from_ssd.release_budget` calls `self.memory_aware_cache.release_reserved_memory`. Release tentatively reserved budget on failure. - Inputs: - `nbytes` (int; required): Required positional or keyword input. - Return annotation: `None` - Calls: self.memory_aware_cache.release_reserved_memory - State reads: self.memory_aware_cache, self.memory_aware_cache.release_reserved_memory ## `vllm_mlx.scheduler.Scheduler._reconstruct_ssd_layers` - Kind: method - Signature: `def _reconstruct_ssd_layers(self, layer_dicts: list[dict]) -> list | None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/scheduler.py#L3462-L3518 - Implementation: Method `Scheduler._reconstruct_ssd_layers` calls `KVCache`, `mx.array`, `ld.get`, `_mx_dtype_from_name`; has 2 explicit return paths. Reconstruct cache objects from deserialized layer dicts. Converts numpy arrays back to MLX arrays and creates KVCache objects. - Inputs: - `layer_dicts` (list[dict]; required): Required positional or keyword input. - Return annotation: `list | None` - Calls: KVCache, mx.array, ld.get, _mx_dtype_from_name, kv.keys.astype, kv.values.astype, setattr, result.append, enumerate, state_arrays[i].astype, ArraysCache, len, logger.warning, list, ld.keys - Return expressions: None; result ## `vllm_mlx.scheduler.Scheduler._reconstruct_ssd_layers._mx_dtype_from_name` - Kind: nested function - Signature: `def _mx_dtype_from_name(name: str)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/scheduler.py#L3473-L3474 - Implementation: Nested Function `Scheduler._reconstruct_ssd_layers._mx_dtype_from_name` calls `getattr`; returns `getattr(mx, name, None)`. Nested Function `Scheduler._reconstruct_ssd_layers._mx_dtype_from_name` calls `getattr`; returns `getattr(mx, name, None)`. - Inputs: - `name` (str; required): Required positional or keyword input. - Return annotation: `not annotated` - Calls: getattr - Return expressions: getattr(mx, name, None) # Module `vllm_mlx.server` Unified OpenAI-compatible API server for vllm-mlx. This module provides a FastAPI server that exposes an OpenAI-compatible API for LLM and MLLM (Multimodal Language Model) inference using MLX on Apple Silicon. Supports two modes: - Simple mode (default): Maximum throughput for single-user scenarios - Batched mode: Continuous batching for multiple concurrent users Features: - Text-only LLM inference (mlx-lm) - Multimodal MLLM inference with images and video (mlx-vlm) - OpenAI-compatible chat/completions API - Streaming responses - MCP (Model Context Protocol) tool integration - Tool calling (Qwen/Llama formats) Usage: # Simple mode (maximum throughput) python -m vllm_mlx.server --model mlx-community/Llama-3.2-3B-Instruct-4bit # Batched mode (for multiple concurrent users) python -m vllm_mlx.server --model mlx-community/Llama-3.2-3B-Instruct-4bit --continuous-batching # With MCP tools python -m vllm_mlx.server --model mlx-community/Qwen3-4B-4bit --mcp-config mcp.json The server provides: - POST /v1/completions - Text completions - POST /v1/chat/completions - Chat completions (with multimodal support) - GET /v1/models - List available models - GET /health - Health check - GET /v1/mcp/tools - List MCP tools - GET /v1/mcp/servers - MCP server status - POST /v1/mcp/execute - Execute MCP tool Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L1-L6916 ## `vllm_mlx.server._resolve_temperature` - Kind: function - Signature: `def _resolve_temperature(request_value: float | None) -> float` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L225-L231 - Implementation: Function `_resolve_temperature` has 3 explicit return paths. Resolve temperature: request > CLI default > fallback. - Inputs: - `request_value` (float | None; required): Required positional or keyword input. - Return annotation: `float` - Return expressions: request_value; _default_temperature; _FALLBACK_TEMPERATURE ## `vllm_mlx.server._resolve_top_p` - Kind: function - Signature: `def _resolve_top_p(request_value: float | None) -> float` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L234-L240 - Implementation: Function `_resolve_top_p` has 3 explicit return paths. Resolve top_p: request > CLI default > fallback. - Inputs: - `request_value` (float | None; required): Required positional or keyword input. - Return annotation: `float` - Return expressions: request_value; _default_top_p; _FALLBACK_TOP_P ## `vllm_mlx.server._resolve_top_k` - Kind: function - Signature: `def _resolve_top_k(request_value: int | None) -> int` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L243-L249 - Implementation: Function `_resolve_top_k` has 3 explicit return paths. Resolve top_k: request > CLI default > fallback. - Inputs: - `request_value` (int | None; required): Required positional or keyword input. - Return annotation: `int` - Return expressions: request_value; _default_top_k; _FALLBACK_TOP_K ## `vllm_mlx.server._resolve_min_p` - Kind: function - Signature: `def _resolve_min_p(request_value: float | None) -> float` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L252-L258 - Implementation: Function `_resolve_min_p` has 3 explicit return paths. Resolve min_p: request > CLI default > fallback. - Inputs: - `request_value` (float | None; required): Required positional or keyword input. - Return annotation: `float` - Return expressions: request_value; _default_min_p; _FALLBACK_MIN_P ## `vllm_mlx.server._resolve_presence_penalty` - Kind: function - Signature: `def _resolve_presence_penalty(request_value: float | None) -> float` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L261-L267 - Implementation: Function `_resolve_presence_penalty` has 3 explicit return paths. Resolve presence_penalty: request > CLI default > fallback. - Inputs: - `request_value` (float | None; required): Required positional or keyword input. - Return annotation: `float` - Return expressions: request_value; _default_presence_penalty; _FALLBACK_PRESENCE_PENALTY ## `vllm_mlx.server._resolve_repetition_penalty` - Kind: function - Signature: `def _resolve_repetition_penalty(request_value: float | None) -> float` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L270-L276 - Implementation: Function `_resolve_repetition_penalty` has 3 explicit return paths. Resolve repetition_penalty: request > CLI default > fallback. - Inputs: - `request_value` (float | None; required): Required positional or keyword input. - Return annotation: `float` - Return expressions: request_value; _default_repetition_penalty; _FALLBACK_REPETITION_PENALTY ## `vllm_mlx.server._resolve_request_max_tokens` - Kind: function - Signature: `def _resolve_request_max_tokens(requested_value: int | None) -> int` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L279-L288 - Implementation: Function `_resolve_request_max_tokens` calls `HTTPException`; can raise `HTTPException`; has 2 explicit return paths. Resolve and validate a request's max_tokens budget. - Inputs: - `requested_value` (int | None; required): Required positional or keyword input. - Return annotation: `int` - Calls: HTTPException - Raises directly: HTTPException - Return expressions: _default_max_tokens; requested_value ## `vllm_mlx.server._resolve_chat_template_kwargs` - Kind: function - Signature: `def _resolve_chat_template_kwargs(request_value: dict[str, object] | None) -> dict[str, object]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L291-L300 - Implementation: Function `_resolve_chat_template_kwargs` calls `resolved.update`; returns `resolved`. Resolve chat template kwargs: request > server default > empty dict. - Inputs: - `request_value` (dict[str, object] | None; required): Required positional or keyword input. - Return annotation: `dict[str, object]` - Calls: resolved.update - Return expressions: resolved ## `vllm_mlx.server.PreparedChatInvocation` - Kind: class - Signature: `class PreparedChatInvocation` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L304-L311 - Implementation: Class `PreparedChatInvocation` declares 0 direct member(s). Fully prepared inputs for a single engine.chat/stream_chat call. - Inputs: - `messages` (list[dict]; required): Required constructor field. - `chat_kwargs` (dict[str, object]; required): Required constructor field. - `response_format` (object | None; required): Required constructor field. - `json_logits_processor` (object | None; required): Required constructor field. - `thinking_processor` (object | None; optional; default `None`): Optional constructor field; defaults to `None`. - Constructs: `vllm_mlx.server.PreparedChatInvocation` - Decorators: dataclass ## `vllm_mlx.server._prepare_chat_messages` - Kind: function - Signature: `def _prepare_chat_messages(engine: BaseEngine, request_messages: list[Message | dict]) -> tuple[list[dict], list, list, list, bool]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L314-L398 - Implementation: Function `_prepare_chat_messages` calls `_validate_remote_media_urls`, `bool`, `getattr`, `hasattr`; returns `(messages, images, videos, audios, has_media)`. Normalize messages and collect media once for both stream/non-stream paths. - Inputs: - `engine` (BaseEngine; required): Required positional or keyword input. - `request_messages` (list[Message | dict]; required): Required positional or keyword input. - Return annotation: `tuple[list[dict], list, list, list, bool]` - Calls: _validate_remote_media_urls, bool, getattr, hasattr, msg.model_dump, dict, raw.items, messages.append, logger.debug, len, msg_dict.get, tc.get, func.get, isinstance, json.loads, _normalize_messages, extract_multimodal_content, canonicalize_system_messages, msg.get, item.get - Return expressions: (messages, images, videos, audios, has_media) ## `vllm_mlx.server._iter_remote_media_urls` - Kind: function - Signature: `def _iter_remote_media_urls(messages: list[Message | dict])` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L401-L429 - Implementation: Function `_iter_remote_media_urls` calls `isinstance`, `msg.get`, `hasattr`, `item.model_dump`; yields values incrementally. Yield remote media URLs from OpenAI-style multimodal message content. - Inputs: - `messages` (list[Message | dict]; required): Required positional or keyword input. - Return annotation: `not annotated` - Calls: isinstance, msg.get, hasattr, item.model_dump, item.dict().items, item.dict, item.get, media_value.get, is_url ## `vllm_mlx.server._validate_remote_media_urls` - Kind: function - Signature: `def _validate_remote_media_urls(messages: list[Message | dict]) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L432-L435 - Implementation: Function `_validate_remote_media_urls` calls `_iter_remote_media_urls`, `_validate_url_safety`. Validate remote media URLs during request preparation. - Inputs: - `messages` (list[Message | dict]; required): Required positional or keyword input. - Return annotation: `None` - Calls: _iter_remote_media_urls, _validate_url_safety ## `vllm_mlx.server._raise_remote_media_http_error` - Kind: function - Signature: `def _raise_remote_media_http_error(exc: UnsafeRemoteURLError) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L438-L444 - Implementation: Function `_raise_remote_media_http_error` calls `logger.warning`, `_sanitize_log_text`, `HTTPException`; can raise `HTTPException`. Log internal URL-safety detail while returning a generic client error. - Inputs: - `exc` (UnsafeRemoteURLError; required): Required positional or keyword input. - Return annotation: `None` - Calls: logger.warning, _sanitize_log_text, HTTPException - Raises directly: HTTPException ## `vllm_mlx.server._prepare_json_logits_processor` - Kind: function - Signature: `def _prepare_json_logits_processor(engine: BaseEngine, messages: list[dict], response_format: object | None, *, tools: list | None, tool_choice: object | None, log_context: str | None=None, thinking_model: bool=False) -> tuple[list[dict], object | None]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L447-L497 - Implementation: Function `_prepare_json_logits_processor` calls `build_json_system_prompt`, `_inject_json_instruction`, `_get_engine_tokenizer`, `build_json_logits_processor`; returns `(messages, json_logits_processor)`. Inject response_format instruction and build constrained decoding processor. - Inputs: - `engine` (BaseEngine; required): Required positional or keyword input. - `messages` (list[dict]; required): Required positional or keyword input. - `response_format` (object | None; required): Required positional or keyword input. - `tools` (list | None; required): Required keyword-only input. - `tool_choice` (object | None; required): Required keyword-only input. - `log_context` (str | None; optional; default `None`): Optional keyword-only input; defaults to `None`. - `thinking_model` (bool; optional; default `False`): Optional keyword-only input; defaults to `False`. - Return annotation: `tuple[list[dict], object | None]` - Calls: build_json_system_prompt, _inject_json_instruction, _get_engine_tokenizer, build_json_logits_processor, logger.warning, logger.info, isinstance, getattr, response_format.get - Return expressions: (messages, json_logits_processor) ## `vllm_mlx.server._build_thinking_processor` - Kind: function - Signature: `def _build_thinking_processor(engine: BaseEngine, thinking_token_budget: int, *, inner: object | None=None, prompt_has_think_tag: bool=True) -> object | None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L500-L554 - Implementation: Function `_build_thinking_processor` calls `_get_engine_tokenizer`, `tokenizer.encode`, `logger.debug`, `getattr`; has 2 explicit return paths. Build a ThinkingAwareLogitsProcessor if the tokenizer has think tokens. - Inputs: - `engine` (BaseEngine; required): Required positional or keyword input. - `thinking_token_budget` (int; required): Required positional or keyword input. - `inner` (object | None; optional; default `None`): Optional keyword-only input; defaults to `None`. - `prompt_has_think_tag` (bool; optional; default `True`): Optional keyword-only input; defaults to `True`. - Return annotation: `object | None` - Calls: _get_engine_tokenizer, tokenizer.encode, logger.debug, getattr, _resolve_no_final_content_token_limit, logger.warning, ThinkingAwareLogitsProcessor, logger.info - Return expressions: None; proc ## `vllm_mlx.server._resolve_no_final_content_token_limit` - Kind: function - Signature: `def _resolve_no_final_content_token_limit() -> int | None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L557-L568 - Implementation: Function `_resolve_no_final_content_token_limit` calls `os.environ.get`, `raw.strip`, `int`, `logger.warning`; has 2 explicit return paths. Function `_resolve_no_final_content_token_limit` calls `os.environ.get`, `raw.strip`, `int`, `logger.warning`; has 2 explicit return paths. - Inputs: none - Return annotation: `int | None` - Calls: os.environ.get, raw.strip, int, logger.warning - Return expressions: None; value ## `vllm_mlx.server._generation_metadata` - Kind: function - Signature: `def _generation_metadata(thinking_processor: object | None) -> GenerationMetadata | None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L571-L583 - Implementation: Function `_generation_metadata` calls `GenerationMetadata`, `getattr`, `bool`; has 2 explicit return paths. Function `_generation_metadata` calls `GenerationMetadata`, `getattr`, `bool`; has 2 explicit return paths. - Inputs: - `thinking_processor` (object | None; required): Required positional or keyword input. - Return annotation: `GenerationMetadata | None` - Calls: GenerationMetadata, getattr, bool - Return expressions: None; GenerationMetadata(no_final_content_watchdog_tokens=getattr(thinking_processor, '_no_final_content_token_limit', None),… ## `vllm_mlx.server._ThinkingAwareLogitsProcessor` - Kind: class - Signature: `class _ThinkingAwareLogitsProcessor` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L586-L697 - Implementation: Class `_ThinkingAwareLogitsProcessor` declares 5 direct member(s). Wrap a ``JSONSchemaLogitsProcessor`` so JSON constraining only activates after the model emits ````, letting it reason freely first. Without this wrapper ``enable_thinking`` is forced to ``False`` when constrained decoding is active, which degrades output for thinking models (Qwen 3.5/3.6, DeepSeek-R1, etc.) — the model produces degenerated whitespace/brace loops instead of valid JSON because it was trained to think before answering. - Inputs: - `inner` (not annotated; required): Required positional or keyword input. - `prompt_has_think_tag` (bool; optional; default `False`): Optional positional or keyword input; defaults to `False`. - Constructs: `vllm_mlx.server._ThinkingAwareLogitsProcessor` ## `vllm_mlx.server._ThinkingAwareLogitsProcessor.__init__` - Kind: method - Signature: `def __init__(self, inner, prompt_has_think_tag: bool=False)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L597-L608 - Implementation: Method `_ThinkingAwareLogitsProcessor.__init__` updates `self._inner`, `self._active`, `self._in_thinking`, `self._waiting_for_json`. Method `_ThinkingAwareLogitsProcessor.__init__` updates `self._inner`, `self._active`, `self._in_thinking`, `self._waiting_for_json`. - Inputs: - `inner` (not annotated; required): Required positional or keyword input. - `prompt_has_think_tag` (bool; optional; default `False`): Optional positional or keyword input; defaults to `False`. - Return annotation: `not annotated` - State writes: self._inner, self._active, self._in_thinking, self._waiting_for_json, self._base_prompt_len, self._json_scan_offset, self._tokenizer ## `vllm_mlx.server._ThinkingAwareLogitsProcessor._scan_for_json_start` - Kind: method - Signature: `def _scan_for_json_start(self, tokens_list, tokens, logits)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L610-L635 - Implementation: Method `_ThinkingAwareLogitsProcessor._scan_for_json_start` updates `self._active`, `self._inner._prompt_len`; calls `len`, `range`, `self._tokenizer.decode`, `any`; has 2 explicit return paths. Scan generated tokens for the first ``{`` or ``[``. Scans from ``_json_scan_offset`` (set when entering the waiting phase) so that thinking-span tokens are never considered. After 50 tokens past the scan offset without a JSON start character the enforcer is force-activated as a safety net. - Inputs: - `tokens_list` (not annotated; required): Required positional or keyword input. - `tokens` (not annotated; required): Required positional or keyword input. - `logits` (not annotated; required): Required positional or keyword input. - Return annotation: `not annotated` - Calls: len, range, self._tokenizer.decode, any, self._inner - State reads: self._json_scan_offset, self._tokenizer.decode, self._tokenizer, self._inner - State writes: self._active, self._inner._prompt_len - Return expressions: self._inner(tokens, logits); logits ## `vllm_mlx.server._ThinkingAwareLogitsProcessor.__call__` - Kind: method - Signature: `def __call__(self, tokens, logits)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L637-L688 - Implementation: Method `_ThinkingAwareLogitsProcessor.__call__` updates `self._base_prompt_len`, `self._in_thinking`, `self._waiting_for_json`, `self._json_scan_offset`; calls `self._inner`, `hasattr`, `tokens.tolist`, `list`; has 3 explicit return paths. Method `_ThinkingAwareLogitsProcessor.__call__` updates `self._base_prompt_len`, `self._in_thinking`, `self._waiting_for_json`, `self._json_scan_offset`; calls `self._inner`, `hasattr`, `tokens.tolist`, `list`; has 3 explicit return paths. - Inputs: - `tokens` (not annotated; required): Required positional or keyword input. - `logits` (not annotated; required): Required positional or keyword input. - Return annotation: `not annotated` - Calls: self._inner, hasattr, tokens.tolist, list, isinstance, len, max, self._scan_for_json_start, self._tokenizer.decode, min - State reads: self._active, self._inner, self._base_prompt_len, self._waiting_for_json, self._scan_for_json_start, self._in_thinking, self._tokenizer.decode, self._tokenizer - State writes: self._base_prompt_len, self._in_thinking, self._waiting_for_json, self._json_scan_offset - Return expressions: self._inner(tokens, logits); self._scan_for_json_start(tokens_list, tokens, logits); logits ## `vllm_mlx.server._ThinkingAwareLogitsProcessor.schema` - Kind: method - Signature: `def schema(self)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L692-L693 - Implementation: Method `_ThinkingAwareLogitsProcessor.schema` returns `self._inner.schema`. Method `_ThinkingAwareLogitsProcessor.schema` returns `self._inner.schema`. - Inputs: none - Return annotation: `not annotated` - Decorators: property - State reads: self._inner.schema, self._inner - Return expressions: self._inner.schema ## `vllm_mlx.server._ThinkingAwareLogitsProcessor._disabled` - Kind: method - Signature: `def _disabled(self)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L696-L697 - Implementation: Method `_ThinkingAwareLogitsProcessor._disabled` returns `self._inner._disabled`. Method `_ThinkingAwareLogitsProcessor._disabled` returns `self._inner._disabled`. - Inputs: none - Return annotation: `not annotated` - Decorators: property - State reads: self._inner._disabled, self._inner - Return expressions: self._inner._disabled ## `vllm_mlx.server._attach_response_format_logits_processor` - Kind: function - Signature: `def _attach_response_format_logits_processor(chat_kwargs: dict, json_logits_processor: object) -> object` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L700-L717 - Implementation: Function `_attach_response_format_logits_processor` calls `dict`, `chat_kwargs.get`, `list`; returns `json_logits_processor`. Attach response_format constraints and keep thinking disabled. response_format content must be constrained from the first generated token. If the processor is hidden behind thinking-state handling, direct JSON emissions can bypass the constraint and run until max_tokens. - Inputs: - `chat_kwargs` (dict; required): Required positional or keyword input. - `json_logits_processor` (object; required): Required positional or keyword input. - Return annotation: `object` - Calls: dict, chat_kwargs.get, list - Return expressions: json_logits_processor ## `vllm_mlx.server._coerce_logit_bias` - Kind: function - Signature: `def _coerce_logit_bias(logit_bias: dict[str, float]) -> dict[int, float]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L720-L730 - Implementation: Function `_coerce_logit_bias` calls `logit_bias.items`, `int`, `float`, `HTTPException`; can raise `HTTPException`; returns `coerced`. Function `_coerce_logit_bias` calls `logit_bias.items`, `int`, `float`, `HTTPException`; can raise `HTTPException`; returns `coerced`. - Inputs: - `logit_bias` (dict[str, float]; required): Required positional or keyword input. - Return annotation: `dict[int, float]` - Calls: logit_bias.items, int, float, HTTPException - Raises directly: HTTPException - Return expressions: coerced ## `vllm_mlx.server._attach_logit_bias_processor` - Kind: function - Signature: `def _attach_logit_bias_processor(chat_kwargs: dict, logit_bias: dict[str, float] | None)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L733-L744 - Implementation: Function `_attach_logit_bias_processor` calls `make_logits_processors`, `_coerce_logit_bias`, `chat_kwargs.get`, `list`; returns `None`. Function `_attach_logit_bias_processor` calls `make_logits_processors`, `_coerce_logit_bias`, `chat_kwargs.get`, `list`; returns `None`. - Inputs: - `chat_kwargs` (dict; required): Required positional or keyword input. - `logit_bias` (dict[str, float] | None; required): Required positional or keyword input. - Return annotation: `not annotated` - Calls: make_logits_processors, _coerce_logit_bias, chat_kwargs.get, list - Return expressions: None ## `vllm_mlx.server._prepare_chat_completion_invocation` - Kind: function - Signature: `def _prepare_chat_completion_invocation(engine: BaseEngine, request: ChatCompletionRequest, effective_max_tokens: int) -> PreparedChatInvocation` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L747-L855 - Implementation: Function `_prepare_chat_completion_invocation` calls `_prepare_chat_messages`, `_prepare_json_logits_processor`, `bool`, `_resolve_temperature`; returns `PreparedChatInvocation(messages=messages, chat_kwargs=chat_kwargs, response_format=response_format, json_logits_process…`. Precompute messages, kwargs, and decoding constraints for chat completions. - Inputs: - `engine` (BaseEngine; required): Required positional or keyword input. - `request` (ChatCompletionRequest; required): Required positional or keyword input. - `effective_max_tokens` (int; required): Required positional or keyword input. - Return annotation: `PreparedChatInvocation` - Calls: _prepare_chat_messages, _prepare_json_logits_processor, bool, _resolve_temperature, _resolve_top_p, _resolve_top_k, _resolve_min_p, _resolve_presence_penalty, _resolve_repetition_penalty, _attach_logit_bias_processor, getattr, _resolve_chat_template_kwargs, convert_tools_for_template, _apply_forced_tool_choice, get_parser_stop_tokens, _attach_response_format_logits_processor, chat_kwargs.get, _build_thinking_processor, list, PreparedChatInvocation - Return expressions: PreparedChatInvocation(messages=messages, chat_kwargs=chat_kwargs, response_format=response_format, json_logits_process… ## `vllm_mlx.server._prepare_anthropic_invocation` - Kind: function - Signature: `def _prepare_anthropic_invocation(engine: BaseEngine, openai_request: ChatCompletionRequest, effective_max_tokens: int) -> PreparedChatInvocation` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L858-L910 - Implementation: Function `_prepare_anthropic_invocation` calls `_prepare_chat_messages`, `_prepare_json_logits_processor`, `bool`, `_resolve_temperature`; returns `PreparedChatInvocation(messages=messages, chat_kwargs=chat_kwargs, response_format=response_format, json_logits_process…`. Precompute messages, kwargs, and decoding constraints for Anthropic API. - Inputs: - `engine` (BaseEngine; required): Required positional or keyword input. - `openai_request` (ChatCompletionRequest; required): Required positional or keyword input. - `effective_max_tokens` (int; required): Required positional or keyword input. - Return annotation: `PreparedChatInvocation` - Calls: _prepare_chat_messages, _prepare_json_logits_processor, bool, _resolve_temperature, _resolve_top_p, _resolve_top_k, _resolve_min_p, _resolve_presence_penalty, _resolve_repetition_penalty, _resolve_chat_template_kwargs, convert_tools_for_template, _apply_forced_tool_choice, _attach_response_format_logits_processor, PreparedChatInvocation - Return expressions: PreparedChatInvocation(messages=messages, chat_kwargs=chat_kwargs, response_format=response_format, json_logits_process… ## `vllm_mlx.server._thinking_disabled` - Kind: function - Signature: `def _thinking_disabled(request, chat_kwargs: dict | None=None) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L934-L950 - Implementation: Function `_thinking_disabled` calls `getattr`, `chat_kwargs.get`, `ctk.get`; has 2 explicit return paths. Return True iff thinking is explicitly disabled for this request. Checks both the request-level ``enable_thinking`` field and the resolved ``chat_template_kwargs`` (which may carry the server-wide default set via ``--default-chat-template-kwargs``). When thinking is disabled the prompt contains no injected ```` block, so the streaming reasoning parser must not default to implicit-thinking mode and swallow plain content into a ``thinking`` block. - Inputs: - `request` (not annotated; required): Required positional or keyword input. - `chat_kwargs` (dict | None; optional; default `None`): Optional positional or keyword input; defaults to `None`. - Return annotation: `bool` - Calls: getattr, chat_kwargs.get, ctk.get - Return expressions: True; False ## `vllm_mlx.server._strip_backslash_before_unicode` - Kind: function - Signature: `def _strip_backslash_before_unicode(obj: object) -> object` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L983-L997 - Implementation: Function `_strip_backslash_before_unicode` calls `isinstance`, `_strip_backslash_before_unicode`, `obj.items`, `re.sub`; has 4 explicit return paths. Remove spurious backslashes before non-ASCII chars in JSON string values. lm-format-enforcer's grammar allows ``\`` (valid JSON escape) followed by non-ASCII characters such as Korean syllables. The model therefore generates ``\빠\르\게`` — valid JSON whose decoded value contains literal backslashes. This helper strips those spurious backslashes so clients receive clean text. - Inputs: - `obj` (object; required): Required positional or keyword input. - Return annotation: `object` - Calls: isinstance, _strip_backslash_before_unicode, obj.items, re.sub - Return expressions: {k: _strip_backslash_before_unicode(v) for k, v in obj.items()}; [_strip_backslash_before_unicode(v) for v in obj]; re.sub('\\\\([^\\x00-\\x7F])', '\\1', obj); obj ## `vllm_mlx.server._sanitize_log_text` - Kind: function - Signature: `def _sanitize_log_text(value: object, limit: int | None=None) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L1000-L1022 - Implementation: Function `_sanitize_log_text` calls `str`, `escaped.append`, `ch.isprintable`, `ord`; has 2 explicit return paths. Escape control characters before logging untrusted text. - Inputs: - `value` (object; required): Required positional or keyword input. - `limit` (int | None; optional; default `None`): Optional positional or keyword input; defaults to `None`. - Return annotation: `str` - Calls: str, escaped.append, ch.isprintable, ord, ''.join, len - Return expressions: sanitized[:limit] + '...'; sanitized ## `vllm_mlx.server._log_and_raise_internal_error` - Kind: function - Signature: `def _log_and_raise_internal_error(log_prefix: str, exc: Exception, detail: str) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L1025-L1028 - Implementation: Function `_log_and_raise_internal_error` calls `logger.error`, `_sanitize_log_text`, `HTTPException`; can raise `HTTPException`. Log a sanitized exception string and raise a generic 500 response. - Inputs: - `log_prefix` (str; required): Required positional or keyword input. - `exc` (Exception; required): Required positional or keyword input. - `detail` (str; required): Required positional or keyword input. - Return annotation: `None` - Calls: logger.error, _sanitize_log_text, HTTPException - Raises directly: HTTPException ## `vllm_mlx.server._raise_engine_busy` - Kind: function - Signature: `def _raise_engine_busy(exc: EngineBusy) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L1031-L1039 - Implementation: Function `_raise_engine_busy` calls `HTTPException`, `str`; can raise `HTTPException`. Translate serialized-engine admission failures into retryable HTTP 503. - Inputs: - `exc` (EngineBusy; required): Required positional or keyword input. - Return annotation: `None` - Calls: HTTPException, str - Raises directly: HTTPException ## `vllm_mlx.server.RequestModelContext` - Kind: class - Signature: `class RequestModelContext` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L1043-L1056 - Implementation: Class `RequestModelContext` declares 1 direct member(s). Request-scoped engine/lease context. - Inputs: - `model_name` (str; required): Required constructor field. - `engine` (BaseEngine; required): Required constructor field. - `lease` (ModelLease | None; optional; default `None`): Optional constructor field; defaults to `None`. - Constructs: `vllm_mlx.server.RequestModelContext` - Decorators: dataclass ## `vllm_mlx.server.RequestModelContext.release` - Kind: method - Signature: `async def release(self) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L1050-L1056 - Implementation: Method `RequestModelContext.release` updates `self.lease`; calls `lease.release`; awaits asynchronous work. Release the registry lease once, if this context owns one. - Inputs: none - Return annotation: `None` - Calls: lease.release - State reads: self.lease - State writes: self.lease ## `vllm_mlx.server._list_available_model_names` - Kind: function - Signature: `def _list_available_model_names() -> list[str]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L1059-L1062 - Implementation: Function `_list_available_model_names` has 2 explicit return paths. Function `_list_available_model_names` has 2 explicit return paths. - Inputs: none - Return annotation: `list[str]` - Return expressions: _model_manager.registered_model_names; [_model_name] if _model_name else [] ## `vllm_mlx.server._response_model_name` - Kind: function - Signature: `def _response_model_name(request_model: str) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L1065-L1067 - Implementation: Function `_response_model_name` returns `_model_name or request_model`. Return the response model field for single-model or registry mode. - Inputs: - `request_model` (str; required): Required positional or keyword input. - Return annotation: `str` - Return expressions: _model_name or request_model ## `vllm_mlx.server._acquire_request_model` - Kind: function - Signature: `async def _acquire_request_model(request_model: str) -> RequestModelContext` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L1070-L1094 - Implementation: Function `_acquire_request_model` calls `_validate_model_name`, `get_engine`, `_detect_native_tool_support`, `_detect_harmony_rendering`; awaits asynchronous work; can raise `HTTPException`; has 2 explicit return paths. Acquire the model/engine that should serve this request. - Inputs: - `request_model` (str; required): Required positional or keyword input. - Return annotation: `RequestModelContext` - Calls: _validate_model_name, get_engine, _detect_native_tool_support, _detect_harmony_rendering, RequestModelContext, _model_manager.acquire, HTTPException, str - Raises directly: HTTPException - Return expressions: RequestModelContext(model_name=_model_name or request_model, engine=engine); RequestModelContext(model_name=request_model, engine=lease.engine, lease=lease) ## `vllm_mlx.server._stream_with_model_context` - Kind: function - Signature: `async def _stream_with_model_context(context: RequestModelContext, stream: AsyncIterator[str]) -> AsyncIterator[str]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L1097-L1106 - Implementation: Function `_stream_with_model_context` calls `context.release`; awaits asynchronous work; yields values incrementally. Ensure model leases survive for the full streaming response. - Inputs: - `context` (RequestModelContext; required): Required positional or keyword input. - `stream` (AsyncIterator[str]; required): Required positional or keyword input. - Return annotation: `AsyncIterator[str]` - Calls: context.release ## `vllm_mlx.server._build_tool_parser` - Kind: function - Signature: `def _build_tool_parser(engine: BaseEngine | None)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L1109-L1123 - Implementation: Function `_build_tool_parser` calls `type`, `ToolParserManager.get_tool_parser`, `_get_engine_tokenizer`, `parser_cls`; has 3 explicit return paths. Create a fresh tool parser instance for a single request/stream. - Inputs: - `engine` (BaseEngine | None; required): Required positional or keyword input. - Return annotation: `not annotated` - Calls: type, ToolParserManager.get_tool_parser, _get_engine_tokenizer, parser_cls - Return expressions: None; parser_cls(tokenizer); parser_cls() ## `vllm_mlx.server._build_reasoning_parser` - Kind: function - Signature: `def _build_reasoning_parser(engine: BaseEngine | None=None)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L1126-L1140 - Implementation: Function `_build_reasoning_parser` calls `getattr`, `get_reasoning_parser`, `parser_cls`, `type(_reasoning_parser)`; has 5 explicit return paths. Create a fresh reasoning parser instance for a single request/stream. - Inputs: - `engine` (BaseEngine | None; optional; default `None`): Optional positional or keyword input; defaults to `None`. - Return annotation: `not annotated` - Calls: getattr, get_reasoning_parser, parser_cls, type(_reasoning_parser), type - Return expressions: parser_cls(tokenizer); parser_cls(); None; type(_reasoning_parser)(tokenizer); type(_reasoning_parser)() ## `vllm_mlx.server._prepare_streaming_reasoning_parser` - Kind: function - Signature: `def _prepare_streaming_reasoning_parser(engine: BaseEngine, request: ChatCompletionRequest | ResponsesRequest | None, chat_kwargs: dict[str, object], *, allowed: bool=True)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L1143-L1156 - Implementation: Function `_prepare_streaming_reasoning_parser` calls `_thinking_disabled`, `_build_reasoning_parser`, `parser.reset_state`; has 2 explicit return paths. Build and reset request-local reasoning state when thinking is enabled. - Inputs: - `engine` (BaseEngine; required): Required positional or keyword input. - `request` (ChatCompletionRequest | ResponsesRequest | None; required): Required positional or keyword input. - `chat_kwargs` (dict[str, object]; required): Required positional or keyword input. - `allowed` (bool; optional; default `True`): Optional keyword-only input; defaults to `True`. - Return annotation: `not annotated` - Calls: _thinking_disabled, _build_reasoning_parser, parser.reset_state - Return expressions: None; parser ## `vllm_mlx.server._prepare_openai_stream_reasoning_state` - Kind: function - Signature: `def _prepare_openai_stream_reasoning_state(engine: BaseEngine, request: ChatCompletionRequest, chat_kwargs: dict[str, object]) -> tuple[object | None, bool]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L1159-L1171 - Implementation: Function `_prepare_openai_stream_reasoning_state` calls `_prepare_streaming_reasoning_parser`, `(engine.model_name or '').lower`, `_thinking_disabled`; returns `(parser, is_thinking_model)`. Return request-local reasoning state and the legacy Nemotron marker state. - Inputs: - `engine` (BaseEngine; required): Required positional or keyword input. - `request` (ChatCompletionRequest; required): Required positional or keyword input. - `chat_kwargs` (dict[str, object]; required): Required positional or keyword input. - Return annotation: `tuple[object | None, bool]` - Calls: _prepare_streaming_reasoning_parser, (engine.model_name or '').lower, _thinking_disabled - Return expressions: (parser, is_thinking_model) ## `vllm_mlx.server._request_tool_definitions` - Kind: function - Signature: `def _request_tool_definitions(request: ChatCompletionRequest) -> list | None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L1174-L1178 - Implementation: Function `_request_tool_definitions` calls `request.model_dump(include={'tools'}).get`, `request.model_dump`; has 2 explicit return paths. Return the request tool schema once for streaming argument coercion. - Inputs: - `request` (ChatCompletionRequest; required): Required positional or keyword input. - Return annotation: `list | None` - Calls: request.model_dump(include={'tools'}).get, request.model_dump - Return expressions: request.model_dump(include={'tools'}).get('tools'); None ## `vllm_mlx.server._streaming_json_fence_stripper` - Kind: function - Signature: `def _streaming_json_fence_stripper(request: ChatCompletionRequest) -> StreamingJsonFenceStripper | None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L1181-L1191 - Implementation: Function `_streaming_json_fence_stripper` calls `getattr`, `isinstance`, `response_format.get`, `StreamingJsonFenceStripper`; has 2 explicit return paths. Create a fence stripper only for JSON-constrained streaming responses. - Inputs: - `request` (ChatCompletionRequest; required): Required positional or keyword input. - Return annotation: `StreamingJsonFenceStripper | None` - Calls: getattr, isinstance, response_format.get, StreamingJsonFenceStripper - Return expressions: StreamingJsonFenceStripper(); None ## `vllm_mlx.server._get_idle_unload_event` - Kind: function - Signature: `def _get_idle_unload_event() -> asyncio.Event` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L1206-L1217 - Implementation: Function `_get_idle_unload_event` calls `asyncio.Event`, `_idle_unload_enabled.set`; returns `_idle_unload_enabled`. Return the idle-unload gate event, creating it on first use. The returned Event is bound to the running loop at creation time. Reset ``_idle_unload_enabled`` to ``None`` when tearing down the server or switching event loops (e.g. in test fixtures). - Inputs: none - Return annotation: `asyncio.Event` - Calls: asyncio.Event, _idle_unload_enabled.set - Return expressions: _idle_unload_enabled ## `vllm_mlx.server._invalidate_tool_parser_cache` - Kind: function - Signature: `def _invalidate_tool_parser_cache(reason: str | None=None) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L1220-L1229 - Implementation: Function `_invalidate_tool_parser_cache` calls `logger.debug`; returns `None`. Drop cached parser state when the serving tokenizer changes. - Inputs: - `reason` (str | None; optional; default `None`): Optional positional or keyword input; defaults to `None`. - Return annotation: `None` - Calls: logger.debug - Return expressions: None ## `vllm_mlx.server._load_prefix_cache_from_disk` - Kind: function - Signature: `def _load_prefix_cache_from_disk(engine: BaseEngine | None=None) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L1232-L1250 - Implementation: Function `_load_prefix_cache_from_disk` calls `_get_cache_dir`, `logger.info`, `target_engine.load_cache_from_disk`, `logger.warning`; returns `None`. Load prefix cache from disk during startup. - Inputs: - `engine` (BaseEngine | None; optional; default `None`): Optional positional or keyword input; defaults to `None`. - Return annotation: `None` - Calls: _get_cache_dir, logger.info, target_engine.load_cache_from_disk, logger.warning, _sanitize_log_text - Return expressions: None ## `vllm_mlx.server._save_prefix_cache_to_disk` - Kind: function - Signature: `def _save_prefix_cache_to_disk(engine: BaseEngine | None=None) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L1253-L1271 - Implementation: Function `_save_prefix_cache_to_disk` calls `_get_cache_dir`, `logger.info`, `target_engine.save_cache_to_disk`, `logger.warning`; returns `None`. Save prefix cache to disk during shutdown. - Inputs: - `engine` (BaseEngine | None; optional; default `None`): Optional positional or keyword input; defaults to `None`. - Return annotation: `None` - Calls: _get_cache_dir, logger.info, target_engine.save_cache_to_disk, logger.warning, _sanitize_log_text - Return expressions: None ## `vllm_mlx.server._get_cache_dir` - Kind: function - Signature: `def _get_cache_dir() -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L1274-L1290 - Implementation: Function `_get_cache_dir` calls `logger.info`, `type`, `str(model_name).replace('/', '--').replace`, `str(model_name).replace`; returns `cache_dir`. Get cache persistence directory based on actual model path. - Inputs: none - Return annotation: `str` - Calls: logger.info, type, str(model_name).replace('/', '--').replace, str(model_name).replace, str, os.path.join, os.path.expanduser - Return expressions: cache_dir ## `vllm_mlx.server._build_engine` - Kind: function - Signature: `def _build_engine(spec: ModelSpec) -> BaseEngine` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L1293-L1323 - Implementation: Function `_build_engine` calls `logger.info`, `BatchedEngine`, `getattr`, `SimpleEngine`; has 2 explicit return paths. Construct an engine instance from a model spec without starting it. - Inputs: - `spec` (ModelSpec; required): Required positional or keyword input. - Return annotation: `BaseEngine` - Calls: logger.info, BatchedEngine, getattr, SimpleEngine - Return expressions: BatchedEngine(model_name=spec.model_name, scheduler_config=spec.scheduler_config, stream_interval=spec.stream_interval,…; SimpleEngine(model_name=spec.model_name, force_mllm=spec.force_mllm, mtp=spec.mtp, prefill_step_size=spec.prefill_step_… ## `vllm_mlx.server._engine_factory` - Kind: function - Signature: `async def _engine_factory(spec: ModelSpec) -> BaseEngine` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L1326-L1328 - Implementation: Function `_engine_factory` calls `_build_engine`; returns `_build_engine(spec)`. Async engine factory used by the residency manager. - Inputs: - `spec` (ModelSpec; required): Required positional or keyword input. - Return annotation: `BaseEngine` - Calls: _build_engine - Return expressions: _build_engine(spec) ## `vllm_mlx.server._run_blocking_engine_cache_io` - Kind: function - Signature: `async def _run_blocking_engine_cache_io(io_fn, engine: BaseEngine) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L1331-L1350 - Implementation: Function `_run_blocking_engine_cache_io` calls `asyncio.create_task`, `asyncio.to_thread`, `asyncio.shield`, `suspend_cancellation`; awaits asynchronous work. Run blocking cache persistence off the event loop. If the caller is canceled while waiting, finish the in-flight thread before propagating cancellation so engine state cannot keep mutating in the background after lifecycle cleanup has started. - Inputs: - `io_fn` (not annotated; required): Required positional or keyword input. - `engine` (BaseEngine; required): Required positional or keyword input. - Return annotation: `None` - Calls: asyncio.create_task, asyncio.to_thread, asyncio.shield, suspend_cancellation, task.done ## `vllm_mlx.server._restore_engine_state` - Kind: function - Signature: `async def _restore_engine_state(spec: ModelSpec, engine: BaseEngine) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L1353-L1356 - Implementation: Function `_restore_engine_state` calls `hasattr`, `_run_blocking_engine_cache_io`; awaits asynchronous work. Restore engine-local state, such as prefix cache, after a cold load. - Inputs: - `spec` (ModelSpec; required): Required positional or keyword input. - `engine` (BaseEngine; required): Required positional or keyword input. - Return annotation: `None` - Calls: hasattr, _run_blocking_engine_cache_io ## `vllm_mlx.server._persist_engine_state` - Kind: function - Signature: `async def _persist_engine_state(spec: ModelSpec, engine: BaseEngine) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L1359-L1362 - Implementation: Function `_persist_engine_state` calls `hasattr`, `_run_blocking_engine_cache_io`; awaits asynchronous work. Persist engine-local state before an idle unload or shutdown unload. - Inputs: - `spec` (ModelSpec; required): Required positional or keyword input. - `engine` (BaseEngine; required): Required positional or keyword input. - Return annotation: `None` - Calls: hasattr, _run_blocking_engine_cache_io ## `vllm_mlx.server._activate_engine` - Kind: function - Signature: `def _activate_engine(engine: BaseEngine | None) -> BaseEngine | None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L1365-L1375 - Implementation: Function `_activate_engine` calls `_invalidate_tool_parser_cache`, `_detect_native_tool_support`, `_detect_harmony_rendering`; returns `_engine`. Set the global engine pointer and refresh parser-sensitive state. - Inputs: - `engine` (BaseEngine | None; required): Required positional or keyword input. - Return annotation: `BaseEngine | None` - Calls: _invalidate_tool_parser_cache, _detect_native_tool_support, _detect_harmony_rendering - Return expressions: _engine ## `vllm_mlx.server._sync_engine_from_residency` - Kind: function - Signature: `def _sync_engine_from_residency() -> BaseEngine | None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L1378-L1388 - Implementation: Function `_sync_engine_from_residency` calls `_activate_engine`, `_residency_manager.get_engine`; has 2 explicit return paths. Sync the global engine pointer from the residency manager state. Safety: all callers run on the single-threaded asyncio event loop and do not yield between reading the residency state and writing ``_engine``, so no additional locking is required. - Inputs: none - Return annotation: `BaseEngine | None` - Calls: _activate_engine, _residency_manager.get_engine - Return expressions: _engine; _activate_engine(_residency_manager.get_engine(_default_model_key)) ## `vllm_mlx.server._get_lifecycle_status` - Kind: function - Signature: `def _get_lifecycle_status() -> dict | None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L1391-L1395 - Implementation: Function `_get_lifecycle_status` calls `_residency_manager.get_status`; has 2 explicit return paths. Get lifecycle status for the default resident if lifecycle is enabled. - Inputs: none - Return annotation: `dict | None` - Calls: _residency_manager.get_status - Return expressions: None; _residency_manager.get_status(_default_model_key) ## `vllm_mlx.server._public_lifecycle_status` - Kind: function - Signature: `def _public_lifecycle_status(lifecycle: dict | None) -> dict | None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L1398-L1410 - Implementation: Function `_public_lifecycle_status` calls `dict`; has 2 explicit return paths. Return residency status safe for unauthenticated public endpoints. - Inputs: - `lifecycle` (dict | None; required): Required positional or keyword input. - Return annotation: `dict | None` - Calls: dict - Return expressions: None; public ## `vllm_mlx.server._lifecycle_loop` - Kind: function - Signature: `async def _lifecycle_loop() -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L1413-L1433 - Implementation: Function `_lifecycle_loop` calls `asyncio.sleep`, `_get_idle_unload_event().wait`, `_get_idle_unload_event`, `_residency_manager.unload_if_idle`; awaits asynchronous work. Background idle-unload loop for the default resident. - Inputs: none - Return annotation: `None` - Calls: asyncio.sleep, _get_idle_unload_event().wait, _get_idle_unload_event, _residency_manager.unload_if_idle, logger.exception, _sync_engine_from_residency, min ## `vllm_mlx.server._acquire_default_engine` - Kind: function - Signature: `async def _acquire_default_engine(*, count_activity: bool=True) -> BaseEngine` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L1436-L1451 - Implementation: Function `_acquire_default_engine` calls `get_engine`, `_residency_manager.acquire`, `_activate_engine`, `HTTPException`; awaits asynchronous work; can raise `HTTPException`; has 2 explicit return paths. Acquire the default engine, auto-loading via the residency manager if needed. - Inputs: - `count_activity` (bool; optional; default `True`): Optional keyword-only input; defaults to `True`. - Return annotation: `BaseEngine` - Calls: get_engine, _residency_manager.acquire, _activate_engine, HTTPException - Raises directly: HTTPException - Return expressions: get_engine(); activated_engine ## `vllm_mlx.server._release_default_engine` - Kind: function - Signature: `async def _release_default_engine(*, count_activity: bool=True) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L1454-L1463 - Implementation: Function `_release_default_engine` calls `_residency_manager.release`, `_sync_engine_from_residency`; awaits asynchronous work; returns `None`. Release the default engine after request processing. - Inputs: - `count_activity` (bool; optional; default `True`): Optional keyword-only input; defaults to `True`. - Return annotation: `None` - Calls: _residency_manager.release, _sync_engine_from_residency - Return expressions: None ## `vllm_mlx.server.lifespan` - Kind: function - Signature: `async def lifespan(app: FastAPI)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L1466-L1589 - Implementation: Function `lifespan` calls `_get_idle_unload_event().clear`, `_get_idle_unload_event`, `_residency_manager.ensure_loaded`, `_sync_engine_from_residency`; awaits asynchronous work; yields values incrementally; can raise `primary_exc`, `cleanup_exc`. FastAPI lifespan for startup/shutdown events. - Inputs: - `app` (FastAPI; required): Required positional or keyword input. - Return annotation: `not annotated` - Calls: _get_idle_unload_event().clear, _get_idle_unload_event, _residency_manager.ensure_loaded, _sync_engine_from_residency, hasattr, _engine.start, _model_manager.preload, _load_prefix_cache_from_disk, load_warmup_file, logger.info, len, warm_prefix_cache, result.get, logger.warning, _sanitize_log_text, asyncio.create_task, _lifecycle_loop, os.environ.get, init_mcp, _get_idle_unload_event().set, _save_prefix_cache_to_disk, _lifecycle_task.cancel, suppress, _mcp_manager.stop, _residency_manager.shutdown, _engine.stop, _model_manager.shutdown, logger.error, type - Raises directly: primary_exc, cleanup_exc ## `vllm_mlx.server._metrics_result_from_status` - Kind: function - Signature: `def _metrics_result_from_status(status_code: int) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L1602-L1610 - Implementation: Function `_metrics_result_from_status` has 4 explicit return paths. Map HTTP-ish status codes to low-cardinality inference results. - Inputs: - `status_code` (int; required): Required positional or keyword input. - Return annotation: `str` - Return expressions: 'client_closed'; 'timeout'; 'error'; 'success' ## `vllm_mlx.server._metrics_path_for_request` - Kind: function - Signature: `def _metrics_path_for_request(request: Request) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L1613-L1626 - Implementation: Function `_metrics_path_for_request` calls `request.scope.get`, `getattr`, `str`, `candidate.matches`; has 2 explicit return paths. Prefer route templates over raw URLs to keep metrics cardinality bounded. - Inputs: - `request` (Request; required): Required positional or keyword input. - Return annotation: `str` - Calls: request.scope.get, getattr, str, candidate.matches - Return expressions: str(path); '__unmatched__' ## `vllm_mlx.server._metrics_middleware` - Kind: function - Signature: `async def _metrics_middleware(request: Request, call_next)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L1630-L1659 - Implementation: Function `_metrics_middleware` calls `call_next`, `_metrics_path_for_request`, `time.perf_counter`, `_metrics.observe_http_start`; awaits asynchronous work; has 2 explicit return paths. Capture generic HTTP request metrics when enabled. - Inputs: - `request` (Request; required): Required positional or keyword input. - `call_next` (not annotated; required): Required positional or keyword input. - Return annotation: `not annotated` - Decorators: app.middleware('http') - Calls: call_next, _metrics_path_for_request, time.perf_counter, _metrics.observe_http_start, _metrics.observe_http_finish - Return expressions: await call_next(request); response ## `vllm_mlx.server.RateLimiter` - Kind: class - Signature: `class RateLimiter` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L1662-L1700 - Implementation: Class `RateLimiter` declares 2 direct member(s). Simple in-memory rate limiter using sliding window. - Inputs: - `requests_per_minute` (int; optional; default `60`): Optional positional or keyword input; defaults to `60`. - `enabled` (bool; optional; default `False`): Optional positional or keyword input; defaults to `False`. - Constructs: `vllm_mlx.server.RateLimiter` ## `vllm_mlx.server.RateLimiter.__init__` - Kind: method - Signature: `def __init__(self, requests_per_minute: int=60, enabled: bool=False)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L1665-L1670 - Implementation: Method `RateLimiter.__init__` updates `self.requests_per_minute`, `self.enabled`, `self.window_size`, `self._requests`; calls `defaultdict`, `threading.Lock`. Method `RateLimiter.__init__` updates `self.requests_per_minute`, `self.enabled`, `self.window_size`, `self._requests`; calls `defaultdict`, `threading.Lock`. - Inputs: - `requests_per_minute` (int; optional; default `60`): Optional positional or keyword input; defaults to `60`. - `enabled` (bool; optional; default `False`): Optional positional or keyword input; defaults to `False`. - Return annotation: `not annotated` - Calls: defaultdict, threading.Lock - State writes: self.requests_per_minute, self.enabled, self.window_size, self._requests, self._lock ## `vllm_mlx.server.RateLimiter.is_allowed` - Kind: method - Signature: `def is_allowed(self, client_id: str) -> tuple[bool, int]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L1672-L1700 - Implementation: Method `RateLimiter.is_allowed` calls `time.time`, `len`, `min`, `int`; has 2 explicit return paths. Check if request is allowed for client. Returns: (is_allowed, retry_after_seconds) - Inputs: - `client_id` (str; required): Required positional or keyword input. - Return annotation: `tuple[bool, int]` - Calls: time.time, len, min, int, max, self._requests[client_id].append - State reads: self.enabled, self.window_size, self._lock, self._requests, self.requests_per_minute - Return expressions: (True, 0); (False, max(1, retry_after)) ## `vllm_mlx.server.check_rate_limit` - Kind: function - Signature: `async def check_rate_limit(request: Request)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L1707-L1720 - Implementation: Function `check_rate_limit` calls `request.headers.get`, `_rate_limiter.is_allowed`, `HTTPException`, `str`; can raise `HTTPException`. Rate limiting dependency. - Inputs: - `request` (Request; required): Required positional or keyword input. - Return annotation: `not annotated` - Calls: request.headers.get, _rate_limiter.is_allowed, HTTPException, str - Raises directly: HTTPException ## `vllm_mlx.server.verify_api_key` - Kind: function - Signature: `async def verify_api_key(credentials: HTTPAuthorizationCredentials=Depends(security))` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L1723-L1742 - Implementation: Function `verify_api_key` calls `logger.warning`, `HTTPException`, `secrets.compare_digest`; can raise `HTTPException`; returns `True`. Verify API key if authentication is enabled. - Inputs: - `credentials` (HTTPAuthorizationCredentials; optional; default `Depends(security)`): Optional positional or keyword input; defaults to `Depends(security)`. - Return annotation: `not annotated` - Calls: logger.warning, HTTPException, secrets.compare_digest - Raises directly: HTTPException - Return expressions: True ## `vllm_mlx.server.get_engine` - Kind: function - Signature: `def get_engine() -> BaseEngine` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L1745-L1749 - Implementation: Function `get_engine` calls `HTTPException`; can raise `HTTPException`; returns `_engine`. Get the loaded engine, raising error if not loaded. - Inputs: none - Return annotation: `BaseEngine` - Calls: HTTPException - Raises directly: HTTPException - Return expressions: _engine ## `vllm_mlx.server._coerce_tool_arguments` - Kind: function - Signature: `def _coerce_tool_arguments(arguments_json: str, tool_name: str, tools: list[dict] | None) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L1752-L1796 - Implementation: Function `_coerce_tool_arguments` calls `isinstance`, `tool.get('function', {}).get`, `tool.get`, `tool['function'].get`; has 2 explicit return paths. Coerce tool call arguments to match the tool schema. If a schema field expects "string" but the model produced an object/array, JSON-stringify the value. This fixes a common LLM failure mode where models output raw JSON objects instead of JSON strings for file content, etc. - Inputs: - `arguments_json` (str; required): Required positional or keyword input. - `tool_name` (str; required): Required positional or keyword input. - `tools` (list[dict] | None; required): Required positional or keyword input. - Return annotation: `str` - Calls: isinstance, tool.get('function', {}).get, tool.get, tool['function'].get, json.loads, schema.get, arguments.items, properties[key].get, json.dumps - Return expressions: arguments_json; json.dumps(arguments, ensure_ascii=False) ## `vllm_mlx.server._validate_model_name` - Kind: function - Signature: `def _validate_model_name(request_model: str) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L1799-L1818 - Implementation: Function `_validate_model_name` calls `_model_manager.has_model`, `', '.join`, `_list_available_model_names`, `HTTPException`; can raise `HTTPException`; returns `None`. Validate that the request model name matches the served model. - Inputs: - `request_model` (str; required): Required positional or keyword input. - Return annotation: `None` - Calls: _model_manager.has_model, ', '.join, _list_available_model_names, HTTPException - Raises directly: HTTPException - Return expressions: None ## `vllm_mlx.server._get_engine_tokenizer` - Kind: function - Signature: `def _get_engine_tokenizer(engine: BaseEngine | None) -> object | None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L1821-L1828 - Implementation: Function `_get_engine_tokenizer` calls `getattr`; has 3 explicit return paths. Return tokenizer-like parser state from the active engine. - Inputs: - `engine` (BaseEngine | None; required): Required positional or keyword input. - Return annotation: `object | None` - Calls: getattr - Return expressions: None; tokenizer; getattr(engine, '_tokenizer', None) ## `vllm_mlx.server._get_or_init_tool_parser` - Kind: function - Signature: `def _get_or_init_tool_parser(engine: BaseEngine | None=None)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L1831-L1841 - Implementation: Function `_get_or_init_tool_parser` calls `ToolParserManager.get_tool_parser`, `_get_engine_tokenizer`, `parser_cls`, `logger.info`; returns `_tool_parser_instance`. Return the cached tool parser, initializing it from the given engine. - Inputs: - `engine` (BaseEngine | None; optional; default `None`): Optional positional or keyword input; defaults to `None`. - Return annotation: `not annotated` - Calls: ToolParserManager.get_tool_parser, _get_engine_tokenizer, parser_cls, logger.info - Return expressions: _tool_parser_instance ## `vllm_mlx.server._parse_tool_calls_with_parser` - Kind: function - Signature: `def _parse_tool_calls_with_parser(output_text: str, request: ChatCompletionRequest | None=None, engine: BaseEngine | None=None) -> tuple[str, list | None]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L1844-L1930 - Implementation: Function `_parse_tool_calls_with_parser` calls `request.model_dump`, `getattr`, `request_dict.get`, `parse_tool_calls`; has 6 explicit return paths. Parse tool calls from model output using the configured parser. If --enable-auto-tool-choice is set with --tool-call-parser, uses the selected parser. Otherwise falls back to the generic parse_tool_calls. Args: output_text: The model output text request: The original request (for context) engine: The request-local engine to use for parser initialization Returns: Tuple of (cleaned_text, tool_calls) - Inputs: - `output_text` (str; required): The model output text - `request` (ChatCompletionRequest | None; optional; default `None`): The original request (for context) - `engine` (BaseEngine | None; optional; default `None`): The request-local engine to use for parser initialization - Return annotation: `tuple[str, list | None]` - Calls: request.model_dump, getattr, request_dict.get, parse_tool_calls, _get_or_init_tool_parser, logger.warning, _sanitize_log_text, _tool_parser_instance.reset, _tool_parser_instance.extract_tool_calls, ToolCall, tc.get, uuid.uuid4, FunctionCall, _coerce_tool_arguments - Return expressions: (output_text, None); parse_tool_calls(output_text, request_dict); (result.content or '', tool_calls); (fallback_text, fallback_calls); (result.content, None); (fallback_text, None) ## `vllm_mlx.server._apply_response_format_or_raise` - Kind: function - Signature: `def _apply_response_format_or_raise(text: str, response_format: object, *, ensure_ascii: bool=False) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L1933-L1952 - Implementation: Function `_apply_response_format_or_raise` calls `apply_response_format_or_error`, `HTTPException`, `_strip_backslash_before_unicode`; can raise `HTTPException`; returns `_strip_backslash_before_unicode(text)`. Return validated JSON content or fail before returning a success response. - Inputs: - `text` (str; required): Required positional or keyword input. - `response_format` (object; required): Required positional or keyword input. - `ensure_ascii` (bool; optional; default `False`): Optional keyword-only input; defaults to `False`. - Return annotation: `str` - Calls: apply_response_format_or_error, HTTPException, _strip_backslash_before_unicode - Raises directly: HTTPException - Return expressions: _strip_backslash_before_unicode(text) ## `vllm_mlx.server._response_format_type` - Kind: function - Signature: `def _response_format_type(response_format: object | None) -> str | None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L1955-L1960 - Implementation: Function `_response_format_type` calls `isinstance`, `response_format.get`, `getattr`; has 3 explicit return paths. Function `_response_format_type` calls `isinstance`, `response_format.get`, `getattr`; has 3 explicit return paths. - Inputs: - `response_format` (object | None; required): Required positional or keyword input. - Return annotation: `str | None` - Calls: isinstance, response_format.get, getattr - Return expressions: None; response_format.get('type'); getattr(response_format, 'type', None) ## `vllm_mlx.server._promote_streaming_response_format_delta` - Kind: function - Signature: `def _promote_streaming_response_format_delta(content: str | None, reasoning: str | None, request: ChatCompletionRequest) -> tuple[str | None, str | None]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L1963-L1981 - Implementation: Function `_promote_streaming_response_format_delta` calls `_response_format_type`, `getattr`; has 2 explicit return paths. Keep response_format JSON on the streaming content channel. Some thinking parsers classify direct JSON output as reasoning when the model emits JSON without an explicit reasoning end marker. For response_format requests, that JSON is the final assistant content. - Inputs: - `content` (str | None; required): Required positional or keyword input. - `reasoning` (str | None; required): Required positional or keyword input. - `request` (ChatCompletionRequest; required): Required positional or keyword input. - Return annotation: `tuple[str | None, str | None]` - Calls: _response_format_type, getattr - Return expressions: (content, reasoning); (reasoning, None) ## `vllm_mlx.server._new_response_item_id` - Kind: function - Signature: `def _new_response_item_id(prefix: str) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L1984-L1986 - Implementation: Function `_new_response_item_id` calls `uuid.uuid4`; returns `f'{prefix}_{uuid.uuid4().hex}'`. Generate stable OpenAI-style item ids. - Inputs: - `prefix` (str; required): Required positional or keyword input. - Return annotation: `str` - Calls: uuid.uuid4 - Return expressions: f'{prefix}_{uuid.uuid4().hex}' ## `vllm_mlx.server._response_content_to_text` - Kind: function - Signature: `def _response_content_to_text(content) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L1989-L2006 - Implementation: Function `_response_content_to_text` calls `isinstance`, `part.get`, `getattr`, `text_parts.append`; has 3 explicit return paths. Normalize Responses API content items into plain text. - Inputs: - `content` (not annotated; required): Required positional or keyword input. - Return annotation: `str` - Calls: isinstance, part.get, getattr, text_parts.append, '\n'.join - Return expressions: ''; content; '\n'.join((part for part in text_parts if part)) ## `vllm_mlx.server._responses_tools_to_chat_tools` - Kind: function - Signature: `def _responses_tools_to_chat_tools(tools: list[ResponseFunctionTool | dict]) -> tuple[list[dict] | None, list[str]]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L2009-L2049 - Implementation: Function `_responses_tools_to_chat_tools` calls `isinstance`, `tool.get`, `unsupported.append`, `type`; has 2 explicit return paths. Convert supported Responses tools and report unsupported tool types. - Inputs: - `tools` (list[ResponseFunctionTool | dict]; required): Required positional or keyword input. - Return annotation: `tuple[list[dict] | None, list[str]]` - Calls: isinstance, tool.get, unsupported.append, type, supported.append - Return expressions: (None, []); (supported or None, unsupported) ## `vllm_mlx.server._responses_input_to_chat_messages` - Kind: function - Signature: `def _responses_input_to_chat_messages(request: ResponsesRequest) -> list[dict]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L2052-L2170 - Implementation: Function `_responses_input_to_chat_messages` calls `_responses_store.get`, `HTTPException`, `messages.extend`, `copy.deepcopy`; can raise `HTTPException`; returns `messages`. Convert Responses API input items into chat-completions-style messages. - Inputs: - `request` (ResponsesRequest; required): Required positional or keyword input. - Return annotation: `list[dict]` - Calls: _responses_store.get, HTTPException, messages.extend, copy.deepcopy, messages.append, isinstance, item.get, _response_content_to_text, _new_response_item_id, '\n'.join, p.get, logger.info, getattr, type - Raises directly: HTTPException - Return expressions: messages ## `vllm_mlx.server._responses_request_to_new_persisted_messages` - Kind: function - Signature: `def _responses_request_to_new_persisted_messages(request: ResponsesRequest) -> list[dict]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L2173-L2181 - Implementation: Function `_responses_request_to_new_persisted_messages` calls `request.model_copy`, `_responses_input_to_chat_messages`; returns `_responses_input_to_chat_messages(request_without_history)`. Persist only the current request's replayable input items. - Inputs: - `request` (ResponsesRequest; required): Required positional or keyword input. - Return annotation: `list[dict]` - Calls: request.model_copy, _responses_input_to_chat_messages - Return expressions: _responses_input_to_chat_messages(request_without_history) ## `vllm_mlx.server._responses_request_to_persisted_messages` - Kind: function - Signature: `def _responses_request_to_persisted_messages(request: ResponsesRequest) -> list[dict]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L2184-L2200 - Implementation: Function `_responses_request_to_persisted_messages` calls `_responses_store.get`, `HTTPException`, `messages.extend`, `copy.deepcopy`; can raise `HTTPException`; returns `messages`. Persist replayable history for chained previous_response_id requests. Responses `instructions` are intentionally not replayed across `previous_response_id`, but replayable message items are. - Inputs: - `request` (ResponsesRequest; required): Required positional or keyword input. - Return annotation: `list[dict]` - Calls: _responses_store.get, HTTPException, messages.extend, copy.deepcopy, _responses_request_to_new_persisted_messages - Raises directly: HTTPException - Return expressions: messages ## `vllm_mlx.server._responses_request_to_chat_request` - Kind: function - Signature: `def _responses_request_to_chat_request(request: ResponsesRequest) -> ChatCompletionRequest` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L2203-L2253 - Implementation: Function `_responses_request_to_chat_request` calls `HTTPException`, `logger.debug`, `_responses_tools_to_chat_tools`, `_responses_input_to_chat_messages`; can raise `HTTPException`; returns `ChatCompletionRequest(model=request.model, messages=[Message(**msg) for msg in messages], temperature=request.temperatu…`. Build a ChatCompletionRequest from a ResponsesRequest. - Inputs: - `request` (ResponsesRequest; required): Required positional or keyword input. - Return annotation: `ChatCompletionRequest` - Calls: HTTPException, logger.debug, _responses_tools_to_chat_tools, _responses_input_to_chat_messages, ', '.join, sorted, set, messages.insert, msg.get, '\n\n'.join, str(msg.get('content', '')).strip, str, ChatCompletionRequest, Message - Raises directly: HTTPException - Return expressions: ChatCompletionRequest(model=request.model, messages=[Message(**msg) for msg in messages], temperature=request.temperatu… ## `vllm_mlx.server._build_responses_output_items` - Kind: function - Signature: `def _build_responses_output_items(text: str | None, reasoning: str | None, tool_calls: list[ToolCall] | None) -> list[ResponseMessageItem | ResponseReasoningItem | ResponseFunctionCallItem]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L2256-L2293 - Implementation: Function `_build_responses_output_items` calls `output_items.append`, `ResponseReasoningItem`, `_new_response_item_id`, `ResponseReasoningTextPart`; returns `output_items`. Convert parsed assistant output into Responses API output items. - Inputs: - `text` (str | None; required): Required positional or keyword input. - `reasoning` (str | None; required): Required positional or keyword input. - `tool_calls` (list[ToolCall] | None; required): Required positional or keyword input. - Return annotation: `list[ResponseMessageItem | ResponseReasoningItem | ResponseFunctionCallItem]` - Calls: output_items.append, ResponseReasoningItem, _new_response_item_id, ResponseReasoningTextPart, ResponseMessageItem, ResponseTextContentPart, ResponseFunctionCallItem - Return expressions: output_items ## `vllm_mlx.server._response_output_items_to_chat_messages` - Kind: function - Signature: `def _response_output_items_to_chat_messages(output_items: list) -> list[dict]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L2296-L2325 - Implementation: Function `_response_output_items_to_chat_messages` calls `isinstance`, `assistant_text_parts.append`, `_response_content_to_text`, `assistant_tool_calls.append`; has 2 explicit return paths. Persist assistant output in chat-completions form for previous_response_id. - Inputs: - `output_items` (list; required): Required positional or keyword input. - Return annotation: `list[dict]` - Calls: isinstance, assistant_text_parts.append, _response_content_to_text, assistant_tool_calls.append, ''.join - Return expressions: []; [{'role': 'assistant', 'content': ''.join(assistant_text_parts), 'tool_calls': assistant_tool_calls or None}] ## `vllm_mlx.server._build_response_object` - Kind: function - Signature: `def _build_response_object(request: ResponsesRequest, output_items: list[ResponseMessageItem | ResponseReasoningItem | ResponseFunctionCallItem], prompt_tokens: int, completion_tokens: int, finish_reason: str | None, response_id: str | None=None) -> ResponseObject` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L2328-L2367 - Implementation: Function `_build_response_object` calls `ResponseObject`, `_new_response_item_id`, `_resolve_top_p`, `_resolve_temperature`; returns `response`. Build a full Responses API object. - Inputs: - `request` (ResponsesRequest; required): Required positional or keyword input. - `output_items` (list[ResponseMessageItem | ResponseReasoningItem | ResponseFunctionCallItem]; required): Required positional or keyword input. - `prompt_tokens` (int; required): Required positional or keyword input. - `completion_tokens` (int; required): Required positional or keyword input. - `finish_reason` (str | None; required): Required positional or keyword input. - `response_id` (str | None; optional; default `None`): Optional positional or keyword input; defaults to `None`. - Return annotation: `ResponseObject` - Calls: ResponseObject, _new_response_item_id, _resolve_top_p, _resolve_temperature, ResponsesUsage, ResponseIncompleteDetails - Return expressions: response ## `vllm_mlx.server._prepare_responses_request` - Kind: function - Signature: `def _prepare_responses_request(request: ResponsesRequest, *, validate_remote_media: bool=True) -> tuple[BaseEngine, ChatCompletionRequest, list[dict], dict]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L2370-L2414 - Implementation: Function `_prepare_responses_request` calls `_validate_model_name`, `get_engine`, `_responses_request_to_chat_request`, `logger.info`; returns `(engine, chat_request, messages, chat_kwargs)`. Prepare a Responses request for execution on the chat engine. - Inputs: - `request` (ResponsesRequest; required): Required positional or keyword input. - `validate_remote_media` (bool; optional; default `True`): Optional keyword-only input; defaults to `True`. - Return annotation: `tuple[BaseEngine, ChatCompletionRequest, list[dict], dict]` - Calls: _validate_model_name, get_engine, _responses_request_to_chat_request, logger.info, isinstance, len, _validate_remote_media_urls, extract_multimodal_content, canonicalize_system_messages, _resolve_temperature, _resolve_top_p, _resolve_chat_template_kwargs, convert_tools_for_template - Return expressions: (engine, chat_request, messages, chat_kwargs) ## `vllm_mlx.server._prepare_streaming_responses_request` - Kind: function - Signature: `def _prepare_streaming_responses_request(request: ResponsesRequest) -> tuple[BaseEngine, ChatCompletionRequest, list[dict], dict]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L2417-L2421 - Implementation: Function `_prepare_streaming_responses_request` calls `_prepare_responses_request`; returns `_prepare_responses_request(request, validate_remote_media=False)`. Prepare a streaming Responses request after eager URL validation. - Inputs: - `request` (ResponsesRequest; required): Required positional or keyword input. - Return annotation: `tuple[BaseEngine, ChatCompletionRequest, list[dict], dict]` - Calls: _prepare_responses_request - Return expressions: _prepare_responses_request(request, validate_remote_media=False) ## `vllm_mlx.server._run_responses_request` - Kind: function - Signature: `async def _run_responses_request(request: ResponsesRequest, raw_request: Request) -> tuple[ResponseObject | None, list[dict]]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L2424-L2477 - Implementation: Function `_run_responses_request` calls `_prepare_responses_request`, `_wait_with_disconnect`, `engine.chat`, `_parse_tool_calls_with_parser`; awaits asynchronous work; has 2 explicit return paths. Execute a Responses API request against the backend chat engine. - Inputs: - `request` (ResponsesRequest; required): Required positional or keyword input. - `raw_request` (Request; required): Required positional or keyword input. - Return annotation: `tuple[ResponseObject | None, list[dict]]` - Calls: _prepare_responses_request, _wait_with_disconnect, engine.chat, _parse_tool_calls_with_parser, _reasoning_parser.extract_reasoning, _build_responses_output_items, clean_output_text, _build_response_object, _responses_request_to_persisted_messages, persisted_messages.extend, _response_output_items_to_chat_messages, copy.deepcopy, response_object.model_copy, len, _responses_store.popitem - Return expressions: (None, []); (response_object, persisted_messages) ## `vllm_mlx.server._stream_responses_request` - Kind: function - Signature: `async def _stream_responses_request(request: ResponsesRequest) -> AsyncIterator[str]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L2480-L2868 - Implementation: Function `_stream_responses_request` calls `_prepare_streaming_responses_request`, `chat_request.model_dump`, `_new_response_item_id`, `_build_response_object`; yields values incrementally. Execute a Responses API request and stream SSE events incrementally. - Inputs: - `request` (ResponsesRequest; required): Required positional or keyword input. - Return annotation: `AsyncIterator[str]` - Calls: _prepare_streaming_responses_request, chat_request.model_dump, _new_response_item_id, _build_response_object, _responses_sse_event, ResponseCreatedEvent, ResponseInProgressEvent, _prepare_streaming_reasoning_parser, _get_streaming_tool_parser, engine.stream_chat, hasattr, reasoning_parser.extract_reasoning_streaming, _start_reasoning_item, ResponseReasoningTextDeltaEvent, _start_text_item, ResponseOutputTextDeltaEvent, SPECIAL_TOKENS_PATTERN.sub, _streaming_tool_markup_possible_after_delta, _extract_streaming_tool_delta, tool_result.get, _parse_tool_calls_with_parser, clean_output_text, ResponseReasoningItem, ResponseReasoningTextPart, ResponseReasoningTextDoneEvent, ResponseContentPartDoneEvent, ResponseOutputItemDoneEvent, ResponseMessageItem, ResponseTextContentPart, ResponseOutputTextDoneEvent, ResponseFunctionCallItem, function_call_items.append, ResponseOutputItemAddedEvent, item.model_copy, ResponseFunctionCallArgumentsDeltaEvent, output_items.append, output_items.extend, _responses_request_to_persisted_messages, persisted_messages.extend, _response_output_items_to_chat_messages, copy.deepcopy, response_object.model_copy, len, _responses_store.popitem, ResponseCompletedEvent ## `vllm_mlx.server._stream_responses_request._start_text_item` - Kind: nested function - Signature: `def _start_text_item() -> list[str]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L2525-L2561 - Implementation: Nested Function `_stream_responses_request._start_text_item` calls `_new_response_item_id`, `events.append`, `_responses_sse_event`, `ResponseOutputItemAddedEvent`; returns `events`. Nested Function `_stream_responses_request._start_text_item` calls `_new_response_item_id`, `events.append`, `_responses_sse_event`, `ResponseOutputItemAddedEvent`; returns `events`. - Inputs: none - Return annotation: `list[str]` - Calls: _new_response_item_id, events.append, _responses_sse_event, ResponseOutputItemAddedEvent, ResponseMessageItem, ResponseContentPartAddedEvent, ResponseTextContentPart - Return expressions: events ## `vllm_mlx.server._stream_responses_request._start_reasoning_item` - Kind: nested function - Signature: `def _start_reasoning_item() -> list[str]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L2563-L2598 - Implementation: Nested Function `_stream_responses_request._start_reasoning_item` calls `_new_response_item_id`, `events.append`, `_responses_sse_event`, `ResponseOutputItemAddedEvent`; returns `events`. Nested Function `_stream_responses_request._start_reasoning_item` calls `_new_response_item_id`, `events.append`, `_responses_sse_event`, `ResponseOutputItemAddedEvent`; returns `events`. - Inputs: none - Return annotation: `list[str]` - Calls: _new_response_item_id, events.append, _responses_sse_event, ResponseOutputItemAddedEvent, ResponseReasoningItem, ResponseContentPartAddedEvent, ResponseReasoningTextPart - Return expressions: events ## `vllm_mlx.server._responses_sse_event` - Kind: function - Signature: `def _responses_sse_event(event_type: str, payload: BaseModel | dict) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L2871-L2878 - Implementation: Function `_responses_sse_event` calls `isinstance`, `payload.model_dump_json`, `json.dumps`; returns `f'event: {event_type}\ndata: {data}\n\n'`. Encode a Responses API SSE event. - Inputs: - `event_type` (str; required): Required positional or keyword input. - `payload` (BaseModel | dict; required): Required positional or keyword input. - Return annotation: `str` - Calls: isinstance, payload.model_dump_json, json.dumps - Return expressions: f'event: {event_type}\ndata: {data}\n\n' ## `vllm_mlx.server._strip_harmony_analysis_blocks` - Kind: function - Signature: `def _strip_harmony_analysis_blocks(text: str) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L2888-L2892 - Implementation: Function `_strip_harmony_analysis_blocks` calls `_HARMONY_ANALYSIS_BLOCK_RE.sub`; returns `_HARMONY_ANALYSIS_BLOCK_RE.sub('', text)`. Remove harmony analysis-channel blocks (and their content) so reasoning text is never handed to the tool parser, while commentary/final text is preserved. - Inputs: - `text` (str; required): Required positional or keyword input. - Return annotation: `str` - Calls: _HARMONY_ANALYSIS_BLOCK_RE.sub - Return expressions: _HARMONY_ANALYSIS_BLOCK_RE.sub('', text) ## `vllm_mlx.server._extract_reasoning_and_tool_calls` - Kind: function - Signature: `def _extract_reasoning_and_tool_calls(output_text: str, request: ChatCompletionRequest | None=None, *, allow_reasoning: bool=True, engine: BaseEngine | None=None) -> tuple[str | None, str | None, list[ToolCall] | None]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L2895-L2951 - Implementation: Function `_extract_reasoning_and_tool_calls` calls `_reasoning_parser.extract_reasoning`, `getattr`, `_strip_harmony_analysis_blocks`, `_parse_tool_calls_with_parser`; returns `(reasoning_text, cleaned_text, tool_calls)`. Extract reasoning first, then parse tool calls from the cleaned content. Non-streaming responses can contain both a reasoning block and structured tool calls in the same final output. If tool parsing runs first and the response contains tools, the caller can no longer reliably recover the reasoning segment because the usual response path skips reasoning parsing once tool_calls is truthy. - Inputs: - `output_text` (str; required): Required positional or keyword input. - `request` (ChatCompletionRequest | None; optional; default `None`): Optional positional or keyword input; defaults to `None`. - `allow_reasoning` (bool; optional; default `True`): Optional keyword-only input; defaults to `True`. - `engine` (BaseEngine | None; optional; default `None`): Optional keyword-only input; defaults to `None`. - Return annotation: `tuple[str | None, str | None, list[ToolCall] | None]` - Calls: _reasoning_parser.extract_reasoning, getattr, _strip_harmony_analysis_blocks, _parse_tool_calls_with_parser, str - Return expressions: (reasoning_text, cleaned_text, tool_calls) ## `vllm_mlx.server._detect_native_tool_support` - Kind: function - Signature: `def _detect_native_tool_support() -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L2954-L2983 - Implementation: Function `_detect_native_tool_support` calls `ToolParserManager.get_tool_parser`, `parser_cls.supports_native_format`, `logger.error`, `ToolParserManager.list_registered`; has 2 explicit return paths. Detect if the active tool parser supports native tool format. Native format means role="tool" messages and tool_calls fields are preserved instead of being converted to text. Returns: True if native format should be preserved - Inputs: none - Return annotation: `bool` - Calls: ToolParserManager.get_tool_parser, parser_cls.supports_native_format, logger.error, ToolParserManager.list_registered, logger.warning, _sanitize_log_text - Return expressions: False; parser_cls.supports_native_format() ## `vllm_mlx.server._detect_harmony_rendering` - Kind: function - Signature: `def _detect_harmony_rendering() -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L2986-L3019 - Implementation: Function `_detect_harmony_rendering` calls `is_harmony_parser_name`, `logger.warning`; has 2 explicit return paths. Detect whether the harmony rendering path should handle prompt building. Returns True when ALL of: - ``--tool-call-parser`` is set to ``harmony`` or ``gpt-oss`` - ``--enable-auto-tool-choice`` is on - the optional ``openai-harmony`` Python package is importable The third condition keeps non-gpt-oss deployments free of an extra runtime dependency: if the package isn't installed, the engine falls back to the standard ``tokenizer.apply_chat_template`` path. The HarmonyToolParser's existing text-flatten behavior also stays in force in that fallback so the response side is unchanged. - Inputs: none - Return annotation: `bool` - Calls: is_harmony_parser_name, logger.warning - Return expressions: False; True ## `vllm_mlx.server._tool_choice_disabled` - Kind: function - Signature: `def _tool_choice_disabled(request: ChatCompletionRequest | None) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L3022-L3031 - Implementation: Function `_tool_choice_disabled` calls `getattr`, `request.model_dump`, `request_dict.get`; has 2 explicit return paths. Return True when tool_choice explicitly disables tool calling. - Inputs: - `request` (ChatCompletionRequest | None; required): Required positional or keyword input. - Return annotation: `bool` - Calls: getattr, request.model_dump, request_dict.get - Return expressions: False; tool_choice == 'none' ## `vllm_mlx.server._get_streaming_tool_parser` - Kind: function - Signature: `def _get_streaming_tool_parser(request: ChatCompletionRequest | None, engine: BaseEngine | None=None)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L3034-L3071 - Implementation: Function `_get_streaming_tool_parser` calls `_tool_choice_disabled`, `_get_engine_tokenizer`, `_build_tool_parser`, `logger.warning`; has 3 explicit return paths. Get a streaming-capable tool parser for this request. Uses the configured parser when auto tool choice is enabled, otherwise falls back to the generic auto parser so streaming still matches the generic non-streaming tool parsing behavior. - Inputs: - `request` (ChatCompletionRequest | None; required): Required positional or keyword input. - `engine` (BaseEngine | None; optional; default `None`): Optional positional or keyword input; defaults to `None`. - Return annotation: `not annotated` - Calls: _tool_choice_disabled, _get_engine_tokenizer, _build_tool_parser, logger.warning, _sanitize_log_text, getattr, ToolParserManager.get_tool_parser, parser_cls, parser.reset - Return expressions: None; _build_tool_parser(engine); parser ## `vllm_mlx.server._extract_streaming_tool_delta` - Kind: function - Signature: `def _extract_streaming_tool_delta(parser, previous_text: str, delta_text: str, request_context: dict) -> tuple[str, dict | None]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L3074-L3088 - Implementation: Function `_extract_streaming_tool_delta` calls `parser.extract_tool_calls_streaming`; returns `(current_text, result)`. Parse one request-local streaming delta and return new accumulated text. - Inputs: - `parser` (not annotated; required): Required positional or keyword input. - `previous_text` (str; required): Required positional or keyword input. - `delta_text` (str; required): Required positional or keyword input. - `request_context` (dict; required): Required positional or keyword input. - Return annotation: `tuple[str, dict | None]` - Calls: parser.extract_tool_calls_streaming - Return expressions: (current_text, result) ## `vllm_mlx.server._stream_request_metadata` - Kind: function - Signature: `def _stream_request_metadata(request: ChatCompletionRequest) -> tuple[dict, list | None, bool]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L3091-L3100 - Implementation: Function `_stream_request_metadata` calls `request.model_dump(include={'tools'}).get`, `request.model_dump`, `bool`; returns `({'tools': tools or []}, tools, include_usage)`. Function `_stream_request_metadata` calls `request.model_dump(include={'tools'}).get`, `request.model_dump`, `bool`; returns `({'tools': tools or []}, tools, include_usage)`. - Inputs: - `request` (ChatCompletionRequest; required): Required positional or keyword input. - Return annotation: `tuple[dict, list | None, bool]` - Calls: request.model_dump(include={'tools'}).get, request.model_dump, bool - Return expressions: ({'tools': tools or []}, tools, include_usage) ## `vllm_mlx.server._parse_streaming_tool_content` - Kind: function - Signature: `def _parse_streaming_tool_content(parser, accumulated_text: str, delta_text: str, request_context: dict) -> tuple[str, dict | None, bool]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L3103-L3116 - Implementation: Function `_parse_streaming_tool_content` calls `_extract_streaming_tool_delta`; returns `(accumulated_text, result, suppress)`. Function `_parse_streaming_tool_content` calls `_extract_streaming_tool_delta`; returns `(accumulated_text, result, suppress)`. - Inputs: - `parser` (not annotated; required): Required positional or keyword input. - `accumulated_text` (str; required): Required positional or keyword input. - `delta_text` (str; required): Required positional or keyword input. - `request_context` (dict; required): Required positional or keyword input. - Return annotation: `tuple[str, dict | None, bool]` - Calls: _extract_streaming_tool_delta - Return expressions: (accumulated_text, result, suppress) ## `vllm_mlx.server._streaming_tool_markup_possible` - Kind: function - Signature: `def _streaming_tool_markup_possible(text: str) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L3119-L3125 - Implementation: Function `_streaming_tool_markup_possible` calls `any`, `_STREAMING_BARE_BRACKET_MARKER.search`, `_STREAMING_BARE_BRACKET_PARTIAL.search`; returns `any((marker in text for marker in _STREAMING_TOOL_MARKERS)) or _STREAMING_BARE_BRACKET_MARKER.search(text) is not None …`. Heuristic marker check to avoid parser work on ordinary text chunks. - Inputs: - `text` (str; required): Required positional or keyword input. - Return annotation: `bool` - Calls: any, _STREAMING_BARE_BRACKET_MARKER.search, _STREAMING_BARE_BRACKET_PARTIAL.search - Return expressions: any((marker in text for marker in _STREAMING_TOOL_MARKERS)) or _STREAMING_BARE_BRACKET_MARKER.search(text) is not None … ## `vllm_mlx.server._streaming_tool_markup_possible_after_delta` - Kind: function - Signature: `def _streaming_tool_markup_possible_after_delta(accumulated_text: str, delta_text: str) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L3128-L3143 - Implementation: Function `_streaming_tool_markup_possible_after_delta` calls `_streaming_tool_markup_possible`; has 2 explicit return paths. Check only the boundary window needed to detect newly appearing tool markup. Streaming paths call this before any marker has been seen. Scanning the full accumulated text on every ordinary chunk is quadratic for long responses, so keep enough trailing context to catch markers split across chunk boundaries. Once markup is possible, callers switch to the parser path with the full accumulated text. - Inputs: - `accumulated_text` (str; required): Required positional or keyword input. - `delta_text` (str; required): Required positional or keyword input. - Return annotation: `bool` - Calls: _streaming_tool_markup_possible - Return expressions: False; _streaming_tool_markup_possible(check_text) ## `vllm_mlx.server.load_embedding_model` - Kind: function - Signature: `def load_embedding_model(model_name: str | None, *, lock: bool=False, reuse_existing: bool=True) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L3146-L3171 - Implementation: Function `load_embedding_model` calls `EmbeddingEngine`, `_embedding_engine.load`; returns `None`. Load or reuse the embedding model engine when configured. - Inputs: - `model_name` (str | None; required): Required positional or keyword input. - `lock` (bool; optional; default `False`): Optional keyword-only input; defaults to `False`. - `reuse_existing` (bool; optional; default `True`): Optional keyword-only input; defaults to `True`. - Return annotation: `None` - Calls: EmbeddingEngine, _embedding_engine.load - Return expressions: None ## `vllm_mlx.server.load_reranker_model` - Kind: function - Signature: `def load_reranker_model(model_name: str | None, *, lock: bool=False, reuse_existing: bool=True) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L3174-L3199 - Implementation: Function `load_reranker_model` calls `RerankEngine`, `_rerank_engine.load`; returns `None`. Load or reuse the reranker model engine when configured. - Inputs: - `model_name` (str | None; required): Required positional or keyword input. - `lock` (bool; optional; default `False`): Optional keyword-only input; defaults to `False`. - `reuse_existing` (bool; optional; default `True`): Optional keyword-only input; defaults to `True`. - Return annotation: `None` - Calls: RerankEngine, _rerank_engine.load - Return expressions: None ## `vllm_mlx.server.load_model` - Kind: function - Signature: `def load_model(model_name: str, use_batching: bool=False, scheduler_config=None, stream_interval: int=1, max_tokens: int=32768, max_request_tokens: int=32768, force_mllm: bool=False, gpu_memory_utilization: float=0.9, served_model_name: str | None=None, trust_remote_code: bool=False, mtp: bool=False, prefill_step_size: int=2048, specprefill_enabled: bool=False, specprefill_threshold: int=8192, specprefill_keep_pct: float=0.3, specprefill_backbone_pct: float=0.0, specprefill_draft_model: str=None, mllm_draft_model: str | None=None, mllm_draft_kind: str | None=None, mllm_draft_block_size: int | None=None, warm_prompts_path: str | None=None, auto_unload_idle_seconds: float=0.0, lazy_load_model: bool=False)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L3202-L3431 - Implementation: Function `load_model` calls `ValueError`, `RuntimeError`, `getattr`, `isinstance`; can raise `ValueError`, `RuntimeError`; returns `None`. Load a model (auto-detects MLLM vs LLM). Args: model_name: HuggingFace model name or local path use_batching: Use continuous batching (BatchedEngine) vs simple mode (SimpleEngine) scheduler_config: Scheduler config for batched mode stream_interval: Tokens to batch before streaming (batched mode only) max_tokens: Default max tokens for generation max_request_tokens: Maximum max_tokens accepted from API clients force_mllm: Force loading as MLLM even if not auto-detected trust_remote_code: Allow HuggingFace remote code execution during model/tokenizer loading mtp: Enable native MTP speculative decoding (SimpleEngine only) prefill_step_size: Chunk size for prompt prefill processing (default: 2048) specprefill_enabled: Enable SpecPrefill (SimpleEngine only) specprefill_threshold: Minimum suffix tokens to trigger SpecPrefill (default: 8192) specprefill_keep_pct: Fraction of tokens to keep (default: 0.3) specprefill_backbone_pct: Fraction of chunks reserved for evenly spaced coverage specprefill_draft_model: Path to small draft model for SpecPrefill scoring mllm_draft_model: Optional MLLM speculative draft/assistant model path. mllm_draft_kind: Optional mlx-vlm draft kind, for example "mtp". mllm_draft_block_size: Optional speculative block size passed to mlx-vlm. auto_unload_idle_seconds: Idle time before auto-unloading the main model. When non-zero, the main model is managed through lifecycle residency instead of being loaded immediately in this function. lazy_load_model: When lifecycle residency is enabled, defer the first resident load until the first request instead of FastAPI lifespan startup. - Inputs: - `model_name` (str; required): HuggingFace model name or local path - `use_batching` (bool; optional; default `False`): Use continuous batching (BatchedEngine) vs simple mode (SimpleEngine) - `scheduler_config` (not annotated; optional; default `None`): Scheduler config for batched mode - `stream_interval` (int; optional; default `1`): Tokens to batch before streaming (batched mode only) - `max_tokens` (int; optional; default `32768`): Default max tokens for generation - `max_request_tokens` (int; optional; default `32768`): Maximum max_tokens accepted from API clients - `force_mllm` (bool; optional; default `False`): Force loading as MLLM even if not auto-detected - `gpu_memory_utilization` (float; optional; default `0.9`): Optional positional or keyword input; defaults to `0.9`. - `served_model_name` (str | None; optional; default `None`): Optional positional or keyword input; defaults to `None`. - `trust_remote_code` (bool; optional; default `False`): Allow HuggingFace remote code execution during model/tokenizer loading - `mtp` (bool; optional; default `False`): Enable native MTP speculative decoding (SimpleEngine only) - `prefill_step_size` (int; optional; default `2048`): Chunk size for prompt prefill processing (default: 2048) - `specprefill_enabled` (bool; optional; default `False`): Enable SpecPrefill (SimpleEngine only) - `specprefill_threshold` (int; optional; default `8192`): Minimum suffix tokens to trigger SpecPrefill (default: 8192) - `specprefill_keep_pct` (float; optional; default `0.3`): Fraction of tokens to keep (default: 0.3) - `specprefill_backbone_pct` (float; optional; default `0.0`): Fraction of chunks reserved for evenly spaced coverage - `specprefill_draft_model` (str; optional; default `None`): Path to small draft model for SpecPrefill scoring - `mllm_draft_model` (str | None; optional; default `None`): Optional MLLM speculative draft/assistant model path. - `mllm_draft_kind` (str | None; optional; default `None`): Optional mlx-vlm draft kind, for example "mtp". - `mllm_draft_block_size` (int | None; optional; default `None`): Optional speculative block size passed to mlx-vlm. - `warm_prompts_path` (str | None; optional; default `None`): Optional positional or keyword input; defaults to `None`. - `auto_unload_idle_seconds` (float; optional; default `0.0`): Idle time before auto-unloading the main model. When non-zero, the main model is managed through lifecycle residency instead of being loaded immediately in this function. - `lazy_load_model` (bool; optional; default `False`): When lifecycle residency is enabled, defer the first resident load until the first request instead of FastAPI lifespan startup. - Return annotation: `not annotated` - Calls: ValueError, RuntimeError, getattr, isinstance, _residency_manager.get_engine, _residency_manager.get_status, existing_status.get, _invalidate_tool_parser_cache, logger.info, ModelSpec, ResidencyManager, _residency_manager.register_model, BatchedEngine, simple_engine_cls, asyncio.get_event_loop, asyncio.new_event_loop, asyncio.set_event_loop, loop.run_until_complete, _engine.start, suppress, loop.shutdown_default_executor, loop.close, previous_loop.is_closed, _detect_native_tool_support, _detect_harmony_rendering - Raises directly: ValueError, RuntimeError - Return expressions: None ## `vllm_mlx.server.load_model_registry` - Kind: function - Signature: `def load_model_registry(config_path: str, *, defaults: RegistryServeDefaults) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L3434-L3457 - Implementation: Function `load_model_registry` calls `load_registry_config`, `ModelManager`, `logger.info`, `len`. Load a registry-backed model manager from YAML configuration. - Inputs: - `config_path` (str; required): Required positional or keyword input. - `defaults` (RegistryServeDefaults; required): Required keyword-only input. - Return annotation: `None` - Calls: load_registry_config, ModelManager, logger.info, len, log_memory_budget_report, build_memory_budget_report ## `vllm_mlx.server.get_usage` - Kind: function - Signature: `def get_usage(output: GenerationOutput) -> Usage` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L3460-L3472 - Implementation: Function `get_usage` calls `hasattr`, `Usage`; returns `Usage(prompt_tokens=total_prompt_tokens, completion_tokens=total_completion_tokens, total_tokens=total_prompt_tokens + …`. Extract usage metrics from GenerationOutput. - Inputs: - `output` (GenerationOutput; required): Required positional or keyword input. - Return annotation: `Usage` - Calls: hasattr, Usage - Return expressions: Usage(prompt_tokens=total_prompt_tokens, completion_tokens=total_completion_tokens, total_tokens=total_prompt_tokens + … ## `vllm_mlx.server.metrics` - Kind: function - Signature: `async def metrics()` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L3476-L3485 - Implementation: Function `metrics` calls `HTTPException`, `_metrics.render_metrics`, `Response`; can raise `HTTPException`; returns `Response(content=payload, headers={'Content-Type': content_type})`. Prometheus scrape endpoint (disabled by default). - Inputs: none - Return annotation: `not annotated` - Decorators: app.get('/metrics') - Calls: HTTPException, _metrics.render_metrics, Response - Raises directly: HTTPException - Return expressions: Response(content=payload, headers={'Content-Type': content_type}) ## `vllm_mlx.server.health` - Kind: function - Signature: `async def health()` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L3489-L3544 - Implementation: Function `health` calls `sum`, `_mcp_manager.get_server_status`, `len`, `_mcp_manager.get_all_tools`; returns `payload`. Health check endpoint. - Inputs: none - Return annotation: `not annotated` - Decorators: app.get('/health') - Calls: sum, _mcp_manager.get_server_status, len, _mcp_manager.get_all_tools, _engine.get_stats, _get_lifecycle_status, lifecycle.get, _list_available_model_names, is_mllm_model, engine_stats.get, payload.update - Return expressions: payload ## `vllm_mlx.server.status` - Kind: function - Signature: `async def status()` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L3548-L3597 - Implementation: Function `status` calls `round`, `_model_manager.list_models`, `_public_lifecycle_status`, `_get_lifecycle_status`; has 3 explicit return paths. Real-time status with per-request details for debugging and monitoring. - Inputs: none - Return annotation: `not annotated` - Decorators: app.get('/v1/status', dependencies=[Depends(verify_api_key)]) - Calls: round, _model_manager.list_models, _public_lifecycle_status, _get_lifecycle_status, _engine.get_stats, stats.get, bg.get - Return expressions: {'status': 'running', 'model_manager': {'memory_budget_gb': round(_model_manager.memory_budget_bytes / 1024 ** 3, 2), '…; {'status': 'not_loaded', 'model': _model_name, 'residency': lifecycle, 'requests': []}; {'status': 'running' if stats.get('running') else 'stopped', 'model': _model_name, 'residency': lifecycle, 'uptime_s': … ## `vllm_mlx.server.cache_stats` - Kind: function - Signature: `async def cache_stats()` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L3601-L3627 - Implementation: Function `cache_stats` calls `hasattr`, `_engine.get_cache_stats`, `get_multimodal_kv_cache_stats`, `get_pixel_values_cache_stats`; has 2 explicit return paths. Get cache statistics for debugging and monitoring. - Inputs: none - Return annotation: `not annotated` - Decorators: app.get('/v1/cache/stats', dependencies=[Depends(verify_api_key)]) - Calls: hasattr, _engine.get_cache_stats, get_multimodal_kv_cache_stats, get_pixel_values_cache_stats, get_pil_cache_stats - Return expressions: {'engine_cache': engine_cache, 'multimodal_kv_cache': get_multimodal_kv_cache_stats(), 'pixel_values_cache': get_pixel_…; {'engine_cache': engine_cache, 'error': 'Cache stats not available (mlx_vlm not loaded)'} ## `vllm_mlx.server.clear_cache` - Kind: function - Signature: `async def clear_cache()` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L3631-L3659 - Implementation: Function `clear_cache` calls `hasattr`, `_engine.clear_runtime_caches`, `logger.warning`, `str`; has 2 explicit return paths. Clear all caches. - Inputs: none - Return annotation: `not annotated` - Decorators: app.delete('/v1/cache', dependencies=[Depends(verify_api_key)]) - Calls: hasattr, _engine.clear_runtime_caches, logger.warning, str, clear_multimodal_kv_cache, clear_pixel_values_cache - Return expressions: {'status': 'cleared', 'engine_cache': cleared_engine, 'caches': ['multimodal_kv', 'pixel_values', 'pil_image']}; {'status': 'cleared', 'engine_cache': cleared_engine, 'error': 'Cache clear not available (mlx_vlm not loaded)'} ## `vllm_mlx.server.clear_prefix_cache` - Kind: function - Signature: `async def clear_prefix_cache()` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L3663-L3713 - Implementation: Function `clear_prefix_cache` calls `hasattr`, `_engine.clear_prefix_cache`, `logger.warning`, `_sanitize_log_text`; has 2 explicit return paths. Clear the text prefix cache used for KV reuse in continuous batching. If the server was started with ``--warm-prompts``, the warm-up is re-run in the background after clear so the next real request still hits the cache. Response returns immediately without waiting for the re-warm to finish. - Inputs: none - Return annotation: `not annotated` - Decorators: app.delete('/v1/cache/prefix', dependencies=[Depends(verify_api_key)]) - Calls: hasattr, _engine.clear_prefix_cache, logger.warning, _sanitize_log_text, asyncio.create_task, _rewarm - Return expressions: {'status': 'no_engine'}; {'status': status, 'rewarm_scheduled': rewarm_scheduled} ## `vllm_mlx.server.clear_prefix_cache._rewarm` - Kind: nested function - Signature: `async def _rewarm()` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L3688-L3707 - Implementation: Nested Function `clear_prefix_cache._rewarm` calls `load_warmup_file`, `warm_prefix_cache`, `logger.info`, `logger.warning`; awaits asynchronous work. Nested Function `clear_prefix_cache._rewarm` calls `load_warmup_file`, `warm_prefix_cache`, `logger.info`, `logger.warning`; awaits asynchronous work. - Inputs: none - Return annotation: `not annotated` - Calls: load_warmup_file, warm_prefix_cache, logger.info, logger.warning, _sanitize_log_text ## `vllm_mlx.server.cancel_request` - Kind: function - Signature: `async def cancel_request(request_id: str)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L3720-L3747 - Implementation: Function `cancel_request` calls `get_engine`, `engine.abort_request`, `logger.exception`, `HTTPException`; awaits asynchronous work; can raise `HTTPException`; returns `{'object': 'request.cancel', 'id': request_id, 'cancelled': True, 'model': _model_name}`. Cancel an active or queued request. The request_id is the chatcmpl-xxx ID from the first SSE streaming chunk. - Inputs: - `request_id` (str; required): Required positional or keyword input. - Return annotation: `not annotated` - Decorators: app.post('/v1/requests/{request_id}/cancel', dependencies=[Depends(verify_api_key), Depends(check_rate_limit)]) - Calls: get_engine, engine.abort_request, logger.exception, HTTPException, logger.info - Raises directly: HTTPException - Return expressions: {'object': 'request.cancel', 'id': request_id, 'cancelled': True, 'model': _model_name} ## `vllm_mlx.server.delete_request` - Kind: function - Signature: `async def delete_request(request_id: str)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L3754-L3756 - Implementation: Function `delete_request` calls `cancel_request`; awaits asynchronous work; returns `await cancel_request(request_id)`. OpenAI-style alias for cancelling an active or queued request. - Inputs: - `request_id` (str; required): Required positional or keyword input. - Return annotation: `not annotated` - Decorators: app.delete('/v1/requests/{request_id}', dependencies=[Depends(verify_api_key), Depends(check_rate_limit)]) - Calls: cancel_request - Return expressions: await cancel_request(request_id) ## `vllm_mlx.server.list_models` - Kind: function - Signature: `async def list_models() -> ModelsResponse` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L3760-L3775 - Implementation: Function `list_models` calls `models.extend`, `ModelInfo`, `_model_manager.list_models`, `models.append`; returns `ModelsResponse(data=models)`. List available models. - Inputs: none - Return annotation: `ModelsResponse` - Decorators: app.get('/v1/models', dependencies=[Depends(verify_api_key)]) - Calls: models.extend, ModelInfo, _model_manager.list_models, models.append, ModelsResponse - Return expressions: ModelsResponse(data=models) ## `vllm_mlx.server.create_embeddings` - Kind: function - Signature: `async def create_embeddings(request: EmbeddingRequest) -> EmbeddingResponse` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L3787-L3908 - Implementation: Function `create_embeddings` calls `_metrics.track_inference`, `resolve_embedding_model_name`, `load_embedding_model`, `isinstance`; can raise `HTTPException`; returns `response`. Create embeddings for the given input text(s). OpenAI-compatible embeddings API supporting single or batch inputs. Single text: ```json { "model": "mlx-community/all-MiniLM-L6-v2-4bit", "input": "The quick brown fox jumps over the lazy dog" } ``` Batch of texts: ```json { "model": "mlx-community/embeddinggemma-300m-6bit", "input": [ "I love machine learning", "Deep learning is fascinating", "Neural networks are powerful" ] } ``` Response: ```json { "object": "list", "data": [ {"object": "embedding", "index": 0, "embedding": [0.023, -0.982, ...]}, {"object": "embedding", "index": 1, "embedding": [0.112, -0.543, ...]}, {"object": "embedding", "index": 2, "embedding": [0.876, 0.221, ...]} ], "model": "mlx-community/embeddinggemma-300m-6bit", "usage": {"prompt_tokens": 24, "total_tokens": 24} } ``` Supported request-time models: - mlx-community/all-MiniLM-L6-v2-4bit (fast, compact) - mlx-community/embeddinggemma-300m-6bit (high quality) - mlx-community/bge-large-en-v1.5-4bit (best for English) - mlx-community/multilingual-e5-small-mlx - mlx-community/multilingual-e5-large-mlx - mlx-community/bert-base-uncased-mlx - mlx-community/ModernBERT-base-mlx Other embedding models must be pinned explicitly with --embedding-model at server startup. - Inputs: - `request` (EmbeddingRequest; required): Required positional or keyword input. - Return annotation: `EmbeddingResponse` - Decorators: app.post('/v1/embeddings', dependencies=[Depends(verify_api_key), Depends(check_rate_limit)]) - Calls: _metrics.track_inference, resolve_embedding_model_name, load_embedding_model, isinstance, HTTPException, time.perf_counter, _embedding_engine.count_tokens, _embedding_engine.embed, logger.info, len, EmbeddingData, enumerate, EmbeddingResponse, EmbeddingUsage, tracker.finish, _metrics_result_from_status, _log_and_raise_internal_error - Raises directly: HTTPException - Return expressions: response ## `vllm_mlx.server.rerank_documents` - Kind: function - Signature: `async def rerank_documents(request: RerankRequest) -> RerankResponse` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L3920-L4038 - Implementation: Function `rerank_documents` calls `HTTPException`, `request.query.strip`, `len`, `isinstance`; awaits asynchronous work; can raise `HTTPException`; returns `RerankResponse(model=model_name, results=results, usage=RerankUsage(total_tokens=total_tokens))`. Rerank documents against a query using a cross-encoder model. Jina/Cohere-compatible reranking API. Accepts a query and a list of documents (strings or {text: ...} objects), returns results sorted by relevance score descending. - Inputs: - `request` (RerankRequest; required): Required positional or keyword input. - Return annotation: `RerankResponse` - Decorators: app.post('/v1/rerank', dependencies=[Depends(verify_api_key), Depends(check_rate_limit)]) - Calls: HTTPException, request.query.strip, len, isinstance, doc_texts.append, original_docs.append, type, time.perf_counter, asyncio.to_thread, logger.info, enumerate, RerankResult, results.append, results.sort, RerankResponse, RerankUsage, logger.error, str - Raises directly: HTTPException - Return expressions: RerankResponse(model=model_name, results=results, usage=RerankUsage(total_tokens=total_tokens)) ## `vllm_mlx.server.list_mcp_tools` - Kind: function - Signature: `async def list_mcp_tools() -> MCPToolsResponse` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L4047-L4063 - Implementation: Function `list_mcp_tools` calls `MCPToolsResponse`, `_mcp_manager.get_all_tools`, `tools.append`, `MCPToolInfo`; has 2 explicit return paths. List all available MCP tools. - Inputs: none - Return annotation: `MCPToolsResponse` - Decorators: app.get('/v1/mcp/tools', dependencies=[Depends(verify_api_key)]) - Calls: MCPToolsResponse, _mcp_manager.get_all_tools, tools.append, MCPToolInfo, len - Return expressions: MCPToolsResponse(tools=[], count=0); MCPToolsResponse(tools=tools, count=len(tools)) ## `vllm_mlx.server.list_mcp_servers` - Kind: function - Signature: `async def list_mcp_servers() -> MCPServersResponse` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L4067-L4084 - Implementation: Function `list_mcp_servers` calls `MCPServersResponse`, `_mcp_manager.get_server_status`, `servers.append`, `MCPServerInfo`; has 2 explicit return paths. Get status of all MCP servers. - Inputs: none - Return annotation: `MCPServersResponse` - Decorators: app.get('/v1/mcp/servers', dependencies=[Depends(verify_api_key)]) - Calls: MCPServersResponse, _mcp_manager.get_server_status, servers.append, MCPServerInfo - Return expressions: MCPServersResponse(servers=[]); MCPServersResponse(servers=servers) ## `vllm_mlx.server.execute_mcp_tool` - Kind: function - Signature: `async def execute_mcp_tool(request: MCPExecuteRequest) -> MCPExecuteResponse` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L4088-L4117 - Implementation: Function `execute_mcp_tool` calls `HTTPException`, `ToolExecutor`, `uuid.uuid4`, `_mcp_executor.execute_tool_calls`; awaits asynchronous work; can raise `HTTPException`; returns `MCPExecuteResponse(tool_name=result.tool_name, content=result.content, is_error=result.is_error, error_message=result.e…`. Execute an MCP tool. - Inputs: - `request` (MCPExecuteRequest; required): Required positional or keyword input. - Return annotation: `MCPExecuteResponse` - Decorators: app.post('/v1/mcp/execute', dependencies=[Depends(verify_api_key)]) - Calls: HTTPException, ToolExecutor, uuid.uuid4, _mcp_executor.execute_tool_calls, MCPExecuteResponse - Raises directly: HTTPException - Return expressions: MCPExecuteResponse(tool_name=result.tool_name, content=result.content, is_error=result.is_error, error_message=result.e… ## `vllm_mlx.server.create_transcription` - Kind: function - Signature: `async def create_transcription(file: UploadFile, model: str='whisper-large-v3', language: str | None=None, response_format: str='json')` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L4130-L4196 - Implementation: Function `create_transcription` calls `_metrics.track_inference`, `resolve_stt_model_name`, `STTEngine`, `_stt_engine.load`; awaits asynchronous work; can raise `HTTPException`; has 2 explicit return paths. Transcribe audio to text (OpenAI Whisper API compatible). Supported models: - whisper-large-v3 (multilingual, best quality) - whisper-large-v3-turbo (faster) - whisper-medium, whisper-small (lighter) - parakeet-tdt-0.6b-v2 (English, fastest) - Inputs: - `file` (UploadFile; required): Required positional or keyword input. - `model` (str; optional; default `'whisper-large-v3'`): Optional positional or keyword input; defaults to `'whisper-large-v3'`. - `language` (str | None; optional; default `None`): Optional positional or keyword input; defaults to `None`. - `response_format` (str; optional; default `'json'`): Optional positional or keyword input; defaults to `'json'`. - Return annotation: `not annotated` - Decorators: app.post('/v1/audio/transcriptions', dependencies=[Depends(verify_api_key)]) - Calls: _metrics.track_inference, resolve_stt_model_name, STTEngine, _stt_engine.load, save_upload_with_limit, _stt_engine.transcribe, os.unlink, tracker.finish, HTTPException, _metrics_result_from_status, _log_and_raise_internal_error - Raises directly: HTTPException - Return expressions: result.text; {'text': result.text, 'language': result.language, 'duration': result.duration} ## `vllm_mlx.server.create_speech` - Kind: function - Signature: `async def create_speech(model: str='kokoro', input: str='', voice: str='af_heart', speed: float=1.0, response_format: str='wav')` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L4200-L4254 - Implementation: Function `create_speech` calls `_metrics.track_inference`, `resolve_tts_model_name`, `validate_tts_input_length`, `TTSEngine`; can raise `HTTPException`; returns `Response(content=audio_bytes, media_type=content_type)`. Generate speech from text (OpenAI TTS API compatible). Supported models: - kokoro (fast, lightweight) - chatterbox (multilingual, expressive) - vibevoice (realtime) - voxcpm (Chinese/English) - Inputs: - `model` (str; optional; default `'kokoro'`): Optional positional or keyword input; defaults to `'kokoro'`. - `input` (str; optional; default `''`): Optional positional or keyword input; defaults to `''`. - `voice` (str; optional; default `'af_heart'`): Optional positional or keyword input; defaults to `'af_heart'`. - `speed` (float; optional; default `1.0`): Optional positional or keyword input; defaults to `1.0`. - `response_format` (str; optional; default `'wav'`): Optional positional or keyword input; defaults to `'wav'`. - Return annotation: `not annotated` - Decorators: app.post('/v1/audio/speech', dependencies=[Depends(verify_api_key)]) - Calls: _metrics.track_inference, resolve_tts_model_name, validate_tts_input_length, TTSEngine, _tts_engine.load, _tts_engine.generate, _tts_engine.to_bytes, tracker.finish, Response, HTTPException, _metrics_result_from_status, _log_and_raise_internal_error - Raises directly: HTTPException - Return expressions: Response(content=audio_bytes, media_type=content_type) ## `vllm_mlx.server.list_voices` - Kind: function - Signature: `async def list_voices(model: str='kokoro')` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L4258-L4267 - Implementation: Function `list_voices` calls `model.lower`; has 3 explicit return paths. List available voices for a TTS model. - Inputs: - `model` (str; optional; default `'kokoro'`): Optional positional or keyword input; defaults to `'kokoro'`. - Return annotation: `not annotated` - Decorators: app.get('/v1/audio/voices', dependencies=[Depends(verify_api_key)]) - Calls: model.lower - Return expressions: {'voices': KOKORO_VOICES}; {'voices': CHATTERBOX_VOICES}; {'voices': ['default']} ## `vllm_mlx.server._ensure_sse_terminal` - Kind: function - Signature: `async def _ensure_sse_terminal(generator: AsyncIterator[str], terminal_frame: str) -> AsyncIterator[str]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L4275-L4296 - Implementation: Function `_ensure_sse_terminal` calls `logger.error`; yields values incrementally. Guarantee that *terminal_frame* is emitted exactly once at the end of *generator*, even if the generator raises mid-stream. If the inner generator already yields the terminal frame on its happy path, the wrapper detects it and avoids double-emission. If the generator raises before reaching the terminal, the wrapper emits it in the ``finally`` block. - Inputs: - `generator` (AsyncIterator[str]; required): Required positional or keyword input. - `terminal_frame` (str; required): Required positional or keyword input. - Return annotation: `AsyncIterator[str]` - Calls: logger.error ## `vllm_mlx.server._find_uvicorn_cycle` - Kind: function - Signature: `def _find_uvicorn_cycle(obj, depth=0, visited=None)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L4299-L4346 - Implementation: Function `_find_uvicorn_cycle` calls `set`, `id`, `visited.add`, `hasattr`; has 3 explicit return paths. Walk through middleware wrappers to find uvicorn's RequestResponseCycle. This relies on uvicorn's internal ``RequestResponseCycle.disconnected`` attribute and Starlette's middleware closure layout. Tested against uvicorn 0.34-0.40 and starlette 0.44-0.46. If either changes the internal layout, this function returns None and disconnect detection silently falls back to timeout-only behaviour. - Inputs: - `obj` (not annotated; required): Required positional or keyword input. - `depth` (not annotated; optional; default `0`): Optional positional or keyword input; defaults to `0`. - `visited` (not annotated; optional; default `None`): Optional positional or keyword input; defaults to `None`. - Return annotation: `not annotated` - Calls: set, id, visited.add, hasattr, isinstance, getattr, _find_uvicorn_cycle - Return expressions: None; obj; result ## `vllm_mlx.server._is_client_disconnected` - Kind: function - Signature: `def _is_client_disconnected(raw_request: Request) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L4349-L4374 - Implementation: Function `_is_client_disconnected` calls `getattr`, `_find_uvicorn_cycle`; has 2 explicit return paths. Reliable client disconnect check. Starlette's ``is_disconnected()`` uses an immediately-cancelled ``anyio.CancelScope`` which prevents the ASGI ``receive()`` from executing — so it always returns False for non-streaming requests. This function bypasses Starlette and reads uvicorn's internal ``disconnected`` flag directly from the ``RequestResponseCycle``, walking through any middleware wrappers via closures. - Inputs: - `raw_request` (Request; required): Required positional or keyword input. - Return annotation: `bool` - Calls: getattr, _find_uvicorn_cycle - Return expressions: True; False ## `vllm_mlx.server._disconnect_guard` - Kind: function - Signature: `async def _disconnect_guard(generator: AsyncIterator[str], raw_request: Request, poll_interval: float=0.5, heartbeat_interval: float=5.0, cleanup=None, timeout: float | None=None) -> AsyncIterator[str]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L4377-L4546 - Implementation: Function `_disconnect_guard` calls `_time.monotonic`, `logger.info`, `generator.__aiter__`, `asyncio.create_task`; awaits asynchronous work; yields values incrementally. Wrap streaming generator to abort on client disconnect. Uses asyncio racing: each __anext__() on the inner generator is raced against a disconnect poller. When neither completes within ``heartbeat_interval`` seconds, an SSE comment is yielded as a heartbeat. This forces an ASGI write which triggers broken-pipe detection — without heartbeats, ``is_disconnected()`` stays False during long prefill because no data is written to the socket. If *timeout* is set, it bounds inactivity from the inner generator, not the total stream lifetime. A stream that continues to produce chunks must be allowed to complete even when generation takes longer than the configured interval. Heartbeats force ASGI writes to detect a disconnected client, but do not count as generator progress. On disconnect, the cancellation propagates to stream_outputs() finally-block → abort_request() → abort_prefill(). - Inputs: - `generator` (AsyncIterator[str]; required): Required positional or keyword input. - `raw_request` (Request; required): Required positional or keyword input. - `poll_interval` (float; optional; default `0.5`): Optional positional or keyword input; defaults to `0.5`. - `heartbeat_interval` (float; optional; default `5.0`): Optional positional or keyword input; defaults to `5.0`. - `cleanup` (not annotated; optional; default `None`): Optional positional or keyword input; defaults to `None`. - `timeout` (float | None; optional; default `None`): Optional positional or keyword input; defaults to `None`. - Return annotation: `AsyncIterator[str]` - Calls: _time.monotonic, logger.info, generator.__aiter__, asyncio.create_task, _wait_disconnect, logger.warning, _elapsed, anext_task.done, anext_task.cancel, asyncio.ensure_future, aiter.__anext__, asyncio.wait, min, anext_task.result, logger.error, type, disconnect_task.done, disconnect_task.cancel, _deferred_generator_close, cleanup, asyncio.iscoroutine ## `vllm_mlx.server._disconnect_guard._elapsed` - Kind: nested function - Signature: `def _elapsed()` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L4407-L4408 - Implementation: Nested Function `_disconnect_guard._elapsed` calls `_time.monotonic`; returns `f'{_time.monotonic() - _t0:.1f}s'`. Nested Function `_disconnect_guard._elapsed` calls `_time.monotonic`; returns `f'{_time.monotonic() - _t0:.1f}s'`. - Inputs: none - Return annotation: `not annotated` - Calls: _time.monotonic - Return expressions: f'{_time.monotonic() - _t0:.1f}s' ## `vllm_mlx.server._disconnect_guard._wait_disconnect` - Kind: nested function - Signature: `async def _wait_disconnect()` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L4417-L4429 - Implementation: Nested Function `_disconnect_guard._wait_disconnect` calls `asyncio.sleep`, `_is_client_disconnected`, `logger.info`, `_elapsed`; awaits asynchronous work; returns `None`. Nested Function `_disconnect_guard._wait_disconnect` calls `asyncio.sleep`, `_is_client_disconnected`, `logger.info`, `_elapsed`; awaits asynchronous work; returns `None`. - Inputs: none - Return annotation: `not annotated` - Calls: asyncio.sleep, _is_client_disconnected, logger.info, _elapsed - Return expressions: None ## `vllm_mlx.server._disconnect_guard._deferred_generator_close` - Kind: nested function - Signature: `async def _deferred_generator_close()` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L4528-L4536 - Implementation: Nested Function `_disconnect_guard._deferred_generator_close` calls `asyncio.sleep`, `_gen_to_close.aclose`, `logger.debug`, `type`; awaits asynchronous work. Nested Function `_disconnect_guard._deferred_generator_close` calls `asyncio.sleep`, `_gen_to_close.aclose`, `logger.debug`, `type`; awaits asynchronous work. - Inputs: none - Return annotation: `not annotated` - Calls: asyncio.sleep, _gen_to_close.aclose, logger.debug, type ## `vllm_mlx.server._wait_with_disconnect` - Kind: function - Signature: `async def _wait_with_disconnect(coro, raw_request: Request, timeout: float, poll_interval: float=0.5, timeout_detail_seconds: float | None=None, cleanup_result=None)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L4549-L4638 - Implementation: Function `_wait_with_disconnect` calls `_time.monotonic`, `asyncio.ensure_future`, `asyncio.create_task`, `_wait_disconnect`; awaits asynchronous work; can raise `HTTPException`; has 2 explicit return paths. Run a coroutine with both timeout and client disconnect detection. For non-streaming requests where _disconnect_guard() can't be used. Races the coroutine against a disconnect poller, same pattern as _disconnect_guard but for awaitable (non-generator) coroutines. - Inputs: - `coro` (not annotated; required): Required positional or keyword input. - `raw_request` (Request; required): Required positional or keyword input. - `timeout` (float; required): Required positional or keyword input. - `poll_interval` (float; optional; default `0.5`): Optional positional or keyword input; defaults to `0.5`. - `timeout_detail_seconds` (float | None; optional; default `None`): Optional positional or keyword input; defaults to `None`. - `cleanup_result` (not annotated; optional; default `None`): Optional positional or keyword input; defaults to `None`. - Return annotation: `not annotated` - Calls: _time.monotonic, asyncio.ensure_future, asyncio.create_task, _wait_disconnect, asyncio.wait, task.cancel, HTTPException, logger.info, task.result, cleanup_result, asyncio.iscoroutine, disconnect_task.done, disconnect_task.cancel, task.done - Raises directly: HTTPException - Return expressions: None; task.result() ## `vllm_mlx.server._wait_with_disconnect._wait_disconnect` - Kind: nested function - Signature: `async def _wait_disconnect()` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L4569-L4581 - Implementation: Nested Function `_wait_with_disconnect._wait_disconnect` calls `asyncio.sleep`, `_is_client_disconnected`, `logger.info`, `_time.monotonic`; awaits asynchronous work; returns `None`. Nested Function `_wait_with_disconnect._wait_disconnect` calls `asyncio.sleep`, `_is_client_disconnected`, `logger.info`, `_time.monotonic`; awaits asynchronous work; returns `None`. - Inputs: none - Return annotation: `not annotated` - Calls: asyncio.sleep, _is_client_disconnected, logger.info, _time.monotonic - Return expressions: None ## `vllm_mlx.server._start_request_budget` - Kind: function - Signature: `def _start_request_budget(timeout: float | None) -> tuple[float, float]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L4641-L4644 - Implementation: Function `_start_request_budget` calls `time.monotonic`; returns `(total_timeout, time.monotonic() + total_timeout)`. Return the total timeout and absolute deadline for a request. - Inputs: - `timeout` (float | None; required): Required positional or keyword input. - Return annotation: `tuple[float, float]` - Calls: time.monotonic - Return expressions: (total_timeout, time.monotonic() + total_timeout) ## `vllm_mlx.server._remaining_request_timeout` - Kind: function - Signature: `def _remaining_request_timeout(total_timeout: float, deadline: float) -> float` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L4647-L4655 - Implementation: Function `_remaining_request_timeout` calls `time.monotonic`, `HTTPException`; can raise `HTTPException`; returns `remaining`. Compute remaining request budget or raise the standard timeout error. - Inputs: - `total_timeout` (float; required): Required positional or keyword input. - `deadline` (float; required): Required positional or keyword input. - Return annotation: `float` - Calls: time.monotonic, HTTPException - Raises directly: HTTPException - Return expressions: remaining ## `vllm_mlx.server._acquire_default_engine_for_request` - Kind: function - Signature: `async def _acquire_default_engine_for_request(raw_request: Request, *, total_timeout: float, deadline: float, count_activity: bool=True, model: str | None=None) -> BaseEngine | None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L4661-L4719 - Implementation: Function `_acquire_default_engine_for_request` calls `_registry_acquire`, `_wait_with_disconnect`, `_remaining_request_timeout`, `_acquire_default_engine`; awaits asynchronous work; has 4 explicit return paths. Acquire the engine for a request, using the model registry when active. When ``_model_manager`` is set (registry mode), acquires the engine for the requested *model* via ``_acquire_request_model``. The resulting ``RequestModelContext`` is stashed in ``_active_request_contexts`` keyed by ``id(raw_request)`` so that the matching ``_release_default_engine`` call can release the lease. In single-model mode the behaviour is unchanged. - Inputs: - `raw_request` (Request; required): Required positional or keyword input. - `total_timeout` (float; required): Required keyword-only input. - `deadline` (float; required): Required keyword-only input. - `count_activity` (bool; optional; default `True`): Optional keyword-only input; defaults to `True`. - `model` (str | None; optional; default `None`): Optional keyword-only input; defaults to `None`. - Return annotation: `BaseEngine | None` - Calls: _registry_acquire, _wait_with_disconnect, _remaining_request_timeout, _acquire_default_engine - Return expressions: await _registry_acquire(); await _wait_with_disconnect(_registry_acquire(), raw_request, timeout=_remaining_request_timeout(total_timeout, deadlin…; await acquire_coro; await _wait_with_disconnect(acquire_coro, raw_request, timeout=_remaining_request_timeout(total_timeout, deadline), tim… ## `vllm_mlx.server._acquire_default_engine_for_request._registry_acquire` - Kind: nested function - Signature: `async def _registry_acquire()` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L4681-L4685 - Implementation: Nested Function `_acquire_default_engine_for_request._registry_acquire` calls `_acquire_request_model`, `id`; awaits asynchronous work; returns `ctx.engine`. Nested Function `_acquire_default_engine_for_request._registry_acquire` calls `_acquire_request_model`, `id`; awaits asynchronous work; returns `ctx.engine`. - Inputs: none - Return annotation: `not annotated` - Calls: _acquire_request_model, id - Return expressions: ctx.engine ## `vllm_mlx.server._acquire_default_engine_for_request._registry_cleanup` - Kind: nested function - Signature: `async def _registry_cleanup(_result)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L4687-L4690 - Implementation: Nested Function `_acquire_default_engine_for_request._registry_cleanup` calls `_active_request_contexts.pop`, `id`, `ctx.release`; awaits asynchronous work. Nested Function `_acquire_default_engine_for_request._registry_cleanup` calls `_active_request_contexts.pop`, `id`, `ctx.release`; awaits asynchronous work. - Inputs: - `_result` (not annotated; required): Required positional or keyword input. - Return annotation: `not annotated` - Calls: _active_request_contexts.pop, id, ctx.release ## `vllm_mlx.server._release_engine_for_request` - Kind: function - Signature: `async def _release_engine_for_request(raw_request: Request | None, *, count_activity: bool=True) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L4722-L4737 - Implementation: Function `_release_engine_for_request` calls `_active_request_contexts.pop`, `id`, `ctx.release`, `_release_default_engine`; awaits asynchronous work; returns `None`. Release the engine acquired for this request. In registry mode, releases the model lease stashed by ``_acquire_default_engine_for_request``. In single-model mode, falls through to the default release path. ``count_activity`` must match the flag used on the matching acquire so idle-unload accounting stays correct. - Inputs: - `raw_request` (Request | None; required): Required positional or keyword input. - `count_activity` (bool; optional; default `True`): Optional keyword-only input; defaults to `True`. - Return annotation: `None` - Calls: _active_request_contexts.pop, id, ctx.release, _release_default_engine - Return expressions: None ## `vllm_mlx.server._make_release_cleanup` - Kind: function - Signature: `def _make_release_cleanup(raw_request: Request | None)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L4740-L4752 - Implementation: Function `_make_release_cleanup` has 2 explicit return paths. Return a cleanup callable suitable for ``_disconnect_guard``. - Inputs: - `raw_request` (Request | None; required): Required positional or keyword input. - Return annotation: `not annotated` - Return expressions: _cleanup; _release_default_engine ## `vllm_mlx.server._make_release_cleanup._cleanup` - Kind: nested function - Signature: `async def _cleanup()` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L4744-L4749 - Implementation: Nested Function `_make_release_cleanup._cleanup` calls `_active_request_contexts.pop`, `id`, `ctx.release`, `_release_default_engine`; awaits asynchronous work. Nested Function `_make_release_cleanup._cleanup` calls `_active_request_contexts.pop`, `id`, `ctx.release`, `_release_default_engine`; awaits asynchronous work. - Inputs: none - Return annotation: `not annotated` - Calls: _active_request_contexts.pop, id, ctx.release, _release_default_engine ## `vllm_mlx.server.create_completion` - Kind: function - Signature: `async def create_completion(request: CompletionRequest, raw_request: Request)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L4763-L4909 - Implementation: Function `create_completion` calls `_validate_model_name`, `_resolve_request_max_tokens`, `_metrics.track_inference`, `isinstance`; awaits asynchronous work; has 3 explicit return paths. Create a text completion. - Inputs: - `request` (CompletionRequest; required): Required positional or keyword input. - `raw_request` (Request; required): Required positional or keyword input. - Return annotation: `not annotated` - Decorators: app.post('/v1/completions', dependencies=[Depends(verify_api_key), Depends(check_rate_limit)]) - Calls: _validate_model_name, _resolve_request_max_tokens, _metrics.track_inference, isinstance, _start_request_budget, sum, len, logger.info, _sanitize_log_text, _acquire_default_engine_for_request, Response, StreamingResponse, _disconnect_guard, _ensure_sse_terminal, stream_completion, _make_release_cleanup, time.perf_counter, enumerate, _resolve_temperature, _resolve_top_p, _resolve_top_k, _resolve_min_p, _resolve_presence_penalty, _resolve_repetition_penalty, getattr, engine.generate, _wait_with_disconnect, _remaining_request_timeout, tracker.finish, _metrics_result_from_status, _raise_engine_busy, choices.append, CompletionChoice, hasattr, CompletionResponse, _response_model_name, Usage, _release_engine_for_request - Return expressions: Response(status_code=499); response; CompletionResponse(model=_response_model_name(request.model), choices=choices, usage=Usage(prompt_tokens=total_prompt_t… ## `vllm_mlx.server.create_chat_completion` - Kind: function - Signature: `async def create_chat_completion(request: ChatCompletionRequest, raw_request: Request)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L4916-L5114 - Implementation: Function `create_chat_completion` calls `_validate_model_name`, `_resolve_request_max_tokens`, `_metrics.track_inference`, `_start_request_budget`; awaits asynchronous work; has 3 explicit return paths. Create a chat completion (supports multimodal content for VLM models). OpenAI-compatible multimodal format for images: ```json messages=[{ "role": "user", "content": [ {"type": "text", "text": "What's in this image?"}, {"type": "image_url", "image_url": {"url": "https://..."}} ] }] ``` Video support: ```json messages=[{ "role": "user", "content": [ {"type": "text", "text": "What happens in this video?"}, {"type": "video_url", "video_url": {"url": "https://example.com/video.mp4"}} ] }] ``` Structured output (JSON mode): ```json response_format={"type": "json_object"} ``` Structured output (JSON Schema): ```json response_format={ "type": "json_schema", "json_schema": { "name": "my_schema", "schema": {"type": "object", "properties": {...}} } } ``` - Inputs: - `request` (ChatCompletionRequest; required): Required positional or keyword input. - `raw_request` (Request; required): Required positional or keyword input. - Return annotation: `not annotated` - Decorators: app.post('/v1/chat/completions', dependencies=[Depends(verify_api_key), Depends(check_rate_limit)]) - Calls: _validate_model_name, _resolve_request_max_tokens, _metrics.track_inference, _start_request_budget, len, isinstance, str, logger.info, _sanitize_log_text, _acquire_default_engine_for_request, Response, _prepare_chat_completion_invocation, tracker.finish, _raise_remote_media_http_error, StreamingResponse, _disconnect_guard, _ensure_sse_terminal, stream_chat_completion, _make_release_cleanup, time.perf_counter, _wait_with_disconnect, engine.chat, _remaining_request_timeout, _metrics_result_from_status, _raise_engine_busy, _extract_reasoning_and_tool_calls, _thinking_disabled, _apply_response_format_or_raise, logger.error, logger.warning, ChatCompletionResponse, _response_model_name, ChatCompletionChoice, AssistantMessage, clean_output_text, Usage, _generation_metadata, _release_engine_for_request - Return expressions: Response(status_code=499); response; ChatCompletionResponse(model=_response_model_name(request.model), choices=[ChatCompletionChoice(message=AssistantMessag… ## `vllm_mlx.server._normalize_messages` - Kind: function - Signature: `def _normalize_messages(messages: list[dict]) -> list[dict]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L5117-L5172 - Implementation: Function `_normalize_messages` calls `messages[0].copy`, `_ROLE_MAP.get`, `isinstance`, `prev.get`; has 2 explicit return paths. Normalize message roles and merge consecutive same-role messages. 1. Maps non-standard roles to standard ones (e.g. ``developer`` -> ``system``). 2. Merges consecutive same-role messages to satisfy chat template constraints (Qwen 3.5, Llama, etc. require alternating roles). Only merges when both messages have string content. Messages with list content (multimodal) are left as-is to preserve image/video attachments. Args: messages: List of message dicts with 'role' and 'content' keys. Returns: New list with normalized roles and consecutive same-role messages merged. - Inputs: - `messages` (list[dict]; required): List of message dicts with 'role' and 'content' keys. - Return annotation: `list[dict]` - Calls: messages[0].copy, _ROLE_MAP.get, isinstance, prev.get, msg.get, logger.debug, len, msg.copy, merged.append, sum, parts.append, logger.info, ', '.join - Return expressions: messages; merged ## `vllm_mlx.server._get_engine_tokenizer` - Kind: function - Signature: `def _get_engine_tokenizer(engine) -> object | None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L5175-L5187 - Implementation: Function `_get_engine_tokenizer` calls `getattr`; has 2 explicit return paths. Return the tokenizer backing ``engine``, if exposed. Different engine classes store the tokenizer under different attributes. We try the common ones and return ``None`` if nothing matches, so that optional features like constrained decoding can degrade gracefully. - Inputs: - `engine` (not annotated; required): Required positional or keyword input. - Return annotation: `object | None` - Calls: getattr - Return expressions: tok; None ## `vllm_mlx.server.create_response` - Kind: function - Signature: `async def create_response(request: ResponsesRequest, raw_request: Request)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L5194-L5214 - Implementation: Function `create_response` calls `_responses_request_to_chat_request`, `_validate_remote_media_urls`, `StreamingResponse`, `_disconnect_guard`; awaits asynchronous work; has 3 explicit return paths. Create a Responses API response. - Inputs: - `request` (ResponsesRequest; required): Required positional or keyword input. - `raw_request` (Request; required): Required positional or keyword input. - Return annotation: `not annotated` - Decorators: app.post('/v1/responses', dependencies=[Depends(verify_api_key), Depends(check_rate_limit)]) - Calls: _responses_request_to_chat_request, _validate_remote_media_urls, StreamingResponse, _disconnect_guard, _stream_responses_request, _run_responses_request, _raise_remote_media_http_error, Response - Return expressions: StreamingResponse(_disconnect_guard(_stream_responses_request(request), raw_request), media_type='text/event-stream'); Response(status_code=499); response_object ## `vllm_mlx.server._get_forced_tool_name` - Kind: function - Signature: `def _get_forced_tool_name(tool_choice) -> str | None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L5217-L5230 - Implementation: Function `_get_forced_tool_name` calls `isinstance`, `tool_choice.get`, `func.get`; has 2 explicit return paths. Extract forced tool name from tool_choice, if any. Returns the function name when tool_choice is a dict like {"type": "function", "function": {"name": "X"}}, or None otherwise. - Inputs: - `tool_choice` (not annotated; required): Required positional or keyword input. - Return annotation: `str | None` - Calls: isinstance, tool_choice.get, func.get - Return expressions: None; func.get('name') ## `vllm_mlx.server._apply_forced_tool_choice` - Kind: function - Signature: `def _apply_forced_tool_choice(tool_choice, tools, messages, chat_kwargs=None)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L5233-L5279 - Implementation: Function `_apply_forced_tool_choice` calls `_get_forced_tool_name`, `_tool_name`, `ValueError`, `_inject_json_instruction`; can raise `ValueError`; returns `(tools, messages)`. Apply forced tool_choice by filtering tools and injecting instructions. Handles: - tool_choice={"type":"function","function":{"name":"X"}} -> filter + instruct - tool_choice="required" -> instruct model to call at least one tool Args: tool_choice: The tool_choice value from the request tools: List of converted tools for the template messages: The message list (will be copied if modified) chat_kwargs: Optional dict to modify (e.g. disable thinking) Returns: Tuple of (tools, messages) - potentially filtered/modified - Inputs: - `tool_choice` (not annotated; required): The tool_choice value from the request - `tools` (not annotated; required): List of converted tools for the template - `messages` (not annotated; required): The message list (will be copied if modified) - `chat_kwargs` (not annotated; optional; default `None`): Optional dict to modify (e.g. disable thinking) - Return annotation: `not annotated` - Calls: _get_forced_tool_name, _tool_name, ValueError, _inject_json_instruction - Raises directly: ValueError - Return expressions: (tools, messages) ## `vllm_mlx.server._tool_name` - Kind: function - Signature: `def _tool_name(tool: dict) -> str | None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L5282-L5287 - Implementation: Function `_tool_name` calls `tool.get`, `isinstance`, `func.get`; has 2 explicit return paths. Extract function name from a tool definition dict. - Inputs: - `tool` (dict; required): Required positional or keyword input. - Return annotation: `str | None` - Calls: tool.get, isinstance, func.get - Return expressions: func.get('name'); None ## `vllm_mlx.server._inject_json_instruction` - Kind: function - Signature: `def _inject_json_instruction(messages: list, instruction: str) -> list` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L5290-L5319 - Implementation: Function `_inject_json_instruction` calls `list`, `enumerate`, `isinstance`, `msg.get`; returns `messages`. Inject JSON instruction into messages. If a system message exists, append to it. Otherwise, prepend a new system message. - Inputs: - `messages` (list; required): Required positional or keyword input. - `instruction` (str; required): Required positional or keyword input. - Return annotation: `list` - Calls: list, enumerate, isinstance, msg.get, getattr, messages.insert - Return expressions: messages ## `vllm_mlx.server._convert_anthropic_stop_reason` - Kind: function - Signature: `def _convert_anthropic_stop_reason(openai_reason: str | None) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L5327-L5335 - Implementation: Function `_convert_anthropic_stop_reason` calls `mapping.get`; returns `mapping.get(openai_reason or '', 'end_turn')`. Convert OpenAI finish_reason to Anthropic stop_reason. - Inputs: - `openai_reason` (str | None; required): Required positional or keyword input. - Return annotation: `str` - Calls: mapping.get - Return expressions: mapping.get(openai_reason or '', 'end_turn') ## `vllm_mlx.server._prepare_anthropic_endpoint_invocation` - Kind: function - Signature: `def _prepare_anthropic_endpoint_invocation(engine: BaseEngine, openai_request: ChatCompletionRequest, effective_max_tokens: int) -> PreparedChatInvocation` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L5338-L5351 - Implementation: Function `_prepare_anthropic_endpoint_invocation` calls `_prepare_anthropic_invocation`, `_raise_remote_media_http_error`; returns `_prepare_anthropic_invocation(engine, openai_request, effective_max_tokens)`. Prepare Anthropic invocation and convert URL-safety errors to 400s. - Inputs: - `engine` (BaseEngine; required): Required positional or keyword input. - `openai_request` (ChatCompletionRequest; required): Required positional or keyword input. - `effective_max_tokens` (int; required): Required positional or keyword input. - Return annotation: `PreparedChatInvocation` - Calls: _prepare_anthropic_invocation, _raise_remote_media_http_error - Return expressions: _prepare_anthropic_invocation(engine, openai_request, effective_max_tokens) ## `vllm_mlx.server.create_anthropic_message` - Kind: function - Signature: `async def create_anthropic_message(request: Request)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L5357-L5578 - Implementation: Function `create_anthropic_message` calls `_metrics.track_inference`, `request.json`, `str`, `request.body`; awaits asynchronous work; has 3 explicit return paths. Anthropic Messages API endpoint. Translates Anthropic-format requests to OpenAI format, runs inference through the existing engine, and converts the response back. Supports both streaming and non-streaming modes. - Inputs: - `request` (Request; required): Required positional or keyword input. - Return annotation: `not annotated` - Decorators: app.post('/v1/messages', dependencies=[Depends(verify_api_key), Depends(check_rate_limit)]) - Calls: _metrics.track_inference, request.json, str, request.body, json.loads, re.sub, AnthropicRequest, _validate_model_name, _resolve_request_max_tokens, len, isinstance, logger.info, _sanitize_log_text, anthropic_to_openai, _start_request_budget, _acquire_default_engine_for_request, Response, _prepare_anthropic_endpoint_invocation, json.dumps, StreamingResponse, _disconnect_guard, _ensure_sse_terminal, _stream_anthropic_messages, _make_release_cleanup, time.perf_counter, _wait_with_disconnect, engine.chat, _remaining_request_timeout, tracker.finish, _metrics_result_from_status, _extract_reasoning_and_tool_calls, _thinking_disabled, _apply_response_format_or_raise, logger.error, logger.warning, clean_output_text, content_blocks.append, AnthropicResponseContentBlock, _convert_anthropic_stop_reason, AnthropicResponse, _response_model_name, AnthropicUsage, anthropic_response.model_dump_json, _release_engine_for_request - Return expressions: Response(status_code=499); response; Response(content=anthropic_response.model_dump_json(exclude_none=True), media_type='application/json') ## `vllm_mlx.server.count_anthropic_tokens` - Kind: function - Signature: `async def count_anthropic_tokens(request: Request)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L5585-L5666 - Implementation: Function `count_anthropic_tokens` calls `request.json`, `body.get`, `isinstance`, `_validate_model_name`; awaits asynchronous work; has 2 explicit return paths. Count tokens for an Anthropic Messages API request. Uses the model's tokenizer for accurate counting. Claude Code calls this endpoint for token budgeting. Note: Don't parse via AnthropicRequest — count_tokens requests from Claude Code don't include max_tokens. - Inputs: - `request` (Request; required): Required positional or keyword input. - Return annotation: `not annotated` - Decorators: app.post('/v1/messages/count_tokens', dependencies=[Depends(verify_api_key), Depends(check_rate_limit)]) - Calls: request.json, body.get, isinstance, _validate_model_name, _start_request_budget, _acquire_default_engine_for_request, Response, len, tokenizer.encode, block.get, msg.get, json.dumps, item.get, tool.get, _release_engine_for_request - Return expressions: Response(status_code=499); {'input_tokens': total_tokens} ## `vllm_mlx.server._emit_content_pieces` - Kind: function - Signature: `def _emit_content_pieces(pieces: list[tuple[str, str]], current_block_type: str | None, block_index: int) -> tuple[list[str], str | None, int]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L5669-L5719 - Implementation: Function `_emit_content_pieces` calls `events.append`, `json.dumps`; returns `(events, current_block_type, block_index)`. Emit Anthropic SSE events for content pieces from the think router. Handles block type transitions (thinking <-> text), emitting content_block_start/stop/delta events as needed. Args: pieces: List of (block_type, text) from StreamingThinkRouter current_block_type: Current open block type, or None block_index: Current block index Returns: Tuple of (events, updated_block_type, updated_block_index) - Inputs: - `pieces` (list[tuple[str, str]]; required): List of (block_type, text) from StreamingThinkRouter - `current_block_type` (str | None; required): Current open block type, or None - `block_index` (int; required): Current block index - Return annotation: `tuple[list[str], str | None, int]` - Calls: events.append, json.dumps - Return expressions: (events, current_block_type, block_index) ## `vllm_mlx.server._stream_anthropic_messages` - Kind: function - Signature: `async def _stream_anthropic_messages(engine: BaseEngine, openai_request: ChatCompletionRequest, anthropic_request: AnthropicRequest, prepared: PreparedChatInvocation, metrics_tracker=None) -> AsyncIterator[str]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L5722-L5995 - Implementation: Function `_stream_anthropic_messages` calls `uuid.uuid4`, `time.perf_counter`, `dict`, `_response_model_name`; yields values incrementally. Stream Anthropic Messages API SSE events. Converts OpenAI streaming chunks to Anthropic event format: message_start -> content_block_start -> content_block_delta* -> content_block_stop -> message_delta -> message_stop When a reasoning parser is active, emits a ``thinking`` content block (index 0) for reasoning tokens and a ``text`` content block (index 1) for the actual response, matching the Anthropic extended thinking format. - Inputs: - `engine` (BaseEngine; required): Required positional or keyword input. - `openai_request` (ChatCompletionRequest; required): Required positional or keyword input. - `anthropic_request` (AnthropicRequest; required): Required positional or keyword input. - `prepared` (PreparedChatInvocation; required): Required positional or keyword input. - `metrics_tracker` (not annotated; optional; default `None`): Optional positional or keyword input; defaults to `None`. - Return annotation: `AsyncIterator[str]` - Calls: uuid.uuid4, time.perf_counter, dict, _response_model_name, json.dumps, _prepare_streaming_reasoning_parser, chat_kwargs.get, _get_streaming_tool_parser, openai_request.model_dump, engine.stream_chat, metrics_tracker.observe_ttft, hasattr, SPECIAL_TOKENS_PATTERN.sub, _streaming_tool_markup_possible_after_delta, _parse_streaming_tool_content, tool_result.get, _TOOL_MARKUP_PATTERN.sub, reasoning_parser.extract_reasoning_streaming, _parse_tool_calls_with_parser, enumerate, json.loads, logger.info, _metrics_result_from_status, metrics_tracker.finish ## `vllm_mlx.server.stream_completion` - Kind: function - Signature: `async def stream_completion(engine: BaseEngine, prompt: str, request: CompletionRequest, max_tokens: int, repetition_penalty: float | None=None, metrics_tracker=None) -> AsyncIterator[str]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L6003-L6084 - Implementation: Function `stream_completion` calls `_resolve_temperature`, `_resolve_top_p`, `_resolve_top_k`, `_resolve_min_p`; yields values incrementally. Stream completion response. - Inputs: - `engine` (BaseEngine; required): Required positional or keyword input. - `prompt` (str; required): Required positional or keyword input. - `request` (CompletionRequest; required): Required positional or keyword input. - `max_tokens` (int; required): Required positional or keyword input. - `repetition_penalty` (float | None; optional; default `None`): Optional positional or keyword input; defaults to `None`. - `metrics_tracker` (not annotated; optional; default `None`): Optional positional or keyword input; defaults to `None`. - Return annotation: `AsyncIterator[str]` - Calls: _resolve_temperature, _resolve_top_p, _resolve_top_k, _resolve_min_p, _resolve_presence_penalty, _resolve_repetition_penalty, getattr, engine.stream_generate, metrics_tracker.observe_ttft, hasattr, uuid.uuid4, int, time.time, _response_model_name, get_usage(output).model_dump, get_usage, json.dumps, _metrics_result_from_status, metrics_tracker.finish ## `vllm_mlx.server.stream_chat_completion` - Kind: function - Signature: `async def stream_chat_completion(engine: BaseEngine, messages: list, request: ChatCompletionRequest, metrics_tracker=None, **kwargs) -> AsyncIterator[str]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L6087-L6512 - Implementation: Function `stream_chat_completion` calls `uuid.uuid4`, `time.perf_counter`, `_stream_request_metadata`, `ChatCompletionChunk`; yields values incrementally. Stream chat completion response. - Inputs: - `engine` (BaseEngine; required): Required positional or keyword input. - `messages` (list; required): Required positional or keyword input. - `request` (ChatCompletionRequest; required): Required positional or keyword input. - `metrics_tracker` (not annotated; optional; default `None`): Optional positional or keyword input; defaults to `None`. - `**kwargs` (not annotated; optional): Additional variadic keyword inputs accepted by this callable. - Return annotation: `AsyncIterator[str]` - Calls: uuid.uuid4, time.perf_counter, _stream_request_metadata, ChatCompletionChunk, _response_model_name, ChatCompletionChunkChoice, ChatCompletionChunkDelta, first_chunk.model_dump_json, _prepare_openai_stream_reasoning_state, _streaming_json_fence_stripper, _get_streaming_tool_parser, engine.stream_chat, metrics_tracker.observe_ttft, hasattr, reasoning_parser.extract_reasoning_streaming, _promote_streaming_response_format_delta, _streaming_tool_markup_possible, _streaming_tool_markup_possible_after_delta, _extract_streaming_tool_delta, chunk.model_dump_json, tc.get, _coerce_tool_arguments, get_usage, tool_result.get, _TOOL_MARKUP_PATTERN.sub, fence_stripper.feed, fence_stripper.finalize, SPECIAL_TOKENS_PATTERN.sub, tool_parser.extract_tool_calls, enumerate, tool_chunk.model_dump_json, getattr, parse_json_output, any, kwargs.get, logger.error, logger.warning, logger.info, Usage, usage_chunk.model_dump_json, _metrics_result_from_status, metrics_tracker.finish ## `vllm_mlx.server.init_mcp` - Kind: function - Signature: `async def init_mcp(config_path: str)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L6520-L6546 - Implementation: Function `init_mcp` calls `load_mcp_config`, `MCPClientManager`, `_mcp_manager.start`, `ToolSandbox`; awaits asynchronous work. Initialize MCP manager from config file. - Inputs: - `config_path` (str; required): Required positional or keyword input. - Return annotation: `not annotated` - Calls: load_mcp_config, MCPClientManager, _mcp_manager.start, ToolSandbox, ToolExecutor, logger.info, len, _mcp_manager.get_all_tools, logger.error, _sanitize_log_text ## `vllm_mlx.server._make_keepalive_http_protocol` - Kind: function - Signature: `def _make_keepalive_http_protocol(idle=10, interval=5, count=3)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L6554-L6589 - Implementation: Function `_make_keepalive_http_protocol` returns `_KeepaliveProtocol`. Create a uvicorn HTTP protocol class with aggressive TCP keepalive. When a client abruptly disconnects (power-off, network loss), the server TCP stack won't notice for ~2 hours (default keepalive). With aggressive keepalive (idle=10s, interval=5s, count=3), dead connections are detected in ~25 seconds, letting ``_wait_with_disconnect()`` abort the request and stop wasting GPU cycles on tokens nobody will receive. - Inputs: - `idle` (not annotated; optional; default `10`): Optional positional or keyword input; defaults to `10`. - `interval` (not annotated; optional; default `5`): Optional positional or keyword input; defaults to `5`. - `count` (not annotated; optional; default `3`): Optional positional or keyword input; defaults to `3`. - Return annotation: `not annotated` - Return expressions: _KeepaliveProtocol ## `vllm_mlx.server._make_keepalive_http_protocol._KeepaliveProtocol` - Kind: nested class - Signature: `class _KeepaliveProtocol(_Base)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L6567-L6587 - Implementation: Nested Class `_make_keepalive_http_protocol._KeepaliveProtocol` derives from `_Base` and declares 1 direct member(s). Nested Class `_make_keepalive_http_protocol._KeepaliveProtocol` derives from `_Base` and declares 1 direct member(s). - Inputs: none - Constructs: `vllm_mlx.server._make_keepalive_http_protocol._KeepaliveProtocol` ## `vllm_mlx.server._make_keepalive_http_protocol._KeepaliveProtocol.connection_made` - Kind: nested function - Signature: `def connection_made(self, transport)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L6568-L6587 - Implementation: Nested Function `_make_keepalive_http_protocol._KeepaliveProtocol.connection_made` calls `super().connection_made`, `super`, `transport.get_extra_info`, `sock.setsockopt`; returns `None`. Nested Function `_make_keepalive_http_protocol._KeepaliveProtocol.connection_made` calls `super().connection_made`, `super`, `transport.get_extra_info`, `sock.setsockopt`; returns `None`. - Inputs: - `transport` (not annotated; required): Required positional or keyword input. - Return annotation: `not annotated` - Calls: super().connection_made, super, transport.get_extra_info, sock.setsockopt, hasattr - Return expressions: None ## `vllm_mlx.server.main` - Kind: function - Signature: `def main()` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L6597-L6708 - Implementation: Function `main` calls `create_parser`, `parser.parse_args`, `_metrics.configure`, `RateLimiter`. Run the server. - Inputs: none - Return annotation: `not annotated` - Calls: create_parser, parser.parse_args, _metrics.configure, RateLimiter, logger.info, logger.warning, get_parser, parser_cls, load_embedding_model, load_model, uvicorn.run, _make_keepalive_http_protocol ## `vllm_mlx.server.create_parser` - Kind: function - Signature: `def create_parser() -> argparse.ArgumentParser` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L6711-L6912 - Implementation: Function `create_parser` calls `argparse.ArgumentParser`, `parser.add_argument`, `make_positive_int_arg_parser`, `list_parsers`; returns `parser`. Create the standalone server CLI parser. - Inputs: none - Return annotation: `argparse.ArgumentParser` - Calls: argparse.ArgumentParser, parser.add_argument, make_positive_int_arg_parser, list_parsers, ', '.join, make_json_object_arg_parser - Return expressions: parser # Module `vllm_mlx.specprefill` SpecPrefill: Attention-based sparse prefill for MLX. Full pipeline for reducing TTFT on long prompts: Step 1 (score_tokens): Use a small draft model to identify important tokens Step 2 (sparse_prefill): Prefill target model with only selected tokens, preserving original positional encoding via manual RoPE Usage: from specprefill import score_tokens, select_chunks, sparse_prefill, cleanup_rope # 1. Score with draft model importance = score_tokens(draft_model, tokens) # 2. Select important token chunks selected = select_chunks(importance, keep_pct=0.3) # 3. Sparse prefill on target model target_cache = make_prompt_cache(target_model) logits = sparse_prefill(target_model, tokens, selected, target_cache) # 4. Generate normally using target_cache... # 5. Cleanup cleanup_rope(target_model) Design notes: - RoPE is relative: Q_m @ K_p^T depends only on (m - p). Selected keys stored contiguously in the cache buffer with correct RoPE angles produce correct attention during decode. - After sparse prefill of N tokens from a total prompt of M, cache.offset = N but decode RoPE needs position M. The _OffsetAdjustedRoPE adds (M - N) to each RoPE offset call, so decode position = N + i + (M - N) = M + i. - GatedDeltaNet (linear attention) layers process sparse tokens through their conv/SSM state normally. This is lossy but acceptable per the SpecPrefill paper — attention layers are the primary long-range mechanism. Reference: arxiv.org/abs/2502.02789 (SpecPrefill: Speculative Prefilling) Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/specprefill.py#L1-L845 ## `vllm_mlx.specprefill._AttentionCapture` - Kind: class - Signature: `class _AttentionCapture` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/specprefill.py#L53-L73 - Implementation: Class `_AttentionCapture` declares 3 direct member(s). Wrapper that captures post-RoPE query vectors and delegates to original. Installed on attention layers during lookahead decode to capture query vectors for importance scoring. Supports multiple architectures via query_extractor callback. - Inputs: - `original` (not annotated; required): Required positional or keyword input. - `buf_idx` (not annotated; required): Required positional or keyword input. - `query_buffer` (not annotated; required): Required positional or keyword input. - `query_extractor` (not annotated; optional; default `None`): Optional positional or keyword input; defaults to `None`. - Constructs: `vllm_mlx.specprefill._AttentionCapture` ## `vllm_mlx.specprefill._AttentionCapture.__init__` - Kind: method - Signature: `def __init__(self, original, buf_idx, query_buffer, query_extractor=None)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/specprefill.py#L61-L65 - Implementation: Method `_AttentionCapture.__init__` updates `self._original`, `self._buf_idx`, `self._query_buffer`, `self._query_extractor`. Method `_AttentionCapture.__init__` updates `self._original`, `self._buf_idx`, `self._query_buffer`, `self._query_extractor`. - Inputs: - `original` (not annotated; required): Required positional or keyword input. - `buf_idx` (not annotated; required): Required positional or keyword input. - `query_buffer` (not annotated; required): Required positional or keyword input. - `query_extractor` (not annotated; optional; default `None`): Optional positional or keyword input; defaults to `None`. - Return annotation: `not annotated` - State writes: self._original, self._buf_idx, self._query_buffer, self._query_extractor ## `vllm_mlx.specprefill._AttentionCapture.__call__` - Kind: method - Signature: `def __call__(self, x, mask=None, cache=None)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/specprefill.py#L67-L70 - Implementation: Method `_AttentionCapture.__call__` calls `self._query_extractor`, `self._query_buffer[self._buf_idx].append`, `self._original`; returns `self._original(x, mask=mask, cache=cache)`. Method `_AttentionCapture.__call__` calls `self._query_extractor`, `self._query_buffer[self._buf_idx].append`, `self._original`; returns `self._original(x, mask=mask, cache=cache)`. - Inputs: - `x` (not annotated; required): Required positional or keyword input. - `mask` (not annotated; optional; default `None`): Optional positional or keyword input; defaults to `None`. - `cache` (not annotated; optional; default `None`): Optional positional or keyword input; defaults to `None`. - Return annotation: `not annotated` - Calls: self._query_extractor, self._query_buffer[self._buf_idx].append, self._original - State reads: self._query_extractor, self._original, self._query_buffer, self._buf_idx - Return expressions: self._original(x, mask=mask, cache=cache) ## `vllm_mlx.specprefill._AttentionCapture.__getattr__` - Kind: method - Signature: `def __getattr__(self, name)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/specprefill.py#L72-L73 - Implementation: Method `_AttentionCapture.__getattr__` calls `getattr`; returns `getattr(self._original, name)`. Method `_AttentionCapture.__getattr__` calls `getattr`; returns `getattr(self._original, name)`. - Inputs: - `name` (not annotated; required): Required positional or keyword input. - Return annotation: `not annotated` - Calls: getattr - State reads: self._original - Return expressions: getattr(self._original, name) ## `vllm_mlx.specprefill._qwen35_extract_queries` - Kind: function - Signature: `def _qwen35_extract_queries(attn, x, cache=None)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/specprefill.py#L76-L92 - Implementation: Function `_qwen35_extract_queries` calls `attn.q_proj`, `mx.split`, `q_out.reshape`, `attn.q_norm(queries).transpose`; returns `queries`. Extract post-RoPE queries from Qwen3.5 attention (gate split + q_norm). Qwen3.5 q_proj output is 2x wider: [queries, gate]. We split, normalize, then apply RoPE. - Inputs: - `attn` (not annotated; required): Required positional or keyword input. - `x` (not annotated; required): Required positional or keyword input. - `cache` (not annotated; optional; default `None`): Optional positional or keyword input; defaults to `None`. - Return annotation: `not annotated` - Calls: attn.q_proj, mx.split, q_out.reshape, attn.q_norm(queries).transpose, attn.q_norm, attn.rope - Return expressions: queries ## `vllm_mlx.specprefill._llama_extract_queries` - Kind: function - Signature: `def _llama_extract_queries(attn, x, cache=None)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/specprefill.py#L95-L113 - Implementation: Function `_llama_extract_queries` calls `getattr`, `attn.q_proj`, `queries.reshape(B, L, n_heads, -1).transpose`, `queries.reshape`; returns `queries`. Extract post-RoPE queries from standard transformer attention. Standard architecture: q_proj → reshape → RoPE. No gate, no q_norm. Works for Llama 3.x, Mistral, Gemma, GPT-OSS, and other GQA models. - Inputs: - `attn` (not annotated; required): Required positional or keyword input. - `x` (not annotated; required): Required positional or keyword input. - `cache` (not annotated; optional; default `None`): Optional positional or keyword input; defaults to `None`. - Return annotation: `not annotated` - Calls: getattr, attn.q_proj, queries.reshape(B, L, n_heads, -1).transpose, queries.reshape, attn.rope - Return expressions: queries ## `vllm_mlx.specprefill._nemotron_h_extract_queries` - Kind: function - Signature: `def _nemotron_h_extract_queries(attn, x, cache=None)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/specprefill.py#L116-L125 - Implementation: Function `_nemotron_h_extract_queries` calls `attn.q_proj(x).reshape(B, L, attn.num_heads, -1).transpose`, `attn.q_proj(x).reshape`, `attn.q_proj`; returns `queries`. Extract queries from Nemotron-H attention (no RoPE, no gate, no q_norm). Nemotron-H attention layers have NO positional encoding — RoPE is absent. Positional modeling comes from Mamba2 layers. Attention is content-based only. - Inputs: - `attn` (not annotated; required): Required positional or keyword input. - `x` (not annotated; required): Required positional or keyword input. - `cache` (not annotated; optional; default `None`): Optional positional or keyword input; defaults to `None`. - Return annotation: `not annotated` - Calls: attn.q_proj(x).reshape(B, L, attn.num_heads, -1).transpose, attn.q_proj(x).reshape, attn.q_proj - Return expressions: queries ## `vllm_mlx.specprefill._patch_attention_for_capture` - Kind: function - Signature: `def _patch_attention_for_capture(model, query_buffer, query_extractor=None)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/specprefill.py#L128-L149 - Implementation: Function `_patch_attention_for_capture` calls `_find_attention_layers`, `len`, `attn_indices.append`, `_get_attn_module`; returns `(originals, attn_indices)`. Replace attention modules on full-attention layers with capture wrappers. Supports both `self_attn` (Qwen3.5/Llama/GPT-OSS) and `mixer` (Nemotron-H block_type="*") attribute conventions. Returns (originals, attn_layer_indices) for cleanup. - Inputs: - `model` (not annotated; required): Required positional or keyword input. - `query_buffer` (not annotated; required): Required positional or keyword input. - `query_extractor` (not annotated; optional; default `None`): Optional positional or keyword input; defaults to `None`. - Return annotation: `not annotated` - Calls: _find_attention_layers, len, attn_indices.append, _get_attn_module, _set_attn_module, _AttentionCapture, originals.append - Return expressions: (originals, attn_indices) ## `vllm_mlx.specprefill._unpatch_attention_capture` - Kind: function - Signature: `def _unpatch_attention_capture(model, originals)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/specprefill.py#L152-L155 - Implementation: Function `_unpatch_attention_capture` calls `_set_attn_module`. Restore original attention modules after capture. - Inputs: - `model` (not annotated; required): Required positional or keyword input. - `originals` (not annotated; required): Required positional or keyword input. - Return annotation: `not annotated` - Calls: _set_attn_module ## `vllm_mlx.specprefill._prefill_draft` - Kind: function - Signature: `def _prefill_draft(model, tokens, cache, step_size=2048, cancel_check=None)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/specprefill.py#L158-L175 - Implementation: Function `_prefill_draft` calls `isinstance`, `mx.array`, `len`, `cancel_check`; returns `logits`. Prefill prompt tokens into cache. Returns logits from last token. - Inputs: - `model` (not annotated; required): Required positional or keyword input. - `tokens` (not annotated; required): Required positional or keyword input. - `cache` (not annotated; required): Required positional or keyword input. - `step_size` (not annotated; optional; default `2048`): Optional positional or keyword input; defaults to `2048`. - `cancel_check` (not annotated; optional; default `None`): Optional positional or keyword input; defaults to `None`. - Return annotation: `not annotated` - Calls: isinstance, mx.array, len, cancel_check, min, model, mx.eval, mx.clear_cache - Return expressions: logits ## `vllm_mlx.specprefill._lookahead_decode` - Kind: function - Signature: `def _lookahead_decode(model, first_logits, cache, n_steps, temp=0.6, top_p=0.95, cancel_check=None)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/specprefill.py#L178-L204 - Implementation: Function `_lookahead_decode` calls `make_sampler`, `cancel_check`, `sampler`, `mx.eval`; returns `generated`. Run n_steps autoregressive decode, returning generated token ids. Query vectors are captured by the monkey-patched attention layers. - Inputs: - `model` (not annotated; required): Required positional or keyword input. - `first_logits` (not annotated; required): Required positional or keyword input. - `cache` (not annotated; required): Required positional or keyword input. - `n_steps` (not annotated; required): Required positional or keyword input. - `temp` (not annotated; optional; default `0.6`): Optional positional or keyword input; defaults to `0.6`. - `top_p` (not annotated; optional; default `0.95`): Optional positional or keyword input; defaults to `0.95`. - `cancel_check` (not annotated; optional; default `None`): Optional positional or keyword input; defaults to `None`. - Return annotation: `not annotated` - Calls: make_sampler, cancel_check, sampler, mx.eval, y.item, range, model, y.reshape, generated.append - Return expressions: generated ## `vllm_mlx.specprefill._avg_pool1d` - Kind: function - Signature: `def _avg_pool1d(x, kernel_size)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/specprefill.py#L207-L223 - Implementation: Function `_avg_pool1d` calls `mx.pad`, `mx.zeros`, `mx.concatenate`, `mx.cumsum`; has 2 explicit return paths. 1D average pooling along last axis via prefix-sum. Args: x: (..., M) input kernel_size: window size (odd for centered) Returns: (..., M) pooled (same size, zero-padded at edges) - Inputs: - `x` (not annotated; required): (..., M) input - `kernel_size` (not annotated; required): window size (odd for centered) - Return annotation: `not annotated` - Calls: mx.pad, mx.zeros, mx.concatenate, mx.cumsum - Return expressions: x; (prefix[..., kernel_size:] - prefix[..., :-kernel_size]) / kernel_size ## `vllm_mlx.specprefill._compute_importance` - Kind: function - Signature: `def _compute_importance(query_buffer, attn_caches, n_prompt, n_attn_heads, n_kv_heads, pool_kernel=13)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/specprefill.py#L226-L271 - Implementation: Function `_compute_importance` calls `enumerate`, `mx.concatenate`, `mx.repeat`, `expanded_keys.transpose`; can raise `RuntimeError`; returns `importance`. Compute per-token importance from captured queries and cached keys. Aggregation (SpecPrefill paper): 1. softmax(Q @ K^T / sqrt(d)) per head, per layer, per lookahead token 2. avg_pool1d smoothing 3. max across (layers × heads) 4. mean across lookahead tokens Returns: (n_prompt,) importance scores. - Inputs: - `query_buffer` (not annotated; required): Required positional or keyword input. - `attn_caches` (not annotated; required): Required positional or keyword input. - `n_prompt` (not annotated; required): Required positional or keyword input. - `n_attn_heads` (not annotated; required): Required positional or keyword input. - `n_kv_heads` (not annotated; required): Required positional or keyword input. - `pool_kernel` (not annotated; optional; default `13`): Optional positional or keyword input; defaults to `13`. - Return annotation: `not annotated` - Calls: enumerate, mx.concatenate, mx.repeat, expanded_keys.transpose, mx.softmax, scores.astype, all_scores.append, weights.squeeze, RuntimeError, _avg_pool1d, mx.max, mx.mean - Raises directly: RuntimeError - Return expressions: importance ## `vllm_mlx.specprefill.score_tokens` - Kind: function - Signature: `def score_tokens(model, tokens, n_lookahead=8, pool_kernel=13, temp=0.6, top_p=0.95, prefill_step_size=2048, query_extractor=None, cancel_check=None)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/specprefill.py#L274-L396 - Implementation: Function `score_tokens` calls `isinstance`, `tokens.tolist`, `len`, `_find_attention_layers`; returns `importance`. Score token importance using attention-based analysis on a draft model. Runs the full scoring pipeline: 1. Prefill the draft model with all tokens 2. N lookahead decode steps, capturing query vectors from attention layers 3. Compute importance: Q_lookahead @ K_prompt^T, aggregated across heads/layers The draft model's cache is created internally and discarded after scoring. Args: model: Draft model (small, fast — e.g. 4B) tokens: list or mx.array of token IDs n_lookahead: decode steps for query capture (default 8) pool_kernel: smoothing kernel for avg_pool1d (default 13, 0=disable) temp: sampling temperature for lookahead (default 0.6) top_p: top-p for lookahead (default 0.95) prefill_step_size: chunk size for draft prefill (default 2048) query_extractor: function(attn, x, cache) → queries tensor. Default: _qwen35_extract_queries. Use _llama_extract_queries for standard Llama/Mistral/Gemma models. Returns: importance: (M,) mx.array of per-token importance scores - Inputs: - `model` (not annotated; required): Draft model (small, fast — e.g. 4B) - `tokens` (not annotated; required): list or mx.array of token IDs - `n_lookahead` (not annotated; optional; default `8`): decode steps for query capture (default 8) - `pool_kernel` (not annotated; optional; default `13`): smoothing kernel for avg_pool1d (default 13, 0=disable) - `temp` (not annotated; optional; default `0.6`): sampling temperature for lookahead (default 0.6) - `top_p` (not annotated; optional; default `0.95`): top-p for lookahead (default 0.95) - `prefill_step_size` (not annotated; optional; default `2048`): chunk size for draft prefill (default 2048) - `query_extractor` (not annotated; optional; default `None`): function(attn, x, cache) → queries tensor. - `cancel_check` (not annotated; optional; default `None`): Optional positional or keyword input; defaults to `None`. - Return annotation: `not annotated` - Calls: isinstance, tokens.tolist, len, _find_attention_layers, _get_attn_module, getattr, _EXTRACTOR_REGISTRY.get, _get_rope, make_prompt_cache, _prefill_draft, range, _patch_attention_for_capture, _lookahead_decode, mx.eval, _unpatch_attention_capture, _build_layer_to_cache_map, cancel_check, _compute_importance, mx.clear_cache - Return expressions: importance ## `vllm_mlx.specprefill.select_chunks` - Kind: function - Signature: `def select_chunks(importance, keep_pct=0.3, chunk_size=32, backbone_pct=0.0)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/specprefill.py#L399-L467 - Implementation: Function `select_chunks` calls `mx.arange`, `math.ceil`, `max`, `range`; has 2 explicit return paths. Select top-k% token chunks by average importance. Args: importance: (M,) per-token importance scores keep_pct: fraction of chunks to keep (default 0.3) chunk_size: tokens per chunk (default 32) backbone_pct: fraction of chunks reserved for evenly-spaced coverage Returns: sorted mx.array of kept token indices - Inputs: - `importance` (not annotated; required): (M,) per-token importance scores - `keep_pct` (not annotated; optional; default `0.3`): fraction of chunks to keep (default 0.3) - `chunk_size` (not annotated; optional; default `32`): tokens per chunk (default 32) - `backbone_pct` (not annotated; optional; default `0.0`): fraction of chunks reserved for evenly-spaced coverage - Return annotation: `not annotated` - Calls: mx.arange, math.ceil, max, range, min, chunk_scores.append, mx.mean(importance[start:end]).item, mx.mean, set, sorted, selected_chunks.update, selected_chunks.add, round, len, _selected_token_count, indices.extend, mx.array - Return expressions: mx.arange(M); mx.array(indices) ## `vllm_mlx.specprefill.select_chunks._selected_token_count` - Kind: nested function - Signature: `def _selected_token_count(chunks)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/specprefill.py#L437-L443 - Implementation: Nested Function `select_chunks._selected_token_count` calls `min`; returns `total`. Nested Function `select_chunks._selected_token_count` calls `min`; returns `total`. - Inputs: - `chunks` (not annotated; required): Required positional or keyword input. - Return annotation: `not annotated` - Calls: min - Return expressions: total ## `vllm_mlx.specprefill.manual_rope` - Kind: function - Signature: `def manual_rope(x, positions, dims, base=10000.0, scale=1.0)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/specprefill.py#L480-L508 - Implementation: Function `manual_rope` calls `mx.arange`, `positions.astype`, `mx.cos`, `mx.sin`; returns `mx.concatenate([rotated, x_pass], axis=-1)`. Apply RoPE at arbitrary (non-contiguous) positions. Uses non-traditional (interleaved) layout matching Qwen3.5: rotates first `dims` dimensions as pairs [0,half), [half,dims), passes through [dims:] unchanged. Args: x: (B, n_heads, L, head_dim) input tensor positions: (L,) position indices (can be non-contiguous) dims: number of dimensions to rotate (head_dim * partial_rotary_factor) base: RoPE base frequency (default 10000.0) scale: position scale divisor (default 1.0, higher = compressed positions) Returns: (B, n_heads, L, head_dim) with RoPE applied - Inputs: - `x` (not annotated; required): (B, n_heads, L, head_dim) input tensor - `positions` (not annotated; required): (L,) position indices (can be non-contiguous) - `dims` (not annotated; required): number of dimensions to rotate (head_dim * partial_rotary_factor) - `base` (not annotated; optional; default `10000.0`): RoPE base frequency (default 10000.0) - `scale` (not annotated; optional; default `1.0`): position scale divisor (default 1.0, higher = compressed positions) - Return annotation: `not annotated` - Calls: mx.arange, positions.astype, mx.cos, mx.sin, mx.concatenate - Return expressions: mx.concatenate([rotated, x_pass], axis=-1) ## `vllm_mlx.specprefill.manual_rope_with_freqs` - Kind: function - Signature: `def manual_rope_with_freqs(x, positions, dims, freqs, pre_scale=1.0)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/specprefill.py#L511-L528 - Implementation: Function `manual_rope_with_freqs` calls `(1.0 / freqs).astype`, `positions[:, None].astype`, `mx.cos`, `mx.sin`; returns `mx.concatenate([rotated, x_pass], axis=-1)`. Apply RoPE at arbitrary positions using pre-computed frequencies. For custom RoPE variants (Llama3, Yarn, SuScaled) that store _freqs. - Inputs: - `x` (not annotated; required): Required positional or keyword input. - `positions` (not annotated; required): Required positional or keyword input. - `dims` (not annotated; required): Required positional or keyword input. - `freqs` (not annotated; required): Required positional or keyword input. - `pre_scale` (not annotated; optional; default `1.0`): Optional positional or keyword input; defaults to `1.0`. - Return annotation: `not annotated` - Calls: (1.0 / freqs).astype, positions[:, None].astype, mx.cos, mx.sin, mx.concatenate - Return expressions: mx.concatenate([rotated, x_pass], axis=-1) ## `vllm_mlx.specprefill._PositionMappedRoPE` - Kind: class - Signature: `class _PositionMappedRoPE` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/specprefill.py#L536-L571 - Implementation: Class `_PositionMappedRoPE` declares 2 direct member(s). Wraps a RoPE module to apply rotation at non-contiguous positions. Used during sparse prefill. The `offset` parameter from the cache tells us which slice of the position array to use for the current chunk: positions = all_positions[(offset - cache_start) : (offset - cache_start) + L] When composing with a pre-populated cache (e.g., system KV cache), cache_start is the initial cache offset so indexing into the position array is correct. - Inputs: - `original_rope` (not annotated; required): Required positional or keyword input. - `all_positions` (not annotated; required): Required positional or keyword input. - `cache_start` (not annotated; optional; default `0`): Optional positional or keyword input; defaults to `0`. - Constructs: `vllm_mlx.specprefill._PositionMappedRoPE` ## `vllm_mlx.specprefill._PositionMappedRoPE.__init__` - Kind: method - Signature: `def __init__(self, original_rope, all_positions, cache_start=0)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/specprefill.py#L547-L561 - Implementation: Method `_PositionMappedRoPE.__init__` updates `self._original`, `self._all_positions`, `self._cache_start`, `self._has_custom_freqs`; calls `hasattr`, `_get_dims`, `_get_pre_scale`. Method `_PositionMappedRoPE.__init__` updates `self._original`, `self._all_positions`, `self._cache_start`, `self._has_custom_freqs`; calls `hasattr`, `_get_dims`, `_get_pre_scale`. - Inputs: - `original_rope` (not annotated; required): Required positional or keyword input. - `all_positions` (not annotated; required): Required positional or keyword input. - `cache_start` (not annotated; optional; default `0`): Optional positional or keyword input; defaults to `0`. - Return annotation: `not annotated` - Calls: hasattr, _get_dims, _get_pre_scale - State reads: self._has_custom_freqs - State writes: self._original, self._all_positions, self._cache_start, self._has_custom_freqs, self._freqs, self._dims, self._pre_scale, self._base, self._scale ## `vllm_mlx.specprefill._PositionMappedRoPE.__call__` - Kind: method - Signature: `def __call__(self, x, offset=0)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/specprefill.py#L563-L571 - Implementation: Method `_PositionMappedRoPE.__call__` calls `manual_rope_with_freqs`, `manual_rope`; has 2 explicit return paths. Method `_PositionMappedRoPE.__call__` calls `manual_rope_with_freqs`, `manual_rope`; has 2 explicit return paths. - Inputs: - `x` (not annotated; required): Required positional or keyword input. - `offset` (not annotated; optional; default `0`): Optional positional or keyword input; defaults to `0`. - Return annotation: `not annotated` - Calls: manual_rope_with_freqs, manual_rope - State reads: self._cache_start, self._all_positions, self._has_custom_freqs, self._dims, self._freqs, self._pre_scale, self._base, self._scale - Return expressions: manual_rope_with_freqs(x, positions, self._dims, self._freqs, pre_scale=self._pre_scale); manual_rope(x, positions, self._dims, base=self._base, scale=self._scale) ## `vllm_mlx.specprefill._OffsetAdjustedRoPE` - Kind: class - Signature: `class _OffsetAdjustedRoPE` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/specprefill.py#L574-L590 - Implementation: Class `_OffsetAdjustedRoPE` declares 2 direct member(s). Wraps a RoPE module to add a constant offset for decode after sparse prefill. After sparse prefill of N tokens from a prompt of M total tokens: cache.offset = N + i (i = decode step) desired RoPE position = M + i adjustment = M - N So: RoPE(x, offset = cache.offset + adjustment) = RoPE(x, M + i) - Inputs: - `original_rope` (not annotated; required): Required positional or keyword input. - `adjustment` (not annotated; required): Required positional or keyword input. - Constructs: `vllm_mlx.specprefill._OffsetAdjustedRoPE` ## `vllm_mlx.specprefill._OffsetAdjustedRoPE.__init__` - Kind: method - Signature: `def __init__(self, original_rope, adjustment)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/specprefill.py#L585-L587 - Implementation: Method `_OffsetAdjustedRoPE.__init__` updates `self._original`, `self._adjustment`. Method `_OffsetAdjustedRoPE.__init__` updates `self._original`, `self._adjustment`. - Inputs: - `original_rope` (not annotated; required): Required positional or keyword input. - `adjustment` (not annotated; required): Required positional or keyword input. - Return annotation: `not annotated` - State writes: self._original, self._adjustment ## `vllm_mlx.specprefill._OffsetAdjustedRoPE.__call__` - Kind: method - Signature: `def __call__(self, x, offset=0)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/specprefill.py#L589-L590 - Implementation: Method `_OffsetAdjustedRoPE.__call__` calls `self._original`; returns `self._original(x, offset=offset + self._adjustment)`. Method `_OffsetAdjustedRoPE.__call__` calls `self._original`; returns `self._original(x, offset=offset + self._adjustment)`. - Inputs: - `x` (not annotated; required): Required positional or keyword input. - `offset` (not annotated; optional; default `0`): Optional positional or keyword input; defaults to `0`. - Return annotation: `not annotated` - Calls: self._original - State reads: self._original, self._adjustment - Return expressions: self._original(x, offset=offset + self._adjustment) ## `vllm_mlx.specprefill._get_dims` - Kind: function - Signature: `def _get_dims(rope_module)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/specprefill.py#L598-L603 - Implementation: Function `_get_dims` calls `hasattr`, `getattr`, `ValueError`, `type`; can raise `ValueError`; returns `getattr(rope_module, attr)`. Extract rotary dimensions from any RoPE variant. - Inputs: - `rope_module` (not annotated; required): Required positional or keyword input. - Return annotation: `not annotated` - Calls: hasattr, getattr, ValueError, type - Raises directly: ValueError - Return expressions: getattr(rope_module, attr) ## `vllm_mlx.specprefill._get_pre_scale` - Kind: function - Signature: `def _get_pre_scale(rope_module)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/specprefill.py#L606-L612 - Implementation: Function `_get_pre_scale` calls `hasattr`; has 3 explicit return paths. Extract pre-scale factor from custom RoPE variants (SuScaled, Yarn). - Inputs: - `rope_module` (not annotated; required): Required positional or keyword input. - Return annotation: `not annotated` - Calls: hasattr - Return expressions: rope_module.mscale; rope_module._scale; 1.0 ## `vllm_mlx.specprefill._find_attention_layers` - Kind: function - Signature: `def _find_attention_layers(model)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/specprefill.py#L615-L630 - Implementation: Function `_find_attention_layers` calls `enumerate`, `hasattr`, `results.append`, `getattr`; returns `results`. Find all full-attention layers across architectures. Supports: - Qwen3.5 / Llama / GPT-OSS: layers with `self_attn` attribute - Nemotron-H: layers with `block_type == "*"` (attention blocks use `mixer`) Returns list of (layer_idx, layer) tuples. - Inputs: - `model` (not annotated; required): Required positional or keyword input. - Return annotation: `not annotated` - Calls: enumerate, hasattr, results.append, getattr - Return expressions: results ## `vllm_mlx.specprefill._get_attn_module` - Kind: function - Signature: `def _get_attn_module(layer)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/specprefill.py#L633-L639 - Implementation: Function `_get_attn_module` calls `hasattr`, `getattr`; has 3 explicit return paths. Get the attention module from a layer (self_attn or mixer). - Inputs: - `layer` (not annotated; required): Required positional or keyword input. - Return annotation: `not annotated` - Calls: hasattr, getattr - Return expressions: layer.self_attn; layer.mixer; None ## `vllm_mlx.specprefill._get_rope` - Kind: function - Signature: `def _get_rope(attn)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/specprefill.py#L642-L647 - Implementation: Function `_get_rope` calls `getattr`; returns `getattr(attn, 'rope', None) or getattr(attn, 'rotary_emb', None)`. Get the RoPE module from an attention layer, or None. mlx_lm models use ``self.rope``; mlx_vlm models use ``self.rotary_emb``. - Inputs: - `attn` (not annotated; required): Required positional or keyword input. - Return annotation: `not annotated` - Calls: getattr - Return expressions: getattr(attn, 'rope', None) or getattr(attn, 'rotary_emb', None) ## `vllm_mlx.specprefill._set_rope` - Kind: function - Signature: `def _set_rope(attn, rope_module)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/specprefill.py#L650-L655 - Implementation: Function `_set_rope` calls `hasattr`. Set the RoPE module on an attention layer. - Inputs: - `attn` (not annotated; required): Required positional or keyword input. - `rope_module` (not annotated; required): Required positional or keyword input. - Return annotation: `not annotated` - Calls: hasattr ## `vllm_mlx.specprefill._set_attn_module` - Kind: function - Signature: `def _set_attn_module(layer, module)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/specprefill.py#L658-L663 - Implementation: Function `_set_attn_module` calls `hasattr`, `getattr`. Set the attention module on a layer (self_attn or mixer). - Inputs: - `layer` (not annotated; required): Required positional or keyword input. - `module` (not annotated; required): Required positional or keyword input. - Return annotation: `not annotated` - Calls: hasattr, getattr ## `vllm_mlx.specprefill._build_layer_to_cache_map` - Kind: function - Signature: `def _build_layer_to_cache_map(model)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/specprefill.py#L666-L690 - Implementation: Function `_build_layer_to_cache_map` calls `any`, `hasattr`, `range`, `len`; has 2 explicit return paths. Build mapping from model layer index to cache index. Standard models (Qwen3.5, Llama, GPT-OSS): one cache entry per layer, so the mapping is identity (layer_idx → layer_idx). Nemotron-H: only M (Mamba2) and * (attention) layers have cache entries. MLP (-) and MoE (E) layers get no cache. The mapping is compacted. Returns dict {layer_idx: cache_idx}. - Inputs: - `model` (not annotated; required): Required positional or keyword input. - Return annotation: `not annotated` - Calls: any, hasattr, range, len, enumerate, getattr - Return expressions: {i: i for i in range(len(model.layers))}; layer_to_cache ## `vllm_mlx.specprefill.sparse_prefill` - Kind: function - Signature: `def sparse_prefill(model, tokens, selected_indices, cache, step_size=2048, position_offset=0, cancel_check=None)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/specprefill.py#L698-L827 - Implementation: Function `sparse_prefill` calls `isinstance`, `mx.array`, `type`, `max`; returns `logits`. Prefill the model cache with selected tokens at their original positions. Runs the model forward on only the selected tokens while preserving their original positional encoding via manual RoPE. After this call, the cache contains KV entries with correct RoPE positions, and attention layers have _OffsetAdjustedRoPE installed for correct decode positioning. Args: model: Language model with .layers property (TextModel or VLM Model) tokens: (M,) all prompt token IDs (mx.array or list) selected_indices: (N,) sorted indices into tokens to keep (mx.array or list) cache: list of KVCache/ArraysCache from make_prompt_cache() step_size: chunk size for processing (default 2048) position_offset: added to selected_indices for RoPE positions (default 0). Use when the cache already has tokens from a prior prefill (e.g., system prompt KV cache with S tokens → position_offset=S). Returns: logits: (1, 1, vocab_size) from the last selected token Side effects: - Populates cache with KV for selected tokens - Installs _OffsetAdjustedRoPE on attention layers for decode - Call cleanup_rope(model) after generation to restore original RoPE - Inputs: - `model` (not annotated; required): Language model with .layers property (TextModel or VLM Model) - `tokens` (not annotated; required): (M,) all prompt token IDs (mx.array or list) - `selected_indices` (not annotated; required): (N,) sorted indices into tokens to keep (mx.array or list) - `cache` (not annotated; required): list of KVCache/ArraysCache from make_prompt_cache() - `step_size` (not annotated; optional; default `2048`): chunk size for processing (default 2048) - `position_offset` (not annotated; optional; default `0`): added to selected_indices for RoPE positions (default 0). Use when the cache already has tokens from a prior prefill (e.g., system prompt KV cache with S tokens → position_offset=S). - `cancel_check` (not annotated; optional; default `None`): Optional positional or keyword input; defaults to `None`. - Return annotation: `not annotated` - Calls: isinstance, mx.array, type, max, getattr, set, range, selected_indices.tolist, sorted, selected_indices.astype, _find_attention_layers, _build_layer_to_cache_map, hasattr, _get_attn_module, _get_rope, _set_rope, _PositionMappedRoPE, int, cancel_check, min, model, mx.eval, mx.clear_cache, _OffsetAdjustedRoPE - Return expressions: logits ## `vllm_mlx.specprefill.cleanup_rope` - Kind: function - Signature: `def cleanup_rope(model)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/specprefill.py#L830-L845 - Implementation: Function `cleanup_rope` calls `_find_attention_layers`, `_get_attn_module`, `_get_rope`, `isinstance`. Restore original RoPE on all attention layers. Call this after generation is complete to remove _OffsetAdjustedRoPE wrappers installed by sparse_prefill(). No-op for architectures without RoPE (e.g. Nemotron-H). - Inputs: - `model` (not annotated; required): Required positional or keyword input. - Return annotation: `not annotated` - Calls: _find_attention_layers, _get_attn_module, _get_rope, isinstance, _set_rope # Module `vllm_mlx.ssd_cache` SSD KV cache tiering for vllm-mlx. This module provides a cold-tier disk cache that sits behind MemoryAwarePrefixCache. Evicted entries spill to NVMe instead of being discarded, and cold-tier fetches reload from disk asynchronously with RAM budget reservation before the read completes. Key design: - SQLite for atomic metadata index (no mutable JSON) - Async writer thread for non-blocking spills - Per-layer serializer interface for hybrid cache types - Atomic temp-file + rename writes for crash consistency - Metrics exposed from day one Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/ssd_cache.py#L1-L1248 ## `vllm_mlx.ssd_cache.SSDCacheConfig` - Kind: class - Signature: `class SSDCacheConfig` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/ssd_cache.py#L43-L78 - Implementation: Class `SSDCacheConfig` declares 2 direct member(s). Configuration for SSD cache tier. Attributes: cache_dir: Directory for SSD cache files. None = auto-detect (~/.cache/vllm-mlx/ssd_cache/{model}/). max_size_gb: Maximum total size of SSD cache in GB. max_entries: Maximum number of entries in SSD cache. file_permissions: Unix permission bits for cache data files. dir_permissions: Unix permission bits for cache directories. spill_queue_size: Max pending spill operations before dropping. retention_seconds: Optional max age for cache entries (None = no expiry). - Inputs: - `cache_dir` (str | None; optional; default `None`): Optional constructor field; defaults to `None`. - `max_size_gb` (float; optional; default `10.0`): Optional constructor field; defaults to `10.0`. - `max_entries` (int; optional; default `10000`): Optional constructor field; defaults to `10000`. - `file_permissions` (int; optional; default `384`): Optional constructor field; defaults to `384`. - `dir_permissions` (int; optional; default `448`): Optional constructor field; defaults to `448`. - `spill_queue_size` (int; optional; default `64`): Optional constructor field; defaults to `64`. - `retention_seconds` (int | None; optional; default `None`): Optional constructor field; defaults to `None`. - Constructs: `vllm_mlx.ssd_cache.SSDCacheConfig` - Decorators: dataclass(frozen=True) ## `vllm_mlx.ssd_cache.SSDCacheConfig.__post_init__` - Kind: method - Signature: `def __post_init__(self) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/ssd_cache.py#L65-L73 - Implementation: Method `SSDCacheConfig.__post_init__` calls `ValueError`; can raise `ValueError`. Method `SSDCacheConfig.__post_init__` calls `ValueError`; can raise `ValueError`. - Inputs: none - Return annotation: `None` - Calls: ValueError - State reads: self.max_size_gb, self.max_entries, self.spill_queue_size - Raises directly: ValueError ## `vllm_mlx.ssd_cache.SSDCacheConfig.max_size_bytes` - Kind: method - Signature: `def max_size_bytes(self) -> int` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/ssd_cache.py#L76-L78 - Implementation: Method `SSDCacheConfig.max_size_bytes` calls `int`; returns `int(self.max_size_gb * _BYTES_PER_GB)`. Maximum cache size in bytes. - Inputs: none - Return annotation: `int` - Decorators: property - Calls: int - State reads: self.max_size_gb - Return expressions: int(self.max_size_gb * _BYTES_PER_GB) ## `vllm_mlx.ssd_cache.SSDCacheStats` - Kind: class - Signature: `class SSDCacheStats` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/ssd_cache.py#L82-L123 - Implementation: Class `SSDCacheStats` declares 1 direct member(s). Statistics for SSD cache tier — exposed from day one. Attributes: spill_count: Number of entries spilled to SSD. spill_bytes: Total bytes written to SSD. ssd_hits: Number of successful SSD cache lookups. ssd_misses: Number of SSD cache lookup misses. reload_latency_sum: Sum of reload latencies in seconds. reload_bytes: Total bytes read from SSD. promotion_failures: Number of failed promotions (RAM budget exhausted). - Inputs: - `spill_count` (int; optional; default `0`): Optional constructor field; defaults to `0`. - `spill_bytes` (int; optional; default `0`): Optional constructor field; defaults to `0`. - `ssd_hits` (int; optional; default `0`): Optional constructor field; defaults to `0`. - `ssd_misses` (int; optional; default `0`): Optional constructor field; defaults to `0`. - `reload_latency_sum` (float; optional; default `0.0`): Optional constructor field; defaults to `0.0`. - `reload_bytes` (int; optional; default `0`): Optional constructor field; defaults to `0`. - `promotion_failures` (int; optional; default `0`): Optional constructor field; defaults to `0`. - Constructs: `vllm_mlx.ssd_cache.SSDCacheStats` - Decorators: dataclass ## `vllm_mlx.ssd_cache.SSDCacheStats.to_dict` - Kind: method - Signature: `def to_dict(self) -> dict` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/ssd_cache.py#L103-L123 - Implementation: Method `SSDCacheStats.to_dict` calls `round`; returns `{'spill_count': self.spill_count, 'spill_bytes': self.spill_bytes, 'ssd_hits': self.ssd_hits, 'ssd_misses': self.ssd_mi…`. Return spill, lookup, reload, and promotion statistics. - Inputs: none - Return annotation: `dict` - Calls: round - State reads: self.ssd_hits, self.ssd_misses, self.reload_latency_sum, self.spill_count, self.spill_bytes, self.reload_bytes, self.promotion_failures - Return expressions: {'spill_count': self.spill_count, 'spill_bytes': self.spill_bytes, 'ssd_hits': self.ssd_hits, 'ssd_misses': self.ssd_mi… ## `vllm_mlx.ssd_cache._tokens_to_blob` - Kind: function - Signature: `def _tokens_to_blob(tokens: tuple[int, ...]) -> bytes` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/ssd_cache.py#L126-L132 - Implementation: Function `_tokens_to_blob` calls `_array.array`, `arr.tobytes`; returns `arr.tobytes()`. Serialize token tuple to a compact binary blob for SQLite storage. Uses the full token sequence as a binary blob for prefix matching. - Inputs: - `tokens` (tuple[int, ...]; required): Required positional or keyword input. - Return annotation: `bytes` - Calls: _array.array, arr.tobytes - Return expressions: arr.tobytes() ## `vllm_mlx.ssd_cache._blob_to_tokens` - Kind: function - Signature: `def _blob_to_tokens(blob: bytes) -> tuple[int, ...]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/ssd_cache.py#L135-L139 - Implementation: Function `_blob_to_tokens` calls `_array.array`, `arr.frombytes`, `tuple`; returns `tuple(arr)`. Deserialize binary blob back to token tuple. - Inputs: - `blob` (bytes; required): Required positional or keyword input. - Return annotation: `tuple[int, ...]` - Calls: _array.array, arr.frombytes, tuple - Return expressions: tuple(arr) ## `vllm_mlx.ssd_cache._tokens_hash` - Kind: function - Signature: `def _tokens_hash(tokens: tuple[int, ...]) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/ssd_cache.py#L142-L144 - Implementation: Function `_tokens_hash` calls `hashlib.sha256(_tokens_to_blob(tokens)).hexdigest`, `hashlib.sha256`, `_tokens_to_blob`; returns `hashlib.sha256(_tokens_to_blob(tokens)).hexdigest()`. Compute SHA-256 hex digest of a token sequence for use as primary key. - Inputs: - `tokens` (tuple[int, ...]; required): Required positional or keyword input. - Return annotation: `str` - Calls: hashlib.sha256(_tokens_to_blob(tokens)).hexdigest, hashlib.sha256, _tokens_to_blob - Return expressions: hashlib.sha256(_tokens_to_blob(tokens)).hexdigest() ## `vllm_mlx.ssd_cache._prefix_hash` - Kind: function - Signature: `def _prefix_hash(tokens: tuple[int, ...]) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/ssd_cache.py#L147-L149 - Implementation: Function `_prefix_hash` calls `_tokens_hash`; returns `_tokens_hash(tokens[:_PREFIX_FILTER_TOKENS])`. Hash the bounded token prefix used to prefilter prefix lookups. - Inputs: - `tokens` (tuple[int, ...]; required): Required positional or keyword input. - Return annotation: `str` - Calls: _tokens_hash - Return expressions: _tokens_hash(tokens[:_PREFIX_FILTER_TOKENS]) ## `vllm_mlx.ssd_cache.SSDIndex` - Kind: class - Signature: `class SSDIndex` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/ssd_cache.py#L152-L405 - Implementation: Class `SSDIndex` declares 14 direct member(s). SQLite-backed index for SSD cache entries. Uses SQLite for atomic metadata operations instead of mutable JSON. The token sequence is stored as a binary blob for prefix-searchable representation. The primary key is a SHA-256 hash of the token sequence. Thread safety: All operations are serialized through a threading.Lock. The SQLite connection uses WAL mode for concurrent read/write safety. - Inputs: - `cache_dir` (str; required): Required positional or keyword input. - Constructs: `vllm_mlx.ssd_cache.SSDIndex` ## `vllm_mlx.ssd_cache.SSDIndex.__init__` - Kind: method - Signature: `def __init__(self, cache_dir: str) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/ssd_cache.py#L165-L173 - Implementation: Method `SSDIndex.__init__` updates `self._cache_dir`, `self._db_lock`, `self._conn`, `self._conn.row_factory`; calls `threading.Lock`, `os.path.join`, `sqlite3.connect`, `self._conn.execute`. Method `SSDIndex.__init__` updates `self._cache_dir`, `self._db_lock`, `self._conn`, `self._conn.row_factory`; calls `threading.Lock`, `os.path.join`, `sqlite3.connect`, `self._conn.execute`. - Inputs: - `cache_dir` (str; required): Required positional or keyword input. - Return annotation: `None` - Calls: threading.Lock, os.path.join, sqlite3.connect, self._conn.execute, self._create_tables - State reads: self._conn.execute, self._conn, self._create_tables - State writes: self._cache_dir, self._db_lock, self._conn, self._conn.row_factory ## `vllm_mlx.ssd_cache.SSDIndex._create_tables` - Kind: method - Signature: `def _create_tables(self) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/ssd_cache.py#L175-L213 - Implementation: Method `SSDIndex._create_tables` calls `self._conn.executescript`, `self._ensure_column`, `self._conn.execute`, `cur.fetchone`. Method `SSDIndex._create_tables` calls `self._conn.executescript`, `self._ensure_column`, `self._conn.execute`, `cur.fetchone`. - Inputs: none - Return annotation: `None` - Calls: self._conn.executescript, self._ensure_column, self._conn.execute, cur.fetchone, self._backfill_prefix_hashes, self._conn.commit - State reads: self._conn.executescript, self._conn, self._ensure_column, self._conn.execute, self._SCHEMA_VERSION, self._backfill_prefix_hashes, self._conn.commit ## `vllm_mlx.ssd_cache.SSDIndex._ensure_column` - Kind: method - Signature: `def _ensure_column(self, table: str, column: str, definition: str) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/ssd_cache.py#L215-L218 - Implementation: Method `SSDIndex._ensure_column` calls `self._conn.execute`, `cur.fetchall`. Method `SSDIndex._ensure_column` calls `self._conn.execute`, `cur.fetchall`. - Inputs: - `table` (str; required): Required positional or keyword input. - `column` (str; required): Required positional or keyword input. - `definition` (str; required): Required positional or keyword input. - Return annotation: `None` - Calls: self._conn.execute, cur.fetchall - State reads: self._conn.execute, self._conn ## `vllm_mlx.ssd_cache.SSDIndex._backfill_prefix_hashes` - Kind: method - Signature: `def _backfill_prefix_hashes(self) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/ssd_cache.py#L220-L230 - Implementation: Method `SSDIndex._backfill_prefix_hashes` calls `self._conn.execute`, `cur.fetchall`, `_blob_to_tokens`, `_prefix_hash`. Method `SSDIndex._backfill_prefix_hashes` calls `self._conn.execute`, `cur.fetchall`, `_blob_to_tokens`, `_prefix_hash`. - Inputs: none - Return annotation: `None` - Calls: self._conn.execute, cur.fetchall, _blob_to_tokens, _prefix_hash - State reads: self._conn.execute, self._conn ## `vllm_mlx.ssd_cache.SSDIndex.insert_entry` - Kind: method - Signature: `def insert_entry(self, tokens_key: tuple[int, ...], file_path: str, memory_bytes: int, num_tokens: int) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/ssd_cache.py#L232-L263 - Implementation: Method `SSDIndex.insert_entry` calls `time.time`, `_tokens_hash`, `_prefix_hash`, `_tokens_to_blob`. Insert or replace a cache entry in the index. - Inputs: - `tokens_key` (tuple[int, ...]; required): Required positional or keyword input. - `file_path` (str; required): Required positional or keyword input. - `memory_bytes` (int; required): Required positional or keyword input. - `num_tokens` (int; required): Required positional or keyword input. - Return annotation: `None` - Calls: time.time, _tokens_hash, _prefix_hash, _tokens_to_blob, self._conn.execute, self._conn.commit - State reads: self._db_lock, self._conn.execute, self._conn, self._conn.commit ## `vllm_mlx.ssd_cache.SSDIndex.lookup_exact` - Kind: method - Signature: `def lookup_exact(self, tokens_key: tuple[int, ...]) -> dict | None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/ssd_cache.py#L265-L280 - Implementation: Method `SSDIndex.lookup_exact` calls `_tokens_hash`, `self._conn.execute`, `cur.fetchone`; has 2 explicit return paths. Look up an exact token sequence. Returns dict or None. - Inputs: - `tokens_key` (tuple[int, ...]; required): Required positional or keyword input. - Return annotation: `dict | None` - Calls: _tokens_hash, self._conn.execute, cur.fetchone - State reads: self._db_lock, self._conn.execute, self._conn - Return expressions: None; {'file_path': row['file_path'], 'memory_bytes': row['memory_bytes'], 'num_tokens': row['num_tokens']} ## `vllm_mlx.ssd_cache.SSDIndex.lookup_prefix` - Kind: method - Signature: `def lookup_prefix(self, query_tokens: tuple[int, ...]) -> list[dict]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/ssd_cache.py#L282-L324 - Implementation: Method `SSDIndex.lookup_prefix` calls `len`, `_tokens_to_blob`, `_tokens_hash`, `range`; has 2 explicit return paths. Find entries whose token sequence is a prefix of query_tokens. Uses a bounded token-prefix hash to avoid scanning all entries, then compares the full stored token blob against the corresponding prefix of query_tokens. Returns list of dicts sorted by num_tokens descending (longest prefix first). - Inputs: - `query_tokens` (tuple[int, ...]; required): Required positional or keyword input. - Return annotation: `list[dict]` - Calls: len, _tokens_to_blob, _tokens_hash, range, min, ','.join, self._conn.execute, cur.fetchall, results.append - State reads: self._db_lock, self._conn.execute, self._conn - Return expressions: []; results ## `vllm_mlx.ssd_cache.SSDIndex.delete_entry` - Kind: method - Signature: `def delete_entry(self, tokens_key: tuple[int, ...]) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/ssd_cache.py#L326-L333 - Implementation: Method `SSDIndex.delete_entry` calls `_tokens_hash`, `self._conn.execute`, `self._conn.commit`. Delete an entry by token sequence. - Inputs: - `tokens_key` (tuple[int, ...]; required): Required positional or keyword input. - Return annotation: `None` - Calls: _tokens_hash, self._conn.execute, self._conn.commit - State reads: self._db_lock, self._conn.execute, self._conn, self._conn.commit ## `vllm_mlx.ssd_cache.SSDIndex.get_lru` - Kind: method - Signature: `def get_lru(self, limit: int=10) -> list[dict]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/ssd_cache.py#L335-L355 - Implementation: Method `SSDIndex.get_lru` calls `self._conn.execute`, `cur.fetchall`, `results.append`; returns `results`. Get the least recently used entries, ordered oldest first. - Inputs: - `limit` (int; optional; default `10`): Optional positional or keyword input; defaults to `10`. - Return annotation: `list[dict]` - Calls: self._conn.execute, cur.fetchall, results.append - State reads: self._db_lock, self._conn.execute, self._conn - Return expressions: results ## `vllm_mlx.ssd_cache.SSDIndex.get_total_bytes` - Kind: method - Signature: `def get_total_bytes(self) -> int` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/ssd_cache.py#L357-L363 - Implementation: Method `SSDIndex.get_total_bytes` calls `self._conn.execute`, `cur.fetchone`; returns `cur.fetchone()[0]`. Get total memory_bytes across all entries. - Inputs: none - Return annotation: `int` - Calls: self._conn.execute, cur.fetchone - State reads: self._db_lock, self._conn.execute, self._conn - Return expressions: cur.fetchone()[0] ## `vllm_mlx.ssd_cache.SSDIndex.get_entry_count` - Kind: method - Signature: `def get_entry_count(self) -> int` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/ssd_cache.py#L365-L369 - Implementation: Method `SSDIndex.get_entry_count` calls `self._conn.execute`, `cur.fetchone`; returns `cur.fetchone()[0]`. Get number of entries in the index. - Inputs: none - Return annotation: `int` - Calls: self._conn.execute, cur.fetchone - State reads: self._db_lock, self._conn.execute, self._conn - Return expressions: cur.fetchone()[0] ## `vllm_mlx.ssd_cache.SSDIndex.touch` - Kind: method - Signature: `def touch(self, tokens_key: tuple[int, ...]) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/ssd_cache.py#L371-L379 - Implementation: Method `SSDIndex.touch` calls `_tokens_hash`, `self._conn.execute`, `time.time`, `self._conn.commit`. Update accessed_at timestamp for an entry (marks as recently used). - Inputs: - `tokens_key` (tuple[int, ...]; required): Required positional or keyword input. - Return annotation: `None` - Calls: _tokens_hash, self._conn.execute, time.time, self._conn.commit - State reads: self._db_lock, self._conn.execute, self._conn, self._conn.commit ## `vllm_mlx.ssd_cache.SSDIndex.all_entries` - Kind: method - Signature: `def all_entries(self) -> list[dict]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/ssd_cache.py#L381-L400 - Implementation: Method `SSDIndex.all_entries` calls `self._conn.execute`, `cur.fetchall`, `results.append`; returns `results`. Return all entries (for startup reconciliation). - Inputs: none - Return annotation: `list[dict]` - Calls: self._conn.execute, cur.fetchall, results.append - State reads: self._db_lock, self._conn.execute, self._conn - Return expressions: results ## `vllm_mlx.ssd_cache.SSDIndex.close` - Kind: method - Signature: `def close(self) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/ssd_cache.py#L402-L405 - Implementation: Method `SSDIndex.close` calls `self._conn.close`. Close the SQLite connection. - Inputs: none - Return annotation: `None` - Calls: self._conn.close - State reads: self._db_lock, self._conn.close, self._conn ## `vllm_mlx.ssd_cache.LayerSerializer` - Kind: class - Signature: `class LayerSerializer(ABC)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/ssd_cache.py#L419-L446 - Implementation: Class `LayerSerializer` derives from `ABC` and declares 3 direct member(s). Interface for per-layer cache serialization. Spill is split across two threads: ``snapshot_layer`` runs on the producer (request handler) thread so the mx→numpy materialization happens where the per-request Stream(gpu, N) is registered; ``serialize_layer`` then runs on the SSD writer thread with numpy only. - Inputs: none - Constructs: `vllm_mlx.ssd_cache.LayerSerializer` ## `vllm_mlx.ssd_cache.LayerSerializer.snapshot_layer` - Kind: method - Signature: `def snapshot_layer(self, layer: Any) -> dict[str, Any]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/ssd_cache.py#L429-L431 - Implementation: Method `LayerSerializer.snapshot_layer` contains no state mutation, call, raise, return, await, or yield. Producer-thread CPU snapshot of an MLX-backed cache layer. - Inputs: - `layer` (Any; required): Required positional or keyword input. - Return annotation: `dict[str, Any]` - Decorators: abstractmethod ## `vllm_mlx.ssd_cache.LayerSerializer.serialize_layer` - Kind: method - Signature: `def serialize_layer(self, snapshot: dict[str, Any], layer_idx: int, file_path: str) -> dict[str, Any]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/ssd_cache.py#L434-L441 - Implementation: Method `LayerSerializer.serialize_layer` contains no state mutation, call, raise, return, await, or yield. Writer-thread: persist a snapshot to safetensors at file_path. Returns metadata dict with at least 'layer_type'. - Inputs: - `snapshot` (dict[str, Any]; required): Required positional or keyword input. - `layer_idx` (int; required): Required positional or keyword input. - `file_path` (str; required): Required positional or keyword input. - Return annotation: `dict[str, Any]` - Decorators: abstractmethod ## `vllm_mlx.ssd_cache.LayerSerializer.deserialize_layer` - Kind: method - Signature: `def deserialize_layer(self, file_path: str, metadata: dict[str, Any]) -> dict` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/ssd_cache.py#L444-L446 - Implementation: Method `LayerSerializer.deserialize_layer` contains no state mutation, call, raise, return, await, or yield. Read a layer back from disk. Returns layer-state dict. - Inputs: - `file_path` (str; required): Required positional or keyword input. - `metadata` (dict[str, Any]; required): Required positional or keyword input. - Return annotation: `dict` - Decorators: abstractmethod ## `vllm_mlx.ssd_cache._mx_to_numpy_safe` - Kind: function - Signature: `def _mx_to_numpy_safe(arr: Any) -> tuple[np.ndarray, str | None]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/ssd_cache.py#L449-L467 - Implementation: Function `_mx_to_numpy_safe` calls `np.array`, `str`, `str(arr.dtype).rsplit`, `arr.astype`; has 2 explicit return paths. mx.array → np.ndarray, upcasting numpy-unsupported dtypes (bf16) to fp32. Returns (numpy_array, original_dtype_name_or_None). The name is only set when an upcast happened, so the SSD-promote path can cast back. - Inputs: - `arr` (Any; required): Required positional or keyword input. - Return annotation: `tuple[np.ndarray, str | None]` - Calls: np.array, str, str(arr.dtype).rsplit, arr.astype, mx.eval - Return expressions: (np.array(arr), None); (np.array(upcast), original_dtype) ## `vllm_mlx.ssd_cache.KVCacheSerializer` - Kind: class - Signature: `class KVCacheSerializer(LayerSerializer)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/ssd_cache.py#L470-L564 - Implementation: Class `KVCacheSerializer` derives from `LayerSerializer` and declares 3 direct member(s). Serializer for KVCache and RotatingKVCache layers. Handles layers with .keys, .values, .offset attributes. RotatingKVCache also has .max_size, .keep, .step, ._idx. - Inputs: none - Constructs: `vllm_mlx.ssd_cache.KVCacheSerializer` ## `vllm_mlx.ssd_cache.KVCacheSerializer.snapshot_layer` - Kind: method - Signature: `def snapshot_layer(self, layer: Any) -> dict[str, Any]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/ssd_cache.py#L481-L515 - Implementation: Method `KVCacheSerializer.snapshot_layer` calls `_mx_to_numpy_safe`, `getattr`, `hasattr`; returns `snapshot`. Copy a KV cache layer into NumPy-backed writer-thread data. - Inputs: - `layer` (Any; required): Required positional or keyword input. - Return annotation: `dict[str, Any]` - Calls: _mx_to_numpy_safe, getattr, hasattr - State reads: self._ROTATING_ATTRS - Return expressions: snapshot ## `vllm_mlx.ssd_cache.KVCacheSerializer.serialize_layer` - Kind: method - Signature: `def serialize_layer(self, snapshot: dict[str, Any], layer_idx: int, file_path: str) -> dict[str, Any]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/ssd_cache.py#L517-L542 - Implementation: Method `KVCacheSerializer.serialize_layer` calls `save_file`; returns `metadata`. Write one KV layer to safetensors and return reconstruction metadata. - Inputs: - `snapshot` (dict[str, Any]; required): Required positional or keyword input. - `layer_idx` (int; required): Required positional or keyword input. - `file_path` (str; required): Required positional or keyword input. - Return annotation: `dict[str, Any]` - Calls: save_file - State reads: self._ROTATING_ATTRS - Return expressions: metadata ## `vllm_mlx.ssd_cache.KVCacheSerializer.deserialize_layer` - Kind: method - Signature: `def deserialize_layer(self, file_path: str, metadata: dict[str, Any]) -> dict` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/ssd_cache.py#L544-L564 - Implementation: Method `KVCacheSerializer.deserialize_layer` calls `load_file`; returns `result`. Load one KV layer as arrays plus cache reconstruction metadata. - Inputs: - `file_path` (str; required): Required positional or keyword input. - `metadata` (dict[str, Any]; required): Required positional or keyword input. - Return annotation: `dict` - Calls: load_file - State reads: self._ROTATING_ATTRS - Return expressions: result ## `vllm_mlx.ssd_cache.ArraysCacheSerializer` - Kind: class - Signature: `class ArraysCacheSerializer(LayerSerializer)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/ssd_cache.py#L567-L627 - Implementation: Class `ArraysCacheSerializer` derives from `LayerSerializer` and declares 3 direct member(s). Serializer for ArraysCache (Mamba/linear attention) layers. Handles layers with .state attribute containing a list of arrays. - Inputs: none - Constructs: `vllm_mlx.ssd_cache.ArraysCacheSerializer` ## `vllm_mlx.ssd_cache.ArraysCacheSerializer.snapshot_layer` - Kind: method - Signature: `def snapshot_layer(self, layer: Any) -> dict[str, Any]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/ssd_cache.py#L573-L587 - Implementation: Method `ArraysCacheSerializer.snapshot_layer` calls `_mx_to_numpy_safe`, `state_np.append`, `original_dtypes.append`, `any`; returns `snapshot`. Copy an arrays-cache state into NumPy-backed writer-thread data. - Inputs: - `layer` (Any; required): Required positional or keyword input. - Return annotation: `dict[str, Any]` - Calls: _mx_to_numpy_safe, state_np.append, original_dtypes.append, any - Return expressions: snapshot ## `vllm_mlx.ssd_cache.ArraysCacheSerializer.serialize_layer` - Kind: method - Signature: `def serialize_layer(self, snapshot: dict[str, Any], layer_idx: int, file_path: str) -> dict[str, Any]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/ssd_cache.py#L589-L610 - Implementation: Method `ArraysCacheSerializer.serialize_layer` calls `enumerate`, `save_file`, `len`; returns `metadata`. Write arrays-cache state to safetensors and return its metadata. - Inputs: - `snapshot` (dict[str, Any]; required): Required positional or keyword input. - `layer_idx` (int; required): Required positional or keyword input. - `file_path` (str; required): Required positional or keyword input. - Return annotation: `dict[str, Any]` - Calls: enumerate, save_file, len - Return expressions: metadata ## `vllm_mlx.ssd_cache.ArraysCacheSerializer.deserialize_layer` - Kind: method - Signature: `def deserialize_layer(self, file_path: str, metadata: dict[str, Any]) -> dict` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/ssd_cache.py#L612-L627 - Implementation: Method `ArraysCacheSerializer.deserialize_layer` calls `load_file`, `range`, `state.append`; returns `result`. Load arrays-cache state and any original dtype hints. - Inputs: - `file_path` (str; required): Required positional or keyword input. - `metadata` (dict[str, Any]; required): Required positional or keyword input. - Return annotation: `dict` - Calls: load_file, range, state.append - Return expressions: result ## `vllm_mlx.ssd_cache.get_serializer_for_layer` - Kind: function - Signature: `def get_serializer_for_layer(layer: Any) -> LayerSerializer` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/ssd_cache.py#L630-L646 - Implementation: Function `get_serializer_for_layer` calls `hasattr`, `KVCacheSerializer`, `isinstance`, `getattr`; can raise `ValueError`; has 2 explicit return paths. Return the appropriate serializer for a cache layer. Dispatches based on duck-typing: - If layer has .keys and .values and .offset -> KVCacheSerializer - If layer has .state and it's a list -> ArraysCacheSerializer Raises ValueError for unsupported layer types. - Inputs: - `layer` (Any; required): Required positional or keyword input. - Return annotation: `LayerSerializer` - Calls: hasattr, KVCacheSerializer, isinstance, getattr, ArraysCacheSerializer, ValueError, type, list, SERIALIZER_SUPPORT_MATRIX.keys - Raises directly: ValueError - Return expressions: KVCacheSerializer(); ArraysCacheSerializer() ## `vllm_mlx.ssd_cache.SSDCacheTier` - Kind: class - Signature: `class SSDCacheTier` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/ssd_cache.py#L649-L1248 - Implementation: Class `SSDCacheTier` declares 15 direct member(s). Cold-tier disk cache for KV cache entries. Manages a SQLite-indexed on-disk cache directory. Evicted RAM entries are spilled here via an async writer thread. Cold-tier fetches reload from disk asynchronously with RAM budget reservation. Directory layout:: cache_dir/ index.db # SQLite metadata index data/ # safetensors files per entry {hash}/ # one directory per entry layer_0.safetensors layer_1.safetensors manifest.json # per-entry layer metadata - Inputs: - `config` (SSDCacheConfig; required): Required positional or keyword input. - Constructs: `vllm_mlx.ssd_cache.SSDCacheTier` ## `vllm_mlx.ssd_cache.SSDCacheTier.__init__` - Kind: method - Signature: `def __init__(self, config: SSDCacheConfig) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/ssd_cache.py#L667-L705 - Implementation: Method `SSDCacheTier.__init__` updates `self._config`, `self._closed`, `self._writer_thread`, `self._cache_dir`; calls `ValueError`, `os.path.join`, `os.makedirs`, `SSDIndex`; can raise `ValueError`. Method `SSDCacheTier.__init__` updates `self._config`, `self._closed`, `self._writer_thread`, `self._cache_dir`; calls `ValueError`, `os.path.join`, `os.makedirs`, `SSDIndex`; can raise `ValueError`. - Inputs: - `config` (SSDCacheConfig; required): Required positional or keyword input. - Return annotation: `None` - Calls: ValueError, os.path.join, os.makedirs, SSDIndex, SSDCacheStats, threading.Lock, queue.Queue, threading.Event, getattr, index.close, logger.exception - State reads: self._cache_dir, self._data_dir - State writes: self._config, self._closed, self._writer_thread, self._cache_dir, self._data_dir, self._index, self._stats, self._lock, self._spill_queue, self._writer_stop - Raises directly: ValueError ## `vllm_mlx.ssd_cache.SSDCacheTier._entry_hash` - Kind: method - Signature: `def _entry_hash(tokens: tuple[int, ...]) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/ssd_cache.py#L708-L710 - Implementation: Method `SSDCacheTier._entry_hash` calls `_tokens_hash`; returns `_tokens_hash(tokens)`. Compute deterministic hash for a token sequence. - Inputs: - `tokens` (tuple[int, ...]; required): Required positional or keyword input. - Return annotation: `str` - Decorators: staticmethod - Calls: _tokens_hash - Return expressions: _tokens_hash(tokens) ## `vllm_mlx.ssd_cache.SSDCacheTier.get_stats` - Kind: method - Signature: `def get_stats(self) -> dict` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/ssd_cache.py#L712-L714 - Implementation: Method `SSDCacheTier.get_stats` calls `self._stats.to_dict`; returns `self._stats.to_dict()`. Return current SSD cache statistics. - Inputs: none - Return annotation: `dict` - Calls: self._stats.to_dict - State reads: self._stats.to_dict, self._stats - Return expressions: self._stats.to_dict() ## `vllm_mlx.ssd_cache.SSDCacheTier.start_writer` - Kind: method - Signature: `def start_writer(self) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/ssd_cache.py#L716-L725 - Implementation: Method `SSDCacheTier.start_writer` updates `self._writer_thread`; calls `self._writer_stop.clear`, `threading.Thread`, `self._writer_thread.start`, `logger.info`; returns `None`. Start the background spill writer thread. - Inputs: none - Return annotation: `None` - Calls: self._writer_stop.clear, threading.Thread, self._writer_thread.start, logger.info - State reads: self._writer_thread, self._writer_stop.clear, self._writer_stop, self._writer_loop, self._writer_thread.start - State writes: self._writer_thread - Return expressions: None ## `vllm_mlx.ssd_cache.SSDCacheTier._writer_loop` - Kind: method - Signature: `def _writer_loop(self) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/ssd_cache.py#L727-L744 - Implementation: Method `SSDCacheTier._writer_loop` calls `self._writer_stop.is_set`, `self._spill_queue.get`, `self._write_entry`, `logger.exception`. Drain spill queue and persist entries. Numpy-only — no MLX here. - Inputs: none - Return annotation: `None` - Calls: self._writer_stop.is_set, self._spill_queue.get, self._write_entry, logger.exception, len - State reads: self._writer_stop.is_set, self._writer_stop, self._spill_queue.get, self._spill_queue, self._write_entry ## `vllm_mlx.ssd_cache.SSDCacheTier.enqueue_spill` - Kind: method - Signature: `def enqueue_spill(self, tokens: tuple[int, ...], cache: list[Any], memory_bytes: int) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/ssd_cache.py#L746-L867 - Implementation: Method `SSDCacheTier.enqueue_spill` calls `any`, `_is_quantized_layer`, `isinstance`, `converted.extend`; has 2 explicit return paths. Enqueue a cache entry for async spill to SSD. Must be called on the producer thread (the request handler that owns the layer's Stream(gpu, N)) — the snapshot below materializes MLX → numpy here so the writer thread never has to. Returns True if enqueued, False if queue is full (entry dropped). - Inputs: - `tokens` (tuple[int, ...]; required): Required positional or keyword input. - `cache` (list[Any]; required): Required positional or keyword input. - `memory_bytes` (int; required): Required positional or keyword input. - Return annotation: `bool` - Calls: any, _is_quantized_layer, isinstance, converted.extend, _dequantize_cache, hasattr, getattr, _KVCache.__new__, mx.dequantize, len, converted.append, str(getattr(k, 'dtype', '')).endswith, str, k.astype, v.astype, mx.eval, logger.info, get_serializer_for_layer, serializer.snapshot_layer, layer_snapshots.append, logger.exception, self._spill_queue.put_nowait, logger.warning - State reads: self._spill_queue.put_nowait, self._spill_queue - Return expressions: False; True ## `vllm_mlx.ssd_cache.SSDCacheTier.enqueue_spill._is_quantized_layer` - Kind: nested function - Signature: `def _is_quantized_layer(layer)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/ssd_cache.py#L772-L776 - Implementation: Nested Function `SSDCacheTier.enqueue_spill._is_quantized_layer` calls `isinstance`, `getattr`; has 2 explicit return paths. Nested Function `SSDCacheTier.enqueue_spill._is_quantized_layer` calls `isinstance`, `getattr`; has 2 explicit return paths. - Inputs: - `layer` (not annotated; required): Required positional or keyword input. - Return annotation: `not annotated` - Calls: isinstance, getattr - Return expressions: True; isinstance(keys, (tuple, list)) ## `vllm_mlx.ssd_cache.SSDCacheTier._write_entry` - Kind: method - Signature: `def _write_entry(self, tokens_key: tuple[int, ...], layer_snapshots: list[tuple[LayerSerializer, dict[str, Any]]], memory_bytes: int) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/ssd_cache.py#L869-L944 - Implementation: Method `SSDCacheTier._write_entry` updates `self._stats.spill_count`, `self._stats.spill_bytes`; calls `self._entry_hash`, `os.path.join`, `os.path.exists`, `shutil.rmtree`. Atomically persist one entry (writer thread; numpy-only input). - Inputs: - `tokens_key` (tuple[int, ...]; required): Required positional or keyword input. - `layer_snapshots` (list[tuple[LayerSerializer, dict[str, Any]]]; required): Required positional or keyword input. - `memory_bytes` (int; required): Required positional or keyword input. - Return annotation: `None` - Calls: self._entry_hash, os.path.join, os.path.exists, shutil.rmtree, os.makedirs, enumerate, serializer.serialize_layer, layer_manifests.append, os.chmod, os.path.getsize, len, open, json.dump, _array.array, arr.tofile, os.rename, self._index.insert_entry, logger.debug, self._enforce_capacity - State reads: self._entry_hash, self._data_dir, self._config.dir_permissions, self._config, self._config.file_permissions, self._index.insert_entry, self._index, self._lock, self._stats, self._enforce_capacity - State writes: self._stats.spill_count, self._stats.spill_bytes ## `vllm_mlx.ssd_cache.SSDCacheTier.lookup_ssd` - Kind: method - Signature: `def lookup_ssd(self, tokens: tuple[int, ...]) -> dict | None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/ssd_cache.py#L946-L958 - Implementation: Method `SSDCacheTier.lookup_ssd` calls `self._index.lookup_exact`; has 2 explicit return paths. Synchronous check whether tokens exist in SSD tier. This is fast (SQLite lookup only, no disk I/O for data). Called from synchronous fetch() to report an SSD candidate. Returns: Dict with entry metadata if found, None otherwise. - Inputs: - `tokens` (tuple[int, ...]; required): Required positional or keyword input. - Return annotation: `dict | None` - Calls: self._index.lookup_exact - State reads: self._index.lookup_exact, self._index - Return expressions: result; None ## `vllm_mlx.ssd_cache.SSDCacheTier.lookup_ssd_prefix` - Kind: method - Signature: `def lookup_ssd_prefix(self, tokens: tuple[int, ...]) -> dict | None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/ssd_cache.py#L960-L968 - Implementation: Method `SSDCacheTier.lookup_ssd_prefix` calls `self._index.lookup_prefix`; has 2 explicit return paths. Find the longest prefix match in the SSD tier. Returns the longest-prefix entry metadata or None. - Inputs: - `tokens` (tuple[int, ...]; required): Required positional or keyword input. - Return annotation: `dict | None` - Calls: self._index.lookup_prefix - State reads: self._index.lookup_prefix, self._index - Return expressions: results[0]; None ## `vllm_mlx.ssd_cache.SSDCacheTier.async_promote` - Kind: method - Signature: `async def async_promote(self, tokens: tuple[int, ...], reserve_budget_fn, release_budget_fn) -> list | None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/ssd_cache.py#L970-L1076 - Implementation: Method `SSDCacheTier.async_promote` updates `self._stats.ssd_misses`, `self._stats.promotion_failures`, `self._stats.ssd_hits`, `self._stats.reload_latency_sum`; calls `self._index.lookup_exact`, `reserve_budget_fn`, `logger.warning`, `time.time`; awaits asynchronous work; has 2 explicit return paths. Promote an entry from SSD to RAM asynchronously. CRITICAL: Reserves RAM budget BEFORE the disk read, to avoid thrash when multiple promotions race. Args: tokens: Token sequence to promote. reserve_budget_fn: Callable(nbytes) -> bool. Must return True if budget is available and reserved, False otherwise. release_budget_fn: Callable(nbytes) -> None. Called to release budget on failure. Returns: List of deserialized cache layers, or None if promotion failed. - Inputs: - `tokens` (tuple[int, ...]; required): Token sequence to promote. - `reserve_budget_fn` (not annotated; required): Callable(nbytes) -> bool. Must return True if budget is available and reserved, False otherwise. - `release_budget_fn` (not annotated; required): Callable(nbytes) -> None. Called to release budget on failure. - Return annotation: `list | None` - Calls: self._index.lookup_exact, reserve_budget_fn, logger.warning, time.time, asyncio.ensure_future, asyncio.to_thread, asyncio.shield, release_budget_fn, logger.exception, sum, os.path.getsize, os.path.join, range, len, os.path.exists, self._index.touch, logger.info - State reads: self._index.lookup_exact, self._index, self._lock, self._stats, self._read_entry, self._data_dir, self._index.touch - State writes: self._stats.ssd_misses, self._stats.promotion_failures, self._stats.ssd_hits, self._stats.reload_latency_sum, self._stats.reload_bytes - Return expressions: None; cache_layers ## `vllm_mlx.ssd_cache.SSDCacheTier._read_entry` - Kind: method - Signature: `def _read_entry(self, tokens: tuple[int, ...], relative_path: str) -> list | None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/ssd_cache.py#L1078-L1121 - Implementation: Method `SSDCacheTier._read_entry` calls `os.path.join`, `open`, `json.load`, `logger.warning`; has 2 explicit return paths. Read a cache entry from disk. Called from thread pool. Returns list of deserialized layer dicts, or None on corruption. - Inputs: - `tokens` (tuple[int, ...]; required): Required positional or keyword input. - `relative_path` (str; required): Required positional or keyword input. - Return annotation: `list | None` - Calls: os.path.join, open, json.load, logger.warning, self._quarantine_entry, KVCacheSerializer, ArraysCacheSerializer, serializer.deserialize_layer, cache_layers.append - State reads: self._data_dir, self._quarantine_entry - Return expressions: None; cache_layers ## `vllm_mlx.ssd_cache.SSDCacheTier._quarantine_entry` - Kind: method - Signature: `def _quarantine_entry(self, tokens: tuple[int, ...], relative_path: str) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/ssd_cache.py#L1123-L1142 - Implementation: Method `SSDCacheTier._quarantine_entry` calls `os.path.join`, `os.path.exists`, `os.makedirs`, `os.path.dirname`. Move a corrupt entry to quarantine and remove from index. - Inputs: - `tokens` (tuple[int, ...]; required): Required positional or keyword input. - `relative_path` (str; required): Required positional or keyword input. - Return annotation: `None` - Calls: os.path.join, os.path.exists, os.makedirs, os.path.dirname, os.rename, logger.warning, self._index.delete_entry - State reads: self._data_dir, self._cache_dir, self._config.dir_permissions, self._config, self._index.delete_entry, self._index ## `vllm_mlx.ssd_cache.SSDCacheTier._enforce_capacity` - Kind: method - Signature: `def _enforce_capacity(self) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/ssd_cache.py#L1144-L1181 - Implementation: Method `SSDCacheTier._enforce_capacity` calls `self._index.get_entry_count`, `self._index.get_total_bytes`, `self._index.get_lru`, `_blob_to_tokens`. Evict oldest SSD entries until within capacity limits. Called after each spill write. Removes entries by LRU order until both entry count and total bytes are within bounds. - Inputs: none - Return annotation: `None` - Calls: self._index.get_entry_count, self._index.get_total_bytes, self._index.get_lru, _blob_to_tokens, os.path.join, os.path.exists, shutil.rmtree, self._index.delete_entry, logger.debug - State reads: self._index.get_entry_count, self._index, self._index.get_total_bytes, self._config.max_entries, self._config, self._config.max_size_bytes, self._index.get_lru, self._data_dir, self._index.delete_entry ## `vllm_mlx.ssd_cache.SSDCacheTier.reconcile` - Kind: method - Signature: `def reconcile(self) -> int` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/ssd_cache.py#L1183-L1229 - Implementation: Method `SSDCacheTier.reconcile` calls `self._index.all_entries`, `os.path.join`, `os.path.isdir`, `os.path.exists`; returns `cleaned`. Reconcile index with files on disk. Removes index entries whose data files are missing. Removes data directories not in the index. Returns number of entries cleaned up. - Inputs: none - Return annotation: `int` - Calls: self._index.all_entries, os.path.join, os.path.isdir, os.path.exists, _blob_to_tokens, self._index.delete_entry, logger.info, os.listdir, entry_name.endswith, shutil.rmtree - State reads: self._index.all_entries, self._index, self._data_dir, self._index.delete_entry - Return expressions: cleaned ## `vllm_mlx.ssd_cache.SSDCacheTier.close` - Kind: method - Signature: `def close(self) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/ssd_cache.py#L1231-L1248 - Implementation: Method `SSDCacheTier.close` updates `self._closed`, `self._writer_thread`; calls `self._writer_stop.set`, `self._spill_queue.put_nowait`, `self._writer_thread.join`, `self._index.close`; returns `None`. Close the SSD cache tier and release resources. - Inputs: none - Return annotation: `None` - Calls: self._writer_stop.set, self._spill_queue.put_nowait, self._writer_thread.join, self._index.close, logger.info - State reads: self._closed, self._writer_stop.set, self._writer_stop, self._writer_thread, self._spill_queue.put_nowait, self._spill_queue, self._writer_thread.join, self._index.close, self._index - State writes: self._closed, self._writer_thread - Return expressions: None # Module `vllm_mlx.text_model_from_vlm` Construct an mlx_lm TextModel from mlx_vlm-loaded model weights. When mlx_vlm loads a model, it strips MTP weights in sanitize(). This module builds a parallel mlx_lm TextModel that: 1. Shares backbone + lm_head weights with the vlm model (zero-copy) 2. Loads MTP weights from safetensors on disk 3. Provides full mlx_lm API: return_hidden, n_confirmed, mtp_forward, make_mtp_cache Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/text_model_from_vlm.py#L1-L272 ## `vllm_mlx.text_model_from_vlm._import_text_model_classes` - Kind: function - Signature: `def _import_text_model_classes(model_type: str)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/text_model_from_vlm.py#L41-L66 - Implementation: Function `_import_text_model_classes` calls `sorted`, `model_type.startswith`, `logger.debug`, `importlib.import_module`; returns `(getattr(module, model_attr), getattr(module, args_attr))`. Return ``(Model, ModelArgs)`` for a text config's ``model_type``. - Inputs: - `model_type` (str; required): Required positional or keyword input. - Return annotation: `not annotated` - Calls: sorted, model_type.startswith, logger.debug, importlib.import_module, getattr - Return expressions: (getattr(module, model_attr), getattr(module, args_attr)) ## `vllm_mlx.text_model_from_vlm.build_text_model` - Kind: function - Signature: `def build_text_model(vlm_model: Any, model_path: str | Path) -> Any | None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/text_model_from_vlm.py#L69-L225 - Implementation: Function `build_text_model` calls `Path`, `(model_path / 'config.json').exists`, `json.loads`, `(model_path / 'config.json').read_text`; has 2 explicit return paths. Build an mlx_lm TextModel from a vlm-loaded model's weights. Args: vlm_model: The mlx_vlm-loaded model (has .language_model attribute) model_path: Path to the model directory (contains config.json + safetensors) Returns: mlx_lm TextModel with MTP support, or None on failure. - Inputs: - `vlm_model` (Any; required): The mlx_vlm-loaded model (has .language_model attribute) - `model_path` (str | Path; required): Path to the model directory (contains config.json + safetensors) - Return annotation: `Any | None` - Calls: Path, (model_path / 'config.json').exists, json.loads, (model_path / 'config.json').read_text, config.get, text_config.get, _import_text_model_classes, logger.debug, TextModelArgs.from_dict, TextModel, mlx.utils.tree_flatten, vlm_lm.parameters, _load_mtp_weights, set, all_weight_names.update, quantization.items, isinstance, nn.quantize, quantization.get, text_model.load_weights, logger.info, len, logger.warning, hasattr, inject_mtp_support, mx.eval, text_model.mtp.parameters, text_model.train, text_model.modules, module.values, logger.error - Return expressions: None; text_model ## `vllm_mlx.text_model_from_vlm.build_text_model._class_predicate` - Kind: nested function - Signature: `def _class_predicate(path, module)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/text_model_from_vlm.py#L127-L137 - Implementation: Nested Function `build_text_model._class_predicate` calls `hasattr`, `per_layer_overrides.items`, `key.endswith`; has 4 explicit return paths. Nested Function `build_text_model._class_predicate` calls `hasattr`, `per_layer_overrides.items`, `key.endswith`; has 4 explicit return paths. - Inputs: - `path` (not annotated; required): Required positional or keyword input. - `module` (not annotated; required): Required positional or keyword input. - Return annotation: `not annotated` - Calls: hasattr, per_layer_overrides.items, key.endswith - Return expressions: False; quantization[path]; override; True ## `vllm_mlx.text_model_from_vlm._load_mtp_weights` - Kind: function - Signature: `def _load_mtp_weights(model_path: Path) -> list[tuple[str, mx.array]]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/text_model_from_vlm.py#L228-L272 - Implementation: Function `_load_mtp_weights` calls `index_file.exists`, `json.loads`, `index_file.read_text`, `index.get`; has 2 explicit return paths. Load MTP weights from safetensors, stripping the language_model. prefix. mlx_vlm's sanitize() strips mtp.* keys during model loading, but the weights are still on disk in the safetensors files. - Inputs: - `model_path` (Path; required): Required positional or keyword input. - Return annotation: `list[tuple[str, mx.array]]` - Calls: index_file.exists, json.loads, index_file.read_text, index.get, weight_map.items, key.startswith, key.replace, mtp_keys.items, shards.setdefault(shard, []).append, shards.setdefault, shards.items, shard_path.exists, logger.warning, mx.load, str, weights.append - Return expressions: []; weights # Module `vllm_mlx.tool_parsers` Tool call parsers for vllm-mlx. This module provides tool call parsing functionality for various model formats. Inspired by vLLM's tool parser architecture but simplified for MLX backend. Available parsers: - auto: Auto-detecting parser that tries all formats (default) - mistral: Mistral models ([TOOL_CALLS] format) - qwen/qwen3: Qwen models ( and [Calling tool:] formats) - llama/llama3/llama4: Llama models ( format) - gemma4/gemma_4: Google Gemma 4 models (<|tool_call>call:name{} format) - hermes/nous: Hermes/NousResearch models - deepseek/deepseek_v3/deepseek_r1: DeepSeek models (unicode tokens) - kimi/kimi_k2/moonshot: Kimi/Moonshot models - granite/granite3: IBM Granite models - nemotron/nemotron3: NVIDIA Nemotron models - xlam: Salesforce xLAM models - functionary/meetkai: MeetKai Functionary models - glm47/glm4: GLM-4.7 and GLM-4.7-Flash models - harmony/gpt-oss: GPT-OSS models (Harmony format with channels) - minimax: MiniMax-M2 models Usage: from vllm_mlx.tool_parsers import ToolParserManager # Get a parser by name parser_cls = ToolParserManager.get_tool_parser("mistral") parser = parser_cls(tokenizer) # Parse tool calls result = parser.extract_tool_calls(model_output) if result.tools_called: for tc in result.tool_calls: print(f"Tool: {tc['name']}, Args: {tc['arguments']}") # List available parsers print(ToolParserManager.list_registered()) Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/__init__.py#L1-L116 ## `vllm_mlx.tool_parsers.get_parser_stop_tokens` - Kind: function - Signature: `def get_parser_stop_tokens(parser_name: str | None, user_stops: list[str] | None) -> list[str]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/__init__.py#L68-L88 - Implementation: Function `get_parser_stop_tokens` calls `list`, `ToolParserManager.get_tool_parser`, `getattr`, `stops.append`; returns `stops`. Merge user-supplied stops with parser-declared extras (deduped). Some models declare end-of-generation tokens beyond the tokenizer's default eos set — e.g. Gemma 4's ``<|tool_response>`` which signals the runtime's turn after a tool call. Parsers expose those via ``extra_stop_tokens``. - Inputs: - `parser_name` (str | None; required): Required positional or keyword input. - `user_stops` (list[str] | None; required): Required positional or keyword input. - Return annotation: `list[str]` - Calls: list, ToolParserManager.get_tool_parser, getattr, stops.append - Return expressions: stops # Module `vllm_mlx.tool_parsers.abstract_tool_parser` Abstract tool parser base class and manager for vllm-mlx. Inspired by vLLM's tool parser architecture but simplified for MLX backend. Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/abstract_tool_parser.py#L1-L286 ## `vllm_mlx.tool_parsers.abstract_tool_parser.ExtractedToolCallInformation` - Kind: class - Signature: `class ExtractedToolCallInformation` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/abstract_tool_parser.py#L27-L37 - Implementation: Class `ExtractedToolCallInformation` declares 0 direct member(s). Information extracted from model output about tool calls. - Inputs: - `tools_called` (bool; required): Required constructor field. - `tool_calls` (list[dict[str, Any]]; required): Required constructor field. - `content` (str | None; optional; default `None`): Optional constructor field; defaults to `None`. - Constructs: `vllm_mlx.tool_parsers.abstract_tool_parser.ExtractedToolCallInformation` - Decorators: dataclass ## `vllm_mlx.tool_parsers.abstract_tool_parser.ToolParser` - Kind: class - Signature: `class ToolParser(ABC)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/abstract_tool_parser.py#L40-L171 - Implementation: Class `ToolParser` derives from `ABC` and declares 7 direct member(s). Abstract base class for tool call parsers. Each parser implementation handles a specific model's tool calling format. - Inputs: - `tokenizer` (PreTrainedTokenizerBase | None; optional; default `None`): The tokenizer for the model (optional, some parsers need it) - Constructs: `vllm_mlx.tool_parsers.abstract_tool_parser.ToolParser` ## `vllm_mlx.tool_parsers.abstract_tool_parser.ToolParser.supports_native_format` - Kind: method - Signature: `def supports_native_format(cls) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/abstract_tool_parser.py#L60-L72 - Implementation: Method `ToolParser.supports_native_format` returns `cls.SUPPORTS_NATIVE_TOOL_FORMAT`. Check if this parser supports native tool message format. Native format means the parser's corresponding model chat template can handle: - role="tool" messages directly (not converted to role="user") - tool_calls field on assistant messages (not converted to text) Returns: True if native format is supported - Inputs: none - Return annotation: `bool` - Decorators: classmethod - State reads: cls.SUPPORTS_NATIVE_TOOL_FORMAT - Return expressions: cls.SUPPORTS_NATIVE_TOOL_FORMAT ## `vllm_mlx.tool_parsers.abstract_tool_parser.ToolParser.strip_think_tags` - Kind: method - Signature: `def strip_think_tags(text: str) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/abstract_tool_parser.py#L75-L101 - Implementation: Method `ToolParser.strip_think_tags` calls `THINK_TAG_PATTERN.sub`, `IMPLICIT_THINK_PATTERN.sub`, `result.strip`; returns `result.strip()`. Strip think tags from text. Handles two scenarios: 1. Full tags: ... in output 2. Only closing tag: ... when was in prompt Used as fallback when no reasoning parser is configured but the model produces thinking tags. This prevents tool parsing failures with models that use thinking tags (e.g., Ring-Mini-Linear-2.0 with hermes). Args: text: Model output that may contain think tags Returns: Text with think tags removed - Inputs: - `text` (str; required): Model output that may contain think tags - Return annotation: `str` - Decorators: staticmethod - Calls: THINK_TAG_PATTERN.sub, IMPLICIT_THINK_PATTERN.sub, result.strip - Return expressions: result.strip() ## `vllm_mlx.tool_parsers.abstract_tool_parser.ToolParser.__init__` - Kind: method - Signature: `def __init__(self, tokenizer: PreTrainedTokenizerBase | None=None)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/abstract_tool_parser.py#L103-L113 - Implementation: Method `ToolParser.__init__` updates `self.model_tokenizer`, `self.current_tool_id`, `self.prev_tool_call_arr`. Initialize the tool parser. Args: tokenizer: The tokenizer for the model (optional, some parsers need it) - Inputs: - `tokenizer` (PreTrainedTokenizerBase | None; optional; default `None`): The tokenizer for the model (optional, some parsers need it) - Return annotation: `not annotated` - State writes: self.model_tokenizer, self.current_tool_id, self.prev_tool_call_arr ## `vllm_mlx.tool_parsers.abstract_tool_parser.ToolParser.vocab` - Kind: method - Signature: `def vocab(self) -> dict[str, int]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/abstract_tool_parser.py#L116-L120 - Implementation: Method `ToolParser.vocab` calls `self.model_tokenizer.get_vocab`; has 2 explicit return paths. Get the tokenizer vocabulary. - Inputs: none - Return annotation: `dict[str, int]` - Decorators: cached_property - Calls: self.model_tokenizer.get_vocab - State reads: self.model_tokenizer, self.model_tokenizer.get_vocab - Return expressions: {}; self.model_tokenizer.get_vocab() ## `vllm_mlx.tool_parsers.abstract_tool_parser.ToolParser.extract_tool_calls` - Kind: method - Signature: `def extract_tool_calls(self, model_output: str, request: dict[str, Any] | None=None) -> ExtractedToolCallInformation` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/abstract_tool_parser.py#L123-L136 - Implementation: Method `ToolParser.extract_tool_calls` can raise `NotImplementedError`. Extract tool calls from a complete model response. Args: model_output: The complete model output string request: Optional request context (for tool definitions, etc.) Returns: ExtractedToolCallInformation with parsed tool calls - Inputs: - `model_output` (str; required): The complete model output string - `request` (dict[str, Any] | None; optional; default `None`): Optional request context (for tool definitions, etc.) - Return annotation: `ExtractedToolCallInformation` - Decorators: abstractmethod - Raises directly: NotImplementedError ## `vllm_mlx.tool_parsers.abstract_tool_parser.ToolParser.extract_tool_calls_streaming` - Kind: method - Signature: `def extract_tool_calls_streaming(self, previous_text: str, current_text: str, delta_text: str, previous_token_ids: Sequence[int] | None=None, current_token_ids: Sequence[int] | None=None, delta_token_ids: Sequence[int] | None=None, request: dict[str, Any] | None=None) -> dict[str, Any] | None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/abstract_tool_parser.py#L138-L166 - Implementation: Method `ToolParser.extract_tool_calls_streaming` returns `None`. Extract tool calls from streaming model output. Override this method for streaming support. Default implementation returns None (no streaming support). Args: previous_text: Text before this delta current_text: Complete text so far delta_text: New text in this chunk previous_token_ids: Token IDs before this delta current_token_ids: All token IDs so far delta_token_ids: New token IDs in this chunk request: Optional request context Returns: Delta message dict with content and/or tool_calls, or None - Inputs: - `previous_text` (str; required): Text before this delta - `current_text` (str; required): Complete text so far - `delta_text` (str; required): New text in this chunk - `previous_token_ids` (Sequence[int] | None; optional; default `None`): Token IDs before this delta - `current_token_ids` (Sequence[int] | None; optional; default `None`): All token IDs so far - `delta_token_ids` (Sequence[int] | None; optional; default `None`): New token IDs in this chunk - `request` (dict[str, Any] | None; optional; default `None`): Optional request context - Return annotation: `dict[str, Any] | None` - Return expressions: None ## `vllm_mlx.tool_parsers.abstract_tool_parser.ToolParser.reset` - Kind: method - Signature: `def reset(self) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/abstract_tool_parser.py#L168-L171 - Implementation: Method `ToolParser.reset` updates `self.current_tool_id`, `self.prev_tool_call_arr`. Reset parser state for a new request. - Inputs: none - Return annotation: `None` - State writes: self.current_tool_id, self.prev_tool_call_arr ## `vllm_mlx.tool_parsers.abstract_tool_parser.ToolParserManager` - Kind: class - Signature: `class ToolParserManager` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/abstract_tool_parser.py#L174-L286 - Implementation: Class `ToolParserManager` declares 5 direct member(s). Central registry for ToolParser implementations. Supports both eager and lazy registration of tool parsers. - Inputs: none - Constructs: `vllm_mlx.tool_parsers.abstract_tool_parser.ToolParserManager` ## `vllm_mlx.tool_parsers.abstract_tool_parser.ToolParserManager.get_tool_parser` - Kind: method - Signature: `def get_tool_parser(cls, name: str) -> type[ToolParser]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/abstract_tool_parser.py#L185-L207 - Implementation: Method `ToolParserManager.get_tool_parser` calls `cls._load_lazy_parser`, `KeyError`, `cls.list_registered`; can raise `KeyError`; has 2 explicit return paths. Retrieve a registered ToolParser class by name. Args: name: Parser name (e.g., 'mistral', 'qwen', 'llama') Returns: The ToolParser class Raises: KeyError: If parser not found - Inputs: - `name` (str; required): Parser name (e.g., 'mistral', 'qwen', 'llama') - Return annotation: `type[ToolParser]` - Decorators: classmethod - Calls: cls._load_lazy_parser, KeyError, cls.list_registered - State reads: cls.tool_parsers, cls.lazy_parsers, cls._load_lazy_parser, cls.list_registered - Raises directly: KeyError - Return expressions: cls.tool_parsers[name]; cls._load_lazy_parser(name) ## `vllm_mlx.tool_parsers.abstract_tool_parser.ToolParserManager._load_lazy_parser` - Kind: method - Signature: `def _load_lazy_parser(cls, name: str) -> type[ToolParser]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/abstract_tool_parser.py#L210-L225 - Implementation: Method `ToolParserManager._load_lazy_parser` calls `importlib.import_module`, `getattr`, `issubclass`, `TypeError`; can raise `TypeError`, `ImportError`; returns `parser_cls`. Import and register a lazily loaded parser. - Inputs: - `name` (str; required): Required positional or keyword input. - Return annotation: `type[ToolParser]` - Decorators: classmethod - Calls: importlib.import_module, getattr, issubclass, TypeError, ImportError - State reads: cls.lazy_parsers, cls.tool_parsers - Raises directly: TypeError, ImportError - Return expressions: parser_cls ## `vllm_mlx.tool_parsers.abstract_tool_parser.ToolParserManager.register_module` - Kind: method - Signature: `def register_module(cls, name: str | list[str], module: type[ToolParser] | None=None, force: bool=True) -> type[ToolParser] | None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/abstract_tool_parser.py#L228-L269 - Implementation: Method `ToolParserManager.register_module` calls `isinstance`, `issubclass`, `TypeError`, `type`; can raise `TypeError`, `KeyError`; has 2 explicit return paths. Register a ToolParser class. Can be used as a decorator or direct call. Usage: @ToolParserManager.register_module("my_parser") class MyToolParser(ToolParser): ... # Or direct registration: ToolParserManager.register_module("my_parser", MyToolParser) - Inputs: - `name` (str | list[str]; required): Required positional or keyword input. - `module` (type[ToolParser] | None; optional; default `None`): Optional positional or keyword input; defaults to `None`. - `force` (bool; optional; default `True`): Optional positional or keyword input; defaults to `True`. - Return annotation: `type[ToolParser] | None` - Decorators: classmethod - Calls: isinstance, issubclass, TypeError, type, KeyError - State reads: cls.tool_parsers - Raises directly: TypeError, KeyError - Return expressions: module; decorator ## `vllm_mlx.tool_parsers.abstract_tool_parser.ToolParserManager.register_module.decorator` - Kind: nested function - Signature: `def decorator(parser_cls: type[ToolParser]) -> type[ToolParser]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/abstract_tool_parser.py#L262-L267 - Implementation: Nested Function `ToolParserManager.register_module.decorator` calls `KeyError`; can raise `KeyError`; returns `parser_cls`. Nested Function `ToolParserManager.register_module.decorator` calls `KeyError`; can raise `KeyError`; returns `parser_cls`. - Inputs: - `parser_cls` (type[ToolParser]; required): Required positional or keyword input. - Return annotation: `type[ToolParser]` - Calls: KeyError - State reads: cls.tool_parsers - Raises directly: KeyError - Return expressions: parser_cls ## `vllm_mlx.tool_parsers.abstract_tool_parser.ToolParserManager.register_lazy_module` - Kind: method - Signature: `def register_lazy_module(cls, name: str, module_path: str, class_name: str) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/abstract_tool_parser.py#L272-L281 - Implementation: Method `ToolParserManager.register_lazy_module` contains no state mutation, call, raise, return, await, or yield. Register a lazy module mapping for deferred loading. Args: name: Parser name to register module_path: Full module path (e.g., 'vllm_mlx.tool_parsers.mistral') class_name: Class name within the module - Inputs: - `name` (str; required): Parser name to register - `module_path` (str; required): Full module path (e.g., 'vllm_mlx.tool_parsers.mistral') - `class_name` (str; required): Class name within the module - Return annotation: `None` - Decorators: classmethod - State reads: cls.lazy_parsers ## `vllm_mlx.tool_parsers.abstract_tool_parser.ToolParserManager.list_registered` - Kind: method - Signature: `def list_registered(cls) -> list[str]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/abstract_tool_parser.py#L284-L286 - Implementation: Method `ToolParserManager.list_registered` calls `sorted`, `set`, `cls.tool_parsers.keys`, `cls.lazy_parsers.keys`; returns `sorted(set(cls.tool_parsers.keys()) | set(cls.lazy_parsers.keys()))`. Return names of all registered tool parsers. - Inputs: none - Return annotation: `list[str]` - Decorators: classmethod - Calls: sorted, set, cls.tool_parsers.keys, cls.lazy_parsers.keys - State reads: cls.tool_parsers.keys, cls.tool_parsers, cls.lazy_parsers.keys, cls.lazy_parsers - Return expressions: sorted(set(cls.tool_parsers.keys()) | set(cls.lazy_parsers.keys())) # Module `vllm_mlx.tool_parsers.auto_tool_parser` Auto-detecting tool call parser for vllm-mlx. Automatically detects and parses tool calls from various model formats. Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/auto_tool_parser.py#L1-L414 ## `vllm_mlx.tool_parsers.auto_tool_parser.generate_tool_id` - Kind: function - Signature: `def generate_tool_id() -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/auto_tool_parser.py#L22-L24 - Implementation: Function `generate_tool_id` calls `uuid.uuid4`; returns `f'call_{uuid.uuid4().hex[:8]}'`. Generate a unique tool call ID. - Inputs: none - Return annotation: `str` - Calls: uuid.uuid4 - Return expressions: f'call_{uuid.uuid4().hex[:8]}' ## `vllm_mlx.tool_parsers.auto_tool_parser.AutoToolParser` - Kind: class - Signature: `class AutoToolParser(ToolParser)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/auto_tool_parser.py#L28-L414 - Implementation: Class `AutoToolParser` derives from `ToolParser` and declares 3 direct member(s). Auto-detecting tool call parser. Tries multiple formats in order: 1. Gemma 4: <|tool_call>call:name{...} 2. Mistral: [TOOL_CALLS] ... 3. Qwen bracket: [Calling tool: func_name({...})] 4. Qwen/Hermes XML: {"name": "...", "arguments": {...}} 5. Llama: {"arg": "value"} 6. Nemotron: ... 7. Raw JSON: {"name": "...", "arguments": {...}} This is the default parser when no specific parser is selected. - Inputs: none - Constructs: `vllm_mlx.tool_parsers.auto_tool_parser.AutoToolParser` - Decorators: ToolParserManager.register_module(['auto', 'generic']) ## `vllm_mlx.tool_parsers.auto_tool_parser.AutoToolParser.extract_tool_calls` - Kind: method - Signature: `def extract_tool_calls(self, model_output: str, request: dict[str, Any] | None=None) -> ExtractedToolCallInformation` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/auto_tool_parser.py#L61-L268 - Implementation: Method `AutoToolParser.extract_tool_calls` calls `Gemma4ToolParser`, `gemma_parser.extract_tool_calls`, `model_output.split`, `parts[0].strip`; has 4 explicit return paths. Extract tool calls by trying all known formats. - Inputs: - `model_output` (str; required): Required positional or keyword input. - `request` (dict[str, Any] | None; optional; default `None`): Optional positional or keyword input; defaults to `None`. - Return annotation: `ExtractedToolCallInformation` - Calls: Gemma4ToolParser, gemma_parser.extract_tool_calls, model_output.split, parts[0].strip, raw.strip, raw.startswith, raw.find, raw[:end_name].strip, tool_calls.append, generate_tool_id, json.loads, isinstance, item.get, json.dumps, str, ExtractedToolCallInformation, self.QWEN_BRACKET_PATTERN.findall, name.strip, self.QWEN_BRACKET_PATTERN.sub('', cleaned_text).strip, self.QWEN_BRACKET_PATTERN.sub, self.BARE_BRACKET_PATTERN.findall, self.BARE_BRACKET_PATTERN.sub('', cleaned_text).strip, self.BARE_BRACKET_PATTERN.sub, self.NEMOTRON_PATTERN.findall, self.NEMOTRON_PARAM_PATTERN.findall, p_name.strip, p_value.strip, self.NEMOTRON_PATTERN.sub('', cleaned_text).strip, self.NEMOTRON_PATTERN.sub, self.QWEN_XML_PATTERN.findall, data.get, self.QWEN_XML_PATTERN.sub('', cleaned_text).strip, self.QWEN_XML_PATTERN.sub, self.LLAMA_PATTERN.findall, self.LLAMA_PATTERN.sub('', cleaned_text).strip, self.LLAMA_PATTERN.sub, self._parse_raw_json_tool_calls, tool_calls.extend - State reads: self.MISTRAL_TOKEN, self.QWEN_BRACKET_PATTERN.findall, self.QWEN_BRACKET_PATTERN, self.QWEN_BRACKET_PATTERN.sub, self.BARE_BRACKET_PATTERN.findall, self.BARE_BRACKET_PATTERN, self.BARE_BRACKET_PATTERN.sub, self.NEMOTRON_PATTERN.findall, self.NEMOTRON_PATTERN, self.NEMOTRON_PARAM_PATTERN.findall, self.NEMOTRON_PARAM_PATTERN, self.NEMOTRON_PATTERN.sub, self.QWEN_XML_PATTERN.findall, self.QWEN_XML_PATTERN, self.QWEN_XML_PATTERN.sub, self.LLAMA_PATTERN.findall, self.LLAMA_PATTERN, self.LLAMA_PATTERN.sub, self._parse_raw_json_tool_calls - Return expressions: result; ExtractedToolCallInformation(tools_called=True, tool_calls=tool_calls, content=content if content else None); ExtractedToolCallInformation(tools_called=True, tool_calls=tool_calls, content=cleaned_text if cleaned_text else None); ExtractedToolCallInformation(tools_called=False, tool_calls=[], content=model_output) ## `vllm_mlx.tool_parsers.auto_tool_parser.AutoToolParser._parse_raw_json_tool_calls` - Kind: method - Signature: `def _parse_raw_json_tool_calls(self, text: str) -> list[dict[str, Any]]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/auto_tool_parser.py#L270-L350 - Implementation: Method `AutoToolParser._parse_raw_json_tool_calls` calls `text.strip`, `text.startswith`, `json.loads`, `isinstance`; has 2 explicit return paths. Parse raw JSON tool calls from text. Handles: - Single JSON object: {"name": "func", "arguments": {...}} - JSON array: [{...}, {...}] - Inputs: - `text` (str; required): Required positional or keyword input. - Return annotation: `list[dict[str, Any]]` - Calls: text.strip, text.startswith, json.loads, isinstance, item.get, tool_calls.append, generate_tool_id, json.dumps, str, enumerate, obj.get - Return expressions: []; tool_calls ## `vllm_mlx.tool_parsers.auto_tool_parser.AutoToolParser.extract_tool_calls_streaming` - Kind: method - Signature: `def extract_tool_calls_streaming(self, previous_text: str, current_text: str, delta_text: str, previous_token_ids: Sequence[int] | None=None, current_token_ids: Sequence[int] | None=None, delta_token_ids: Sequence[int] | None=None, request: dict[str, Any] | None=None) -> dict[str, Any] | None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/auto_tool_parser.py#L352-L414 - Implementation: Method `AutoToolParser.extract_tool_calls_streaming` calls `any`, `self.BARE_BRACKET_PARTIAL_PATTERN.search`, `self.BARE_BRACKET_PATTERN.search`, `self.extract_tool_calls`; has 3 explicit return paths. Extract tool calls from streaming model output. Uses simple heuristics to detect when a tool call might be complete. - Inputs: - `previous_text` (str; required): Required positional or keyword input. - `current_text` (str; required): Required positional or keyword input. - `delta_text` (str; required): Required positional or keyword input. - `previous_token_ids` (Sequence[int] | None; optional; default `None`): Optional positional or keyword input; defaults to `None`. - `current_token_ids` (Sequence[int] | None; optional; default `None`): Optional positional or keyword input; defaults to `None`. - `delta_token_ids` (Sequence[int] | None; optional; default `None`): Optional positional or keyword input; defaults to `None`. - `request` (dict[str, Any] | None; optional; default `None`): Optional positional or keyword input; defaults to `None`. - Return annotation: `dict[str, Any] | None` - Calls: any, self.BARE_BRACKET_PARTIAL_PATTERN.search, self.BARE_BRACKET_PATTERN.search, self.extract_tool_calls, enumerate - State reads: self.MISTRAL_TOKEN, self.BARE_BRACKET_PARTIAL_PATTERN.search, self.BARE_BRACKET_PARTIAL_PATTERN, self.BARE_BRACKET_PATTERN.search, self.BARE_BRACKET_PATTERN, self.extract_tool_calls - Return expressions: None; {'content': delta_text}; {'tool_calls': [{'index': i, 'id': tc['id'], 'type': 'function', 'function': {'name': tc['name'], 'arguments': tc['argu… # Module `vllm_mlx.tool_parsers.deepseek_tool_parser` DeepSeek tool call parser for vllm-mlx. Handles DeepSeek V3 and R1 tool calling formats: - <｜tool▁calls▁begin｜>...<｜tool▁calls▁end｜> wrapper - <｜tool▁call▁begin｜>function<｜tool▁sep｜>name ```json {...} ```<｜tool▁call▁end｜> Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/deepseek_tool_parser.py#L1-L170 ## `vllm_mlx.tool_parsers.deepseek_tool_parser.generate_tool_id` - Kind: function - Signature: `def generate_tool_id() -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/deepseek_tool_parser.py#L26-L28 - Implementation: Function `generate_tool_id` calls `uuid.uuid4`; returns `f'call_{uuid.uuid4().hex[:8]}'`. Generate a unique tool call ID. - Inputs: none - Return annotation: `str` - Calls: uuid.uuid4 - Return expressions: f'call_{uuid.uuid4().hex[:8]}' ## `vllm_mlx.tool_parsers.deepseek_tool_parser.DeepSeekToolParser` - Kind: class - Signature: `class DeepSeekToolParser(ToolParser)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/deepseek_tool_parser.py#L32-L170 - Implementation: Class `DeepSeekToolParser` derives from `ToolParser` and declares 2 direct member(s). Tool call parser for DeepSeek V3 and R1 models. Supports DeepSeek's tool call format with special unicode tokens: <｜tool▁calls▁begin｜> <｜tool▁call▁begin｜>function<｜tool▁sep｜>get_weather ```json {"city": "Paris"} ```<｜tool▁call▁end｜> <｜tool▁calls▁end｜> Used when --enable-auto-tool-choice --tool-call-parser deepseek are set. - Inputs: none - Constructs: `vllm_mlx.tool_parsers.deepseek_tool_parser.DeepSeekToolParser` - Decorators: ToolParserManager.register_module(['deepseek', 'deepseek_v3', 'deepseek_r1']) ## `vllm_mlx.tool_parsers.deepseek_tool_parser.DeepSeekToolParser.extract_tool_calls` - Kind: method - Signature: `def extract_tool_calls(self, model_output: str, request: dict[str, Any] | None=None) -> ExtractedToolCallInformation` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/deepseek_tool_parser.py#L69-L133 - Implementation: Method `DeepSeekToolParser.extract_tool_calls` calls `ExtractedToolCallInformation`, `model_output.find`, `model_output[:content_end].strip`, `self.TOOL_CALL_PATTERN.findall`; has 2 explicit return paths. Extract tool calls from DeepSeek model output. - Inputs: - `model_output` (str; required): Required positional or keyword input. - `request` (dict[str, Any] | None; optional; default `None`): Optional positional or keyword input; defaults to `None`. - Return annotation: `ExtractedToolCallInformation` - Calls: ExtractedToolCallInformation, model_output.find, model_output[:content_end].strip, self.TOOL_CALL_PATTERN.findall, json.loads, tool_calls.append, generate_tool_id, func_name.strip, func_args.strip, self.TOOL_CALL_SIMPLE_PATTERN.findall - State reads: self.TOOL_CALLS_START, self.TOOL_CALL_PATTERN.findall, self.TOOL_CALL_PATTERN, self.TOOL_CALL_SIMPLE_PATTERN.findall, self.TOOL_CALL_SIMPLE_PATTERN - Return expressions: ExtractedToolCallInformation(tools_called=False, tool_calls=[], content=model_output); ExtractedToolCallInformation(tools_called=True, tool_calls=tool_calls, content=content) ## `vllm_mlx.tool_parsers.deepseek_tool_parser.DeepSeekToolParser.extract_tool_calls_streaming` - Kind: method - Signature: `def extract_tool_calls_streaming(self, previous_text: str, current_text: str, delta_text: str, previous_token_ids: Sequence[int] | None=None, current_token_ids: Sequence[int] | None=None, delta_token_ids: Sequence[int] | None=None, request: dict[str, Any] | None=None) -> dict[str, Any] | None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/deepseek_tool_parser.py#L135-L170 - Implementation: Method `DeepSeekToolParser.extract_tool_calls_streaming` calls `self.extract_tool_calls`, `enumerate`; has 3 explicit return paths. Extract tool calls from streaming DeepSeek model output. - Inputs: - `previous_text` (str; required): Required positional or keyword input. - `current_text` (str; required): Required positional or keyword input. - `delta_text` (str; required): Required positional or keyword input. - `previous_token_ids` (Sequence[int] | None; optional; default `None`): Optional positional or keyword input; defaults to `None`. - `current_token_ids` (Sequence[int] | None; optional; default `None`): Optional positional or keyword input; defaults to `None`. - `delta_token_ids` (Sequence[int] | None; optional; default `None`): Optional positional or keyword input; defaults to `None`. - `request` (dict[str, Any] | None; optional; default `None`): Optional positional or keyword input; defaults to `None`. - Return annotation: `dict[str, Any] | None` - Calls: self.extract_tool_calls, enumerate - State reads: self.TOOL_CALLS_START, self.TOOL_CALL_END, self.TOOL_CALLS_END, self.extract_tool_calls - Return expressions: {'content': delta_text}; {'tool_calls': [{'index': i, 'id': tc['id'], 'type': 'function', 'function': {'name': tc['name'], 'arguments': tc['argu…; None # Module `vllm_mlx.tool_parsers.functionary_tool_parser` Functionary tool call parser for vllm-mlx. Handles MeetKai Functionary models' tool calling format. Similar to OpenAI function calling with JSON arguments. Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/functionary_tool_parser.py#L1-L193 ## `vllm_mlx.tool_parsers.functionary_tool_parser.generate_tool_id` - Kind: function - Signature: `def generate_tool_id() -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/functionary_tool_parser.py#L22-L24 - Implementation: Function `generate_tool_id` calls `uuid.uuid4`; returns `f'call_{uuid.uuid4().hex[:8]}'`. Generate a unique tool call ID. - Inputs: none - Return annotation: `str` - Calls: uuid.uuid4 - Return expressions: f'call_{uuid.uuid4().hex[:8]}' ## `vllm_mlx.tool_parsers.functionary_tool_parser.FunctionaryToolParser` - Kind: class - Signature: `class FunctionaryToolParser(ToolParser)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/functionary_tool_parser.py#L28-L193 - Implementation: Class `FunctionaryToolParser` derives from `ToolParser` and declares 2 direct member(s). Tool call parser for MeetKai Functionary models. Supports Functionary's tool call format similar to OpenAI: - Uses special tokens to mark tool calls - Arguments are JSON strings Formats supported: - <|from|>assistant <|recipient|>func_name <|content|>{"args": ...} - {"args": ...} Used when --enable-auto-tool-choice --tool-call-parser functionary are set. - Inputs: none - Constructs: `vllm_mlx.tool_parsers.functionary_tool_parser.FunctionaryToolParser` - Decorators: ToolParserManager.register_module(['functionary', 'meetkai']) ## `vllm_mlx.tool_parsers.functionary_tool_parser.FunctionaryToolParser.extract_tool_calls` - Kind: method - Signature: `def extract_tool_calls(self, model_output: str, request: dict[str, Any] | None=None) -> ExtractedToolCallInformation` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/functionary_tool_parser.py#L61-L153 - Implementation: Method `FunctionaryToolParser.extract_tool_calls` calls `self.RECIPIENT_PATTERN.findall`, `func_name.lower`, `json.loads`, `tool_calls.append`; has 2 explicit return paths. Extract tool calls from Functionary model output. - Inputs: - `model_output` (str; required): Required positional or keyword input. - `request` (dict[str, Any] | None; optional; default `None`): Optional positional or keyword input; defaults to `None`. - Return annotation: `ExtractedToolCallInformation` - Calls: self.RECIPIENT_PATTERN.findall, func_name.lower, json.loads, tool_calls.append, generate_tool_id, self.RECIPIENT_PATTERN.sub, re.sub('<\\|from\\|>assistant\\s*', '', cleaned_text).strip, re.sub, self.FUNCTION_PATTERN.findall, func_name.strip, self.FUNCTION_PATTERN.sub('', cleaned_text).strip, self.FUNCTION_PATTERN.sub, self.JSON_ARRAY_PATTERN.match, model_output.strip, isinstance, call.get, json.dumps, str, ExtractedToolCallInformation - State reads: self.RECIPIENT_PATTERN.findall, self.RECIPIENT_PATTERN, self.RECIPIENT_PATTERN.sub, self.FUNCTION_PATTERN.findall, self.FUNCTION_PATTERN, self.FUNCTION_PATTERN.sub, self.JSON_ARRAY_PATTERN.match, self.JSON_ARRAY_PATTERN - Return expressions: ExtractedToolCallInformation(tools_called=True, tool_calls=tool_calls, content=cleaned_text if cleaned_text else None); ExtractedToolCallInformation(tools_called=False, tool_calls=[], content=model_output) ## `vllm_mlx.tool_parsers.functionary_tool_parser.FunctionaryToolParser.extract_tool_calls_streaming` - Kind: method - Signature: `def extract_tool_calls_streaming(self, previous_text: str, current_text: str, delta_text: str, previous_token_ids: Sequence[int] | None=None, current_token_ids: Sequence[int] | None=None, delta_token_ids: Sequence[int] | None=None, request: dict[str, Any] | None=None) -> dict[str, Any] | None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/functionary_tool_parser.py#L155-L193 - Implementation: Method `FunctionaryToolParser.extract_tool_calls_streaming` calls `any`, `self.extract_tool_calls`, `enumerate`; has 3 explicit return paths. Extract tool calls from streaming Functionary model output. - Inputs: - `previous_text` (str; required): Required positional or keyword input. - `current_text` (str; required): Required positional or keyword input. - `delta_text` (str; required): Required positional or keyword input. - `previous_token_ids` (Sequence[int] | None; optional; default `None`): Optional positional or keyword input; defaults to `None`. - `current_token_ids` (Sequence[int] | None; optional; default `None`): Optional positional or keyword input; defaults to `None`. - `delta_token_ids` (Sequence[int] | None; optional; default `None`): Optional positional or keyword input; defaults to `None`. - `request` (dict[str, Any] | None; optional; default `None`): Optional positional or keyword input; defaults to `None`. - Return annotation: `dict[str, Any] | None` - Calls: any, self.extract_tool_calls, enumerate - State reads: self.extract_tool_calls - Return expressions: {'content': delta_text}; {'tool_calls': [{'index': i, 'id': tc['id'], 'type': 'function', 'function': {'name': tc['name'], 'arguments': tc['argu…; None # Module `vllm_mlx.tool_parsers.gemma4_tool_parser` Gemma 4 tool call parser for vllm-mlx. Handles Gemma 4's native tool call format: <|tool_call>call:func_name{<|"|>key<|"|>: <|"|>value<|"|>, num: 42} Gemma 4 uses special tokens instead of JSON: - <|tool_call> / delimit tool call blocks - <|"|> replaces " for string values - Keys are unquoted bare identifiers - Multiple call:name{...} can appear in a single block Fallback forms (issue #80): under long system prompts + multi-turn + several tools, Gemma 4 frequently abandons the canonical brace form and instead emits its call as plain text in `content`, using Python-style call syntax: e4b: <|tool_call>call:radarr_get_movies(search="Dune") e2b: ```tool_code radarr_get_movies(search="Dune") ``` e2b: tool_code = radarr_get_movies(search="Dune") print(tool_code) These are parsed by a fallback layer (ast-based) when the canonical parse finds no calls, so the host can still dispatch the tool. Reference: mlx-lm PR #1105, vllm PR #38837 Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/gemma4_tool_parser.py#L1-L513 ## `vllm_mlx.tool_parsers.gemma4_tool_parser._find_balanced_brace` - Kind: function - Signature: `def _find_balanced_brace(text: str, start: int) -> int` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/gemma4_tool_parser.py#L92-L125 - Implementation: Function `_find_balanced_brace` calls `len`, `text.startswith`; has 2 explicit return paths. Find the index of the closing } that balances the { at `start`. Before counting braces, <|"|>-delimited strings are conceptually opaque -- we skip over <|"|>...<|"|> regions so that braces inside string values (e.g. code snippets) don't affect depth counting. Args: text: The string to search (may contain <|"|> tokens) start: Index of the opening { Returns: Index of the matching } in the ORIGINAL text, or -1 if not found - Inputs: - `text` (str; required): The string to search (may contain <|"|> tokens) - `start` (int; required): Index of the opening { - Return annotation: `int` - Calls: len, text.startswith - Return expressions: -1; i ## `vllm_mlx.tool_parsers.gemma4_tool_parser._find_balanced_paren` - Kind: function - Signature: `def _find_balanced_paren(text: str, start: int) -> int` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/gemma4_tool_parser.py#L128-L159 - Implementation: Function `_find_balanced_paren` calls `len`; has 2 explicit return paths. Find the index of the closing ) that balances the ( at `start`. Python string literals ('...'/"...") are treated as opaque so that parens inside string argument values don't affect depth counting. Returns the index of the matching ) in `text`, or -1 if not found. - Inputs: - `text` (str; required): Required positional or keyword input. - `start` (int; required): Required positional or keyword input. - Return annotation: `int` - Calls: len - Return expressions: -1; i ## `vllm_mlx.tool_parsers.gemma4_tool_parser._quote_bare_value` - Kind: function - Signature: `def _quote_bare_value(m: re.Match) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/gemma4_tool_parser.py#L162-L168 - Implementation: Function `_quote_bare_value` calls `m.group`; has 2 explicit return paths. Substitution callback for _BARE_VALUE — quotes bare identifiers that are not JSON literals (true/false/null). - Inputs: - `m` (re.Match; required): Required positional or keyword input. - Return annotation: `str` - Calls: m.group - Return expressions: m.group(0); f'{ws}"{word}"' ## `vllm_mlx.tool_parsers.gemma4_tool_parser._gemma4_args_to_json` - Kind: function - Signature: `def _gemma4_args_to_json(text: str) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/gemma4_tool_parser.py#L171-L207 - Implementation: Function `_gemma4_args_to_json` calls `_STRING_DELIM_RE.sub`, `_BARE_KEY.sub`, `_BARE_VALUE.sub`, `_PLACEHOLDER_RE.sub`; returns `text`. Convert Gemma 4 tool call args to valid JSON. Four-step conversion (ORDER MATTERS): 1. Extract <|"|>-delimited strings into numbered \x00N\x00 placeholders. This protects string contents from step 2's bare-key quoting -- without this, a string value like "key: value" would be corrupted. 2. Quote bare keys (word: -> "word":) now that strings are safe. 3. Quote bare string VALUES that the template emitted without <|"|> wrappers. Happens with nullable/enum schemas where the STRING branch of the template isn't taken. 4. Restore placeholders as properly JSON-escaped strings via json.dumps(). Uses a single re.sub pass (O(len(text))) instead of per-placeholder replace. - Inputs: - `text` (str; required): Required positional or keyword input. - Return annotation: `str` - Calls: _STRING_DELIM_RE.sub, _BARE_KEY.sub, _BARE_VALUE.sub, _PLACEHOLDER_RE.sub - Return expressions: text ## `vllm_mlx.tool_parsers.gemma4_tool_parser._gemma4_args_to_json._capture` - Kind: nested function - Signature: `def _capture(m: re.Match) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/gemma4_tool_parser.py#L187-L189 - Implementation: Nested Function `_gemma4_args_to_json._capture` calls `strings.append`, `m.group`, `len`; returns `f'\x00{len(strings) - 1}\x00'`. Nested Function `_gemma4_args_to_json._capture` calls `strings.append`, `m.group`, `len`; returns `f'\x00{len(strings) - 1}\x00'`. - Inputs: - `m` (re.Match; required): Required positional or keyword input. - Return annotation: `str` - Calls: strings.append, m.group, len - Return expressions: f'\x00{len(strings) - 1}\x00' ## `vllm_mlx.tool_parsers.gemma4_tool_parser._gemma4_args_to_json._restore` - Kind: nested function - Signature: `def _restore(m: re.Match) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/gemma4_tool_parser.py#L201-L203 - Implementation: Nested Function `_gemma4_args_to_json._restore` calls `int`, `m.group`, `len`, `json.dumps`; returns `json.dumps(strings[idx]) if idx < len(strings) else m.group(0)`. Nested Function `_gemma4_args_to_json._restore` calls `int`, `m.group`, `len`, `json.dumps`; returns `json.dumps(strings[idx]) if idx < len(strings) else m.group(0)`. - Inputs: - `m` (re.Match; required): Required positional or keyword input. - Return annotation: `str` - Calls: int, m.group, len, json.dumps - Return expressions: json.dumps(strings[idx]) if idx < len(strings) else m.group(0) ## `vllm_mlx.tool_parsers.gemma4_tool_parser._call_node_to_tool` - Kind: function - Signature: `def _call_node_to_tool(call: ast.Call) -> tuple[str, dict[str, Any]] | None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/gemma4_tool_parser.py#L210-L233 - Implementation: Function `_call_node_to_tool` calls `isinstance`, `ast.literal_eval`; has 2 explicit return paths. Map a Python `ast.Call` node to (function_name, kwargs_dict). Only keyword arguments are mapped (Gemma emits its tool calls as kwargs); positional args are ignored because the parameter names aren't recoverable. Returns None if the name or any argument value isn't a plain literal. - Inputs: - `call` (ast.Call; required): Required positional or keyword input. - Return annotation: `tuple[str, dict[str, Any]] | None` - Calls: isinstance, ast.literal_eval - Return expressions: None; (name, args) ## `vllm_mlx.tool_parsers.gemma4_tool_parser._parse_python_call` - Kind: function - Signature: `def _parse_python_call(src: str) -> tuple[str, dict[str, Any]] | None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/gemma4_tool_parser.py#L236-L244 - Implementation: Function `_parse_python_call` calls `ast.parse`, `src.strip`, `isinstance`, `_call_node_to_tool`; has 2 explicit return paths. Parse a single `fn(...)` Python call expression into (name, kwargs). - Inputs: - `src` (str; required): Required positional or keyword input. - Return annotation: `tuple[str, dict[str, Any]] | None` - Calls: ast.parse, src.strip, isinstance, _call_node_to_tool - Return expressions: None; _call_node_to_tool(node.body) ## `vllm_mlx.tool_parsers.gemma4_tool_parser._parse_calls_from_code` - Kind: function - Signature: `def _parse_calls_from_code(code: str) -> list[tuple[str, dict[str, Any]]]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/gemma4_tool_parser.py#L247-L259 - Implementation: Function `_parse_calls_from_code` calls `ast.parse`, `textwrap.dedent(code).strip`, `textwrap.dedent`, `isinstance`; returns `results`. Parse every top-level `fn(...)` call statement in a code-fence body. - Inputs: - `code` (str; required): Required positional or keyword input. - Return annotation: `list[tuple[str, dict[str, Any]]]` - Calls: ast.parse, textwrap.dedent(code).strip, textwrap.dedent, isinstance, _call_node_to_tool, results.append - Return expressions: results ## `vllm_mlx.tool_parsers.gemma4_tool_parser._strip_spans` - Kind: function - Signature: `def _strip_spans(text: str, spans: list[tuple[int, int]]) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/gemma4_tool_parser.py#L262-L275 - Implementation: Function `_strip_spans` calls `sorted`, `max`, `out.append`, `''.join`; has 2 explicit return paths. Remove the given [start, end) spans from `text` (handles overlaps). - Inputs: - `text` (str; required): Required positional or keyword input. - `spans` (list[tuple[int, int]]; required): Required positional or keyword input. - Return annotation: `str` - Calls: sorted, max, out.append, ''.join - Return expressions: text; ''.join(out) ## `vllm_mlx.tool_parsers.gemma4_tool_parser.generate_tool_id` - Kind: function - Signature: `def generate_tool_id() -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/gemma4_tool_parser.py#L278-L280 - Implementation: Function `generate_tool_id` calls `uuid.uuid4`; returns `f'call_{uuid.uuid4().hex[:8]}'`. Generate a unique tool call ID. - Inputs: none - Return annotation: `str` - Calls: uuid.uuid4 - Return expressions: f'call_{uuid.uuid4().hex[:8]}' ## `vllm_mlx.tool_parsers.gemma4_tool_parser.Gemma4ToolParser` - Kind: class - Signature: `class Gemma4ToolParser(ToolParser)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/gemma4_tool_parser.py#L284-L513 - Implementation: Class `Gemma4ToolParser` derives from `ToolParser` and declares 5 direct member(s). Tool call parser for Gemma 4 models. Parses: <|tool_call>call:func{<|"|>key<|"|>: <|"|>val<|"|>} Used when --enable-auto-tool-choice --tool-call-parser gemma4 are set. - Inputs: none - Constructs: `vllm_mlx.tool_parsers.gemma4_tool_parser.Gemma4ToolParser` - Decorators: ToolParserManager.register_module('gemma4') ## `vllm_mlx.tool_parsers.gemma4_tool_parser.Gemma4ToolParser.extract_tool_calls` - Kind: method - Signature: `def extract_tool_calls(self, model_output: str, request: dict[str, Any] | None=None) -> ExtractedToolCallInformation` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/gemma4_tool_parser.py#L300-L324 - Implementation: Method `Gemma4ToolParser.extract_tool_calls` calls `self.strip_think_tags`, `self._extract_canonical`, `ExtractedToolCallInformation`, `self._extract_fallback`; has 3 explicit return paths. Extract tool calls from a complete Gemma 4 model response. - Inputs: - `model_output` (str; required): Required positional or keyword input. - `request` (dict[str, Any] | None; optional; default `None`): Optional positional or keyword input; defaults to `None`. - Return annotation: `ExtractedToolCallInformation` - Calls: self.strip_think_tags, self._extract_canonical, ExtractedToolCallInformation, self._extract_fallback - State reads: self.strip_think_tags, self._extract_canonical, self._extract_fallback - Return expressions: ExtractedToolCallInformation(tools_called=True, tool_calls=tool_calls, content=content_before); fallback; ExtractedToolCallInformation(tools_called=False, tool_calls=[], content=model_output) ## `vllm_mlx.tool_parsers.gemma4_tool_parser.Gemma4ToolParser._extract_canonical` - Kind: method - Signature: `def _extract_canonical(self, cleaned: str) -> tuple[list[dict[str, Any]], str | None]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/gemma4_tool_parser.py#L326-L382 - Implementation: Method `Gemma4ToolParser._extract_canonical` calls `cleaned.find`, `cleaned[:start_idx].strip`, `len`, `_CALL_PREFIX.search`; has 2 explicit return paths. Parse the canonical <|tool_call>call:fn{...} form. Returns (tool_calls, content_before). tool_calls is empty when the canonical markers/braces aren't present. - Inputs: - `cleaned` (str; required): Required positional or keyword input. - Return annotation: `tuple[list[dict[str, Any]], str | None]` - Calls: cleaned.find, cleaned[:start_idx].strip, len, _CALL_PREFIX.search, m.group, m.end, _find_balanced_brace, _gemma4_args_to_json, json.loads, tool_calls.append, generate_tool_id, logger.warning - Return expressions: ([], None); (tool_calls, content_before) ## `vllm_mlx.tool_parsers.gemma4_tool_parser.Gemma4ToolParser._extract_fallback` - Kind: method - Signature: `def _extract_fallback(self, cleaned: str) -> ExtractedToolCallInformation | None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/gemma4_tool_parser.py#L384-L463 - Implementation: Method `Gemma4ToolParser._extract_fallback` calls `_TOOL_CODE_FENCE_RE.finditer`, `_parse_calls_from_code`, `m.group`, `tool_calls.append`; has 2 explicit return paths. Parse the Python-style fallback forms (issue #80). Handles ```tool_code``` blocks (bare `fn(...)` calls) and the parenthesized `call:fn(...)` form. Returns None if neither is present. - Inputs: - `cleaned` (str; required): Required positional or keyword input. - Return annotation: `ExtractedToolCallInformation | None` - Calls: _TOOL_CODE_FENCE_RE.finditer, _parse_calls_from_code, m.group, tool_calls.append, generate_tool_id, json.dumps, spans.append, m.start, m.end, _CALL_PAREN_RE.finditer, any, _find_balanced_paren, _parse_python_call, _TOOL_CODE_ASSIGN_RE.finditer, _strip_spans, content.replace(TOOL_CALL_START, '').replace, content.replace, re.sub, content.strip, ExtractedToolCallInformation - Return expressions: None; ExtractedToolCallInformation(tools_called=True, tool_calls=tool_calls, content=content) ## `vllm_mlx.tool_parsers.gemma4_tool_parser.Gemma4ToolParser._format_streaming` - Kind: method - Signature: `def _format_streaming(self, result: ExtractedToolCallInformation) -> dict[str, Any]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/gemma4_tool_parser.py#L465-L480 - Implementation: Method `Gemma4ToolParser._format_streaming` calls `enumerate`; returns `{'tool_calls': [{'index': i, 'id': tc['id'], 'type': 'function', 'function': {'name': tc['name'], 'arguments': tc['argu…`. Render extracted tool calls into the streaming delta shape. - Inputs: - `result` (ExtractedToolCallInformation; required): Required positional or keyword input. - Return annotation: `dict[str, Any]` - Calls: enumerate - Return expressions: {'tool_calls': [{'index': i, 'id': tc['id'], 'type': 'function', 'function': {'name': tc['name'], 'arguments': tc['argu… ## `vllm_mlx.tool_parsers.gemma4_tool_parser.Gemma4ToolParser.extract_tool_calls_streaming` - Kind: method - Signature: `def extract_tool_calls_streaming(self, previous_text: str, current_text: str, delta_text: str, previous_token_ids: Sequence[int] | None=None, current_token_ids: Sequence[int] | None=None, delta_token_ids: Sequence[int] | None=None, request: dict[str, Any] | None=None) -> dict[str, Any] | None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/gemma4_tool_parser.py#L482-L513 - Implementation: Method `Gemma4ToolParser.extract_tool_calls_streaming` calls `bool`, `_FALLBACK_MARKER_RE.search`, `self.extract_tool_calls`, `self._format_streaming`; has 3 explicit return paths. Extract tool calls from streaming Gemma 4 model output. - Inputs: - `previous_text` (str; required): Required positional or keyword input. - `current_text` (str; required): Required positional or keyword input. - `delta_text` (str; required): Required positional or keyword input. - `previous_token_ids` (Sequence[int] | None; optional; default `None`): Optional positional or keyword input; defaults to `None`. - `current_token_ids` (Sequence[int] | None; optional; default `None`): Optional positional or keyword input; defaults to `None`. - `delta_token_ids` (Sequence[int] | None; optional; default `None`): Optional positional or keyword input; defaults to `None`. - `request` (dict[str, Any] | None; optional; default `None`): Optional positional or keyword input; defaults to `None`. - Return annotation: `dict[str, Any] | None` - Calls: bool, _FALLBACK_MARKER_RE.search, self.extract_tool_calls, self._format_streaming - State reads: self.extract_tool_calls, self._format_streaming - Return expressions: {'content': delta_text}; self._format_streaming(result); None # Module `vllm_mlx.tool_parsers.glm47_tool_parser` GLM-4.7 tool call parser for vllm-mlx. Handles GLM-4.7-Flash style tool calling format. Based on vLLM's glm47_moe_tool_parser.py Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/glm47_tool_parser.py#L1-L184 ## `vllm_mlx.tool_parsers.glm47_tool_parser.generate_tool_id` - Kind: function - Signature: `def generate_tool_id() -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/glm47_tool_parser.py#L22-L24 - Implementation: Function `generate_tool_id` calls `uuid.uuid4`; returns `f'call_{uuid.uuid4().hex[:8]}'`. Generate a unique tool call ID. - Inputs: none - Return annotation: `str` - Calls: uuid.uuid4 - Return expressions: f'call_{uuid.uuid4().hex[:8]}' ## `vllm_mlx.tool_parsers.glm47_tool_parser.Glm47ToolParser` - Kind: class - Signature: `class Glm47ToolParser(ToolParser)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/glm47_tool_parser.py#L28-L184 - Implementation: Class `Glm47ToolParser` derives from `ToolParser` and declares 4 direct member(s). Tool call parser for GLM-4.7 and GLM-4.7-Flash models. Supports GLM-4.7 tool call format: function_name param1value1 param2value2 Used when --enable-auto-tool-choice --tool-call-parser glm47 are set. - Inputs: none - Constructs: `vllm_mlx.tool_parsers.glm47_tool_parser.Glm47ToolParser` - Decorators: ToolParserManager.register_module(['glm47', 'glm4']) ## `vllm_mlx.tool_parsers.glm47_tool_parser.Glm47ToolParser._deserialize` - Kind: method - Signature: `def _deserialize(self, value: str) -> Any` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/glm47_tool_parser.py#L57-L67 - Implementation: Method `Glm47ToolParser._deserialize` calls `value.strip`, `json.loads`; has 2 explicit return paths. Convert string value to appropriate Python type. Uses json.loads for type coercion, falls back to raw string. - Inputs: - `value` (str; required): Required positional or keyword input. - Return annotation: `Any` - Calls: value.strip, json.loads - Return expressions: json.loads(value); value ## `vllm_mlx.tool_parsers.glm47_tool_parser.Glm47ToolParser._get_tool_names` - Kind: method - Signature: `def _get_tool_names(self, request: dict[str, Any] | None) -> set[str]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/glm47_tool_parser.py#L69-L77 - Implementation: Method `Glm47ToolParser._get_tool_names` calls `set`, `t.get('function', {}).get`, `t.get`, `request.get`; has 2 explicit return paths. Extract valid tool names from the request. - Inputs: - `request` (dict[str, Any] | None; required): Required positional or keyword input. - Return annotation: `set[str]` - Calls: set, t.get('function', {}).get, t.get, request.get, isinstance - Return expressions: set(); {t.get('function', {}).get('name', '') for t in request.get('tools', []) if isinstance(t, dict)} ## `vllm_mlx.tool_parsers.glm47_tool_parser.Glm47ToolParser.extract_tool_calls` - Kind: method - Signature: `def extract_tool_calls(self, model_output: str, request: dict[str, Any] | None=None) -> ExtractedToolCallInformation` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/glm47_tool_parser.py#L79-L137 - Implementation: Method `Glm47ToolParser.extract_tool_calls` calls `self.strip_think_tags`, `self._get_tool_names`, `self.FUNC_DETAIL_PATTERN.findall`, `match[0].strip`; has 2 explicit return paths. Extract tool calls from a complete GLM-4.7 model response. - Inputs: - `model_output` (str; required): Required positional or keyword input. - `request` (dict[str, Any] | None; optional; default `None`): Optional positional or keyword input; defaults to `None`. - Return annotation: `ExtractedToolCallInformation` - Calls: self.strip_think_tags, self._get_tool_names, self.FUNC_DETAIL_PATTERN.findall, match[0].strip, len, self.ARG_PATTERN.findall, arg_key.strip, self._deserialize, tool_calls.append, generate_tool_id, json.dumps, ExtractedToolCallInformation - State reads: self.strip_think_tags, self._get_tool_names, self.FUNC_DETAIL_PATTERN.findall, self.FUNC_DETAIL_PATTERN, self.ARG_PATTERN.findall, self.ARG_PATTERN, self._deserialize - Return expressions: ExtractedToolCallInformation(tools_called=True, tool_calls=tool_calls, content=None); ExtractedToolCallInformation(tools_called=False, tool_calls=[], content=cleaned_text) ## `vllm_mlx.tool_parsers.glm47_tool_parser.Glm47ToolParser.extract_tool_calls_streaming` - Kind: method - Signature: `def extract_tool_calls_streaming(self, previous_text: str, current_text: str, delta_text: str, previous_token_ids: Sequence[int] | None=None, current_token_ids: Sequence[int] | None=None, delta_token_ids: Sequence[int] | None=None, request: dict[str, Any] | None=None) -> dict[str, Any] | None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/glm47_tool_parser.py#L139-L184 - Implementation: Method `Glm47ToolParser.extract_tool_calls_streaming` calls `self.extract_tool_calls`, `enumerate`, `self.strip_think_tags`; has 3 explicit return paths. Extract tool calls from streaming GLM-4.7 model output. - Inputs: - `previous_text` (str; required): Required positional or keyword input. - `current_text` (str; required): Required positional or keyword input. - `delta_text` (str; required): Required positional or keyword input. - `previous_token_ids` (Sequence[int] | None; optional; default `None`): Optional positional or keyword input; defaults to `None`. - `current_token_ids` (Sequence[int] | None; optional; default `None`): Optional positional or keyword input; defaults to `None`. - `delta_token_ids` (Sequence[int] | None; optional; default `None`): Optional positional or keyword input; defaults to `None`. - `request` (dict[str, Any] | None; optional; default `None`): Optional positional or keyword input; defaults to `None`. - Return annotation: `dict[str, Any] | None` - Calls: self.extract_tool_calls, enumerate, self.strip_think_tags - State reads: self.extract_tool_calls, self.strip_think_tags - Return expressions: None; {'tool_calls': [{'index': i, 'id': tc['id'], 'type': 'function', 'function': {'name': tc['name'], 'arguments': tc['argu…; {'content': clean_delta} # Module `vllm_mlx.tool_parsers.granite_tool_parser` Granite tool call parser for vllm-mlx. Handles IBM Granite models' tool calling format: - <|tool_call|> or followed by JSON array - [{"name": "func", "arguments": {...}}] Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/granite_tool_parser.py#L1-L147 ## `vllm_mlx.tool_parsers.granite_tool_parser.generate_tool_id` - Kind: function - Signature: `def generate_tool_id() -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/granite_tool_parser.py#L22-L24 - Implementation: Function `generate_tool_id` calls `uuid.uuid4`; returns `f'call_{uuid.uuid4().hex[:8]}'`. Generate a unique tool call ID. - Inputs: none - Return annotation: `str` - Calls: uuid.uuid4 - Return expressions: f'call_{uuid.uuid4().hex[:8]}' ## `vllm_mlx.tool_parsers.granite_tool_parser.GraniteToolParser` - Kind: class - Signature: `class GraniteToolParser(ToolParser)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/granite_tool_parser.py#L28-L147 - Implementation: Class `GraniteToolParser` derives from `ToolParser` and declares 2 direct member(s). Tool call parser for IBM Granite models. Supports Granite's tool call format: <|tool_call|>[{"name": "get_weather", "arguments": {"city": "Paris"}}] Or Granite 3.1: [{"name": "get_weather", "arguments": {"city": "Paris"}}] Used when --enable-auto-tool-choice --tool-call-parser granite are set. - Inputs: none - Constructs: `vllm_mlx.tool_parsers.granite_tool_parser.GraniteToolParser` - Decorators: ToolParserManager.register_module(['granite', 'granite3']) ## `vllm_mlx.tool_parsers.granite_tool_parser.GraniteToolParser.extract_tool_calls` - Kind: method - Signature: `def extract_tool_calls(self, model_output: str, request: dict[str, Any] | None=None) -> ExtractedToolCallInformation` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/granite_tool_parser.py#L47-L105 - Implementation: Method `GraniteToolParser.extract_tool_calls` calls `model_output.strip`, `stripped.startswith`, `stripped[len(self.BOT_TOKEN):].lstrip`, `len`; has 2 explicit return paths. Extract tool calls from Granite model output. - Inputs: - `model_output` (str; required): Required positional or keyword input. - `request` (dict[str, Any] | None; optional; default `None`): Optional positional or keyword input; defaults to `None`. - Return annotation: `ExtractedToolCallInformation` - Calls: model_output.strip, stripped.startswith, stripped[len(self.BOT_TOKEN):].lstrip, len, stripped[len(self.BOT_STRING):].lstrip, ExtractedToolCallInformation, json.loads, isinstance, call.get, tool_calls.append, generate_tool_id, json.dumps, str - State reads: self.BOT_TOKEN, self.BOT_STRING - Return expressions: ExtractedToolCallInformation(tools_called=False, tool_calls=[], content=model_output); ExtractedToolCallInformation(tools_called=True, tool_calls=tool_calls, content=None) ## `vllm_mlx.tool_parsers.granite_tool_parser.GraniteToolParser.extract_tool_calls_streaming` - Kind: method - Signature: `def extract_tool_calls_streaming(self, previous_text: str, current_text: str, delta_text: str, previous_token_ids: Sequence[int] | None=None, current_token_ids: Sequence[int] | None=None, delta_token_ids: Sequence[int] | None=None, request: dict[str, Any] | None=None) -> dict[str, Any] | None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/granite_tool_parser.py#L107-L147 - Implementation: Method `GraniteToolParser.extract_tool_calls_streaming` calls `current_text.strip`, `stripped.startswith`, `self.extract_tool_calls`, `enumerate`; has 3 explicit return paths. Extract tool calls from streaming Granite model output. - Inputs: - `previous_text` (str; required): Required positional or keyword input. - `current_text` (str; required): Required positional or keyword input. - `delta_text` (str; required): Required positional or keyword input. - `previous_token_ids` (Sequence[int] | None; optional; default `None`): Optional positional or keyword input; defaults to `None`. - `current_token_ids` (Sequence[int] | None; optional; default `None`): Optional positional or keyword input; defaults to `None`. - `delta_token_ids` (Sequence[int] | None; optional; default `None`): Optional positional or keyword input; defaults to `None`. - `request` (dict[str, Any] | None; optional; default `None`): Optional positional or keyword input; defaults to `None`. - Return annotation: `dict[str, Any] | None` - Calls: current_text.strip, stripped.startswith, self.extract_tool_calls, enumerate - State reads: self.BOT_TOKEN, self.BOT_STRING, self.extract_tool_calls - Return expressions: {'content': delta_text}; {'tool_calls': [{'index': i, 'id': tc['id'], 'type': 'function', 'function': {'name': tc['name'], 'arguments': tc['argu…; None # Module `vllm_mlx.tool_parsers.harmony_tool_parser` Harmony tool call parser for GPT-OSS models. Harmony uses control tokens and channels for tool calling: <|channel|>commentary to=functions.get_weather <|constrain|>json <|message|>{"location": "San Francisco"} <|call|> The final response is in the 'final' channel: <|channel|>final <|message|>The weather is 72F. <|return|> Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/harmony_tool_parser.py#L1-L253 ## `vllm_mlx.tool_parsers.harmony_tool_parser._generate_tool_id` - Kind: function - Signature: `def _generate_tool_id() -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/harmony_tool_parser.py#L32-L34 - Implementation: Function `_generate_tool_id` calls `uuid.uuid4`; returns `f'call_{uuid.uuid4().hex[:8]}'`. Generate a unique tool call ID. - Inputs: none - Return annotation: `str` - Calls: uuid.uuid4 - Return expressions: f'call_{uuid.uuid4().hex[:8]}' ## `vllm_mlx.tool_parsers.harmony_tool_parser.HarmonyToolParser` - Kind: class - Signature: `class HarmonyToolParser(ToolParser)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/harmony_tool_parser.py#L57-L219 - Implementation: Class `HarmonyToolParser` derives from `ToolParser` and declares 3 direct member(s). Tool call parser for GPT-OSS models using Harmony format. Harmony uses control tokens and 3 channels: - analysis: internal reasoning (handled by reasoning parser) - commentary: tool calls addressed with to=functions.{name} - final: user-facing response Used when --enable-auto-tool-choice --tool-call-parser harmony are set. - Inputs: none - Constructs: `vllm_mlx.tool_parsers.harmony_tool_parser.HarmonyToolParser` - Decorators: ToolParserManager.register_module(['harmony', 'gpt-oss']) ## `vllm_mlx.tool_parsers.harmony_tool_parser.HarmonyToolParser.extract_tool_calls` - Kind: method - Signature: `def extract_tool_calls(self, model_output: str, request: dict[str, Any] | None=None) -> ExtractedToolCallInformation` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/harmony_tool_parser.py#L71-L140 - Implementation: Method `HarmonyToolParser.extract_tool_calls` calls `_COMMENTARY_BLOCK_PATTERN.finditer`, `match.group`, `match.group(2).strip`, `json.loads`; has 2 explicit return paths. Extract tool calls from a complete Harmony model response. Parses commentary channel blocks for tool calls and the final channel for the user-facing content. - Inputs: - `model_output` (str; required): Required positional or keyword input. - `request` (dict[str, Any] | None; optional; default `None`): Optional positional or keyword input; defaults to `None`. - Return annotation: `ExtractedToolCallInformation` - Calls: _COMMENTARY_BLOCK_PATTERN.finditer, match.group, match.group(2).strip, json.loads, tool_calls.append, _generate_tool_id, isinstance, json.dumps, str, _FINAL_BLOCK_PATTERN.search, final_match.group(1).strip, final_match.group, ExtractedToolCallInformation, _strip_control_tokens - Return expressions: ExtractedToolCallInformation(tools_called=True, tool_calls=tool_calls, content=content); ExtractedToolCallInformation(tools_called=False, tool_calls=[], content=content) ## `vllm_mlx.tool_parsers.harmony_tool_parser.HarmonyToolParser.extract_tool_calls_streaming` - Kind: method - Signature: `def extract_tool_calls_streaming(self, previous_text: str, current_text: str, delta_text: str, previous_token_ids: Sequence[int] | None=None, current_token_ids: Sequence[int] | None=None, delta_token_ids: Sequence[int] | None=None, request: dict[str, Any] | None=None) -> dict[str, Any] | None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/harmony_tool_parser.py#L142-L214 - Implementation: Method `HarmonyToolParser.extract_tool_calls_streaming` updates `self._emitted_streaming_signatures`; calls `hasattr`, `set`, `any`, `self.extract_tool_calls`; has 3 explicit return paths. Extract tool calls from streaming Harmony model output. A commentary block completes when an explicit terminator arrives (<|call|>, <|end|>, <|return|>, <|start|>) or when the model moves on to the <|channel|>final block; the completed call is emitted once (deduplicated by name + arguments). Final-channel content is emitted as regular content deltas and plain text passes through unchanged. - Inputs: - `previous_text` (str; required): Required positional or keyword input. - `current_text` (str; required): Required positional or keyword input. - `delta_text` (str; required): Required positional or keyword input. - `previous_token_ids` (Sequence[int] | None; optional; default `None`): Optional positional or keyword input; defaults to `None`. - `current_token_ids` (Sequence[int] | None; optional; default `None`): Optional positional or keyword input; defaults to `None`. - `delta_token_ids` (Sequence[int] | None; optional; default `None`): Optional positional or keyword input; defaults to `None`. - `request` (dict[str, Any] | None; optional; default `None`): Optional positional or keyword input; defaults to `None`. - Return annotation: `dict[str, Any] | None` - Calls: hasattr, set, any, self.extract_tool_calls, enumerate, self._emitted_streaming_signatures.add, emitted.append, current_text.rfind, current_text.find, len, msg_content.replace('<|return|>', '').strip, msg_content.replace, _is_control_token - State reads: self.extract_tool_calls, self._emitted_streaming_signatures, self._emitted_streaming_signatures.add - State writes: self._emitted_streaming_signatures - Return expressions: {'content': delta_text}; {'tool_calls': emitted}; None ## `vllm_mlx.tool_parsers.harmony_tool_parser.HarmonyToolParser.reset` - Kind: method - Signature: `def reset(self) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/harmony_tool_parser.py#L216-L219 - Implementation: Method `HarmonyToolParser.reset` updates `self._emitted_streaming_signatures`; calls `super().reset`, `super`, `set`. Reset parser state for a new request. - Inputs: none - Return annotation: `None` - Calls: super().reset, super, set - State writes: self._emitted_streaming_signatures ## `vllm_mlx.tool_parsers.harmony_tool_parser._strip_control_tokens` - Kind: function - Signature: `def _strip_control_tokens(text: str) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/harmony_tool_parser.py#L222-L240 - Implementation: Function `_strip_control_tokens` calls `result.replace`, `re.sub`, `result.strip`; returns `result.strip()`. Remove Harmony control tokens from text. - Inputs: - `text` (str; required): Required positional or keyword input. - Return annotation: `str` - Calls: result.replace, re.sub, result.strip - Return expressions: result.strip() ## `vllm_mlx.tool_parsers.harmony_tool_parser._is_control_token` - Kind: function - Signature: `def _is_control_token(text: str) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/harmony_tool_parser.py#L243-L253 - Implementation: Function `_is_control_token` calls `text.strip`; returns `text.strip() in {'<|start|>', '<|end|>', '<|message|>', '<|channel|>', '<|constrain|>', '<|return|>', '<|call|>'}`. Check if text is a Harmony control token. - Inputs: - `text` (str; required): Required positional or keyword input. - Return annotation: `bool` - Calls: text.strip - Return expressions: text.strip() in {'<|start|>', '<|end|>', '<|message|>', '<|channel|>', '<|constrain|>', '<|return|>', '<|call|>'} # Module `vllm_mlx.tool_parsers.hermes_tool_parser` Hermes/Nous tool call parser for vllm-mlx. Handles Hermes-style tool calling format used by NousResearch models. Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/hermes_tool_parser.py#L1-L336 ## `vllm_mlx.tool_parsers.hermes_tool_parser.generate_tool_id` - Kind: function - Signature: `def generate_tool_id() -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/hermes_tool_parser.py#L22-L24 - Implementation: Function `generate_tool_id` calls `uuid.uuid4`; returns `f'call_{uuid.uuid4().hex[:8]}'`. Generate a unique tool call ID. - Inputs: none - Return annotation: `str` - Calls: uuid.uuid4 - Return expressions: f'call_{uuid.uuid4().hex[:8]}' ## `vllm_mlx.tool_parsers.hermes_tool_parser._parse_param_value` - Kind: function - Signature: `def _parse_param_value(val: str) -> Any` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/hermes_tool_parser.py#L27-L49 - Implementation: Function `_parse_param_value` calls `json.loads`, `ast.literal_eval`, `isinstance`, `sorted`; has 3 explicit return paths. Parse a tool call parameter value, handling both JSON and Python literals. Tries json.loads first. If that fails, falls back to ast.literal_eval for Python literal syntax (single quotes, True/False, None). Converts sets to lists and rejects types that are not JSON-serializable (complex, bytes) to avoid crashes during json.dumps later. - Inputs: - `val` (str; required): Required positional or keyword input. - Return annotation: `Any` - Calls: json.loads, ast.literal_eval, isinstance, sorted, json.dumps - Return expressions: json.loads(val); val; python_val ## `vllm_mlx.tool_parsers.hermes_tool_parser.HermesToolParser` - Kind: class - Signature: `class HermesToolParser(ToolParser)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/hermes_tool_parser.py#L53-L336 - Implementation: Class `HermesToolParser` derives from `ToolParser` and declares 3 direct member(s). Tool call parser for Hermes/Nous models. Supports Hermes tool call format: - {"name": "func", "arguments": {...}} - Sometimes with additional reasoning in - Fallback: raw JSON {"name": "func", "arguments": {...}} (for models that omit tags) Used when --enable-auto-tool-choice --tool-call-parser hermes are set. - Inputs: none - Constructs: `vllm_mlx.tool_parsers.hermes_tool_parser.HermesToolParser` - Decorators: ToolParserManager.register_module(['hermes', 'nous']) ## `vllm_mlx.tool_parsers.hermes_tool_parser.HermesToolParser.extract_tool_calls` - Kind: method - Signature: `def extract_tool_calls(self, model_output: str, request: dict[str, Any] | None=None) -> ExtractedToolCallInformation` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/hermes_tool_parser.py#L92-L245 - Implementation: Method `HermesToolParser.extract_tool_calls` calls `self.strip_think_tags`, `self.REASONING_PATTERN.findall`, `self.REASONING_PATTERN.sub`, `self.TOOL_CALL_PATTERN.findall`; has 2 explicit return paths. Extract tool calls from a complete Hermes model response. - Inputs: - `model_output` (str; required): Required positional or keyword input. - `request` (dict[str, Any] | None; optional; default `None`): Optional positional or keyword input; defaults to `None`. - Return annotation: `ExtractedToolCallInformation` - Calls: self.strip_think_tags, self.REASONING_PATTERN.findall, self.REASONING_PATTERN.sub, self.TOOL_CALL_PATTERN.findall, json.loads, data.get, tool_calls.append, generate_tool_id, isinstance, json.dumps, str, self.TOOL_CALL_PATTERN.sub('', cleaned_text).strip, self.TOOL_CALL_PATTERN.sub, self.NEMOTRON_PATTERN.findall, self.PARAM_PATTERN.findall, p_name.strip, _parse_param_value, p_value.strip, name.strip, self.NEMOTRON_PATTERN.sub('', cleaned_text).strip, self.NEMOTRON_PATTERN.sub, self.BARE_FUNCTION_PATTERN.findall, self.BARE_FUNCTION_PATTERN.sub('', cleaned_text).strip, self.BARE_FUNCTION_PATTERN.sub, self.TOOL_CALL_LENIENT_PATTERN.findall, self.TOOL_CALL_LENIENT_PATTERN.sub('', cleaned_text, count=1).strip, self.TOOL_CALL_LENIENT_PATTERN.sub, self.RAW_JSON_TOOL_PATTERN.findall, t.get('function', {}).get, t.get, request.get, self.RAW_JSON_TOOL_PATTERN.sub('', cleaned_text, count=1).strip, self.RAW_JSON_TOOL_PATTERN.sub, ' '.join, ExtractedToolCallInformation - State reads: self.strip_think_tags, self.REASONING_PATTERN.findall, self.REASONING_PATTERN, self.REASONING_PATTERN.sub, self.TOOL_CALL_PATTERN.findall, self.TOOL_CALL_PATTERN, self.TOOL_CALL_PATTERN.sub, self.NEMOTRON_PATTERN.findall, self.NEMOTRON_PATTERN, self.PARAM_PATTERN.findall, self.PARAM_PATTERN, self.NEMOTRON_PATTERN.sub, self.BARE_FUNCTION_PATTERN.findall, self.BARE_FUNCTION_PATTERN, self.BARE_FUNCTION_PATTERN.sub, self.TOOL_CALL_LENIENT_PATTERN.findall, self.TOOL_CALL_LENIENT_PATTERN, self.TOOL_CALL_LENIENT_PATTERN.sub, self.RAW_JSON_TOOL_PATTERN.findall, self.RAW_JSON_TOOL_PATTERN, self.RAW_JSON_TOOL_PATTERN.sub - Return expressions: ExtractedToolCallInformation(tools_called=True, tool_calls=tool_calls, content=cleaned_text if cleaned_text else None); ExtractedToolCallInformation(tools_called=False, tool_calls=[], content=cleaned_text) ## `vllm_mlx.tool_parsers.hermes_tool_parser.HermesToolParser._format_streaming_tool_calls` - Kind: method - Signature: `def _format_streaming_tool_calls(tool_calls: list[dict], start_index: int=0) -> dict[str, Any]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/hermes_tool_parser.py#L248-L265 - Implementation: Method `HermesToolParser._format_streaming_tool_calls` calls `enumerate`; returns `{'tool_calls': [{'index': start_index + i, 'id': tc['id'], 'type': 'function', 'function': {'name': tc['name'], 'argume…`. Format tool calls for streaming response. - Inputs: - `tool_calls` (list[dict]; required): Required positional or keyword input. - `start_index` (int; optional; default `0`): Optional positional or keyword input; defaults to `0`. - Return annotation: `dict[str, Any]` - Decorators: staticmethod - Calls: enumerate - Return expressions: {'tool_calls': [{'index': start_index + i, 'id': tc['id'], 'type': 'function', 'function': {'name': tc['name'], 'argume… ## `vllm_mlx.tool_parsers.hermes_tool_parser.HermesToolParser.extract_tool_calls_streaming` - Kind: method - Signature: `def extract_tool_calls_streaming(self, previous_text: str, current_text: str, delta_text: str, previous_token_ids: Sequence[int] | None=None, current_token_ids: Sequence[int] | None=None, delta_token_ids: Sequence[int] | None=None, request: dict[str, Any] | None=None) -> dict[str, Any] | None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/hermes_tool_parser.py#L267-L336 - Implementation: Method `HermesToolParser.extract_tool_calls_streaming` calls `current_text.count`, `previous_text.count`, `self.extract_tool_calls`, `self._format_streaming_tool_calls`; has 5 explicit return paths. Extract tool calls from streaming Hermes model output. Uses tag counting to correctly handle multiple sequential tool calls. - Inputs: - `previous_text` (str; required): Required positional or keyword input. - `current_text` (str; required): Required positional or keyword input. - `delta_text` (str; required): Required positional or keyword input. - `previous_token_ids` (Sequence[int] | None; optional; default `None`): Optional positional or keyword input; defaults to `None`. - `current_token_ids` (Sequence[int] | None; optional; default `None`): Optional positional or keyword input; defaults to `None`. - `delta_token_ids` (Sequence[int] | None; optional; default `None`): Optional positional or keyword input; defaults to `None`. - `request` (dict[str, Any] | None; optional; default `None`): Optional positional or keyword input; defaults to `None`. - Return annotation: `dict[str, Any] | None` - Calls: current_text.count, previous_text.count, self.extract_tool_calls, self._format_streaming_tool_calls, delta_text.rstrip().endswith, delta_text.rstrip - State reads: self.extract_tool_calls, self._format_streaming_tool_calls - Return expressions: None; self._format_streaming_tool_calls(new_calls, start_index=prev_close_count); {'content': delta_text}; self._format_streaming_tool_calls(new_calls, start_index=prev_func_close); self._format_streaming_tool_calls(result.tool_calls) # Module `vllm_mlx.tool_parsers.kimi_tool_parser` Kimi/Moonshot tool call parser for vllm-mlx. Handles Kimi K2 and related models' tool calling format: - <|tool_calls_section_begin|>...<|tool_calls_section_end|> - <|tool_call_begin|>func_name:0<|tool_call_argument_begin|>{...}<|tool_call_end|> Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/kimi_tool_parser.py#L1-L160 ## `vllm_mlx.tool_parsers.kimi_tool_parser.generate_tool_id` - Kind: function - Signature: `def generate_tool_id() -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/kimi_tool_parser.py#L23-L25 - Implementation: Function `generate_tool_id` calls `uuid.uuid4`; returns `f'call_{uuid.uuid4().hex[:8]}'`. Generate a unique tool call ID. - Inputs: none - Return annotation: `str` - Calls: uuid.uuid4 - Return expressions: f'call_{uuid.uuid4().hex[:8]}' ## `vllm_mlx.tool_parsers.kimi_tool_parser.KimiToolParser` - Kind: class - Signature: `class KimiToolParser(ToolParser)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/kimi_tool_parser.py#L29-L160 - Implementation: Class `KimiToolParser` derives from `ToolParser` and declares 3 direct member(s). Tool call parser for Kimi K2 and Moonshot models. Supports Kimi's tool call format: <|tool_calls_section_begin|> <|tool_call_begin|>func:0<|tool_call_argument_begin|>{...}<|tool_call_end|> <|tool_calls_section_end|> Used when --enable-auto-tool-choice --tool-call-parser kimi are set. - Inputs: none - Constructs: `vllm_mlx.tool_parsers.kimi_tool_parser.KimiToolParser` - Decorators: ToolParserManager.register_module(['kimi', 'kimi_k2', 'moonshot']) ## `vllm_mlx.tool_parsers.kimi_tool_parser.KimiToolParser._has_tool_section` - Kind: method - Signature: `def _has_tool_section(self, text: str) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/kimi_tool_parser.py#L59-L65 - Implementation: Method `KimiToolParser._has_tool_section` returns `self.TOOL_CALLS_START in text or self.TOOL_CALLS_START_ALT in text or self.TOOL_CALL_START in text`. Check if text contains tool section markers. - Inputs: - `text` (str; required): Required positional or keyword input. - Return annotation: `bool` - State reads: self.TOOL_CALLS_START, self.TOOL_CALLS_START_ALT, self.TOOL_CALL_START - Return expressions: self.TOOL_CALLS_START in text or self.TOOL_CALLS_START_ALT in text or self.TOOL_CALL_START in text ## `vllm_mlx.tool_parsers.kimi_tool_parser.KimiToolParser.extract_tool_calls` - Kind: method - Signature: `def extract_tool_calls(self, model_output: str, request: dict[str, Any] | None=None) -> ExtractedToolCallInformation` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/kimi_tool_parser.py#L67-L124 - Implementation: Method `KimiToolParser.extract_tool_calls` calls `self._has_tool_section`, `ExtractedToolCallInformation`, `model_output.find`, `model_output[:idx].strip`; has 2 explicit return paths. Extract tool calls from Kimi model output. - Inputs: - `model_output` (str; required): Required positional or keyword input. - `request` (dict[str, Any] | None; optional; default `None`): Optional positional or keyword input; defaults to `None`. - Return annotation: `ExtractedToolCallInformation` - Calls: self._has_tool_section, ExtractedToolCallInformation, model_output.find, model_output[:idx].strip, self.TOOL_CALL_PATTERN.findall, func_id.split, func_name.split, json.loads, tool_calls.append, generate_tool_id, func_name.strip, func_args.strip - State reads: self._has_tool_section, self.TOOL_CALLS_START, self.TOOL_CALLS_START_ALT, self.TOOL_CALL_PATTERN.findall, self.TOOL_CALL_PATTERN - Return expressions: ExtractedToolCallInformation(tools_called=False, tool_calls=[], content=model_output); ExtractedToolCallInformation(tools_called=True, tool_calls=tool_calls, content=content) ## `vllm_mlx.tool_parsers.kimi_tool_parser.KimiToolParser.extract_tool_calls_streaming` - Kind: method - Signature: `def extract_tool_calls_streaming(self, previous_text: str, current_text: str, delta_text: str, previous_token_ids: Sequence[int] | None=None, current_token_ids: Sequence[int] | None=None, delta_token_ids: Sequence[int] | None=None, request: dict[str, Any] | None=None) -> dict[str, Any] | None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/kimi_tool_parser.py#L126-L160 - Implementation: Method `KimiToolParser.extract_tool_calls_streaming` calls `self._has_tool_section`, `self.extract_tool_calls`, `enumerate`; has 3 explicit return paths. Extract tool calls from streaming Kimi model output. - Inputs: - `previous_text` (str; required): Required positional or keyword input. - `current_text` (str; required): Required positional or keyword input. - `delta_text` (str; required): Required positional or keyword input. - `previous_token_ids` (Sequence[int] | None; optional; default `None`): Optional positional or keyword input; defaults to `None`. - `current_token_ids` (Sequence[int] | None; optional; default `None`): Optional positional or keyword input; defaults to `None`. - `delta_token_ids` (Sequence[int] | None; optional; default `None`): Optional positional or keyword input; defaults to `None`. - `request` (dict[str, Any] | None; optional; default `None`): Optional positional or keyword input; defaults to `None`. - Return annotation: `dict[str, Any] | None` - Calls: self._has_tool_section, self.extract_tool_calls, enumerate - State reads: self._has_tool_section, self.TOOL_CALL_END, self.extract_tool_calls - Return expressions: {'content': delta_text}; {'tool_calls': [{'index': i, 'id': tc['id'], 'type': 'function', 'function': {'name': tc['name'], 'arguments': tc['argu…; None # Module `vllm_mlx.tool_parsers.llama_tool_parser` Llama tool call parser for vllm-mlx. Handles Llama's tool calling format: - XML style: {"arg": "value"} Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/llama_tool_parser.py#L1-L128 ## `vllm_mlx.tool_parsers.llama_tool_parser.generate_tool_id` - Kind: function - Signature: `def generate_tool_id() -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/llama_tool_parser.py#L22-L24 - Implementation: Function `generate_tool_id` calls `uuid.uuid4`; returns `f'call_{uuid.uuid4().hex[:8]}'`. Generate a unique tool call ID. - Inputs: none - Return annotation: `str` - Calls: uuid.uuid4 - Return expressions: f'call_{uuid.uuid4().hex[:8]}' ## `vllm_mlx.tool_parsers.llama_tool_parser.LlamaToolParser` - Kind: class - Signature: `class LlamaToolParser(ToolParser)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/llama_tool_parser.py#L28-L128 - Implementation: Class `LlamaToolParser` derives from `ToolParser` and declares 2 direct member(s). Tool call parser for Llama models. Supports Llama tool call format: - {"arg": "value"} Used when --enable-auto-tool-choice --tool-call-parser llama are set. - Inputs: none - Constructs: `vllm_mlx.tool_parsers.llama_tool_parser.LlamaToolParser` - Decorators: ToolParserManager.register_module(['llama', 'llama3', 'llama4']) ## `vllm_mlx.tool_parsers.llama_tool_parser.LlamaToolParser.extract_tool_calls` - Kind: method - Signature: `def extract_tool_calls(self, model_output: str, request: dict[str, Any] | None=None) -> ExtractedToolCallInformation` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/llama_tool_parser.py#L44-L90 - Implementation: Method `LlamaToolParser.extract_tool_calls` calls `self.FUNCTION_PATTERN.findall`, `json.loads`, `tool_calls.append`, `generate_tool_id`; has 2 explicit return paths. Extract tool calls from a complete Llama model response. - Inputs: - `model_output` (str; required): Required positional or keyword input. - `request` (dict[str, Any] | None; optional; default `None`): Optional positional or keyword input; defaults to `None`. - Return annotation: `ExtractedToolCallInformation` - Calls: self.FUNCTION_PATTERN.findall, json.loads, tool_calls.append, generate_tool_id, name.strip, isinstance, json.dumps, str, self.FUNCTION_PATTERN.sub('', cleaned_text).strip, self.FUNCTION_PATTERN.sub, ExtractedToolCallInformation - State reads: self.FUNCTION_PATTERN.findall, self.FUNCTION_PATTERN, self.FUNCTION_PATTERN.sub - Return expressions: ExtractedToolCallInformation(tools_called=True, tool_calls=tool_calls, content=cleaned_text if cleaned_text else None); ExtractedToolCallInformation(tools_called=False, tool_calls=[], content=model_output) ## `vllm_mlx.tool_parsers.llama_tool_parser.LlamaToolParser.extract_tool_calls_streaming` - Kind: method - Signature: `def extract_tool_calls_streaming(self, previous_text: str, current_text: str, delta_text: str, previous_token_ids: Sequence[int] | None=None, current_token_ids: Sequence[int] | None=None, delta_token_ids: Sequence[int] | None=None, request: dict[str, Any] | None=None) -> dict[str, Any] | None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/llama_tool_parser.py#L92-L128 - Implementation: Method `LlamaToolParser.extract_tool_calls_streaming` calls `self.extract_tool_calls`, `enumerate`; has 3 explicit return paths. Extract tool calls from streaming Llama model output. - Inputs: - `previous_text` (str; required): Required positional or keyword input. - `current_text` (str; required): Required positional or keyword input. - `delta_text` (str; required): Required positional or keyword input. - `previous_token_ids` (Sequence[int] | None; optional; default `None`): Optional positional or keyword input; defaults to `None`. - `current_token_ids` (Sequence[int] | None; optional; default `None`): Optional positional or keyword input; defaults to `None`. - `delta_token_ids` (Sequence[int] | None; optional; default `None`): Optional positional or keyword input; defaults to `None`. - `request` (dict[str, Any] | None; optional; default `None`): Optional positional or keyword input; defaults to `None`. - Return annotation: `dict[str, Any] | None` - Calls: self.extract_tool_calls, enumerate - State reads: self.extract_tool_calls - Return expressions: {'content': delta_text}; {'tool_calls': [{'index': i, 'id': tc['id'], 'type': 'function', 'function': {'name': tc['name'], 'arguments': tc['argu…; None # Module `vllm_mlx.tool_parsers.minimax_tool_parser` MiniMax tool call parser for vllm-mlx. Parses the MiniMax-M2 native XML tool call format: param-value Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/minimax_tool_parser.py#L1-L178 ## `vllm_mlx.tool_parsers.minimax_tool_parser.generate_tool_id` - Kind: function - Signature: `def generate_tool_id() -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/minimax_tool_parser.py#L26-L29 - Implementation: Function `generate_tool_id` calls `uuid.uuid4`; returns `f'call_{uuid.uuid4().hex[:8]}'`. Return a short OpenAI-compatible identifier for a parsed tool call. - Inputs: none - Return annotation: `str` - Calls: uuid.uuid4 - Return expressions: f'call_{uuid.uuid4().hex[:8]}' ## `vllm_mlx.tool_parsers.minimax_tool_parser.MiniMaxToolParser` - Kind: class - Signature: `class MiniMaxToolParser(ToolParser)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/minimax_tool_parser.py#L33-L178 - Implementation: Class `MiniMaxToolParser` derives from `ToolParser` and declares 5 direct member(s). Parser for MiniMax-M2 tool call format. Format: value - Inputs: none - Constructs: `vllm_mlx.tool_parsers.minimax_tool_parser.MiniMaxToolParser` - Decorators: ToolParserManager.register_module(['minimax', 'minimax_m2']) ## `vllm_mlx.tool_parsers.minimax_tool_parser.MiniMaxToolParser._extract_invokes` - Kind: method - Signature: `def _extract_invokes(self, text: str) -> list[dict[str, Any]]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/minimax_tool_parser.py#L54-L78 - Implementation: Method `MiniMaxToolParser._extract_invokes` calls `self.INVOKE_PATTERN.findall`, `self.PARAM_PATTERN.findall`, `p_value.strip`, `json.loads`; returns `tool_calls`. Extract tool calls from invoke elements, with or without wrapper. - Inputs: - `text` (str; required): Required positional or keyword input. - Return annotation: `list[dict[str, Any]]` - Calls: self.INVOKE_PATTERN.findall, self.PARAM_PATTERN.findall, p_value.strip, json.loads, tool_calls.append, generate_tool_id, func_name.strip, json.dumps - State reads: self.INVOKE_PATTERN.findall, self.INVOKE_PATTERN, self.PARAM_PATTERN.findall, self.PARAM_PATTERN - Return expressions: tool_calls ## `vllm_mlx.tool_parsers.minimax_tool_parser.MiniMaxToolParser.extract_tool_calls` - Kind: method - Signature: `def extract_tool_calls(self, model_output: str, request: dict[str, Any] | None=None) -> ExtractedToolCallInformation` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/minimax_tool_parser.py#L80-L121 - Implementation: Method `MiniMaxToolParser.extract_tool_calls` calls `self.TOOL_CALL_BLOCK.findall`, `tool_calls.extend`, `self._extract_invokes`, `self.TOOL_CALL_BLOCK.sub('', model_output).strip`; has 3 explicit return paths. Extract wrapped or bare MiniMax invoke elements from complete output. - Inputs: - `model_output` (str; required): Required positional or keyword input. - `request` (dict[str, Any] | None; optional; default `None`): Optional positional or keyword input; defaults to `None`. - Return annotation: `ExtractedToolCallInformation` - Calls: self.TOOL_CALL_BLOCK.findall, tool_calls.extend, self._extract_invokes, self.TOOL_CALL_BLOCK.sub('', model_output).strip, self.TOOL_CALL_BLOCK.sub, self.THINK_PATTERN.sub('', cleaned).strip, self.THINK_PATTERN.sub, re.sub('\\[e~\\[.*$', '', cleaned).strip, re.sub, ExtractedToolCallInformation, bool, self.INVOKE_PATTERN.sub('', model_output).strip, self.INVOKE_PATTERN.sub, cleaned.replace('', '').strip, cleaned.replace - State reads: self.TOOL_CALL_BLOCK.findall, self.TOOL_CALL_BLOCK, self._extract_invokes, self.TOOL_CALL_BLOCK.sub, self.THINK_PATTERN.sub, self.THINK_PATTERN, self.INVOKE_PATTERN.sub, self.INVOKE_PATTERN - Return expressions: ExtractedToolCallInformation(tools_called=bool(tool_calls), tool_calls=tool_calls, content=cleaned if cleaned else None); ExtractedToolCallInformation(tools_called=True, tool_calls=tool_calls, content=cleaned if cleaned else None); ExtractedToolCallInformation(tools_called=False, tool_calls=[], content=model_output) ## `vllm_mlx.tool_parsers.minimax_tool_parser.MiniMaxToolParser._has_tool_start` - Kind: method - Signature: `def _has_tool_start(self, text: str) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/minimax_tool_parser.py#L123-L127 - Implementation: Method `MiniMaxToolParser._has_tool_start` calls `self.INVOKE_PATTERN.search`; returns `'' in text or (' bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/minimax_tool_parser.py#L129-L140 - Implementation: Method `MiniMaxToolParser._has_tool_end` has 3 explicit return paths. Check if a tool call block just completed. - Inputs: - `current` (str; required): Required positional or keyword input. - `previous` (str; required): Required positional or keyword input. - Return annotation: `bool` - Return expressions: '' in current and '' not in previous; True; False ## `vllm_mlx.tool_parsers.minimax_tool_parser.MiniMaxToolParser.extract_tool_calls_streaming` - Kind: method - Signature: `def extract_tool_calls_streaming(self, previous_text: str, current_text: str, delta_text: str, previous_token_ids: Sequence[int] | None=None, current_token_ids: Sequence[int] | None=None, delta_token_ids: Sequence[int] | None=None, request: dict[str, Any] | None=None) -> dict[str, Any] | None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/minimax_tool_parser.py#L142-L178 - Implementation: Method `MiniMaxToolParser.extract_tool_calls_streaming` calls `self._has_tool_start`, `self._has_tool_end`, `self.extract_tool_calls`, `enumerate`; has 3 explicit return paths. Emit content deltas or a completed MiniMax tool-call delta. - Inputs: - `previous_text` (str; required): Required positional or keyword input. - `current_text` (str; required): Required positional or keyword input. - `delta_text` (str; required): Required positional or keyword input. - `previous_token_ids` (Sequence[int] | None; optional; default `None`): Optional positional or keyword input; defaults to `None`. - `current_token_ids` (Sequence[int] | None; optional; default `None`): Optional positional or keyword input; defaults to `None`. - `delta_token_ids` (Sequence[int] | None; optional; default `None`): Optional positional or keyword input; defaults to `None`. - `request` (dict[str, Any] | None; optional; default `None`): Optional positional or keyword input; defaults to `None`. - Return annotation: `dict[str, Any] | None` - Calls: self._has_tool_start, self._has_tool_end, self.extract_tool_calls, enumerate - State reads: self._has_tool_start, self._has_tool_end, self.extract_tool_calls - Return expressions: {'content': delta_text}; {'tool_calls': [{'index': i, 'id': tc['id'], 'type': 'function', 'function': {'name': tc['name'], 'arguments': tc['argu…; None # Module `vllm_mlx.tool_parsers.mistral_tool_parser` Mistral tool call parser for vllm-mlx. Handles Mistral's tool calling format: - Format: ``[TOOL_CALLS] [{"name": "func", "arguments": {...}}]`` - Or newer: ``[TOOL_CALLS]func_name{"arg": "value"}`` - Or newest (Ministral 3, Devstral Small 2, Dec 2025 tokenizers): ``[TOOL_CALLS]func_name[ARGS]{"arg": "value"}`` Confirmed directly in these models' chat_template.jinja: ``{{- '[TOOL_CALLS]' + tool['function']['name'] + '[ARGS]' + arguments }}`` Used with models like Mistral-7B-Instruct, Devstral, Ministral 3, etc. Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/mistral_tool_parser.py#L1-L512 ## `vllm_mlx.tool_parsers.mistral_tool_parser.generate_mistral_tool_id` - Kind: function - Signature: `def generate_mistral_tool_id() -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/mistral_tool_parser.py#L34-L40 - Implementation: Function `generate_mistral_tool_id` calls `''.join`, `choices`; returns `''.join(choices(ALPHANUMERIC, k=9))`. Generate a random Mistral-compatible tool call ID. Mistral Tool Call IDs must be alphanumeric with a length of 9. - Inputs: none - Return annotation: `str` - Calls: ''.join, choices - Return expressions: ''.join(choices(ALPHANUMERIC, k=9)) ## `vllm_mlx.tool_parsers.mistral_tool_parser._is_plain_tool_name` - Kind: function - Signature: `def _is_plain_tool_name(name: str) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/mistral_tool_parser.py#L43-L45 - Implementation: Function `_is_plain_tool_name` calls `bool`, `_TOOL_NAME_PATTERN.match`; returns `bool(_TOOL_NAME_PATTERN.match(name))`. Return True for names that are safe to dispatch as function calls. - Inputs: - `name` (str; required): Required positional or keyword input. - Return annotation: `bool` - Calls: bool, _TOOL_NAME_PATTERN.match - Return expressions: bool(_TOOL_NAME_PATTERN.match(name)) ## `vllm_mlx.tool_parsers.mistral_tool_parser.MistralToolParser` - Kind: class - Signature: `class MistralToolParser(ToolParser)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/mistral_tool_parser.py#L49-L512 - Implementation: Class `MistralToolParser` derives from `ToolParser` and declares 8 direct member(s). Tool call parser for Mistral models. Supports both old and new Mistral tool call formats: - Old (< v11): ``[TOOL_CALLS] [{"name": "add", "arguments": {"a": 1, "b": 2}}]`` - New (>= v11): ``[TOOL_CALLS]add{"a": 1, "b": 2}`` Used when --enable-auto-tool-choice --tool-call-parser mistral are set. - Inputs: - `tokenizer` (not annotated; optional; default `None`): Optional positional or keyword input; defaults to `None`. - Constructs: `vllm_mlx.tool_parsers.mistral_tool_parser.MistralToolParser` - Decorators: ToolParserManager.register_module('mistral') ## `vllm_mlx.tool_parsers.mistral_tool_parser.MistralToolParser.__init__` - Kind: method - Signature: `def __init__(self, tokenizer=None)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/mistral_tool_parser.py#L68-L90 - Implementation: Method `MistralToolParser.__init__` updates `self.bot_token_id`, `self._args_started`, `self._args_in_string`, `self._args_escaped`; calls `super().__init__`, `super`, `self.vocab.get`. Method `MistralToolParser.__init__` updates `self.bot_token_id`, `self._args_started`, `self._args_in_string`, `self._args_escaped`; calls `super().__init__`, `super`, `self.vocab.get`. - Inputs: - `tokenizer` (not annotated; optional; default `None`): Optional positional or keyword input; defaults to `None`. - Return annotation: `not annotated` - Calls: super().__init__, super, self.vocab.get - State reads: self.vocab, self.vocab.get, self.BOT_TOKEN - State writes: self.bot_token_id, self._args_started, self._args_in_string, self._args_escaped, self._name_buffer, self._name_buffer_overflow, self._current_tool_call_id, self._tool_call_id_emitted ## `vllm_mlx.tool_parsers.mistral_tool_parser.MistralToolParser.reset` - Kind: method - Signature: `def reset(self) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/mistral_tool_parser.py#L92-L102 - Implementation: Method `MistralToolParser.reset` updates `self._args_started`, `self._args_in_string`, `self._args_escaped`, `self._name_buffer`; calls `super().reset`, `super`. Reset shared and Mistral-specific streaming tool-call state. - Inputs: none - Return annotation: `None` - Calls: super().reset, super - State writes: self._args_started, self._args_in_string, self._args_escaped, self._name_buffer, self._name_buffer_overflow, self._current_tool_call_id, self._tool_call_id_emitted ## `vllm_mlx.tool_parsers.mistral_tool_parser.MistralToolParser._start_new_tool_call` - Kind: method - Signature: `def _start_new_tool_call(self) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/mistral_tool_parser.py#L104-L114 - Implementation: Method `MistralToolParser._start_new_tool_call` updates `self.current_tool_id`, `self._args_started`, `self._args_in_string`, `self._args_escaped`; calls `generate_mistral_tool_id`. Begin a new streaming tool call: bump the index and reset the per-call name/arguments and id state. - Inputs: none - Return annotation: `None` - Calls: generate_mistral_tool_id - State writes: self.current_tool_id, self._args_started, self._args_in_string, self._args_escaped, self._name_buffer, self._name_buffer_overflow, self._current_tool_call_id, self._tool_call_id_emitted ## `vllm_mlx.tool_parsers.mistral_tool_parser.MistralToolParser._scan_args_for_new_call` - Kind: method - Signature: `def _scan_args_for_new_call(self, text: str) -> int` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/mistral_tool_parser.py#L116-L146 - Implementation: Method `MistralToolParser._scan_args_for_new_call` updates `self._args_escaped`, `self._args_in_string`; calls `len`, `text.startswith`; has 2 explicit return paths. Scan an argument delta, updating the persistent JSON string state, and return the position of the first [TOOL_CALLS] marker that sits outside a string (a new call), or -1 when there is none. Quote state is carried across deltas so a marker inside a quoted value (e.g. ``{"city": "[TOOL_CALLS]rm"}``) stays argument data while a marker between two calls opens the next index. - Inputs: - `text` (str; required): Required positional or keyword input. - Return annotation: `int` - Calls: len, text.startswith - State reads: self._args_escaped, self._args_in_string, self.BOT_TOKEN - State writes: self._args_escaped, self._args_in_string - Return expressions: i; -1 ## `vllm_mlx.tool_parsers.mistral_tool_parser.MistralToolParser._split_on_tool_call_markers` - Kind: method - Signature: `def _split_on_tool_call_markers(self, text: str) -> list[str]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/mistral_tool_parser.py#L148-L192 - Implementation: Method `MistralToolParser._split_on_tool_call_markers` calls `text.find`, `len`, `text.startswith`, `parts.append`; has 2 explicit return paths. Split on [TOOL_CALLS] occurrences that are outside JSON strings. A marker appearing inside a quoted string value is argument data, not a new call — splitting there would let untrusted model output forge a second dispatchable call. The quote-state scan starts at the first marker, not at index 0: the text before the first marker is prose, not JSON, so an odd number of double quotes there must not leave ``in_string`` set when the marker arrives (that would hide the call entirely). - Inputs: - `text` (str; required): Required positional or keyword input. - Return annotation: `list[str]` - Calls: text.find, len, text.startswith, parts.append - State reads: self.BOT_TOKEN - Return expressions: [text]; parts ## `vllm_mlx.tool_parsers.mistral_tool_parser.MistralToolParser.extract_tool_calls` - Kind: method - Signature: `def extract_tool_calls(self, model_output: str, request: dict[str, Any] | None=None) -> ExtractedToolCallInformation` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/mistral_tool_parser.py#L194-L332 - Implementation: Method `MistralToolParser.extract_tool_calls` calls `ExtractedToolCallInformation`, `self._split_on_tool_call_markers`, `content_and_raw_tool_calls[0].strip`, `raw_tool_call.strip`; has 2 explicit return paths. Extract tool calls from a complete Mistral model response. Args: model_output: The complete model output string request: Optional request context Returns: ExtractedToolCallInformation with parsed tool calls - Inputs: - `model_output` (str; required): The complete model output string - `request` (dict[str, Any] | None; optional; default `None`): Optional request context - Return annotation: `ExtractedToolCallInformation` - Calls: ExtractedToolCallInformation, self._split_on_tool_call_markers, content_and_raw_tool_calls[0].strip, raw_tool_call.strip, raw_tool_call.find, raw_tool_call.startswith, raw_tool_call[:args_idx].strip, len, _is_plain_tool_name, json.loads, tool_calls.append, generate_mistral_tool_id, raw_tool_call[:end_name].strip, isinstance, item.get, json.dumps, str, self.TOOL_CALL_REGEX.search, match.group, (content + ' ' + raw_tool_call).strip - State reads: self.BOT_TOKEN, self._split_on_tool_call_markers, self.ARGS_TOKEN, self.TOOL_CALL_REGEX.search, self.TOOL_CALL_REGEX - Return expressions: ExtractedToolCallInformation(tools_called=False, tool_calls=[], content=model_output); ExtractedToolCallInformation(tools_called=True, tool_calls=tool_calls, content=content if content else None) ## `vllm_mlx.tool_parsers.mistral_tool_parser.MistralToolParser.extract_tool_calls_streaming` - Kind: method - Signature: `def extract_tool_calls_streaming(self, previous_text: str, current_text: str, delta_text: str, previous_token_ids: Sequence[int] | None=None, current_token_ids: Sequence[int] | None=None, delta_token_ids: Sequence[int] | None=None, request: dict[str, Any] | None=None) -> dict[str, Any] | None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/mistral_tool_parser.py#L334-L453 - Implementation: Method `MistralToolParser.extract_tool_calls_streaming` updates `self._tool_call_id_emitted`; calls `self._scan_args_for_new_call`, `self._start_new_tool_call`, `self._parse_streaming_tool_delta`, `len`; has 6 explicit return paths. Extract tool calls from streaming Mistral model output. For streaming, we detect when [TOOL_CALLS] appears and start accumulating tool call data. - Inputs: - `previous_text` (str; required): Required positional or keyword input. - `current_text` (str; required): Required positional or keyword input. - `delta_text` (str; required): Required positional or keyword input. - `previous_token_ids` (Sequence[int] | None; optional; default `None`): Optional positional or keyword input; defaults to `None`. - `current_token_ids` (Sequence[int] | None; optional; default `None`): Optional positional or keyword input; defaults to `None`. - `delta_token_ids` (Sequence[int] | None; optional; default `None`): Optional positional or keyword input; defaults to `None`. - `request` (dict[str, Any] | None; optional; default `None`): Optional positional or keyword input; defaults to `None`. - Return annotation: `dict[str, Any] | None` - Calls: self._scan_args_for_new_call, self._start_new_tool_call, self._parse_streaming_tool_delta, len, result.get, delta_text.split, self.BOT_TOKEN.join - State reads: self._args_started, self._scan_args_for_new_call, self.current_tool_id, self._start_new_tool_call, self._parse_streaming_tool_delta, self.BOT_TOKEN, self._current_tool_call_id, self.BOT_TOKEN.join, self._name_buffer_overflow, self._tool_call_id_emitted - State writes: self._tool_call_id_emitted - Return expressions: {'tool_calls': [{'index': self.current_tool_id, 'type': 'function', 'function': {'arguments': delta_text}}]}; result if result else None; {'content': delta_text}; {'content': tool_delta['content']}; {'tool_calls': [tool_call]}; None ## `vllm_mlx.tool_parsers.mistral_tool_parser.MistralToolParser._parse_streaming_tool_delta` - Kind: method - Signature: `def _parse_streaming_tool_delta(self, text: str) -> dict[str, str] | None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/mistral_tool_parser.py#L455-L512 - Implementation: Method `MistralToolParser._parse_streaming_tool_delta` updates `self._name_buffer`, `self._args_started`, `self._name_buffer_overflow`; calls `self._name_buffer.find`, `len`, `self._name_buffer[:idx].strip`; has 4 explicit return paths. Parse a streaming delta for tool call information. Once the name/arguments boundary (the `[ARGS]` marker, or a bare `{` for older checkpoints) has been seen for the current tool call, every subsequent delta is argument text and is never re-classified — JSON string content (bare keys/values like `city` or `Paris`) has no distinguishing leading punctuation, so re-evaluating each delta in isolation (the previous approach) misclassified mid-argument fragments as more of the function name. - Inputs: - `text` (str; required): Required positional or keyword input. - Return annotation: `dict[str, str] | None` - Calls: self._name_buffer.find, len, self._name_buffer[:idx].strip - State reads: self._args_started, self._name_buffer.find, self._name_buffer, self.ARGS_TOKEN, self._NAME_BUFFER_LIMIT - State writes: self._name_buffer, self._args_started, self._name_buffer_overflow - Return expressions: None; {'arguments': text}; result if result else None; {'content': overflowed} # Module `vllm_mlx.tool_parsers.nemotron_tool_parser` Nemotron tool call parser for vllm-mlx. Handles NVIDIA Nemotron models' tool calling format: - v Supports Nemotron-3-Nano-30B-A3B and similar models. Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/nemotron_tool_parser.py#L1-L166 ## `vllm_mlx.tool_parsers.nemotron_tool_parser.generate_tool_id` - Kind: function - Signature: `def generate_tool_id() -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/nemotron_tool_parser.py#L24-L26 - Implementation: Function `generate_tool_id` calls `uuid.uuid4`; returns `f'call_{uuid.uuid4().hex[:8]}'`. Generate a unique tool call ID. - Inputs: none - Return annotation: `str` - Calls: uuid.uuid4 - Return expressions: f'call_{uuid.uuid4().hex[:8]}' ## `vllm_mlx.tool_parsers.nemotron_tool_parser.NemotronToolParser` - Kind: class - Signature: `class NemotronToolParser(ToolParser)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/nemotron_tool_parser.py#L30-L166 - Implementation: Class `NemotronToolParser` derives from `ToolParser` and declares 2 direct member(s). Tool call parser for NVIDIA Nemotron models. Supports Nemotron's tool call format: Paris Also supports JSON arguments: {"city": "Paris"} Used when --enable-auto-tool-choice --tool-call-parser nemotron are set. - Inputs: none - Constructs: `vllm_mlx.tool_parsers.nemotron_tool_parser.NemotronToolParser` - Decorators: ToolParserManager.register_module(['nemotron', 'nemotron3']) ## `vllm_mlx.tool_parsers.nemotron_tool_parser.NemotronToolParser.extract_tool_calls` - Kind: method - Signature: `def extract_tool_calls(self, model_output: str, request: dict[str, Any] | None=None) -> ExtractedToolCallInformation` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/nemotron_tool_parser.py#L55-L130 - Implementation: Method `NemotronToolParser.extract_tool_calls` calls `ExtractedToolCallInformation`, `self.TOOL_CALL_PATTERN.findall`, `func_name.strip`, `content.strip`; has 2 explicit return paths. Extract tool calls from Nemotron model output. - Inputs: - `model_output` (str; required): Required positional or keyword input. - `request` (dict[str, Any] | None; optional; default `None`): Optional positional or keyword input; defaults to `None`. - Return annotation: `ExtractedToolCallInformation` - Calls: ExtractedToolCallInformation, self.TOOL_CALL_PATTERN.findall, func_name.strip, content.strip, content.startswith, json.loads, tool_calls.append, generate_tool_id, self.PARAM_PATTERN.findall, param_name.strip, param_value.strip, json.dumps, self.TOOL_CALL_PATTERN.sub('', cleaned_text).strip, self.TOOL_CALL_PATTERN.sub - State reads: self.TOOL_CALL_PATTERN.findall, self.TOOL_CALL_PATTERN, self.PARAM_PATTERN.findall, self.PARAM_PATTERN, self.TOOL_CALL_PATTERN.sub - Return expressions: ExtractedToolCallInformation(tools_called=False, tool_calls=[], content=model_output); ExtractedToolCallInformation(tools_called=True, tool_calls=tool_calls, content=cleaned_text if cleaned_text else None) ## `vllm_mlx.tool_parsers.nemotron_tool_parser.NemotronToolParser.extract_tool_calls_streaming` - Kind: method - Signature: `def extract_tool_calls_streaming(self, previous_text: str, current_text: str, delta_text: str, previous_token_ids: Sequence[int] | None=None, current_token_ids: Sequence[int] | None=None, delta_token_ids: Sequence[int] | None=None, request: dict[str, Any] | None=None) -> dict[str, Any] | None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/nemotron_tool_parser.py#L132-L166 - Implementation: Method `NemotronToolParser.extract_tool_calls_streaming` calls `self.extract_tool_calls`, `enumerate`; has 3 explicit return paths. Extract tool calls from streaming Nemotron model output. - Inputs: - `previous_text` (str; required): Required positional or keyword input. - `current_text` (str; required): Required positional or keyword input. - `delta_text` (str; required): Required positional or keyword input. - `previous_token_ids` (Sequence[int] | None; optional; default `None`): Optional positional or keyword input; defaults to `None`. - `current_token_ids` (Sequence[int] | None; optional; default `None`): Optional positional or keyword input; defaults to `None`. - `delta_token_ids` (Sequence[int] | None; optional; default `None`): Optional positional or keyword input; defaults to `None`. - `request` (dict[str, Any] | None; optional; default `None`): Optional positional or keyword input; defaults to `None`. - Return annotation: `dict[str, Any] | None` - Calls: self.extract_tool_calls, enumerate - State reads: self.extract_tool_calls - Return expressions: {'content': delta_text}; {'tool_calls': [{'index': i, 'id': tc['id'], 'type': 'function', 'function': {'name': tc['name'], 'arguments': tc['argu…; None # Module `vllm_mlx.tool_parsers.poolside_v1_tool_parser` Tool parser for the Poolside v1 Laguna chat-template format. Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/poolside_v1_tool_parser.py#L1-L362 ## `vllm_mlx.tool_parsers.poolside_v1_tool_parser._consume_stream_state` - Kind: function - Signature: `def _consume_stream_state(parser, pending: dict[int, dict[str, Any]], valid_names: set[str], request: dict[str, Any] | None) -> tuple[bool, str]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/poolside_v1_tool_parser.py#L16-L30 - Implementation: Function `_consume_stream_state` calls `parser._consume_text_before_tool`, `parser._consume_tool_name`, `parser._consume_string_value`, `parser._consume_pending_key`; has 5 explicit return paths. Function `_consume_stream_state` calls `parser._consume_text_before_tool`, `parser._consume_tool_name`, `parser._consume_string_value`, `parser._consume_pending_key`; has 5 explicit return paths. - Inputs: - `parser` (not annotated; required): Required positional or keyword input. - `pending` (dict[int, dict[str, Any]]; required): Required positional or keyword input. - `valid_names` (set[str]; required): Required positional or keyword input. - `request` (dict[str, Any] | None; required): Required positional or keyword input. - Return annotation: `tuple[bool, str]` - Calls: parser._consume_text_before_tool, parser._consume_tool_name, parser._consume_string_value, parser._consume_pending_key, parser._consume_tool_body - Return expressions: parser._consume_text_before_tool(); (parser._consume_tool_name(pending, valid_names), ''); (parser._consume_string_value(pending), ''); (parser._consume_pending_key(pending, request), ''); (parser._consume_tool_body(pending), '') ## `vllm_mlx.tool_parsers.poolside_v1_tool_parser.PoolsideV1ToolParser` - Kind: class - Signature: `class PoolsideV1ToolParser(Glm47ToolParser)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/poolside_v1_tool_parser.py#L34-L362 - Implementation: Class `PoolsideV1ToolParser` derives from `Glm47ToolParser` and declares 18 direct member(s). Parse Laguna tool calls and stream schema-declared strings incrementally. - Inputs: - `tokenizer` (not annotated; optional; default `None`): Optional positional or keyword input; defaults to `None`. - Constructs: `vllm_mlx.tool_parsers.poolside_v1_tool_parser.PoolsideV1ToolParser` - Decorators: ToolParserManager.register_module('poolside_v1') ## `vllm_mlx.tool_parsers.poolside_v1_tool_parser.PoolsideV1ToolParser.__init__` - Kind: method - Signature: `def __init__(self, tokenizer=None)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/poolside_v1_tool_parser.py#L45-L47 - Implementation: Method `PoolsideV1ToolParser.__init__` calls `super().__init__`, `super`, `self.reset`. Method `PoolsideV1ToolParser.__init__` calls `super().__init__`, `super`, `self.reset`. - Inputs: - `tokenizer` (not annotated; optional; default `None`): Optional positional or keyword input; defaults to `None`. - Return annotation: `not annotated` - Calls: super().__init__, super, self.reset - State reads: self.reset ## `vllm_mlx.tool_parsers.poolside_v1_tool_parser.PoolsideV1ToolParser.reset` - Kind: method - Signature: `def reset(self) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/poolside_v1_tool_parser.py#L49-L62 - Implementation: Method `PoolsideV1ToolParser.reset` updates `self._buffer`, `self._in_tool_call`, `self._current_tool_name`, `self._pending_key`; calls `super().reset`, `super`. Reset Laguna parser buffers and per-call argument state. - Inputs: none - Return annotation: `None` - Calls: super().reset, super - State writes: self._buffer, self._in_tool_call, self._current_tool_name, self._pending_key, self._streaming_string_value, self._reject_current, self._tool_ids, self._args_started, self._args_closed, self._seen_keys ## `vllm_mlx.tool_parsers.poolside_v1_tool_parser.PoolsideV1ToolParser._string_argument_names` - Kind: method - Signature: `def _string_argument_names(request: dict[str, Any] | None, tool_name: str) -> set[str]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/poolside_v1_tool_parser.py#L65-L85 - Implementation: Method `PoolsideV1ToolParser._string_argument_names` calls `set`, `request.get`, `isinstance`, `tool.get`; has 2 explicit return paths. Method `PoolsideV1ToolParser._string_argument_names` calls `set`, `request.get`, `isinstance`, `tool.get`; has 2 explicit return paths. - Inputs: - `request` (dict[str, Any] | None; required): Required positional or keyword input. - `tool_name` (str; required): Required positional or keyword input. - Return annotation: `set[str]` - Decorators: staticmethod - Calls: set, request.get, isinstance, tool.get, function.get, parameters.get, properties.items, schema.get - Return expressions: set(); {name for name, schema in properties.items() if isinstance(schema, dict) and schema.get('type') == 'string'} ## `vllm_mlx.tool_parsers.poolside_v1_tool_parser.PoolsideV1ToolParser._escape_string_content` - Kind: method - Signature: `def _escape_string_content(value: str) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/poolside_v1_tool_parser.py#L88-L89 - Implementation: Method `PoolsideV1ToolParser._escape_string_content` calls `json.dumps`; returns `json.dumps(value, ensure_ascii=False)[1:-1]`. Method `PoolsideV1ToolParser._escape_string_content` calls `json.dumps`; returns `json.dumps(value, ensure_ascii=False)[1:-1]`. - Inputs: - `value` (str; required): Required positional or keyword input. - Return annotation: `str` - Decorators: staticmethod - Calls: json.dumps - Return expressions: json.dumps(value, ensure_ascii=False)[1:-1] ## `vllm_mlx.tool_parsers.poolside_v1_tool_parser.PoolsideV1ToolParser._hold_partial_suffix` - Kind: method - Signature: `def _hold_partial_suffix(buffer: str, marker: str) -> tuple[str, str]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/poolside_v1_tool_parser.py#L92-L96 - Implementation: Method `PoolsideV1ToolParser._hold_partial_suffix` calls `range`, `min`, `len`, `buffer.endswith`; has 2 explicit return paths. Method `PoolsideV1ToolParser._hold_partial_suffix` calls `range`, `min`, `len`, `buffer.endswith`; has 2 explicit return paths. - Inputs: - `buffer` (str; required): Required positional or keyword input. - `marker` (str; required): Required positional or keyword input. - Return annotation: `tuple[str, str]` - Decorators: staticmethod - Calls: range, min, len, buffer.endswith - Return expressions: (buffer[:-size], buffer[-size:]); (buffer, '') ## `vllm_mlx.tool_parsers.poolside_v1_tool_parser.PoolsideV1ToolParser.extract_tool_calls` - Kind: method - Signature: `def extract_tool_calls(self, model_output: str, request: dict[str, Any] | None=None) -> ExtractedToolCallInformation` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/poolside_v1_tool_parser.py#L98-L138 - Implementation: Method `PoolsideV1ToolParser.extract_tool_calls` calls `self.strip_think_tags`, `self._get_tool_names`, `self.FUNC_DETAIL_PATTERN.finditer`, `match.group(1).strip`; has 2 explicit return paths. Extract complete Laguna tool blocks and preserve remaining content. - Inputs: - `model_output` (str; required): Required positional or keyword input. - `request` (dict[str, Any] | None; optional; default `None`): Optional positional or keyword input; defaults to `None`. - Return annotation: `ExtractedToolCallInformation` - Calls: self.strip_think_tags, self._get_tool_names, self.FUNC_DETAIL_PATTERN.finditer, match.group(1).strip, match.group, self._string_argument_names, self.ARG_PATTERN.findall, raw_key.strip, self._deserialize, raw_value.strip, tool_calls.append, generate_tool_id, json.dumps, cleaned_text.find, content.strip, ExtractedToolCallInformation, self._UNCLOSED_TOOL_CALL.sub('', cleaned_text).strip, self._UNCLOSED_TOOL_CALL.sub - State reads: self.strip_think_tags, self._get_tool_names, self.FUNC_DETAIL_PATTERN.finditer, self.FUNC_DETAIL_PATTERN, self._string_argument_names, self.ARG_PATTERN.findall, self.ARG_PATTERN, self._deserialize, self._START, self._UNCLOSED_TOOL_CALL.sub, self._UNCLOSED_TOOL_CALL - Return expressions: ExtractedToolCallInformation(True, tool_calls, content); ExtractedToolCallInformation(False, [], content) ## `vllm_mlx.tool_parsers.poolside_v1_tool_parser.PoolsideV1ToolParser._begin_tool_call` - Kind: method - Signature: `def _begin_tool_call(self) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/poolside_v1_tool_parser.py#L140-L150 - Implementation: Method `PoolsideV1ToolParser._begin_tool_call` updates `self.current_tool_id`, `self._in_tool_call`, `self._current_tool_name`, `self._pending_key`; calls `self._tool_ids.append`, `generate_tool_id`, `self._args_started.append`, `self._args_closed.append`. Method `PoolsideV1ToolParser._begin_tool_call` updates `self.current_tool_id`, `self._in_tool_call`, `self._current_tool_name`, `self._pending_key`; calls `self._tool_ids.append`, `generate_tool_id`, `self._args_started.append`, `self._args_closed.append`. - Inputs: none - Return annotation: `None` - Calls: self._tool_ids.append, generate_tool_id, self._args_started.append, self._args_closed.append, self._seen_keys.append, set - State reads: self._tool_ids.append, self._tool_ids, self._args_started.append, self._args_started, self._args_closed.append, self._args_closed, self._seen_keys.append, self._seen_keys - State writes: self.current_tool_id, self._in_tool_call, self._current_tool_name, self._pending_key, self._streaming_string_value, self._reject_current ## `vllm_mlx.tool_parsers.poolside_v1_tool_parser.PoolsideV1ToolParser._finish_tool_call` - Kind: method - Signature: `def _finish_tool_call(self) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/poolside_v1_tool_parser.py#L152-L157 - Implementation: Method `PoolsideV1ToolParser._finish_tool_call` updates `self._in_tool_call`, `self._current_tool_name`, `self._pending_key`, `self._streaming_string_value`. Method `PoolsideV1ToolParser._finish_tool_call` updates `self._in_tool_call`, `self._current_tool_name`, `self._pending_key`, `self._streaming_string_value`. - Inputs: none - Return annotation: `None` - State writes: self._in_tool_call, self._current_tool_name, self._pending_key, self._streaming_string_value, self._reject_current ## `vllm_mlx.tool_parsers.poolside_v1_tool_parser.PoolsideV1ToolParser._delta` - Kind: method - Signature: `def _delta(self, pending: dict[int, dict[str, Any]], *, name: str | None=None, arguments: str='') -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/poolside_v1_tool_parser.py#L159-L179 - Implementation: Method `PoolsideV1ToolParser._delta` calls `pending.setdefault`; returns `None`. Method `PoolsideV1ToolParser._delta` calls `pending.setdefault`; returns `None`. - Inputs: - `pending` (dict[int, dict[str, Any]]; required): Required positional or keyword input. - `name` (str | None; optional; default `None`): Optional keyword-only input; defaults to `None`. - `arguments` (str; optional; default `''`): Optional keyword-only input; defaults to `''`. - Return annotation: `None` - Calls: pending.setdefault - State reads: self._reject_current, self.current_tool_id, self._tool_ids - Return expressions: None ## `vllm_mlx.tool_parsers.poolside_v1_tool_parser.PoolsideV1ToolParser._argument_prefix` - Kind: method - Signature: `def _argument_prefix(self, key: str) -> str | None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/poolside_v1_tool_parser.py#L181-L188 - Implementation: Method `PoolsideV1ToolParser._argument_prefix` calls `seen.add`, `json.dumps`; has 2 explicit return paths. Method `PoolsideV1ToolParser._argument_prefix` calls `seen.add`, `json.dumps`; has 2 explicit return paths. - Inputs: - `key` (str; required): Required positional or keyword input. - Return annotation: `str | None` - Calls: seen.add, json.dumps - State reads: self._seen_keys, self.current_tool_id, self._args_started - Return expressions: None; separator + json.dumps(key, ensure_ascii=False) + ': ' ## `vllm_mlx.tool_parsers.poolside_v1_tool_parser.PoolsideV1ToolParser._close_arguments` - Kind: method - Signature: `def _close_arguments(self) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/poolside_v1_tool_parser.py#L190-L194 - Implementation: Method `PoolsideV1ToolParser._close_arguments` has 2 explicit return paths. Method `PoolsideV1ToolParser._close_arguments` has 2 explicit return paths. - Inputs: none - Return annotation: `str` - State reads: self._args_closed, self.current_tool_id, self._args_started - Return expressions: ''; '}' if self._args_started[self.current_tool_id] else '{}' ## `vllm_mlx.tool_parsers.poolside_v1_tool_parser.PoolsideV1ToolParser._discard_through_tool_end` - Kind: method - Signature: `def _discard_through_tool_end(self) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/poolside_v1_tool_parser.py#L196-L202 - Implementation: Method `PoolsideV1ToolParser._discard_through_tool_end` updates `self._buffer`; calls `self._buffer.find`, `len`, `self._finish_tool_call`; has 2 explicit return paths. Method `PoolsideV1ToolParser._discard_through_tool_end` updates `self._buffer`; calls `self._buffer.find`, `len`, `self._finish_tool_call`; has 2 explicit return paths. - Inputs: none - Return annotation: `bool` - Calls: self._buffer.find, len, self._finish_tool_call - State reads: self._buffer.find, self._buffer, self._END, self._finish_tool_call - State writes: self._buffer - Return expressions: False; True ## `vllm_mlx.tool_parsers.poolside_v1_tool_parser.PoolsideV1ToolParser._consume_text_before_tool` - Kind: method - Signature: `def _consume_text_before_tool(self) -> tuple[bool, str]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/poolside_v1_tool_parser.py#L204-L214 - Implementation: Method `PoolsideV1ToolParser._consume_text_before_tool` updates `self._buffer`; calls `self._buffer.find`, `self._hold_partial_suffix`, `len`, `self._begin_tool_call`; has 2 explicit return paths. Consume plain text or enter the next ```` state. - Inputs: none - Return annotation: `tuple[bool, str]` - Calls: self._buffer.find, self._hold_partial_suffix, len, self._begin_tool_call - State reads: self._buffer.find, self._buffer, self._START, self._hold_partial_suffix, self._begin_tool_call - State writes: self._buffer - Return expressions: (False, emitted); (True, content) ## `vllm_mlx.tool_parsers.poolside_v1_tool_parser.PoolsideV1ToolParser._consume_tool_name` - Kind: method - Signature: `def _consume_tool_name(self, pending: dict[int, dict[str, Any]], valid_names: set[str]) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/poolside_v1_tool_parser.py#L216-L247 - Implementation: Method `PoolsideV1ToolParser._consume_tool_name` updates `self._buffer`, `self._reject_current`, `self._current_tool_name`; calls `self._buffer.find`, `min`, `self._buffer[:cut].strip`, `self._buffer.startswith`; has 3 explicit return paths. Consume a tool name, or wait for enough input to identify it. - Inputs: - `pending` (dict[int, dict[str, Any]]; required): Required positional or keyword input. - `valid_names` (set[str]; required): Required positional or keyword input. - Return annotation: `bool` - Calls: self._buffer.find, min, self._buffer[:cut].strip, self._buffer.startswith, self._discard_through_tool_end, self._delta - State reads: self._buffer.find, self._buffer, self._KEY_START, self._END, self._buffer.startswith, self._discard_through_tool_end, self._delta - State writes: self._buffer, self._reject_current, self._current_tool_name - Return expressions: False; self._discard_through_tool_end(); True ## `vllm_mlx.tool_parsers.poolside_v1_tool_parser.PoolsideV1ToolParser._consume_string_value` - Kind: method - Signature: `def _consume_string_value(self, pending: dict[int, dict[str, Any]]) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/poolside_v1_tool_parser.py#L249-L267 - Implementation: Method `PoolsideV1ToolParser._consume_string_value` updates `self._buffer`, `self._streaming_string_value`, `self._pending_key`, `self._reject_current`; calls `self._buffer.find`, `self._escape_string_content`, `len`, `self._delta`; has 3 explicit return paths. Consume a string argument value, retaining incomplete suffixes. - Inputs: - `pending` (dict[int, dict[str, Any]]; required): Required positional or keyword input. - Return annotation: `bool` - Calls: self._buffer.find, self._escape_string_content, len, self._delta, self._discard_through_tool_end, self._hold_partial_suffix - State reads: self._buffer.find, self._buffer, self._VALUE_END, self._escape_string_content, self._delta, self._END, self._discard_through_tool_end, self._hold_partial_suffix - State writes: self._buffer, self._streaming_string_value, self._pending_key, self._reject_current - Return expressions: True; self._discard_through_tool_end(); False ## `vllm_mlx.tool_parsers.poolside_v1_tool_parser.PoolsideV1ToolParser._consume_pending_key` - Kind: method - Signature: `def _consume_pending_key(self, pending: dict[int, dict[str, Any]], request: dict[str, Any] | None) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/poolside_v1_tool_parser.py#L269-L305 - Implementation: Method `PoolsideV1ToolParser._consume_pending_key` updates `self._reject_current`, `self._buffer`, `self._pending_key`, `self._streaming_string_value`; calls `self._buffer.find`, `self._discard_through_tool_end`, `len`, `self._pending_key.strip`; has 3 explicit return paths. Consume the value for the currently buffered argument key. - Inputs: - `pending` (dict[int, dict[str, Any]]; required): Required positional or keyword input. - `request` (dict[str, Any] | None; required): Required positional or keyword input. - Return annotation: `bool` - Calls: self._buffer.find, self._discard_through_tool_end, len, self._pending_key.strip, self._argument_prefix, self._string_argument_names, self._delta, self._buffer[:value_end].strip, json.dumps, self._deserialize - State reads: self._buffer.find, self._buffer, self._VALUE_START, self._END, self._discard_through_tool_end, self._pending_key, self._pending_key.strip, self._argument_prefix, self._string_argument_names, self._current_tool_name, self._delta, self._VALUE_END, self._deserialize - State writes: self._reject_current, self._buffer, self._pending_key, self._streaming_string_value - Return expressions: self._discard_through_tool_end(); False; True ## `vllm_mlx.tool_parsers.poolside_v1_tool_parser.PoolsideV1ToolParser._consume_tool_body` - Kind: method - Signature: `def _consume_tool_body(self, pending: dict[int, dict[str, Any]]) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/poolside_v1_tool_parser.py#L307-L328 - Implementation: Method `PoolsideV1ToolParser._consume_tool_body` updates `self._buffer`, `self._pending_key`; calls `self._buffer.find`, `len`, `self._delta`, `self._close_arguments`; has 2 explicit return paths. Consume an argument key or close the current tool call. - Inputs: - `pending` (dict[int, dict[str, Any]]; required): Required positional or keyword input. - Return annotation: `bool` - Calls: self._buffer.find, len, self._delta, self._close_arguments, self._finish_tool_call - State reads: self._buffer.find, self._buffer, self._END, self._KEY_START, self._delta, self._close_arguments, self._finish_tool_call, self._KEY_END - State writes: self._buffer, self._pending_key - Return expressions: True; False ## `vllm_mlx.tool_parsers.poolside_v1_tool_parser.PoolsideV1ToolParser.extract_tool_calls_streaming` - Kind: method - Signature: `def extract_tool_calls_streaming(self, previous_text: str, current_text: str, delta_text: str, previous_token_ids: Sequence[int] | None=None, current_token_ids: Sequence[int] | None=None, delta_token_ids: Sequence[int] | None=None, request: dict[str, Any] | None=None) -> dict[str, Any] | None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/poolside_v1_tool_parser.py#L330-L362 - Implementation: Method `PoolsideV1ToolParser.extract_tool_calls_streaming` updates `self._buffer`; calls `self._get_tool_names`, `_consume_stream_state`, `list`, `pending.values`; returns `payload or None`. Incrementally emit Laguna content and schema-aware tool arguments. - Inputs: - `previous_text` (str; required): Required positional or keyword input. - `current_text` (str; required): Required positional or keyword input. - `delta_text` (str; required): Required positional or keyword input. - `previous_token_ids` (Sequence[int] | None; optional; default `None`): Optional positional or keyword input; defaults to `None`. - `current_token_ids` (Sequence[int] | None; optional; default `None`): Optional positional or keyword input; defaults to `None`. - `delta_token_ids` (Sequence[int] | None; optional; default `None`): Optional positional or keyword input; defaults to `None`. - `request` (dict[str, Any] | None; optional; default `None`): Optional positional or keyword input; defaults to `None`. - Return annotation: `dict[str, Any] | None` - Calls: self._get_tool_names, _consume_stream_state, list, pending.values - State reads: self._get_tool_names - State writes: self._buffer - Return expressions: payload or None # Module `vllm_mlx.tool_parsers.qwen3_xml_tool_parser` Qwen 3.5 XML tool call parser for vllm-mlx. Handles Qwen 3.5's XML parameter format: value1 Authority: Qwen 3.5 HF chat template, vLLM PR #25028 (from Qwen API team). Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/qwen3_xml_tool_parser.py#L1-L1559 ## `vllm_mlx.tool_parsers.qwen3_xml_tool_parser.DeltaFunctionCall` - Kind: class - Signature: `class DeltaFunctionCall` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/qwen3_xml_tool_parser.py#L54-L58 - Implementation: Class `DeltaFunctionCall` declares 0 direct member(s). Incremental function name and argument payload used by the XML parser. - Inputs: - `name` (Optional[str]; optional; default `None`): Optional constructor field; defaults to `None`. - `arguments` (str; optional; default `''`): Optional constructor field; defaults to `''`. - Constructs: `vllm_mlx.tool_parsers.qwen3_xml_tool_parser.DeltaFunctionCall` - Decorators: dataclass ## `vllm_mlx.tool_parsers.qwen3_xml_tool_parser.DeltaToolCall` - Kind: class - Signature: `class DeltaToolCall` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/qwen3_xml_tool_parser.py#L62-L68 - Implementation: Class `DeltaToolCall` declares 0 direct member(s). Incremental indexed tool call produced by the XML parser shim. - Inputs: - `index` (int; optional; default `0`): Optional constructor field; defaults to `0`. - `id` (Optional[str]; optional; default `None`): Optional constructor field; defaults to `None`. - `type` (str; optional; default `'function'`): Optional constructor field; defaults to `'function'`. - `function` (Optional[DeltaFunctionCall]; optional; default `None`): Optional constructor field; defaults to `None`. - Constructs: `vllm_mlx.tool_parsers.qwen3_xml_tool_parser.DeltaToolCall` - Decorators: dataclass ## `vllm_mlx.tool_parsers.qwen3_xml_tool_parser.DeltaMessage` - Kind: class - Signature: `class DeltaMessage` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/qwen3_xml_tool_parser.py#L72-L78 - Implementation: Class `DeltaMessage` declares 0 direct member(s). Incremental content, reasoning, and tool calls from the parser shim. - Inputs: - `content` (Optional[str]; optional; default `None`): Optional constructor field; defaults to `None`. - `tool_calls` (Optional[list[DeltaToolCall]]; optional; default `None`): Optional constructor field; defaults to `None`. - `role` (Optional[str]; optional; default `None`): Optional constructor field; defaults to `None`. - `reasoning_content` (Optional[str]; optional; default `None`): Optional constructor field; defaults to `None`. - Constructs: `vllm_mlx.tool_parsers.qwen3_xml_tool_parser.DeltaMessage` - Decorators: dataclass ## `vllm_mlx.tool_parsers.qwen3_xml_tool_parser._FunctionDef` - Kind: class - Signature: `class _FunctionDef` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/qwen3_xml_tool_parser.py#L85-L99 - Implementation: Class `_FunctionDef` declares 3 direct member(s). Wrap a function definition dict for attribute access. - Inputs: - `d` (dict; required): Required positional or keyword input. - Constructs: `vllm_mlx.tool_parsers.qwen3_xml_tool_parser._FunctionDef` ## `vllm_mlx.tool_parsers.qwen3_xml_tool_parser._FunctionDef.__init__` - Kind: method - Signature: `def __init__(self, d: dict)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/qwen3_xml_tool_parser.py#L90-L91 - Implementation: Method `_FunctionDef.__init__` updates `self._d`. Method `_FunctionDef.__init__` updates `self._d`. - Inputs: - `d` (dict; required): Required positional or keyword input. - Return annotation: `not annotated` - State writes: self._d ## `vllm_mlx.tool_parsers.qwen3_xml_tool_parser._FunctionDef.name` - Kind: method - Signature: `def name(self) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/qwen3_xml_tool_parser.py#L94-L95 - Implementation: Method `_FunctionDef.name` calls `self._d.get`; returns `self._d.get('name', '')`. Method `_FunctionDef.name` calls `self._d.get`; returns `self._d.get('name', '')`. - Inputs: none - Return annotation: `str` - Decorators: property - Calls: self._d.get - State reads: self._d.get, self._d - Return expressions: self._d.get('name', '') ## `vllm_mlx.tool_parsers.qwen3_xml_tool_parser._FunctionDef.parameters` - Kind: method - Signature: `def parameters(self) -> dict` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/qwen3_xml_tool_parser.py#L98-L99 - Implementation: Method `_FunctionDef.parameters` calls `self._d.get`; returns `self._d.get('parameters', {})`. Method `_FunctionDef.parameters` calls `self._d.get`; returns `self._d.get('parameters', {})`. - Inputs: none - Return annotation: `dict` - Decorators: property - Calls: self._d.get - State reads: self._d.get, self._d - Return expressions: self._d.get('parameters', {}) ## `vllm_mlx.tool_parsers.qwen3_xml_tool_parser._ToolDef` - Kind: class - Signature: `class _ToolDef` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/qwen3_xml_tool_parser.py#L102-L117 - Implementation: Class `_ToolDef` declares 3 direct member(s). Wrap a tool definition dict for attribute access. - Inputs: - `d` (dict; required): Required positional or keyword input. - Constructs: `vllm_mlx.tool_parsers.qwen3_xml_tool_parser._ToolDef` ## `vllm_mlx.tool_parsers.qwen3_xml_tool_parser._ToolDef.__init__` - Kind: method - Signature: `def __init__(self, d: dict)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/qwen3_xml_tool_parser.py#L107-L109 - Implementation: Method `_ToolDef.__init__` updates `self._d`, `self._func`; calls `_FunctionDef`, `d.get`. Method `_ToolDef.__init__` updates `self._d`, `self._func`; calls `_FunctionDef`, `d.get`. - Inputs: - `d` (dict; required): Required positional or keyword input. - Return annotation: `not annotated` - Calls: _FunctionDef, d.get - State writes: self._d, self._func ## `vllm_mlx.tool_parsers.qwen3_xml_tool_parser._ToolDef.type` - Kind: method - Signature: `def type(self) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/qwen3_xml_tool_parser.py#L112-L113 - Implementation: Method `_ToolDef.type` calls `self._d.get`; returns `self._d.get('type', 'function')`. Method `_ToolDef.type` calls `self._d.get`; returns `self._d.get('type', 'function')`. - Inputs: none - Return annotation: `str` - Decorators: property - Calls: self._d.get - State reads: self._d.get, self._d - Return expressions: self._d.get('type', 'function') ## `vllm_mlx.tool_parsers.qwen3_xml_tool_parser._ToolDef.function` - Kind: method - Signature: `def function(self) -> _FunctionDef` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/qwen3_xml_tool_parser.py#L116-L117 - Implementation: Method `_ToolDef.function` returns `self._func`. Method `_ToolDef.function` returns `self._func`. - Inputs: none - Return annotation: `_FunctionDef` - Decorators: property - State reads: self._func - Return expressions: self._func ## `vllm_mlx.tool_parsers.qwen3_xml_tool_parser.StreamingXMLToolCallParser` - Kind: class - Signature: `class StreamingXMLToolCallParser` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/qwen3_xml_tool_parser.py#L126-L1427 - Implementation: Class `StreamingXMLToolCallParser` declares 27 direct member(s). Streaming XML parser for Qwen 3.5 ```` format. Architecture: 1. **Preprocessing** (``_preprocess_before_xml_parse``): scans raw text for ```` tags, extracts type hints from tool schemas, and rewrites the XML into expat-parseable form. 2. **Expat parsing**: an incremental ``xml.parsers.expat`` parser fires ``start_element`` / ``end_element`` / ``character_data`` callbacks. 3. **Type coercion** (``_coerce_param_value``): converts string values to int/float/bool/object/array based on JSON Schema type hints. Complex types use a deferred ``ast.literal_eval`` + ``json.loads`` fallback. 4. **Auto-closing**: if the model truncates output mid-tag, the parser synthesizes closing tags so partial tool calls are still extractable. Streaming: ``update(delta)`` feeds incremental text. Completed tool calls are emitted via ``get_streaming_output()`` as they close. State resets between tool calls within the same response. - Inputs: none - Constructs: `vllm_mlx.tool_parsers.qwen3_xml_tool_parser.StreamingXMLToolCallParser` ## `vllm_mlx.tool_parsers.qwen3_xml_tool_parser.StreamingXMLToolCallParser.__init__` - Kind: method - Signature: `def __init__(self)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/qwen3_xml_tool_parser.py#L146-L156 - Implementation: Method `StreamingXMLToolCallParser.__init__` updates `self.tools`, `self.tool_call_start_token`, `self.tool_call_end_token`, `self.function_start_token`; calls `self.reset_streaming_state`. Method `StreamingXMLToolCallParser.__init__` updates `self.tools`, `self.tool_call_start_token`, `self.tool_call_end_token`, `self.function_start_token`; calls `self.reset_streaming_state`. - Inputs: none - Return annotation: `not annotated` - Calls: self.reset_streaming_state - State reads: self.reset_streaming_state - State writes: self.tools, self.tool_call_start_token, self.tool_call_end_token, self.function_start_token, self.function_end_token, self.parameter_start_token, self.parameter_end_token ## `vllm_mlx.tool_parsers.qwen3_xml_tool_parser.StreamingXMLToolCallParser.reset_streaming_state` - Kind: method - Signature: `def reset_streaming_state(self)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/qwen3_xml_tool_parser.py#L158-L208 - Implementation: Method `StreamingXMLToolCallParser.reset_streaming_state` updates `self.deltas`, `self.tool_call_index`, `self.current_call_id`, `self.last_completed_call_id`; calls `ParserCreate`, `self.setup_parser`. Reset streaming parsing state - Inputs: none - Return annotation: `not annotated` - Calls: ParserCreate, self.setup_parser - State reads: self.setup_parser - State writes: self.deltas, self.tool_call_index, self.current_call_id, self.last_completed_call_id, self.current_function_name, self.current_function_open, self.implicit_tool_call_wrapper, self._pending_implicit_delta, self._pending_implicit_raw_text, self._pending_implicit_text_buffer, self._current_raw_element, self.parameters, self.current_param_name, self.current_param_value, self.current_param_value_converted, self.current_param_is_first, self.should_emit_end_newline, self.start_quote_emitted, self.streaming_buffer, self.last_processed_pos, self.text_content_buffer, self._pre_inside_parameter, self._pre_param_buffer, self._pre_current_param_name, self.defer_current_parameter, self.deferred_param_raw_value, self.parser ## `vllm_mlx.tool_parsers.qwen3_xml_tool_parser.StreamingXMLToolCallParser.parse_single_streaming_chunks` - Kind: method - Signature: `def parse_single_streaming_chunks(self, xml_chunk: str) -> DeltaMessage` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/qwen3_xml_tool_parser.py#L210-L330 - Implementation: Method `StreamingXMLToolCallParser.parse_single_streaming_chunks` updates `self.streaming_buffer`, `self.text_content_buffer`; calls `len`, `self._process_complete_xml_elements`, `xml_chunk.count`, `sum`; has 3 explicit return paths. Parse single streaming XML chunk and return Delta response This is the actual streaming interface that receives chunks one by one and maintains internal state Args: xml_chunk: Single XML chunk string Returns: DeltaMessage: Contains delta information generated by this chunk, returns empty response if no complete elements - Inputs: - `xml_chunk` (str; required): Single XML chunk string - Return annotation: `DeltaMessage` - Calls: len, self._process_complete_xml_elements, xml_chunk.count, sum, isinstance, self._end_element, any, logger.warning, self._merge_new_deltas_to_single_response, DeltaMessage, self._emit_delta - State reads: self.deltas, self._process_complete_xml_elements, self.current_call_id, self.function_end_token, self.current_param_name, self._end_element, self.current_function_name, self.tool_call_end_token, self._merge_new_deltas_to_single_response, self.text_content_buffer, self.tool_call_index, self._emit_delta - State writes: self.streaming_buffer, self.text_content_buffer - Return expressions: result_delta; text_delta; DeltaMessage(content=None) ## `vllm_mlx.tool_parsers.qwen3_xml_tool_parser.StreamingXMLToolCallParser._escape_xml_special_chars` - Kind: method - Signature: `def _escape_xml_special_chars(self, text: str) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/qwen3_xml_tool_parser.py#L332-L351 - Implementation: Method `StreamingXMLToolCallParser._escape_xml_special_chars` calls `xml_escapes.items`, `text.replace`; returns `text`. Escape XML special characters Args: text: Original text Returns: Escaped text - Inputs: - `text` (str; required): Original text - Return annotation: `str` - Calls: xml_escapes.items, text.replace - Return expressions: text ## `vllm_mlx.tool_parsers.qwen3_xml_tool_parser.StreamingXMLToolCallParser._process_complete_xml_elements` - Kind: method - Signature: `def _process_complete_xml_elements(self) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/qwen3_xml_tool_parser.py#L353-L438 - Implementation: Method `StreamingXMLToolCallParser._process_complete_xml_elements` updates `self.last_processed_pos`, `self.text_content_buffer`, `self._current_raw_element`; calls `len`, `self._find_next_complete_element`, `self._should_skip_element`, `self._preprocess_xml_chunk`; returns `found_any`. Process complete XML elements in buffer Returns: bool: Whether complete elements were found and processed - Inputs: none - Return annotation: `bool` - Calls: len, self._find_next_complete_element, self._should_skip_element, self._preprocess_xml_chunk, preprocessed_element.strip().startswith, preprocessed_element.strip, DeltaMessage, self._emit_delta, self._end_element, DeltaToolCall, DeltaFunctionCall, self._reset_xml_parser_after_tool_call, self.parser.Parse, logger.warning - State reads: self.last_processed_pos, self.streaming_buffer, self._find_next_complete_element, self._should_skip_element, self._preprocess_xml_chunk, self.tool_call_index, self.text_content_buffer, self._emit_delta, self.current_call_id, self.current_param_name, self._end_element, self.current_function_open, self.current_function_name, self._reset_xml_parser_after_tool_call, self.parser.Parse, self.parser - State writes: self.last_processed_pos, self.text_content_buffer, self._current_raw_element - Return expressions: found_any ## `vllm_mlx.tool_parsers.qwen3_xml_tool_parser.StreamingXMLToolCallParser._should_skip_element` - Kind: method - Signature: `def _should_skip_element(self, element: str) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/qwen3_xml_tool_parser.py#L440-L474 - Implementation: Method `StreamingXMLToolCallParser._should_skip_element` updates `self.text_content_buffer`; calls `element.startswith`; has 3 explicit return paths. Determine whether an element should be skipped Args: element: Element to evaluate Returns: bool: True means should skip, False means should process - Inputs: - `element` (str; required): Element to evaluate - Return annotation: `bool` - Calls: element.startswith - State reads: self.tool_call_start_token, self.function_start_token, self.parameter_start_token, self.current_call_id - State writes: self.text_content_buffer - Return expressions: False; True; not element ## `vllm_mlx.tool_parsers.qwen3_xml_tool_parser.StreamingXMLToolCallParser._looks_like_partial_tool_open` - Kind: method - Signature: `def _looks_like_partial_tool_open(self, fragment: str) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/qwen3_xml_tool_parser.py#L488-L501 - Implementation: Method `StreamingXMLToolCallParser._looks_like_partial_tool_open` calls `fragment.startswith`, `prefix.startswith`; has 2 explicit return paths. True if `fragment` could complete into a tool-related XML tag. Covers two shapes: * `fragment` is a prefix of a known tag head (e.g. ````. - Inputs: - `fragment` (str; required): Required positional or keyword input. - Return annotation: `bool` - Calls: fragment.startswith, prefix.startswith - State reads: self._TOOL_TAG_PREFIXES - Return expressions: False; True ## `vllm_mlx.tool_parsers.qwen3_xml_tool_parser.StreamingXMLToolCallParser._find_next_complete_element` - Kind: method - Signature: `def _find_next_complete_element(self, start_pos: int) -> tuple[Optional[str], int]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/qwen3_xml_tool_parser.py#L503-L569 - Implementation: Method `StreamingXMLToolCallParser._find_next_complete_element` calls `buffer.startswith`, `buffer.find`, `self._looks_like_partial_tool_open`, `len`; has 6 explicit return paths. Find next complete XML element from specified position Args: start_pos: Position to start searching Returns: (Complete element string, element end position), returns (None, start_pos) if no complete element found - Inputs: - `start_pos` (int; required): Position to start searching - Return annotation: `tuple[Optional[str], int]` - Calls: buffer.startswith, buffer.find, self._looks_like_partial_tool_open, len - State reads: self.streaming_buffer, self._looks_like_partial_tool_open, self.current_call_id - Return expressions: (None, start_pos); (buffer[:tag_end], start_pos + tag_end); (buffer[:tag_end2 + 1], start_pos + tag_end2 + 1); (buffer, start_pos + len(buffer)); (text_content, start_pos + next_tag_pos); (remaining, start_pos + len(remaining)) ## `vllm_mlx.tool_parsers.qwen3_xml_tool_parser.StreamingXMLToolCallParser._merge_new_deltas_to_single_response` - Kind: method - Signature: `def _merge_new_deltas_to_single_response(self, initial_count: int) -> DeltaMessage` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/qwen3_xml_tool_parser.py#L571-L633 - Implementation: Method `StreamingXMLToolCallParser._merge_new_deltas_to_single_response` calls `len`, `DeltaMessage`, `merged_tool_calls.append`; has 3 explicit return paths. Merge newly generated deltas from this processing into a single DeltaMessage Args: initial_count: Delta count before processing Returns: Merged DeltaMessage containing all newly generated delta information - Inputs: - `initial_count` (int; required): Delta count before processing - Return annotation: `DeltaMessage` - Calls: len, DeltaMessage, merged_tool_calls.append - State reads: self.deltas - Return expressions: DeltaMessage(content=None); new_deltas[0]; DeltaMessage(content=merged_content if merged_content else None, tool_calls=merged_tool_calls) ## `vllm_mlx.tool_parsers.qwen3_xml_tool_parser.StreamingXMLToolCallParser._preprocess_xml_chunk` - Kind: method - Signature: `def _preprocess_xml_chunk(self, chunk: str) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/qwen3_xml_tool_parser.py#L635-L748 - Implementation: Method `StreamingXMLToolCallParser._preprocess_xml_chunk` updates `self.defer_current_parameter`, `self.deferred_param_raw_value`, `self._pre_inside_parameter`, `self._pre_param_buffer`; calls `chunk.startswith`, `re.sub`, `processed.startswith`, `self._escape_xml_special_chars`; has 4 explicit return paths. Preprocess XML chunk, handle non-standard formats, and escape special characters Args: chunk: Original XML chunk Returns: Processed XML chunk - Inputs: - `chunk` (str; required): Original XML chunk - Return annotation: `str` - Calls: chunk.startswith, re.sub, processed.startswith, self._escape_xml_special_chars, self._get_param_type, param_type.startswith, re.match, m.group - State reads: self.tool_call_start_token, self.tool_call_end_token, self.function_start_token, self.function_end_token, self.parameter_start_token, self.parameter_end_token, self._pre_inside_parameter, self._pre_param_buffer, self._escape_xml_special_chars, self._pre_current_param_name, self._get_param_type - State writes: self.defer_current_parameter, self.deferred_param_raw_value, self._pre_inside_parameter, self._pre_param_buffer, self._pre_current_param_name - Return expressions: f'{safe_text}'; self._escape_xml_special_chars(original_chunk); ''; processed ## `vllm_mlx.tool_parsers.qwen3_xml_tool_parser.StreamingXMLToolCallParser._emit_delta` - Kind: method - Signature: `def _emit_delta(self, delta: DeltaMessage)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/qwen3_xml_tool_parser.py#L750-L752 - Implementation: Method `StreamingXMLToolCallParser._emit_delta` calls `self.deltas.append`. Emit Delta response (streaming output) - Inputs: - `delta` (DeltaMessage; required): Required positional or keyword input. - Return annotation: `not annotated` - Calls: self.deltas.append - State reads: self.deltas.append, self.deltas ## `vllm_mlx.tool_parsers.qwen3_xml_tool_parser.StreamingXMLToolCallParser._auto_close_open_parameter_if_needed` - Kind: method - Signature: `def _auto_close_open_parameter_if_needed(self, incoming_tag: Optional[str]=None)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/qwen3_xml_tool_parser.py#L754-L777 - Implementation: Method `StreamingXMLToolCallParser._auto_close_open_parameter_if_needed` calls `self._end_element`. Before starting to process new elements, if there are unclosed tags from before, automatically complete their endings to the parser. - If there are unclosed parameters, it's equivalent to feeding `` - When about to start a new function or tool_call, if there are unclosed functions, complete ``. - When about to start a new tool_call, if there are unclosed tool_calls, complete ``. - Inputs: - `incoming_tag` (Optional[str]; optional; default `None`): Optional positional or keyword input; defaults to `None`. - Return annotation: `not annotated` - Calls: self._end_element - State reads: self.current_param_name, self._end_element, self.current_function_name, self.current_call_id ## `vllm_mlx.tool_parsers.qwen3_xml_tool_parser.StreamingXMLToolCallParser._start_element` - Kind: method - Signature: `def _start_element(self, name: str, attrs: dict[str, str])` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/qwen3_xml_tool_parser.py#L779-L888 - Implementation: Method `StreamingXMLToolCallParser._start_element` updates `self.parameters`, `self.current_call_id`, `self.current_param_is_first`, `self.tool_call_index`; calls `self._auto_close_open_parameter_if_needed`, `self._get_next_call_id`, `name.startswith`, `self._start_element`; returns `None`. Handle XML start element events - Inputs: - `name` (str; required): Required positional or keyword input. - `attrs` (dict[str, str]; required): Required positional or keyword input. - Return annotation: `not annotated` - Calls: self._auto_close_open_parameter_if_needed, self._get_next_call_id, name.startswith, self._start_element, self._extract_function_name, DeltaMessage, DeltaToolCall, DeltaFunctionCall, self._emit_delta, self._flush_pending_implicit_delta, self._extract_parameter_name - State reads: self._auto_close_open_parameter_if_needed, self._get_next_call_id, self.current_call_id, self._start_element, self._extract_function_name, self.tool_call_index, self._current_raw_element, self._emit_delta, self._flush_pending_implicit_delta, self._extract_parameter_name, self.parameters - State writes: self.parameters, self.current_call_id, self.current_param_is_first, self.tool_call_index, self.implicit_tool_call_wrapper, self.current_function_name, self.current_function_open, self._pending_implicit_delta, self._pending_implicit_raw_text, self._pending_implicit_text_buffer, self.current_param_name, self.current_param_value, self.current_param_value_converted, self.start_quote_emitted - Return expressions: None ## `vllm_mlx.tool_parsers.qwen3_xml_tool_parser.StreamingXMLToolCallParser._flush_pending_implicit_delta` - Kind: method - Signature: `def _flush_pending_implicit_delta(self) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/qwen3_xml_tool_parser.py#L890-L900 - Implementation: Method `StreamingXMLToolCallParser._flush_pending_implicit_delta` updates `self._pending_implicit_delta`, `self._pending_implicit_raw_text`, `self._pending_implicit_text_buffer`; calls `self._emit_delta`. Emit a deferred bare- delta now that the call is confirmed. - Inputs: none - Return annotation: `None` - Calls: self._emit_delta - State reads: self._pending_implicit_delta, self._emit_delta - State writes: self._pending_implicit_delta, self._pending_implicit_raw_text, self._pending_implicit_text_buffer ## `vllm_mlx.tool_parsers.qwen3_xml_tool_parser.StreamingXMLToolCallParser._abandon_pending_implicit_tool_call` - Kind: method - Signature: `def _abandon_pending_implicit_tool_call(self) -> tuple[str, str]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/qwen3_xml_tool_parser.py#L902-L928 - Implementation: Method `StreamingXMLToolCallParser._abandon_pending_implicit_tool_call` updates `self._pending_implicit_delta`, `self._pending_implicit_raw_text`, `self._pending_implicit_text_buffer`, `self.implicit_tool_call_wrapper`; calls `self._reset_xml_parser_after_tool_call`; returns `(raw_text, buffered_text)`. Roll back a deferred bare- auto-open: prose followed. Drops the pending function-name delta and unwinds the synthesised wrapper state so no tool_call is ever emitted for this fragment. Returns ``(raw_text, buffered_text)`` so the caller can restore the original prose (raw `` tag + any whitespace held while waiting on commitment) as user-visible content. - Inputs: none - Return annotation: `tuple[str, str]` - Calls: self._reset_xml_parser_after_tool_call - State reads: self._pending_implicit_raw_text, self._pending_implicit_text_buffer, self.tool_call_index, self.current_call_id, self._reset_xml_parser_after_tool_call - State writes: self._pending_implicit_delta, self._pending_implicit_raw_text, self._pending_implicit_text_buffer, self.implicit_tool_call_wrapper, self.current_function_name, self.current_function_open, self.tool_call_index, self.last_completed_call_id, self.current_call_id - Return expressions: (raw_text, buffered_text) ## `vllm_mlx.tool_parsers.qwen3_xml_tool_parser.StreamingXMLToolCallParser._char_data` - Kind: method - Signature: `def _char_data(self, data: str)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/qwen3_xml_tool_parser.py#L930-L1025 - Implementation: Method `StreamingXMLToolCallParser._char_data` updates `self._pending_implicit_text_buffer`, `self.should_emit_end_newline`, `self.current_param_value`, `self.start_quote_emitted`; calls `data.strip`, `self._abandon_pending_implicit_tool_call`, `self._emit_delta`, `DeltaMessage`; returns `None`. Handle XML character data events - Inputs: - `data` (str; required): Required positional or keyword input. - Return annotation: `not annotated` - Calls: data.strip, self._abandon_pending_implicit_tool_call, self._emit_delta, DeltaMessage, original_data.endswith, self._get_param_type, data.startswith, DeltaToolCall, DeltaFunctionCall, self._convert_param_value, self._convert_for_json_streaming, len - State reads: self._pending_implicit_delta, self.current_param_name, self._abandon_pending_implicit_tool_call, self._emit_delta, self.defer_current_parameter, self.should_emit_end_newline, self._get_param_type, self.current_param_value, self.start_quote_emitted, self.tool_call_index, self.current_call_id, self._convert_param_value, self._convert_for_json_streaming, self.current_param_value_converted - State writes: self._pending_implicit_text_buffer, self.should_emit_end_newline, self.current_param_value, self.start_quote_emitted, self.current_param_value_converted - Return expressions: None ## `vllm_mlx.tool_parsers.qwen3_xml_tool_parser.StreamingXMLToolCallParser._end_element` - Kind: method - Signature: `def _end_element(self, name: str)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/qwen3_xml_tool_parser.py#L1027-L1206 - Implementation: Method `StreamingXMLToolCallParser._end_element` updates `self.should_emit_end_newline`, `self.current_param_name`, `self.current_param_value`, `self.current_param_value_converted`; calls `name.startswith`, `self._auto_close_open_parameter_if_needed`, `ast.literal_eval`, `json.dumps`; returns `None`. Handle XML end element events - Inputs: - `name` (str; required): Required positional or keyword input. - Return annotation: `not annotated` - Calls: name.startswith, self._auto_close_open_parameter_if_needed, ast.literal_eval, json.dumps, DeltaMessage, DeltaToolCall, DeltaFunctionCall, self._emit_delta, self._get_param_type, self._convert_param_value, self._flush_pending_implicit_delta, self._end_element, self.text_content_buffer.strip, self._reset_xml_parser_after_tool_call - State reads: self.current_param_name, self._auto_close_open_parameter_if_needed, self.current_param_value, self.defer_current_parameter, self.deferred_param_raw_value, self.should_emit_end_newline, self.tool_call_index, self.current_call_id, self._emit_delta, self.parameters, self._get_param_type, self._convert_param_value, self.start_quote_emitted, self._flush_pending_implicit_delta, self.implicit_tool_call_wrapper, self._end_element, self.current_function_open, self.text_content_buffer.strip, self.text_content_buffer, self._reset_xml_parser_after_tool_call - State writes: self.should_emit_end_newline, self.current_param_name, self.current_param_value, self.current_param_value_converted, self.start_quote_emitted, self.defer_current_parameter, self.deferred_param_raw_value, self.current_function_open, self.implicit_tool_call_wrapper - Return expressions: None ## `vllm_mlx.tool_parsers.qwen3_xml_tool_parser.StreamingXMLToolCallParser.setup_parser` - Kind: method - Signature: `def setup_parser(self)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/qwen3_xml_tool_parser.py#L1208-L1213 - Implementation: Method `StreamingXMLToolCallParser.setup_parser` updates `self.parser.buffer_text`, `self.parser.StartElementHandler`, `self.parser.EndElementHandler`, `self.parser.CharacterDataHandler`. Set up XML parser event handlers - Inputs: none - Return annotation: `not annotated` - State reads: self.parser, self._start_element, self._end_element, self._char_data - State writes: self.parser.buffer_text, self.parser.StartElementHandler, self.parser.EndElementHandler, self.parser.CharacterDataHandler ## `vllm_mlx.tool_parsers.qwen3_xml_tool_parser.StreamingXMLToolCallParser.set_tools` - Kind: method - Signature: `def set_tools(self, tools: Union[list[ChatCompletionToolsParam], None])` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/qwen3_xml_tool_parser.py#L1215-L1217 - Implementation: Method `StreamingXMLToolCallParser.set_tools` updates `self.tools`. Set tool configuration information - Inputs: - `tools` (Union[list[ChatCompletionToolsParam], None]; required): Required positional or keyword input. - Return annotation: `not annotated` - State writes: self.tools ## `vllm_mlx.tool_parsers.qwen3_xml_tool_parser.StreamingXMLToolCallParser._get_next_call_id` - Kind: method - Signature: `def _get_next_call_id(self)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/qwen3_xml_tool_parser.py#L1219-L1221 - Implementation: Method `StreamingXMLToolCallParser._get_next_call_id` calls `uuid.uuid4`; returns `f'call_{uuid.uuid4().hex[:24]}'`. Generate unique call ID - Inputs: none - Return annotation: `not annotated` - Calls: uuid.uuid4 - Return expressions: f'call_{uuid.uuid4().hex[:24]}' ## `vllm_mlx.tool_parsers.qwen3_xml_tool_parser.StreamingXMLToolCallParser._extract_function_name` - Kind: method - Signature: `def _extract_function_name(self, name: str, attrs: dict[str, str]) -> Optional[str]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/qwen3_xml_tool_parser.py#L1223-L1233 - Implementation: Method `StreamingXMLToolCallParser._extract_function_name` calls `name.split`, `len`; has 3 explicit return paths. Extract function name from various formats - Inputs: - `name` (str; required): Required positional or keyword input. - `attrs` (dict[str, str]; required): Required positional or keyword input. - Return annotation: `Optional[str]` - Calls: name.split, len - Return expressions: attrs['name']; parts[1]; None ## `vllm_mlx.tool_parsers.qwen3_xml_tool_parser.StreamingXMLToolCallParser._extract_parameter_name` - Kind: method - Signature: `def _extract_parameter_name(self, name: str, attrs: dict[str, str]) -> Optional[str]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/qwen3_xml_tool_parser.py#L1235-L1247 - Implementation: Method `StreamingXMLToolCallParser._extract_parameter_name` calls `name.split`, `len`; has 3 explicit return paths. Extract parameter name from various formats - Inputs: - `name` (str; required): Required positional or keyword input. - `attrs` (dict[str, str]; required): Required positional or keyword input. - Return annotation: `Optional[str]` - Calls: name.split, len - Return expressions: attrs['name']; parts[1]; None ## `vllm_mlx.tool_parsers.qwen3_xml_tool_parser.StreamingXMLToolCallParser._get_param_type` - Kind: method - Signature: `def _get_param_type(self, param_name: str) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/qwen3_xml_tool_parser.py#L1249-L1287 - Implementation: Method `StreamingXMLToolCallParser._get_param_type` calls `hasattr`, `isinstance`, `self.repair_param_type`, `str`; has 3 explicit return paths. Get parameter type based on tool configuration, defaults to string Args: param_name: Parameter name Returns: Parameter type - Inputs: - `param_name` (str; required): Parameter name - Return annotation: `str` - Calls: hasattr, isinstance, self.repair_param_type, str, properties[param_name].get, param_config.get - State reads: self.tools, self.current_function_name, self.repair_param_type - Return expressions: 'string'; self.repair_param_type(str(properties[param_name].get('type', 'string'))); self.repair_param_type(str(param_config.get('type', 'string'))) ## `vllm_mlx.tool_parsers.qwen3_xml_tool_parser.StreamingXMLToolCallParser.repair_param_type` - Kind: method - Signature: `def repair_param_type(self, param_type: str) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/qwen3_xml_tool_parser.py#L1289-L1315 - Implementation: Method `StreamingXMLToolCallParser.repair_param_type` calls `param_type.startswith`; has 2 explicit return paths. Repair unknown parameter types by treating them as string Args: param_type: Parameter type Returns: Repaired parameter type - Inputs: - `param_type` (str; required): Parameter type - Return annotation: `str` - Calls: param_type.startswith - Return expressions: param_type; 'string' ## `vllm_mlx.tool_parsers.qwen3_xml_tool_parser.StreamingXMLToolCallParser._convert_param_value` - Kind: method - Signature: `def _convert_param_value(self, param_value: str, param_type: str) -> Any` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/qwen3_xml_tool_parser.py#L1317-L1371 - Implementation: Method `StreamingXMLToolCallParser._convert_param_value` calls `param_value.lower`, `param_type.strip().lower`, `param_type.strip`, `param_type.startswith`; has 5 explicit return paths. Convert value based on parameter type Args: param_value: Parameter value param_type: Parameter type Returns: Converted value - Inputs: - `param_value` (str; required): Parameter value - `param_type` (str; required): Parameter type - Return annotation: `Any` - Calls: param_value.lower, param_type.strip().lower, param_type.strip, param_type.startswith, int, logger.warning, float - State reads: self.current_param_name, self.current_function_name - Return expressions: None; param_value; int(param_value); float_param_value if float_param_value - int(float_param_value) != 0 else int(float_param_value); param_value == 'true' ## `vllm_mlx.tool_parsers.qwen3_xml_tool_parser.StreamingXMLToolCallParser._convert_for_json_streaming` - Kind: method - Signature: `def _convert_for_json_streaming(self, converted_value: Any, param_type: str) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/qwen3_xml_tool_parser.py#L1373-L1395 - Implementation: Method `StreamingXMLToolCallParser._convert_for_json_streaming` calls `json.dumps`, `isinstance`; has 4 explicit return paths. Convert converted_value based on whether it's empty and if type is string Args: converted_value: Converted value param_type: Parameter type Returns: Converted string for streaming output - Inputs: - `converted_value` (Any; required): Converted value - `param_type` (str; required): Parameter type - Return annotation: `str` - Calls: json.dumps, isinstance - Return expressions: ''; json.dumps(converted_value, ensure_ascii=False)[1:-1]; json.dumps(converted_value, ensure_ascii=False); converted_value ## `vllm_mlx.tool_parsers.qwen3_xml_tool_parser.StreamingXMLToolCallParser._reset_xml_parser_after_tool_call` - Kind: method - Signature: `def _reset_xml_parser_after_tool_call(self)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/qwen3_xml_tool_parser.py#L1397-L1427 - Implementation: Method `StreamingXMLToolCallParser._reset_xml_parser_after_tool_call` updates `self.parser`, `self.last_completed_call_id`, `self.current_call_id`, `self.current_function_name`; calls `ParserCreate`, `self.setup_parser`. Each tool_call is treated as a separate XML document, so we need to reset the parser after each tool_call. - Inputs: none - Return annotation: `not annotated` - Calls: ParserCreate, self.setup_parser - State reads: self.setup_parser, self.current_call_id - State writes: self.parser, self.last_completed_call_id, self.current_call_id, self.current_function_name, self.current_function_open, self.parameters, self.current_param_name, self.current_param_value, self.current_param_value_converted, self.current_param_is_first, self.should_emit_end_newline, self.start_quote_emitted, self.text_content_buffer, self._pre_inside_parameter, self._pre_param_buffer, self._pre_current_param_name, self.defer_current_parameter, self.deferred_param_raw_value ## `vllm_mlx.tool_parsers.qwen3_xml_tool_parser.Qwen3XMLToolParser` - Kind: class - Signature: `class Qwen3XMLToolParser(ToolParser)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/qwen3_xml_tool_parser.py#L1442-L1559 - Implementation: Class `Qwen3XMLToolParser` derives from `ToolParser` and declares 4 direct member(s). XML tool call parser for Qwen 3.5 models, adapted for vllm-mlx. Core parsing logic from vLLM PR #25028 (Qwen API team). Uses expat-based streaming XML parser with type coercion, deferred parsing for complex types, and auto-closing of malformed XML. - Inputs: - `tokenizer` (not annotated; optional; default `None`): Optional positional or keyword input; defaults to `None`. - Constructs: `vllm_mlx.tool_parsers.qwen3_xml_tool_parser.Qwen3XMLToolParser` - Decorators: ToolParserManager.register_module(['qwen3_xml', 'qwen3.5', 'qwen3_coder']) ## `vllm_mlx.tool_parsers.qwen3_xml_tool_parser.Qwen3XMLToolParser.__init__` - Kind: method - Signature: `def __init__(self, tokenizer=None)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/qwen3_xml_tool_parser.py#L1454-L1460 - Implementation: Method `Qwen3XMLToolParser.__init__` updates `self._xml_parser`; calls `super().__init__`, `super`, `StreamingXMLToolCallParser`, `logger.info`. Method `Qwen3XMLToolParser.__init__` updates `self._xml_parser`; calls `super().__init__`, `super`, `StreamingXMLToolCallParser`, `logger.info`. - Inputs: - `tokenizer` (not annotated; optional; default `None`): Optional positional or keyword input; defaults to `None`. - Return annotation: `not annotated` - Calls: super().__init__, super, StreamingXMLToolCallParser, logger.info - State reads: self.__class__.__name__, self.__class__ - State writes: self._xml_parser ## `vllm_mlx.tool_parsers.qwen3_xml_tool_parser.Qwen3XMLToolParser._wrap_tools` - Kind: method - Signature: `def _wrap_tools(request: dict[str, Any] | None) -> list[_ToolDef] | None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/qwen3_xml_tool_parser.py#L1463-L1467 - Implementation: Method `Qwen3XMLToolParser._wrap_tools` calls `request.get`, `_ToolDef`; has 2 explicit return paths. Convert tool definition dicts to _ToolDef wrappers for attribute access. - Inputs: - `request` (dict[str, Any] | None; required): Required positional or keyword input. - Return annotation: `list[_ToolDef] | None` - Decorators: staticmethod - Calls: request.get, _ToolDef - Return expressions: [_ToolDef(t) for t in request['tools']]; None ## `vllm_mlx.tool_parsers.qwen3_xml_tool_parser.Qwen3XMLToolParser.extract_tool_calls` - Kind: method - Signature: `def extract_tool_calls(self, model_output: str, request: dict[str, Any] | None=None) -> ExtractedToolCallInformation` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/qwen3_xml_tool_parser.py#L1469-L1507 - Implementation: Method `Qwen3XMLToolParser.extract_tool_calls` calls `self.strip_think_tags`, `self._xml_parser.reset_streaming_state`, `self._wrap_tools`, `self._xml_parser.set_tools`; has 2 explicit return paths. Extract tool calls from complete Qwen 3.5 output. - Inputs: - `model_output` (str; required): Required positional or keyword input. - `request` (dict[str, Any] | None; optional; default `None`): Optional positional or keyword input; defaults to `None`. - Return annotation: `ExtractedToolCallInformation` - Calls: self.strip_think_tags, self._xml_parser.reset_streaming_state, self._wrap_tools, self._xml_parser.set_tools, self._xml_parser.parse_single_streaming_chunks, ExtractedToolCallInformation, tool_calls.append, uuid.uuid4, len - State reads: self.strip_think_tags, self._xml_parser.reset_streaming_state, self._xml_parser, self._wrap_tools, self._xml_parser.set_tools, self._xml_parser.parse_single_streaming_chunks - Return expressions: ExtractedToolCallInformation(tools_called=False, tool_calls=[], content=result.content if result.content else model_out…; ExtractedToolCallInformation(tools_called=len(tool_calls) > 0, tool_calls=tool_calls, content=result.content) ## `vllm_mlx.tool_parsers.qwen3_xml_tool_parser.Qwen3XMLToolParser.extract_tool_calls_streaming` - Kind: method - Signature: `def extract_tool_calls_streaming(self, previous_text: str, current_text: str, delta_text: str, previous_token_ids: Sequence[int] | None=None, current_token_ids: Sequence[int] | None=None, delta_token_ids: Sequence[int] | None=None, request: dict[str, Any] | None=None) -> dict[str, Any] | None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/qwen3_xml_tool_parser.py#L1509-L1559 - Implementation: Method `Qwen3XMLToolParser.extract_tool_calls_streaming` calls `self._xml_parser.reset_streaming_state`, `self._wrap_tools`, `self._xml_parser.set_tools`, `self._xml_parser.parse_single_streaming_chunks`; has 3 explicit return paths. Extract tool calls from streaming Qwen 3.5 output. Returns dict with 'tool_calls' and/or 'content' keys, or None to suppress the chunk. - Inputs: - `previous_text` (str; required): Required positional or keyword input. - `current_text` (str; required): Required positional or keyword input. - `delta_text` (str; required): Required positional or keyword input. - `previous_token_ids` (Sequence[int] | None; optional; default `None`): Optional positional or keyword input; defaults to `None`. - `current_token_ids` (Sequence[int] | None; optional; default `None`): Optional positional or keyword input; defaults to `None`. - `delta_token_ids` (Sequence[int] | None; optional; default `None`): Optional positional or keyword input; defaults to `None`. - `request` (dict[str, Any] | None; optional; default `None`): Optional positional or keyword input; defaults to `None`. - Return annotation: `dict[str, Any] | None` - Calls: self._xml_parser.reset_streaming_state, self._wrap_tools, self._xml_parser.set_tools, self._xml_parser.parse_single_streaming_chunks, tool_calls.append - State reads: self._xml_parser.reset_streaming_state, self._xml_parser, self._wrap_tools, self._xml_parser.set_tools, self._xml_parser.parse_single_streaming_chunks - Return expressions: None; {'tool_calls': tool_calls}; {'content': result.content} # Module `vllm_mlx.tool_parsers.qwen_tool_parser` Qwen tool call parser for vllm-mlx. Handles Qwen's tool calling formats: - XML style: {"name": "func", "arguments": {...}} - Bracket style: [Calling tool: func_name({"arg": "value"})] - Function style: value Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/qwen_tool_parser.py#L1-L351 ## `vllm_mlx.tool_parsers.qwen_tool_parser._parse_param_value` - Kind: function - Signature: `def _parse_param_value(val: str) -> Any` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/qwen_tool_parser.py#L25-L40 - Implementation: Function `_parse_param_value` calls `json.loads`, `ast.literal_eval`, `isinstance`, `sorted`; has 3 explicit return paths. Parse a parameter value, handling JSON literals and plain strings. - Inputs: - `val` (str; required): Required positional or keyword input. - Return annotation: `Any` - Calls: json.loads, ast.literal_eval, isinstance, sorted, json.dumps - Return expressions: json.loads(val); val; python_val ## `vllm_mlx.tool_parsers.qwen_tool_parser.generate_tool_id` - Kind: function - Signature: `def generate_tool_id() -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/qwen_tool_parser.py#L43-L45 - Implementation: Function `generate_tool_id` calls `uuid.uuid4`; returns `f'call_{uuid.uuid4().hex[:8]}'`. Generate a unique tool call ID. - Inputs: none - Return annotation: `str` - Calls: uuid.uuid4 - Return expressions: f'call_{uuid.uuid4().hex[:8]}' ## `vllm_mlx.tool_parsers.qwen_tool_parser.QwenToolParser` - Kind: class - Signature: `class QwenToolParser(ToolParser)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/qwen_tool_parser.py#L49-L351 - Implementation: Class `QwenToolParser` derives from `ToolParser` and declares 6 direct member(s). Tool call parser for Qwen models. Supports multiple Qwen tool call formats: - XML: {"name": "func", "arguments": {...}} - Bracket: [Calling tool: func_name({"arg": "value"})] - Function: value Used when --enable-auto-tool-choice --tool-call-parser qwen are set. - Inputs: none - Constructs: `vllm_mlx.tool_parsers.qwen_tool_parser.QwenToolParser` - Decorators: ToolParserManager.register_module(['qwen', 'qwen3']) ## `vllm_mlx.tool_parsers.qwen_tool_parser.QwenToolParser.extract_tool_calls` - Kind: method - Signature: `def extract_tool_calls(self, model_output: str, request: dict[str, Any] | None=None) -> ExtractedToolCallInformation` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/qwen_tool_parser.py#L78-L197 - Implementation: Method `QwenToolParser.extract_tool_calls` calls `self.strip_think_tags`, `self.BRACKET_PATTERN.findall`, `json.loads`, `tool_calls.append`; has 2 explicit return paths. Extract tool calls from a complete Qwen model response. - Inputs: - `model_output` (str; required): Required positional or keyword input. - `request` (dict[str, Any] | None; optional; default `None`): Optional positional or keyword input; defaults to `None`. - Return annotation: `ExtractedToolCallInformation` - Calls: self.strip_think_tags, self.BRACKET_PATTERN.findall, json.loads, tool_calls.append, generate_tool_id, name.strip, isinstance, json.dumps, str, self.BRACKET_PATTERN.sub('', cleaned_text).strip, self.BRACKET_PATTERN.sub, self.XML_PATTERN.findall, data.get, self.XML_PATTERN.sub('', cleaned_text).strip, self.XML_PATTERN.sub, self.FUNCTION_PATTERN.findall, params_block.strip, params_block_stripped.startswith, self.PARAM_PATTERN.findall, p_name.strip, _parse_param_value, p_value.strip, self.FUNCTION_PATTERN.sub('', cleaned_text).strip, self.FUNCTION_PATTERN.sub, self.EMPTY_TOOL_CALL.sub('', cleaned_text).strip, self.EMPTY_TOOL_CALL.sub, self._strip_unclosed_markup, ExtractedToolCallInformation - State reads: self.strip_think_tags, self.BRACKET_PATTERN.findall, self.BRACKET_PATTERN, self.BRACKET_PATTERN.sub, self.XML_PATTERN.findall, self.XML_PATTERN, self.XML_PATTERN.sub, self.FUNCTION_PATTERN.findall, self.FUNCTION_PATTERN, self.PARAM_PATTERN.findall, self.PARAM_PATTERN, self.FUNCTION_PATTERN.sub, self.EMPTY_TOOL_CALL.sub, self.EMPTY_TOOL_CALL, self._strip_unclosed_markup - Return expressions: ExtractedToolCallInformation(tools_called=True, tool_calls=tool_calls, content=cleaned_text if cleaned_text else None); ExtractedToolCallInformation(tools_called=False, tool_calls=[], content=stripped) ## `vllm_mlx.tool_parsers.qwen_tool_parser.QwenToolParser._strip_unclosed_markup` - Kind: method - Signature: `def _strip_unclosed_markup(text: str) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/qwen_tool_parser.py#L200-L228 - Implementation: Method `QwenToolParser._strip_unclosed_markup` calls `len`, `text.rfind`, `min`, `text[:earliest].rstrip`; has 2 explicit return paths. Strip a trailing unclosed tool-call marker (truncated output). When generation hits max_tokens mid-tool-call, a partial ````/`` bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/qwen_tool_parser.py#L235-L237 - Implementation: Method `QwenToolParser._has_partial_marker` calls `self._get_partial_marker_len`; returns `self._get_partial_marker_len(text) > 0`. Check if text ends with an incomplete tool call marker prefix. - Inputs: - `text` (str; required): Required positional or keyword input. - Return annotation: `bool` - Calls: self._get_partial_marker_len - State reads: self._get_partial_marker_len - Return expressions: self._get_partial_marker_len(text) > 0 ## `vllm_mlx.tool_parsers.qwen_tool_parser.QwenToolParser._get_partial_marker_len` - Kind: method - Signature: `def _get_partial_marker_len(self, text: str) -> int` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/qwen_tool_parser.py#L239-L248 - Implementation: Method `QwenToolParser._get_partial_marker_len` calls `range`, `len`, `tail.endswith`; returns `best`. Return the length of a partial tool call marker suffix at end of text. - Inputs: - `text` (str; required): Required positional or keyword input. - Return annotation: `int` - Calls: range, len, tail.endswith - State reads: self._PARTIAL_MARKERS - Return expressions: best ## `vllm_mlx.tool_parsers.qwen_tool_parser.QwenToolParser._was_buffering` - Kind: method - Signature: `def _was_buffering(self, previous_text: str) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/qwen_tool_parser.py#L250-L252 - Implementation: Method `QwenToolParser._was_buffering` calls `self._has_partial_marker`; returns `self._has_partial_marker(previous_text)`. Check if the previous call was buffering a partial marker. - Inputs: - `previous_text` (str; required): Required positional or keyword input. - Return annotation: `bool` - Calls: self._has_partial_marker - State reads: self._has_partial_marker - Return expressions: self._has_partial_marker(previous_text) ## `vllm_mlx.tool_parsers.qwen_tool_parser.QwenToolParser.extract_tool_calls_streaming` - Kind: method - Signature: `def extract_tool_calls_streaming(self, previous_text: str, current_text: str, delta_text: str, previous_token_ids: Sequence[int] | None=None, current_token_ids: Sequence[int] | None=None, delta_token_ids: Sequence[int] | None=None, request: dict[str, Any] | None=None) -> dict[str, Any] | None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/qwen_tool_parser.py#L254-L351 - Implementation: Method `QwenToolParser.extract_tool_calls_streaming` calls `self._has_partial_marker`, `self._get_partial_marker_len`, `len`, `self._was_buffering`; has 6 explicit return paths. Extract tool calls from streaming Qwen model output. - Inputs: - `previous_text` (str; required): Required positional or keyword input. - `current_text` (str; required): Required positional or keyword input. - `delta_text` (str; required): Required positional or keyword input. - `previous_token_ids` (Sequence[int] | None; optional; default `None`): Optional positional or keyword input; defaults to `None`. - `current_token_ids` (Sequence[int] | None; optional; default `None`): Optional positional or keyword input; defaults to `None`. - `delta_token_ids` (Sequence[int] | None; optional; default `None`): Optional positional or keyword input; defaults to `None`. - `request` (dict[str, Any] | None; optional; default `None`): Optional positional or keyword input; defaults to `None`. - Return annotation: `dict[str, Any] | None` - Calls: self._has_partial_marker, self._get_partial_marker_len, len, self._was_buffering, range, previous_text.endswith, current_text.count, previous_text.count, self.extract_tool_calls, enumerate - State reads: self._has_partial_marker, self._get_partial_marker_len, self._was_buffering, self._PARTIAL_MARKERS, self.extract_tool_calls - Return expressions: {'content': delta_text[:safe_chars]}; None; {'content': prefix + delta_text}; {'content': delta_text}; {'tool_calls': [{'index': prev_func_close + i, 'id': tc['id'], 'type': 'function', 'function': {'name': tc['name'], 'ar…; {'tool_calls': [{'index': i, 'id': tc['id'], 'type': 'function', 'function': {'name': tc['name'], 'arguments': tc['argu… # Module `vllm_mlx.tool_parsers.xlam_tool_parser` xLAM tool call parser for vllm-mlx. Handles Salesforce xLAM models' tool calling format which supports: - JSON arrays of tool calls - Tool calls in markdown code blocks - Tool calls after reasoning blocks Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/xlam_tool_parser.py#L1-L177 ## `vllm_mlx.tool_parsers.xlam_tool_parser.generate_tool_id` - Kind: function - Signature: `def generate_tool_id() -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/xlam_tool_parser.py#L24-L26 - Implementation: Function `generate_tool_id` calls `uuid.uuid4`; returns `f'call_{uuid.uuid4().hex[:8]}'`. Generate a unique tool call ID. - Inputs: none - Return annotation: `str` - Calls: uuid.uuid4 - Return expressions: f'call_{uuid.uuid4().hex[:8]}' ## `vllm_mlx.tool_parsers.xlam_tool_parser.xLAMToolParser` - Kind: class - Signature: `class xLAMToolParser(ToolParser)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/xlam_tool_parser.py#L30-L177 - Implementation: Class `xLAMToolParser` derives from `ToolParser` and declares 3 direct member(s). Tool call parser for Salesforce xLAM models. Supports multiple formats: - JSON array: [{"name": "func", "arguments": {...}}] - Markdown code blocks: ```json [...] ``` - After thinking: [...] Used when --enable-auto-tool-choice --tool-call-parser xlam are set. - Inputs: none - Constructs: `vllm_mlx.tool_parsers.xlam_tool_parser.xLAMToolParser` - Decorators: ToolParserManager.register_module('xlam') ## `vllm_mlx.tool_parsers.xlam_tool_parser.xLAMToolParser._try_extract_json` - Kind: method - Signature: `def _try_extract_json(self, text: str) -> tuple[str | None, list | None]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/xlam_tool_parser.py#L47-L91 - Implementation: Method `xLAMToolParser._try_extract_json` calls `pattern.findall`, `json.loads`, `match.strip`, `isinstance`; has 3 explicit return paths. Try to extract JSON tool calls from text. Returns: Tuple of (content, tool_calls_list) - Inputs: - `text` (str; required): Required positional or keyword input. - Return annotation: `tuple[str | None, list | None]` - Calls: pattern.findall, json.loads, match.strip, isinstance, pattern.sub('', text).strip, pattern.sub, self.THINKING_PATTERN.search, thinking_match.group(1).strip, thinking_match.group, text[:thinking_match.start() + len('')].strip, thinking_match.start, len, text.strip, text.startswith - State reads: self.CODE_BLOCK_PATTERN, self.TOOL_CALLS_TAG_PATTERN, self.THINKING_PATTERN.search, self.THINKING_PATTERN - Return expressions: (content if content else None, parsed); (None, parsed); (text, None) ## `vllm_mlx.tool_parsers.xlam_tool_parser.xLAMToolParser.extract_tool_calls` - Kind: method - Signature: `def extract_tool_calls(self, model_output: str, request: dict[str, Any] | None=None) -> ExtractedToolCallInformation` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/xlam_tool_parser.py#L93-L131 - Implementation: Method `xLAMToolParser.extract_tool_calls` calls `self._try_extract_json`, `ExtractedToolCallInformation`, `isinstance`, `call.get`; has 3 explicit return paths. Extract tool calls from xLAM model output. - Inputs: - `model_output` (str; required): Required positional or keyword input. - `request` (dict[str, Any] | None; optional; default `None`): Optional positional or keyword input; defaults to `None`. - Return annotation: `ExtractedToolCallInformation` - Calls: self._try_extract_json, ExtractedToolCallInformation, isinstance, call.get, tool_calls.append, generate_tool_id, json.dumps, str - State reads: self._try_extract_json - Return expressions: ExtractedToolCallInformation(tools_called=False, tool_calls=[], content=content or model_output); ExtractedToolCallInformation(tools_called=True, tool_calls=tool_calls, content=content); ExtractedToolCallInformation(tools_called=False, tool_calls=[], content=model_output) ## `vllm_mlx.tool_parsers.xlam_tool_parser.xLAMToolParser.extract_tool_calls_streaming` - Kind: method - Signature: `def extract_tool_calls_streaming(self, previous_text: str, current_text: str, delta_text: str, previous_token_ids: Sequence[int] | None=None, current_token_ids: Sequence[int] | None=None, delta_token_ids: Sequence[int] | None=None, request: dict[str, Any] | None=None) -> dict[str, Any] | None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/tool_parsers/xlam_tool_parser.py#L133-L177 - Implementation: Method `xLAMToolParser.extract_tool_calls_streaming` calls `any`, `current_text.strip`, `stripped.startswith`, `self.extract_tool_calls`; has 3 explicit return paths. Extract tool calls from streaming xLAM model output. - Inputs: - `previous_text` (str; required): Required positional or keyword input. - `current_text` (str; required): Required positional or keyword input. - `delta_text` (str; required): Required positional or keyword input. - `previous_token_ids` (Sequence[int] | None; optional; default `None`): Optional positional or keyword input; defaults to `None`. - `current_token_ids` (Sequence[int] | None; optional; default `None`): Optional positional or keyword input; defaults to `None`. - `delta_token_ids` (Sequence[int] | None; optional; default `None`): Optional positional or keyword input; defaults to `None`. - `request` (dict[str, Any] | None; optional; default `None`): Optional positional or keyword input; defaults to `None`. - Return annotation: `dict[str, Any] | None` - Calls: any, current_text.strip, stripped.startswith, self.extract_tool_calls, enumerate - State reads: self.extract_tool_calls - Return expressions: {'content': delta_text}; {'tool_calls': [{'index': i, 'id': tc['id'], 'type': 'function', 'function': {'name': tc['name'], 'arguments': tc['argu…; None # Module `vllm_mlx.utils` Utility modules for vllm-mlx. Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/utils/__init__.py#L1-L13 # Module `vllm_mlx.utils.chat_templates` Chat templates for various models. This module contains Jinja2 chat templates for models that don't include them in their tokenizer configuration. Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/utils/chat_templates.py#L1-L225 # Module `vllm_mlx.utils.download` Resumable model download with retry/timeout support. Pre-downloads models via huggingface_hub.snapshot_download() with configurable timeout and retry logic before passing to mlx-lm/mlx-vlm. Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/utils/download.py#L1-L144 ## `vllm_mlx.utils.download.DownloadConfig` - Kind: class - Signature: `class DownloadConfig` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/utils/download.py#L45-L51 - Implementation: Class `DownloadConfig` declares 0 direct member(s). Configuration for model download behavior. - Inputs: - `download_timeout` (int; optional; default `300`): Optional constructor field; defaults to `300`. - `max_retries` (int; optional; default `3`): Optional constructor field; defaults to `3`. - `retry_backoff_base` (float; optional; default `2.0`): Optional constructor field; defaults to `2.0`. - `offline` (bool; optional; default `False`): Optional constructor field; defaults to `False`. - Constructs: `vllm_mlx.utils.download.DownloadConfig` - Decorators: dataclass ## `vllm_mlx.utils.download.ensure_model_downloaded` - Kind: function - Signature: `def ensure_model_downloaded(model_name: str, config: DownloadConfig | None=None, is_mllm: bool=False) -> Path` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/utils/download.py#L54-L144 - Implementation: Function `ensure_model_downloaded` calls `DownloadConfig`, `Path`, `model_path.exists`, `logger.info`; can raise `RuntimeError`; has 2 explicit return paths. Ensure a model is available locally, downloading with retry if needed. Args: model_name: HuggingFace model name or local path. config: Download configuration. Uses defaults if None. is_mllm: If True, use MLLM download patterns (broader file set). Returns: Path to the local model directory. Raises: RuntimeError: If download fails after all retries. KeyboardInterrupt: Propagated immediately without retry. - Inputs: - `model_name` (str; required): HuggingFace model name or local path. - `config` (DownloadConfig | None; optional; default `None`): Download configuration. Uses defaults if None. - `is_mllm` (bool; optional; default `False`): If True, use MLLM download patterns (broader file set). - Return annotation: `Path` - Calls: DownloadConfig, Path, model_path.exists, logger.info, snapshot_download, RuntimeError, os.environ.get, str, range, logger.warning, time.sleep, logger.error, os.environ.pop - Raises directly: RuntimeError - Return expressions: model_path; result # Module `vllm_mlx.utils.harmony_render` Harmony-format prompt rendering for GPT-OSS via ``openai-harmony``. GPT-OSS models are trained with OpenAI's harmony wire format (channeled ``<|start|>assistant<|channel|>commentary ...<|call|>`` tool calls, ``<|start|>functions.X to=assistant<|channel|>commentary<|message|>...`` tool results, etc.). Rendering harmony correctly from OpenAI-style chat messages is delicate: prior assistant ``tool_calls`` must arrive at the template as structural objects, not the bracket-text fallback that ``api.utils.extract_multimodal_content()`` produces for non-native parsers. This module bypasses the Jinja chat template entirely for harmony-active engines: it converts the OpenAI-format ``messages`` (plus ``tools``) to an ``openai_harmony.Conversation`` and asks the library — the canonical renderer maintained by OpenAI — to serialize it. That sidesteps both the text-flattening upstream and any template-vs-training-format drift. The library is an optional dependency. ``HAS_HARMONY`` reflects import success so the rest of the engine can fall back to ``apply_chat_template`` when the package is absent. See https://github.com/waybarrios/vllm-mlx/issues/568 for the original report and the patch shape Thump604 outlined. Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/utils/harmony_render.py#L1-L303 ## `vllm_mlx.utils.harmony_render._harmony_encoding` - Kind: function - Signature: `def _harmony_encoding() -> Any` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/utils/harmony_render.py#L45-L53 - Implementation: Function `_harmony_encoding` calls `_oh.load_harmony_encoding`; returns `_oh.load_harmony_encoding(_oh.HarmonyEncodingName.HARMONY_GPT_OSS)`. Load the harmony encoding once and reuse it across requests. ``load_harmony_encoding`` reads the harmony tokenizer assets, so calling it on every ``render_messages`` invocation would add latency to the per-request prompt build. Callers reach this only after the HAS_HARMONY guard in ``render_messages``, so ``_oh`` is never None here. - Inputs: none - Return annotation: `Any` - Decorators: lru_cache(maxsize=1) - Calls: _oh.load_harmony_encoding - Return expressions: _oh.load_harmony_encoding(_oh.HarmonyEncodingName.HARMONY_GPT_OSS) ## `vllm_mlx.utils.harmony_render.is_harmony_parser_name` - Kind: function - Signature: `def is_harmony_parser_name(parser_name: str | None) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/utils/harmony_render.py#L56-L61 - Implementation: Function `is_harmony_parser_name` returns `parser_name in {'harmony', 'gpt-oss'}`. Return True when the active --tool-call-parser is a harmony alias. ``HarmonyToolParser`` registers under both ``"harmony"`` and ``"gpt-oss"``. - Inputs: - `parser_name` (str | None; required): Required positional or keyword input. - Return annotation: `bool` - Return expressions: parser_name in {'harmony', 'gpt-oss'} ## `vllm_mlx.utils.harmony_render._build_tools` - Kind: function - Signature: `def _build_tools(tools: list[dict] | None) -> list[Any] | None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/utils/harmony_render.py#L64-L80 - Implementation: Function `_build_tools` calls `t.get`, `fn.get`, `tool_descs.append`, `_oh.ToolDescription.new`; has 2 explicit return paths. Function `_build_tools` calls `t.get`, `fn.get`, `tool_descs.append`, `_oh.ToolDescription.new`; has 2 explicit return paths. - Inputs: - `tools` (list[dict] | None; required): Required positional or keyword input. - Return annotation: `list[Any] | None` - Calls: t.get, fn.get, tool_descs.append, _oh.ToolDescription.new - Return expressions: None; tool_descs or None ## `vllm_mlx.utils.harmony_render._content_to_text` - Kind: function - Signature: `def _content_to_text(content: Any) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/utils/harmony_render.py#L83-L97 - Implementation: Function `_content_to_text` calls `isinstance`, `item.get`, `parts.append`, `'\n'.join`; has 4 explicit return paths. Flatten OpenAI content (str | list[dict]) to plain text. - Inputs: - `content` (Any; required): Required positional or keyword input. - Return annotation: `str` - Calls: isinstance, item.get, parts.append, '\n'.join, str - Return expressions: ''; content; '\n'.join(parts); str(content) ## `vllm_mlx.utils.harmony_render._convert_message` - Kind: function - Signature: `def _convert_message(msg: dict) -> list[Any]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/utils/harmony_render.py#L100-L182 - Implementation: Function `_convert_message` calls `msg.get`, `_content_to_text`, `out.append`, `_oh.Message.from_role_and_content`; has 2 explicit return paths. Convert one OpenAI-format message to one or more ``openai_harmony.Message``. A single assistant turn can carry multiple tool_calls; harmony represents each as its own commentary-channel message addressed to ``functions.X``. Prior reasoning lives in an analysis-channel message that precedes the tool calls. - Inputs: - `msg` (dict; required): Required positional or keyword input. - Return annotation: `list[Any]` - Calls: msg.get, _content_to_text, out.append, _oh.Message.from_role_and_content, tool_name.startswith, _oh.Message, _oh.Author.new, _oh.TextContent, str, tc.get, fn.get, isinstance, json.dumps - Return expressions: []; out ## `vllm_mlx.utils.harmony_render._resolve_tool_names` - Kind: function - Signature: `def _resolve_tool_names(messages: list[dict]) -> list[dict]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/utils/harmony_render.py#L185-L214 - Implementation: Function `_resolve_tool_names` calls `isinstance`, `out.append`, `m.get`, `tc.get`; returns `out`. Stamp ``name=functions.X`` on each ``role=tool`` message by tracing back the most recent assistant ``tool_call_id`` -> function name. - Inputs: - `messages` (list[dict]; required): Required positional or keyword input. - Return annotation: `list[dict]` - Calls: isinstance, out.append, m.get, tc.get, fn.get, dict, new_m.get, by_call_id.get - Return expressions: out ## `vllm_mlx.utils.harmony_render.render_messages` - Kind: function - Signature: `def render_messages(messages: list[dict], tools: list[dict] | None=None, reasoning_effort: str | None=None) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/utils/harmony_render.py#L217-L303 - Implementation: Function `render_messages` calls `RuntimeError`, `_resolve_tool_names`, `isinstance`, `other_msgs.append`; can raise `RuntimeError`; returns `enc.decode(token_ids)`. Render OpenAI-format messages as a harmony-format prompt string. Raises ``RuntimeError`` if ``openai-harmony`` is not importable; callers should pre-check with :data:`HAS_HARMONY` and fall back to ``tokenizer.apply_chat_template`` when False. Args: messages: OpenAI chat-completions messages. tools: OpenAI-format tools list (each item ``{"type":"function","function":{...}}``). reasoning_effort: ``"low"``, ``"medium"``, or ``"high"``. Defaults to medium. Returns: Decoded harmony prompt with the trailing ``<|start|>assistant`` marker ready for the model to begin generation. - Inputs: - `messages` (list[dict]; required): OpenAI chat-completions messages. - `tools` (list[dict] | None; optional; default `None`): OpenAI-format tools list (each item ``{"type":"function","function":{...}}``). - `reasoning_effort` (str | None; optional; default `None`): ``"low"``, ``"medium"``, or ``"high"``. Defaults to medium. - Return annotation: `str` - Calls: RuntimeError, _resolve_tool_names, isinstance, other_msgs.append, m.get, system_msgs.append, developer_msgs.append, _build_tools, h_messages.extend, _convert_message, _oh.SystemContent.new, getattr, reasoning_effort.upper, sys_content.with_reasoning_effort, h_messages.append, _oh.Message.from_role_and_content, _oh.DeveloperContent.new().with_function_tools, _oh.DeveloperContent.new, _oh.Conversation.from_messages, _harmony_encoding, enc.render_conversation_for_completion, enc.decode - Raises directly: RuntimeError - Return expressions: enc.decode(token_ids) # Module `vllm_mlx.utils.mamba_cache` BatchMambaCache implementation for continuous batching with Mamba models. mlx-lm's BatchGenerator requires cache objects to have an `extract` method, but MambaCache (which extends ArraysCache) doesn't have one. This module provides a BatchMambaCache wrapper that adds batching support. Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/utils/mamba_cache.py#L1-L215 ## `vllm_mlx.utils.mamba_cache.BatchMambaCache` - Kind: class - Signature: `class BatchMambaCache(MambaCache)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/utils/mamba_cache.py#L24-L96 - Implementation: Class `BatchMambaCache` derives from `MambaCache` and declares 3 direct member(s). Batch-aware MambaCache for continuous batching. This extends MambaCache to support batch operations required by mlx-lm's BatchGenerator, specifically the `extract` method. - Inputs: - `left_padding` (Optional[List[int]]; optional; default `None`): Amount of left padding for each sequence in batch - `size` (int; optional; default `2`): Number of state arrays (default 2 for Mamba models) - Constructs: `vllm_mlx.utils.mamba_cache.BatchMambaCache` ## `vllm_mlx.utils.mamba_cache.BatchMambaCache.__init__` - Kind: method - Signature: `def __init__(self, left_padding: Optional[List[int]]=None, size: int=2)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/utils/mamba_cache.py#L32-L43 - Implementation: Method `BatchMambaCache.__init__` updates `self._batch_size`; calls `super().__init__`, `super`, `len`. Initialize BatchMambaCache. Args: left_padding: Amount of left padding for each sequence in batch size: Number of state arrays (default 2 for Mamba models) - Inputs: - `left_padding` (Optional[List[int]]; optional; default `None`): Amount of left padding for each sequence in batch - `size` (int; optional; default `2`): Number of state arrays (default 2 for Mamba models) - Return annotation: `not annotated` - Calls: super().__init__, super, len - State writes: self._batch_size ## `vllm_mlx.utils.mamba_cache.BatchMambaCache.extract` - Kind: method - Signature: `def extract(self, idx: int) -> MambaCache` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/utils/mamba_cache.py#L45-L63 - Implementation: Method `BatchMambaCache.extract` calls `len`, `MambaCache`, `mx.contiguous`; returns `cache`. Extract a single cache from the batch. Args: idx: Index of the sequence to extract Returns: A new MambaCache with the extracted state - Inputs: - `idx` (int; required): Index of the sequence to extract - Return annotation: `MambaCache` - Calls: len, MambaCache, mx.contiguous - State reads: self.cache - Return expressions: cache ## `vllm_mlx.utils.mamba_cache.BatchMambaCache.merge` - Kind: method - Signature: `def merge(cls, caches: List[MambaCache]) -> 'BatchMambaCache'` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/utils/mamba_cache.py#L66-L96 - Implementation: Method `BatchMambaCache.merge` calls `cls`, `len`, `range`, `merged_cache.cache.append`; has 2 explicit return paths. Merge multiple MambaCache objects into a BatchMambaCache. Args: caches: List of MambaCache objects to merge Returns: A new BatchMambaCache containing all caches - Inputs: - `caches` (List[MambaCache]; required): List of MambaCache objects to merge - Return annotation: `'BatchMambaCache'` - Decorators: classmethod - Calls: cls, len, range, merged_cache.cache.append, mx.concatenate - Return expressions: cls([]); merged_cache ## `vllm_mlx.utils.mamba_cache.patch_mlx_lm_for_mamba` - Kind: function - Signature: `def patch_mlx_lm_for_mamba()` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/utils/mamba_cache.py#L99-L194 - Implementation: Function `patch_mlx_lm_for_mamba` calls `importlib.import_module`, `logger.info`. Patch mlx-lm to support MambaCache in BatchGenerator. This modifies the _make_cache function to handle MambaCache by converting it to BatchMambaCache. - Inputs: none - Return annotation: `not annotated` - Calls: importlib.import_module, logger.info ## `vllm_mlx.utils.mamba_cache.patch_mlx_lm_for_mamba._patched_make_cache` - Kind: nested function - Signature: `def _patched_make_cache(model, left_padding, max_kv_size=None)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/utils/mamba_cache.py#L126-L166 - Implementation: Nested Function `patch_mlx_lm_for_mamba._patched_make_cache` calls `hasattr`, `model.make_cache`, `to_batch_cache`, `BatchRotatingKVCache`; has 3 explicit return paths. Convert a list of regular caches into their corresponding batch-aware caches, with support for MambaCache. Args: model: The model to create cache for left_padding: Left padding for batch max_kv_size: Maximum KV cache size (mlx-lm 0.30.6+) - Inputs: - `model` (not annotated; required): The model to create cache for - `left_padding` (not annotated; required): Left padding for batch - `max_kv_size` (not annotated; optional; default `None`): Maximum KV cache size (mlx-lm 0.30.6+) - Return annotation: `not annotated` - Calls: hasattr, model.make_cache, to_batch_cache, BatchRotatingKVCache, BatchKVCache - Return expressions: [to_batch_cache(c) for c in cache]; [BatchRotatingKVCache(max_kv_size, left_padding) for _ in model.layers]; [BatchKVCache(left_padding) for _ in model.layers] ## `vllm_mlx.utils.mamba_cache.patch_mlx_lm_for_mamba._patched_make_cache.to_batch_cache` - Kind: nested function - Signature: `def to_batch_cache(c)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/utils/mamba_cache.py#L137-L155 - Implementation: Nested Function `patch_mlx_lm_for_mamba._patched_make_cache.to_batch_cache` calls `isinstance`, `BatchKVCache`, `BatchMambaCache`, `mx.array`; can raise `ValueError`; has 5 explicit return paths. Nested Function `patch_mlx_lm_for_mamba._patched_make_cache.to_batch_cache` calls `isinstance`, `BatchKVCache`, `BatchMambaCache`, `mx.array`; can raise `ValueError`; has 5 explicit return paths. - Inputs: - `c` (not annotated; required): Required positional or keyword input. - Return annotation: `not annotated` - Calls: isinstance, BatchKVCache, BatchMambaCache, mx.array, ValueError, BatchRotatingKVCache, CacheList, to_batch_cache, type - Raises directly: ValueError - Return expressions: BatchKVCache(left_padding); BatchMambaCache(left_padding); c; BatchRotatingKVCache(c.max_size, left_padding); CacheList(*(to_batch_cache(sub_c) for sub_c in c.caches)) ## `vllm_mlx.utils.mamba_cache.patch_mlx_lm_for_mamba._patched_merge_caches` - Kind: nested function - Signature: `def _patched_merge_caches(caches)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/utils/mamba_cache.py#L174-L190 - Implementation: Nested Function `patch_mlx_lm_for_mamba._patched_merge_caches` calls `range`, `len`, `isinstance`, `BatchKVCache.merge`; can raise `ValueError`; returns `batch_cache`. Merge caches with MambaCache support. - Inputs: - `caches` (not annotated; required): Required positional or keyword input. - Return annotation: `not annotated` - Calls: range, len, isinstance, BatchKVCache.merge, BatchRotatingKVCache.merge, BatchMambaCache.merge, ValueError, type, batch_cache.append - Raises directly: ValueError - Return expressions: batch_cache ## `vllm_mlx.utils.mamba_cache.ensure_mamba_support` - Kind: function - Signature: `def ensure_mamba_support()` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/utils/mamba_cache.py#L201-L215 - Implementation: Function `ensure_mamba_support` calls `logger.info`. Ensure MambaCache batching support is enabled. NOTE: Disabled for mlx-lm >= 0.30.6 where ArraysCache natively supports all batch operations (extract, merge, filter, prepare). The old patch replaced ArraysCache with BatchMambaCache, which broke hybrid models (Qwen3.5) that mix ArraysCache + KVCache layers. - Inputs: none - Return annotation: `not annotated` - Calls: logger.info # Module `vllm_mlx.utils.tokenizer` Tokenizer utilities with fallback support for non-standard tokenizers. Some models (e.g., Nemotron) use non-standard tokenizer configurations that transformers doesn't recognize. This module provides fallback loading directly from tokenizer.json. Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/utils/tokenizer.py#L1-L280 ## `vllm_mlx.utils.tokenizer._needs_tokenizer_fallback` - Kind: function - Signature: `def _needs_tokenizer_fallback(model_name: str) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/utils/tokenizer.py#L25-L28 - Implementation: Function `_needs_tokenizer_fallback` calls `model_name.lower`, `any`, `pattern.lower`; returns `any((pattern.lower() in model_lower for pattern in FALLBACK_MODELS))`. Check if model needs tokenizer fallback. - Inputs: - `model_name` (str; required): Required positional or keyword input. - Return annotation: `bool` - Calls: model_name.lower, any, pattern.lower - Return expressions: any((pattern.lower() in model_lower for pattern in FALLBACK_MODELS)) ## `vllm_mlx.utils.tokenizer._needs_strict_false` - Kind: function - Signature: `def _needs_strict_false(model_name: str) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/utils/tokenizer.py#L31-L49 - Implementation: Function `_needs_strict_false` calls `_download`, `load_config`; has 2 explicit return paths. Check if model needs strict=False loading (VLM models with extra weights). VLM models (e.g., Qwen3.5) have vision_tower weights that don't match the text-only model class. Loading with strict=True fails and wastes memory by loading all weights (~100 GB) before raising ValueError. Detect these models up-front to avoid the double-load penalty. - Inputs: - `model_name` (str; required): Required positional or keyword input. - Return annotation: `bool` - Calls: _download, load_config - Return expressions: False; True ## `vllm_mlx.utils.tokenizer.load_model_with_fallback` - Kind: function - Signature: `def load_model_with_fallback(model_name: str, tokenizer_config: dict=None)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/utils/tokenizer.py#L52-L111 - Implementation: Function `load_model_with_fallback` calls `_needs_tokenizer_fallback`, `logger.info`, `_load_with_tokenizer_fallback`, `_needs_strict_false`; has 3 explicit return paths. Load model and tokenizer with fallback for non-standard tokenizers. Args: model_name: HuggingFace model name or local path tokenizer_config: Optional tokenizer configuration Returns: Tuple of (model, tokenizer) - Inputs: - `model_name` (str; required): HuggingFace model name or local path - `tokenizer_config` (dict; optional; default `None`): Optional tokenizer configuration - Return annotation: `not annotated` - Calls: _needs_tokenizer_fallback, logger.info, _load_with_tokenizer_fallback, _needs_strict_false, _load_strict_false, load, str, logger.warning, gc.collect, _try_inject_mtp_post_load - Return expressions: _load_with_tokenizer_fallback(model_name); _load_strict_false(model_name, tokenizer_config); (model, tokenizer) ## `vllm_mlx.utils.tokenizer._load_strict_false` - Kind: function - Signature: `def _load_strict_false(model_name: str, tokenizer_config: dict=None)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/utils/tokenizer.py#L114-L153 - Implementation: Function `_load_strict_false` calls `_download`, `load_model`, `tree_flatten`, `model.parameters`; returns `(model, tokenizer)`. Load model with strict=False to discard extra weights. Handles models with extra parameters that the text-only model class doesn't define (e.g., vision tower weights in VLM models like Qwen3.5, or MTP layers). The model's own sanitize() handles key remapping (e.g., language_model.* prefix), and strict=False silently drops unmatched keys. - Inputs: - `model_name` (str; required): Required positional or keyword input. - `tokenizer_config` (dict; optional; default `None`): Optional positional or keyword input; defaults to `None`. - Return annotation: `not annotated` - Calls: _download, load_model, tree_flatten, model.parameters, len, sum, mx.all(v == 0).item, mx.all, logger.info, hasattr, mx.mean(emb.astype(mx.float32)).item, mx.mean, emb.astype, load_tokenizer, config.get, _try_inject_mtp - Return expressions: (model, tokenizer) ## `vllm_mlx.utils.tokenizer._try_inject_mtp` - Kind: function - Signature: `def _try_inject_mtp(model, model_path, config)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/utils/tokenizer.py#L156-L176 - Implementation: Function `_try_inject_mtp` calls `config.get`, `text_config.get`, `inject_mtp_support`; returns `None`. Inject MTP support if model has MTP config + weights. - Inputs: - `model` (not annotated; required): Required positional or keyword input. - `model_path` (not annotated; required): Required positional or keyword input. - `config` (not annotated; required): Required positional or keyword input. - Return annotation: `not annotated` - Calls: config.get, text_config.get, inject_mtp_support - Return expressions: None ## `vllm_mlx.utils.tokenizer._try_inject_mtp_post_load` - Kind: function - Signature: `def _try_inject_mtp_post_load(model, model_name)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/utils/tokenizer.py#L179-L215 - Implementation: Function `_try_inject_mtp_post_load` calls `_download`, `Path`, `config_path.exists`, `open`; returns `None`. Check if MTP weights exist but were stripped by sanitize(), and inject. - Inputs: - `model` (not annotated; required): Required positional or keyword input. - `model_name` (not annotated; required): Required positional or keyword input. - Return annotation: `not annotated` - Calls: _download, Path, config_path.exists, open, json.load, config.get, text_config.get, hasattr, getattr, mtp_file.exists, logger.info, _try_inject_mtp - Return expressions: None ## `vllm_mlx.utils.tokenizer._load_with_tokenizer_fallback` - Kind: function - Signature: `def _load_with_tokenizer_fallback(model_name: str)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/utils/tokenizer.py#L218-L280 - Implementation: Function `_load_with_tokenizer_fallback` calls `logger.info`, `ensure_model_downloaded`, `load_model`, `tokenizer_json.exists`; can raise `ValueError`; returns `(model, tokenizer)`. Load model with fallback tokenizer for non-standard models like Nemotron. - Inputs: - `model_name` (str; required): Required positional or keyword input. - Return annotation: `not annotated` - Calls: logger.info, ensure_model_downloaded, load_model, tokenizer_json.exists, Tokenizer.from_file, str, tokenizer_config_path.exists, open, json.load, config.get, PreTrainedTokenizerFast, _needs_tokenizer_fallback, ValueError - Raises directly: ValueError - Return expressions: (model, tokenizer) # Module `vllm_mlx.utils.truncation` Shared resolution of the tokenizer truncation length for embedding and reranker models. The input token limit follows each model's own context window (``max_position_embeddings``) instead of a hard-coded 512. A finite tokenizer limit further constrains that architecture value, while HuggingFace's huge unset placeholder is ignored. This keeps position-table offsets safe for RoBERTa-family models without losing model-derived limits for sentinel values. Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/utils/truncation.py#L1-L83 ## `vllm_mlx.utils.truncation._config_get` - Kind: function - Signature: `def _config_get(config: Any, key: str) -> Any` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/utils/truncation.py#L25-L31 - Implementation: Function `_config_get` calls `isinstance`, `config.get`, `getattr`; has 3 explicit return paths. Read ``key`` from a model config that may be a dict or an object. - Inputs: - `config` (Any; required): Required positional or keyword input. - `key` (str; required): Required positional or keyword input. - Return annotation: `Any` - Calls: isinstance, config.get, getattr - Return expressions: None; config.get(key); getattr(config, key, None) ## `vllm_mlx.utils.truncation.inner_tokenizer` - Kind: function - Signature: `def inner_tokenizer(tokenizer: Any) -> Any` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/utils/truncation.py#L34-L36 - Implementation: Function `inner_tokenizer` calls `getattr`; returns `getattr(tokenizer, '_tokenizer', tokenizer)`. Unwrap a wrapping tokenizer to its inner ``_tokenizer`` when present. - Inputs: - `tokenizer` (Any; required): Required positional or keyword input. - Return annotation: `Any` - Calls: getattr - Return expressions: getattr(tokenizer, '_tokenizer', tokenizer) ## `vllm_mlx.utils.truncation._positive_int` - Kind: function - Signature: `def _positive_int(value: Any) -> int | None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/utils/truncation.py#L39-L42 - Implementation: Function `_positive_int` calls `isinstance`; has 2 explicit return paths. Function `_positive_int` calls `isinstance`; has 2 explicit return paths. - Inputs: - `value` (Any; required): Required positional or keyword input. - Return annotation: `int | None` - Calls: isinstance - Return expressions: value; None ## `vllm_mlx.utils.truncation.resolve_max_length` - Kind: function - Signature: `def resolve_max_length(config: Any, tokenizer: Any, *, default: int=MAX_LENGTH_DEFAULT, sentinel_threshold: int=TOKENIZER_SENTINEL_THRESHOLD) -> int` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/utils/truncation.py#L45-L83 - Implementation: Function `resolve_max_length` calls `_positive_int`, `_config_get`, `getattr`, `inner_tokenizer`; returns `resolved`. Resolve the tokenizer truncation length for a model. Source order: 1. ``config.max_position_embeddings`` — an explicit value supplied by the model's own architecture. 2. A finite ``tokenizer.model_max_length`` further constrains the architecture value. This matters for RoBERTa-family models, whose position table includes reserved padding positions. 3. ``default``, when neither source yields a usable value. Args: config: Model config as a dict (reranker) or object (embeddings). tokenizer: The tokenizer (possibly wrapping an inner ``_tokenizer``). default: Fallback when no usable value is found. sentinel_threshold: Tokenizer-derived values at or above this are treated as an unset HuggingFace sentinel, not a real length. Returns: The truncation length to pass as ``max_length``. - Inputs: - `config` (Any; required): Model config as a dict (reranker) or object (embeddings). - `tokenizer` (Any; required): The tokenizer (possibly wrapping an inner ``_tokenizer``). - `default` (int; optional; default `MAX_LENGTH_DEFAULT`): Fallback when no usable value is found. - `sentinel_threshold` (int; optional; default `TOKENIZER_SENTINEL_THRESHOLD`): Tokenizer-derived values at or above this are treated as an unset HuggingFace sentinel, not a real length. - Return annotation: `int` - Calls: _positive_int, _config_get, getattr, inner_tokenizer, min - Return expressions: resolved # Module `vllm_mlx.vision_embedding_cache` Vision Embedding Cache for MLLM continuous batching. This module provides caching for vision embeddings to avoid redundant computation when the same images are processed multiple times. Cache Levels: 1. Pixel Values Cache - Caches processed image tensors (prepare_inputs output) 2. Vision Encoding Cache - Caches VLM forward pass output (logits + cache state) Performance Impact: - Without cache: ~2s per image for vision encoding - With cache hit: ~0.01s (100x speedup for repeated images) Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/vision_embedding_cache.py#L1-L413 ## `vllm_mlx.vision_embedding_cache.VisionCacheStats` - Kind: class - Signature: `class VisionCacheStats` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/vision_embedding_cache.py#L30-L66 - Implementation: Class `VisionCacheStats` declares 3 direct member(s). Statistics for vision cache performance. - Inputs: - `pixel_cache_hits` (int; optional; default `0`): Optional constructor field; defaults to `0`. - `pixel_cache_misses` (int; optional; default `0`): Optional constructor field; defaults to `0`. - `encoding_cache_hits` (int; optional; default `0`): Optional constructor field; defaults to `0`. - `encoding_cache_misses` (int; optional; default `0`): Optional constructor field; defaults to `0`. - `total_time_saved` (float; optional; default `0.0`): Optional constructor field; defaults to `0.0`. - `total_images_processed` (int; optional; default `0`): Optional constructor field; defaults to `0`. - Constructs: `vllm_mlx.vision_embedding_cache.VisionCacheStats` - Decorators: dataclass ## `vllm_mlx.vision_embedding_cache.VisionCacheStats.pixel_hit_rate` - Kind: method - Signature: `def pixel_hit_rate(self) -> float` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/vision_embedding_cache.py#L41-L45 - Implementation: Method `VisionCacheStats.pixel_hit_rate` returns `self.pixel_cache_hits / total if total > 0 else 0.0`. Return successful pixel-cache lookups divided by all pixel lookups. - Inputs: none - Return annotation: `float` - Decorators: property - State reads: self.pixel_cache_hits, self.pixel_cache_misses - Return expressions: self.pixel_cache_hits / total if total > 0 else 0.0 ## `vllm_mlx.vision_embedding_cache.VisionCacheStats.encoding_hit_rate` - Kind: method - Signature: `def encoding_hit_rate(self) -> float` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/vision_embedding_cache.py#L48-L52 - Implementation: Method `VisionCacheStats.encoding_hit_rate` returns `self.encoding_cache_hits / total if total > 0 else 0.0`. Return successful encoding lookups divided by all encoding lookups. - Inputs: none - Return annotation: `float` - Decorators: property - State reads: self.encoding_cache_hits, self.encoding_cache_misses - Return expressions: self.encoding_cache_hits / total if total > 0 else 0.0 ## `vllm_mlx.vision_embedding_cache.VisionCacheStats.to_dict` - Kind: method - Signature: `def to_dict(self) -> dict` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/vision_embedding_cache.py#L54-L66 - Implementation: Method `VisionCacheStats.to_dict` returns `{'pixel_cache_hits': self.pixel_cache_hits, 'pixel_cache_misses': self.pixel_cache_misses, 'pixel_hit_rate': self.pixel…`. Return pixel, encoding, timing, and image counters. - Inputs: none - Return annotation: `dict` - State reads: self.pixel_cache_hits, self.pixel_cache_misses, self.pixel_hit_rate, self.encoding_cache_hits, self.encoding_cache_misses, self.encoding_hit_rate, self.total_time_saved, self.total_images_processed - Return expressions: {'pixel_cache_hits': self.pixel_cache_hits, 'pixel_cache_misses': self.pixel_cache_misses, 'pixel_hit_rate': self.pixel… ## `vllm_mlx.vision_embedding_cache.PixelCacheEntry` - Kind: class - Signature: `class PixelCacheEntry` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/vision_embedding_cache.py#L70-L78 - Implementation: Class `PixelCacheEntry` declares 0 direct member(s). Cached pixel values from prepare_inputs. - Inputs: - `pixel_values` (mx.array; required): Required constructor field. - `input_ids` (mx.array; required): Required constructor field. - `attention_mask` (Optional[mx.array]; required): Required constructor field. - `image_grid_thw` (Optional[mx.array]; required): Required constructor field. - `extra_kwargs` (Dict[str, Any]; required): Required constructor field. - `processing_time` (float; optional; default `0.0`): Optional constructor field; defaults to `0.0`. - Constructs: `vllm_mlx.vision_embedding_cache.PixelCacheEntry` - Decorators: dataclass ## `vllm_mlx.vision_embedding_cache.PixelOnlyCacheEntry` - Kind: class - Signature: `class PixelOnlyCacheEntry` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/vision_embedding_cache.py#L82-L92 - Implementation: Class `PixelOnlyCacheEntry` declares 0 direct member(s). Cached pixel values only (prompt-independent). This cache stores only the image-dependent data that doesn't change with different prompts. Useful when the same images are used with different prompts. - Inputs: - `pixel_values` (mx.array; required): Required constructor field. - `image_grid_thw` (Optional[mx.array]; required): Required constructor field. - `processing_time` (float; optional; default `0.0`): Optional constructor field; defaults to `0.0`. - Constructs: `vllm_mlx.vision_embedding_cache.PixelOnlyCacheEntry` - Decorators: dataclass ## `vllm_mlx.vision_embedding_cache.EncodingCacheEntry` - Kind: class - Signature: `class EncodingCacheEntry` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/vision_embedding_cache.py#L96-L102 - Implementation: Class `EncodingCacheEntry` declares 0 direct member(s). Cached vision encoding output. - Inputs: - `logits` (mx.array; required): Required constructor field. - `first_token` (int; required): Required constructor field. - `logprobs` (mx.array; required): Required constructor field. - `encoding_time` (float; optional; default `0.0`): Optional constructor field; defaults to `0.0`. - Constructs: `vllm_mlx.vision_embedding_cache.EncodingCacheEntry` - Decorators: dataclass ## `vllm_mlx.vision_embedding_cache.compute_image_hash` - Kind: function - Signature: `def compute_image_hash(image_path: str) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/vision_embedding_cache.py#L105-L124 - Implementation: Function `compute_image_hash` calls `Path`, `path.exists`, `path.is_file`, `open`; has 3 explicit return paths. Compute hash of image content. For files: hash the actual content For URLs/base64: hash the string - Inputs: - `image_path` (str; required): Required positional or keyword input. - Return annotation: `str` - Calls: Path, path.exists, path.is_file, open, f.read, hashlib.sha256(content).hexdigest, hashlib.sha256, hashlib.sha256(image_path.encode()).hexdigest, image_path.encode, hashlib.sha256(str(image_path).encode()).hexdigest, str(image_path).encode, str - Return expressions: hashlib.sha256(content).hexdigest()[:16]; hashlib.sha256(image_path.encode()).hexdigest()[:16]; hashlib.sha256(str(image_path).encode()).hexdigest()[:16] ## `vllm_mlx.vision_embedding_cache.compute_images_hash` - Kind: function - Signature: `def compute_images_hash(images: List[str]) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/vision_embedding_cache.py#L127-L132 - Implementation: Function `compute_images_hash` calls `sorted`, `compute_image_hash`, `hashlib.sha256('_'.join(hashes).encode()).hexdigest`, `hashlib.sha256`; has 2 explicit return paths. Compute combined hash for multiple images. - Inputs: - `images` (List[str]; required): Required positional or keyword input. - Return annotation: `str` - Calls: sorted, compute_image_hash, hashlib.sha256('_'.join(hashes).encode()).hexdigest, hashlib.sha256, '_'.join(hashes).encode, '_'.join - Return expressions: 'no_images'; hashlib.sha256('_'.join(hashes).encode()).hexdigest()[:16] ## `vllm_mlx.vision_embedding_cache.VisionEmbeddingCache` - Kind: class - Signature: `class VisionEmbeddingCache` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/vision_embedding_cache.py#L135-L413 - Implementation: Class `VisionEmbeddingCache` declares 12 direct member(s). Two-level cache for vision processing in MLLM. Level 1 (Pixel Cache): - Caches output of prepare_inputs() (pixel_values, input_ids, etc.) - Key: hash(images) + hash(prompt) - Saves: Image loading, resizing, normalization time (~0.5-1s) Level 2 (Encoding Cache): - Caches output of VLM forward pass (logits, first token) - Key: hash(images) + hash(prompt) - Saves: Vision encoder computation time (~1-2s) Example: >>> cache = VisionEmbeddingCache(max_pixel_entries=50, max_encoding_entries=20) >>> >>> # First request - cache miss >>> pixel_entry = cache.get_pixel_cache(images, prompt) >>> if pixel_entry is None: ... # Process images... ... cache.set_pixel_cache(images, prompt, pixel_values, ...) >>> >>> # Second request with same image - cache hit! >>> pixel_entry = cache.get_pixel_cache(images, prompt) # Returns cached data - Inputs: - `max_pixel_entries` (int; optional; default `100`): Max entries in pixel cache (LRU eviction) - `max_encoding_entries` (int; optional; default `50`): Max entries in encoding cache - `enabled` (bool; optional; default `True`): Whether caching is enabled - Constructs: `vllm_mlx.vision_embedding_cache.VisionEmbeddingCache` ## `vllm_mlx.vision_embedding_cache.VisionEmbeddingCache.__init__` - Kind: method - Signature: `def __init__(self, max_pixel_entries: int=100, max_encoding_entries: int=50, enabled: bool=True)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/vision_embedding_cache.py#L162-L185 - Implementation: Method `VisionEmbeddingCache.__init__` updates `self.max_pixel_entries`, `self.max_encoding_entries`, `self.enabled`, `self._pixel_cache`; calls `OrderedDict`, `VisionCacheStats`. Initialize the vision embedding cache. Args: max_pixel_entries: Max entries in pixel cache (LRU eviction) max_encoding_entries: Max entries in encoding cache enabled: Whether caching is enabled - Inputs: - `max_pixel_entries` (int; optional; default `100`): Max entries in pixel cache (LRU eviction) - `max_encoding_entries` (int; optional; default `50`): Max entries in encoding cache - `enabled` (bool; optional; default `True`): Whether caching is enabled - Return annotation: `not annotated` - Calls: OrderedDict, VisionCacheStats - State writes: self.max_pixel_entries, self.max_encoding_entries, self.enabled, self._pixel_cache, self._pixel_only_cache, self._encoding_cache, self.stats ## `vllm_mlx.vision_embedding_cache.VisionEmbeddingCache._make_key` - Kind: method - Signature: `def _make_key(self, images: List[str], prompt: str) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/vision_embedding_cache.py#L187-L192 - Implementation: Method `VisionEmbeddingCache._make_key` calls `compute_images_hash`, `hashlib.sha256(prompt.encode()).hexdigest`, `hashlib.sha256`, `prompt.encode`; returns `f'{img_hash}_{prompt_hash}'`. Create cache key from images and prompt. - Inputs: - `images` (List[str]; required): Required positional or keyword input. - `prompt` (str; required): Required positional or keyword input. - Return annotation: `str` - Calls: compute_images_hash, hashlib.sha256(prompt.encode()).hexdigest, hashlib.sha256, prompt.encode - Return expressions: f'{img_hash}_{prompt_hash}' ## `vllm_mlx.vision_embedding_cache.VisionEmbeddingCache._make_image_only_key` - Kind: method - Signature: `def _make_image_only_key(self, images: List[str]) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/vision_embedding_cache.py#L194-L196 - Implementation: Method `VisionEmbeddingCache._make_image_only_key` calls `compute_images_hash`; returns `compute_images_hash(images)`. Create cache key from images only (prompt-independent). - Inputs: - `images` (List[str]; required): Required positional or keyword input. - Return annotation: `str` - Calls: compute_images_hash - Return expressions: compute_images_hash(images) ## `vllm_mlx.vision_embedding_cache.VisionEmbeddingCache.get_pixel_cache` - Kind: method - Signature: `def get_pixel_cache(self, images: List[str], prompt: str) -> Optional[PixelCacheEntry]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/vision_embedding_cache.py#L200-L229 - Implementation: Method `VisionEmbeddingCache.get_pixel_cache` updates `self.stats.pixel_cache_hits`, `self.stats.total_time_saved`, `self.stats.pixel_cache_misses`; calls `self._make_key`, `self._pixel_cache.pop`, `logger.debug`; has 2 explicit return paths. Get cached pixel values for images+prompt. Returns: PixelCacheEntry if found, None otherwise - Inputs: - `images` (List[str]; required): Required positional or keyword input. - `prompt` (str; required): Required positional or keyword input. - Return annotation: `Optional[PixelCacheEntry]` - Calls: self._make_key, self._pixel_cache.pop, logger.debug - State reads: self.enabled, self._make_key, self._pixel_cache, self._pixel_cache.pop, self.stats - State writes: self.stats.pixel_cache_hits, self.stats.total_time_saved, self.stats.pixel_cache_misses - Return expressions: None; entry ## `vllm_mlx.vision_embedding_cache.VisionEmbeddingCache.set_pixel_cache` - Kind: method - Signature: `def set_pixel_cache(self, images: List[str], prompt: str, pixel_values: mx.array, input_ids: mx.array, attention_mask: Optional[mx.array]=None, image_grid_thw: Optional[mx.array]=None, extra_kwargs: Optional[Dict[str, Any]]=None, processing_time: float=0.0) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/vision_embedding_cache.py#L231-L264 - Implementation: Method `VisionEmbeddingCache.set_pixel_cache` updates `self.stats.total_images_processed`; calls `self._make_key`, `len`, `next`, `iter`; returns `None`. Store pixel values in cache. - Inputs: - `images` (List[str]; required): Required positional or keyword input. - `prompt` (str; required): Required positional or keyword input. - `pixel_values` (mx.array; required): Required positional or keyword input. - `input_ids` (mx.array; required): Required positional or keyword input. - `attention_mask` (Optional[mx.array]; optional; default `None`): Optional positional or keyword input; defaults to `None`. - `image_grid_thw` (Optional[mx.array]; optional; default `None`): Optional positional or keyword input; defaults to `None`. - `extra_kwargs` (Optional[Dict[str, Any]]; optional; default `None`): Optional positional or keyword input; defaults to `None`. - `processing_time` (float; optional; default `0.0`): Optional positional or keyword input; defaults to `0.0`. - Return annotation: `None` - Calls: self._make_key, len, next, iter, logger.debug, PixelCacheEntry - State reads: self.enabled, self._make_key, self._pixel_cache, self.max_pixel_entries, self.stats - State writes: self.stats.total_images_processed - Return expressions: None ## `vllm_mlx.vision_embedding_cache.VisionEmbeddingCache.get_pixel_values` - Kind: method - Signature: `def get_pixel_values(self, images: List[str]) -> Optional[PixelOnlyCacheEntry]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/vision_embedding_cache.py#L268-L299 - Implementation: Method `VisionEmbeddingCache.get_pixel_values` updates `self.stats.pixel_cache_hits`, `self.stats.total_time_saved`, `self.stats.pixel_cache_misses`; calls `self._make_image_only_key`, `self._pixel_only_cache.pop`, `logger.debug`; has 2 explicit return paths. Get cached pixel values for images (prompt-independent). This is useful when the same images are used with different prompts. Only the pixel_values and image_grid_thw are cached (no input_ids). Returns: PixelOnlyCacheEntry if found, None otherwise - Inputs: - `images` (List[str]; required): Required positional or keyword input. - Return annotation: `Optional[PixelOnlyCacheEntry]` - Calls: self._make_image_only_key, self._pixel_only_cache.pop, logger.debug - State reads: self.enabled, self._make_image_only_key, self._pixel_only_cache, self._pixel_only_cache.pop, self.stats - State writes: self.stats.pixel_cache_hits, self.stats.total_time_saved, self.stats.pixel_cache_misses - Return expressions: None; entry ## `vllm_mlx.vision_embedding_cache.VisionEmbeddingCache.set_pixel_values` - Kind: method - Signature: `def set_pixel_values(self, images: List[str], pixel_values: mx.array, image_grid_thw: Optional[mx.array]=None, processing_time: float=0.0) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/vision_embedding_cache.py#L301-L326 - Implementation: Method `VisionEmbeddingCache.set_pixel_values` calls `self._make_image_only_key`, `len`, `next`, `iter`; returns `None`. Store pixel values in cache (prompt-independent). - Inputs: - `images` (List[str]; required): Required positional or keyword input. - `pixel_values` (mx.array; required): Required positional or keyword input. - `image_grid_thw` (Optional[mx.array]; optional; default `None`): Optional positional or keyword input; defaults to `None`. - `processing_time` (float; optional; default `0.0`): Optional positional or keyword input; defaults to `0.0`. - Return annotation: `None` - Calls: self._make_image_only_key, len, next, iter, logger.debug, PixelOnlyCacheEntry - State reads: self.enabled, self._make_image_only_key, self._pixel_only_cache, self.max_pixel_entries - Return expressions: None ## `vllm_mlx.vision_embedding_cache.VisionEmbeddingCache.get_encoding_cache` - Kind: method - Signature: `def get_encoding_cache(self, images: List[str], prompt: str) -> Optional[EncodingCacheEntry]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/vision_embedding_cache.py#L330-L358 - Implementation: Method `VisionEmbeddingCache.get_encoding_cache` updates `self.stats.encoding_cache_hits`, `self.stats.total_time_saved`, `self.stats.encoding_cache_misses`; calls `self._make_key`, `self._encoding_cache.pop`, `logger.debug`; has 2 explicit return paths. Get cached vision encoding output. Returns: EncodingCacheEntry if found, None otherwise - Inputs: - `images` (List[str]; required): Required positional or keyword input. - `prompt` (str; required): Required positional or keyword input. - Return annotation: `Optional[EncodingCacheEntry]` - Calls: self._make_key, self._encoding_cache.pop, logger.debug - State reads: self.enabled, self._make_key, self._encoding_cache, self._encoding_cache.pop, self.stats - State writes: self.stats.encoding_cache_hits, self.stats.total_time_saved, self.stats.encoding_cache_misses - Return expressions: None; entry ## `vllm_mlx.vision_embedding_cache.VisionEmbeddingCache.set_encoding_cache` - Kind: method - Signature: `def set_encoding_cache(self, images: List[str], prompt: str, logits: mx.array, first_token: int, logprobs: mx.array, encoding_time: float=0.0) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/vision_embedding_cache.py#L360-L388 - Implementation: Method `VisionEmbeddingCache.set_encoding_cache` calls `self._make_key`, `len`, `next`, `iter`; returns `None`. Store vision encoding output in cache. - Inputs: - `images` (List[str]; required): Required positional or keyword input. - `prompt` (str; required): Required positional or keyword input. - `logits` (mx.array; required): Required positional or keyword input. - `first_token` (int; required): Required positional or keyword input. - `logprobs` (mx.array; required): Required positional or keyword input. - `encoding_time` (float; optional; default `0.0`): Optional positional or keyword input; defaults to `0.0`. - Return annotation: `None` - Calls: self._make_key, len, next, iter, logger.debug, EncodingCacheEntry - State reads: self.enabled, self._make_key, self._encoding_cache, self.max_encoding_entries - Return expressions: None ## `vllm_mlx.vision_embedding_cache.VisionEmbeddingCache.get_stats` - Kind: method - Signature: `def get_stats(self) -> dict` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/vision_embedding_cache.py#L392-L398 - Implementation: Method `VisionEmbeddingCache.get_stats` calls `self.stats.to_dict`, `len`; returns `stats`. Get cache statistics. - Inputs: none - Return annotation: `dict` - Calls: self.stats.to_dict, len - State reads: self.stats.to_dict, self.stats, self._pixel_cache, self._pixel_only_cache, self._encoding_cache - Return expressions: stats ## `vllm_mlx.vision_embedding_cache.VisionEmbeddingCache.clear` - Kind: method - Signature: `def clear(self) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/vision_embedding_cache.py#L400-L405 - Implementation: Method `VisionEmbeddingCache.clear` updates `self.stats`; calls `self._pixel_cache.clear`, `self._pixel_only_cache.clear`, `self._encoding_cache.clear`, `VisionCacheStats`. Clear all caches and reset stats. - Inputs: none - Return annotation: `None` - Calls: self._pixel_cache.clear, self._pixel_only_cache.clear, self._encoding_cache.clear, VisionCacheStats - State reads: self._pixel_cache.clear, self._pixel_cache, self._pixel_only_cache.clear, self._pixel_only_cache, self._encoding_cache.clear, self._encoding_cache - State writes: self.stats ## `vllm_mlx.vision_embedding_cache.VisionEmbeddingCache.__repr__` - Kind: method - Signature: `def __repr__(self) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/vision_embedding_cache.py#L407-L413 - Implementation: Method `VisionEmbeddingCache.__repr__` calls `len`; returns `f' str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/vllm_platform.py#L24-L35 - Implementation: Function `_get_apple_chip_name` calls `subprocess.run`, `result.stdout.strip`; has 2 explicit return paths. Get the name of the Apple Silicon chip. - Inputs: none - Return annotation: `str` - Calls: subprocess.run, result.stdout.strip - Return expressions: result.stdout.strip(); 'Apple Silicon' ## `vllm_mlx.vllm_platform._get_unified_memory_size` - Kind: function - Signature: `def _get_unified_memory_size() -> int` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/vllm_platform.py#L38-L50 - Implementation: Function `_get_unified_memory_size` calls `subprocess.run`, `int`, `result.stdout.strip`; has 2 explicit return paths. Get the total unified memory size in bytes. - Inputs: none - Return annotation: `int` - Calls: subprocess.run, int, result.stdout.strip - Return expressions: int(result.stdout.strip()); 8 * 1024 * 1024 * 1024 ## `vllm_mlx.vllm_platform._is_mlx_available` - Kind: function - Signature: `def _is_mlx_available() -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/vllm_platform.py#L53-L63 - Implementation: Function `_is_mlx_available` calls `mx.array`, `logger.debug`; has 2 explicit return paths. Check if MLX is available and working. - Inputs: none - Return annotation: `bool` - Calls: mx.array, logger.debug - Return expressions: True; False ## `vllm_mlx.vllm_platform._is_apple_silicon` - Kind: function - Signature: `def _is_apple_silicon() -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/vllm_platform.py#L66-L68 - Implementation: Function `_is_apple_silicon` calls `platform.machine`; returns `sys.platform == 'darwin' and platform.machine() == 'arm64'`. Check if running on Apple Silicon. - Inputs: none - Return annotation: `bool` - Calls: platform.machine - Return expressions: sys.platform == 'darwin' and platform.machine() == 'arm64' ## `vllm_mlx.vllm_platform.MLXPlatform` - Kind: class - Signature: `class MLXPlatform` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/vllm_platform.py#L71-L351 - Implementation: Class `MLXPlatform` declares 30 direct member(s). Platform implementation for Apple Silicon using MLX. This platform uses Apple's MLX framework for GPU-accelerated inference on Apple Silicon Macs. It integrates with mlx-lm for LLM inference and mlx-vlm for vision-language models. Key features: - Unified memory model (no CPU<->GPU transfers) - Native Metal GPU acceleration - Optimized kernels for Apple Silicon - Support for quantized models (4-bit, 8-bit) - Inputs: none - Constructs: `vllm_mlx.vllm_platform.MLXPlatform` ## `vllm_mlx.vllm_platform.MLXPlatform._enum` - Kind: method - Signature: `def _enum(self)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/vllm_platform.py#L90-L93 - Implementation: Method `MLXPlatform._enum` returns `PlatformEnum.OOT`. Method `MLXPlatform._enum` returns `PlatformEnum.OOT`. - Inputs: none - Return annotation: `not annotated` - Decorators: property - Return expressions: PlatformEnum.OOT ## `vllm_mlx.vllm_platform.MLXPlatform.supported_dtypes` - Kind: method - Signature: `def supported_dtypes(self) -> list[torch.dtype]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/vllm_platform.py#L122-L133 - Implementation: Method `MLXPlatform.supported_dtypes` calls `mx.array`; has 2 explicit return paths. Return supported dtypes for MLX. - Inputs: none - Return annotation: `list[torch.dtype]` - Decorators: property - Calls: mx.array - Return expressions: [torch.bfloat16, torch.float16, torch.float32]; [torch.float16, torch.float32] ## `vllm_mlx.vllm_platform.MLXPlatform.is_cuda` - Kind: method - Signature: `def is_cuda(self) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/vllm_platform.py#L135-L138 - Implementation: Method `MLXPlatform.is_cuda` returns `False`. Return ``False`` because this platform does not use CUDA. - Inputs: none - Return annotation: `bool` - Return expressions: False ## `vllm_mlx.vllm_platform.MLXPlatform.is_rocm` - Kind: method - Signature: `def is_rocm(self) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/vllm_platform.py#L140-L143 - Implementation: Method `MLXPlatform.is_rocm` returns `False`. Return ``False`` because this platform does not use ROCm. - Inputs: none - Return annotation: `bool` - Return expressions: False ## `vllm_mlx.vllm_platform.MLXPlatform.is_tpu` - Kind: method - Signature: `def is_tpu(self) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/vllm_platform.py#L145-L148 - Implementation: Method `MLXPlatform.is_tpu` returns `False`. Return ``False`` because this platform is not a TPU backend. - Inputs: none - Return annotation: `bool` - Return expressions: False ## `vllm_mlx.vllm_platform.MLXPlatform.is_xpu` - Kind: method - Signature: `def is_xpu(self) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/vllm_platform.py#L150-L153 - Implementation: Method `MLXPlatform.is_xpu` returns `False`. Return ``False`` because this platform does not use Intel XPU. - Inputs: none - Return annotation: `bool` - Return expressions: False ## `vllm_mlx.vllm_platform.MLXPlatform.is_cpu` - Kind: method - Signature: `def is_cpu(self) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/vllm_platform.py#L155-L158 - Implementation: Method `MLXPlatform.is_cpu` returns `False`. Return ``False`` because MLX targets Apple GPU acceleration here. - Inputs: none - Return annotation: `bool` - Return expressions: False ## `vllm_mlx.vllm_platform.MLXPlatform.is_mlx` - Kind: method - Signature: `def is_mlx(self) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/vllm_platform.py#L160-L163 - Implementation: Method `MLXPlatform.is_mlx` returns `True`. Return ``True`` to identify the MLX platform plugin. - Inputs: none - Return annotation: `bool` - Return expressions: True ## `vllm_mlx.vllm_platform.MLXPlatform.is_out_of_tree` - Kind: method - Signature: `def is_out_of_tree(self) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/vllm_platform.py#L165-L168 - Implementation: Method `MLXPlatform.is_out_of_tree` returns `True`. Return ``True`` because MLX is registered as a vLLM plugin. - Inputs: none - Return annotation: `bool` - Return expressions: True ## `vllm_mlx.vllm_platform.MLXPlatform.is_cuda_alike` - Kind: method - Signature: `def is_cuda_alike(self) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/vllm_platform.py#L170-L173 - Implementation: Method `MLXPlatform.is_cuda_alike` returns `False`. Return ``False`` because MLX does not implement CUDA semantics. - Inputs: none - Return annotation: `bool` - Return expressions: False ## `vllm_mlx.vllm_platform.MLXPlatform.is_sleep_mode_available` - Kind: method - Signature: `def is_sleep_mode_available(self) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/vllm_platform.py#L175-L178 - Implementation: Method `MLXPlatform.is_sleep_mode_available` returns `False`. Return ``False`` because vLLM sleep mode is unavailable on MLX. - Inputs: none - Return annotation: `bool` - Return expressions: False ## `vllm_mlx.vllm_platform.MLXPlatform.get_device_name` - Kind: method - Signature: `def get_device_name(cls, device_id: int=0) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/vllm_platform.py#L181-L183 - Implementation: Method `MLXPlatform.get_device_name` calls `_get_apple_chip_name`; returns `_get_apple_chip_name()`. Get the Apple Silicon chip name. - Inputs: - `device_id` (int; optional; default `0`): Optional positional or keyword input; defaults to `0`. - Return annotation: `str` - Decorators: classmethod - Calls: _get_apple_chip_name - Return expressions: _get_apple_chip_name() ## `vllm_mlx.vllm_platform.MLXPlatform.get_device_uuid` - Kind: method - Signature: `def get_device_uuid(cls, device_id: int=0) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/vllm_platform.py#L186-L188 - Implementation: Method `MLXPlatform.get_device_uuid` returns `'mlx-0'`. Get device UUID (not applicable for MLX). - Inputs: - `device_id` (int; optional; default `0`): Optional positional or keyword input; defaults to `0`. - Return annotation: `str` - Decorators: classmethod - Return expressions: 'mlx-0' ## `vllm_mlx.vllm_platform.MLXPlatform.get_device_total_memory` - Kind: method - Signature: `def get_device_total_memory(cls, device_id: int=0) -> int` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/vllm_platform.py#L191-L193 - Implementation: Method `MLXPlatform.get_device_total_memory` calls `_get_unified_memory_size`; returns `_get_unified_memory_size()`. Get total unified memory in bytes. - Inputs: - `device_id` (int; optional; default `0`): Optional positional or keyword input; defaults to `0`. - Return annotation: `int` - Decorators: classmethod - Calls: _get_unified_memory_size - Return expressions: _get_unified_memory_size() ## `vllm_mlx.vllm_platform.MLXPlatform.inference_mode` - Kind: method - Signature: `def inference_mode(cls)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/vllm_platform.py#L196-L200 - Implementation: Method `MLXPlatform.inference_mode` calls `torch.no_grad`; returns `torch.no_grad()`. Return inference mode context manager. - Inputs: none - Return annotation: `not annotated` - Decorators: classmethod - Calls: torch.no_grad - Return expressions: torch.no_grad() ## `vllm_mlx.vllm_platform.MLXPlatform.set_device` - Kind: method - Signature: `def set_device(cls, device: torch.device) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/vllm_platform.py#L203-L206 - Implementation: Method `MLXPlatform.set_device` contains no state mutation, call, raise, return, await, or yield. Set the device (no-op for MLX, uses default device). - Inputs: - `device` (torch.device; required): Required positional or keyword input. - Return annotation: `None` - Decorators: classmethod ## `vllm_mlx.vllm_platform.MLXPlatform.seed_everything` - Kind: method - Signature: `def seed_everything(cls, seed: int | None=None) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/vllm_platform.py#L209-L225 - Implementation: Method `MLXPlatform.seed_everything` calls `random.seed`, `np.random.seed`, `torch.manual_seed`, `mx.random.seed`. Set random seeds for reproducibility. - Inputs: - `seed` (int | None; optional; default `None`): Optional positional or keyword input; defaults to `None`. - Return annotation: `None` - Decorators: classmethod - Calls: random.seed, np.random.seed, torch.manual_seed, mx.random.seed ## `vllm_mlx.vllm_platform.MLXPlatform.import_kernels` - Kind: method - Signature: `def import_kernels(cls) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/vllm_platform.py#L228-L231 - Implementation: Method `MLXPlatform.import_kernels` contains no state mutation, call, raise, return, await, or yield. Import MLX kernels (no custom C kernels). - Inputs: none - Return annotation: `None` - Decorators: classmethod ## `vllm_mlx.vllm_platform.MLXPlatform.get_attn_backend_cls` - Kind: method - Signature: `def get_attn_backend_cls(cls, selected_backend, head_size: int, dtype: torch.dtype, kv_cache_dtype, block_size: int, use_mla: bool, has_sink: bool, use_sparse: bool, attn_type: str | None=None) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/vllm_platform.py#L234-L248 - Implementation: Method `MLXPlatform.get_attn_backend_cls` returns `'vllm_mlx.attention.MLXAttentionBackend'`. Return MLX attention backend class path. - Inputs: - `selected_backend` (not annotated; required): Required positional or keyword input. - `head_size` (int; required): Required positional or keyword input. - `dtype` (torch.dtype; required): Required positional or keyword input. - `kv_cache_dtype` (not annotated; required): Required positional or keyword input. - `block_size` (int; required): Required positional or keyword input. - `use_mla` (bool; required): Required positional or keyword input. - `has_sink` (bool; required): Required positional or keyword input. - `use_sparse` (bool; required): Required positional or keyword input. - `attn_type` (str | None; optional; default `None`): Optional positional or keyword input; defaults to `None`. - Return annotation: `str` - Decorators: classmethod - Return expressions: 'vllm_mlx.attention.MLXAttentionBackend' ## `vllm_mlx.vllm_platform.MLXPlatform.check_and_update_config` - Kind: method - Signature: `def check_and_update_config(cls, vllm_config: 'VllmConfig') -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/vllm_platform.py#L251-L280 - Implementation: Method `MLXPlatform.check_and_update_config` calls `logger.info`, `_get_apple_chip_name`, `_get_unified_memory_size`, `hasattr`. Check and update vLLM configuration for MLX. - Inputs: - `vllm_config` ('VllmConfig'; required): Required positional or keyword input. - Return annotation: `None` - Decorators: classmethod - Calls: logger.info, _get_apple_chip_name, _get_unified_memory_size, hasattr, logger.warning ## `vllm_mlx.vllm_platform.MLXPlatform.verify_model_arch` - Kind: method - Signature: `def verify_model_arch(cls, model_arch: str) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/vllm_platform.py#L283-L294 - Implementation: Method `MLXPlatform.verify_model_arch` calls `hint.lower`, `model_arch.lower`, `logger.warning`. Verify model architecture is supported on MLX. - Inputs: - `model_arch` (str; required): Required positional or keyword input. - Return annotation: `None` - Decorators: classmethod - Calls: hint.lower, model_arch.lower, logger.warning ## `vllm_mlx.vllm_platform.MLXPlatform.verify_quantization` - Kind: method - Signature: `def verify_quantization(cls, quant: str) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/vllm_platform.py#L297-L304 - Implementation: Method `MLXPlatform.verify_quantization` calls `ValueError`; can raise `ValueError`. Verify quantization method is supported. - Inputs: - `quant` (str; required): Required positional or keyword input. - Return annotation: `None` - Decorators: classmethod - Calls: ValueError - Raises directly: ValueError ## `vllm_mlx.vllm_platform.MLXPlatform.is_pin_memory_available` - Kind: method - Signature: `def is_pin_memory_available(cls) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/vllm_platform.py#L307-L309 - Implementation: Method `MLXPlatform.is_pin_memory_available` returns `False`. Pin memory not needed with unified memory. - Inputs: none - Return annotation: `bool` - Decorators: classmethod - Return expressions: False ## `vllm_mlx.vllm_platform.MLXPlatform.get_current_memory_usage` - Kind: method - Signature: `def get_current_memory_usage(cls, device=None) -> float` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/vllm_platform.py#L312-L323 - Implementation: Method `MLXPlatform.get_current_memory_usage` calls `psutil.Process`, `float`, `process.memory_info`; has 2 explicit return paths. Get current memory usage in bytes. - Inputs: - `device` (not annotated; optional; default `None`): Optional positional or keyword input; defaults to `None`. - Return annotation: `float` - Decorators: classmethod - Calls: psutil.Process, float, process.memory_info - Return expressions: float(process.memory_info().rss); 0.0 ## `vllm_mlx.vllm_platform.MLXPlatform.supports_fp8` - Kind: method - Signature: `def supports_fp8(cls) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/vllm_platform.py#L326-L328 - Implementation: Method `MLXPlatform.supports_fp8` returns `False`. FP8 not supported on MLX. - Inputs: none - Return annotation: `bool` - Decorators: classmethod - Return expressions: False ## `vllm_mlx.vllm_platform.MLXPlatform.use_custom_allreduce` - Kind: method - Signature: `def use_custom_allreduce(cls) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/vllm_platform.py#L331-L333 - Implementation: Method `MLXPlatform.use_custom_allreduce` returns `False`. Custom allreduce not available. - Inputs: none - Return annotation: `bool` - Decorators: classmethod - Return expressions: False ## `vllm_mlx.vllm_platform.MLXPlatform.support_static_graph_mode` - Kind: method - Signature: `def support_static_graph_mode(cls) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/vllm_platform.py#L336-L338 - Implementation: Method `MLXPlatform.support_static_graph_mode` returns `False`. Static graph mode (CUDA graphs) not supported. - Inputs: none - Return annotation: `bool` - Decorators: classmethod - Return expressions: False ## `vllm_mlx.vllm_platform.MLXPlatform.get_device_communicator_cls` - Kind: method - Signature: `def get_device_communicator_cls(cls) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/vllm_platform.py#L341-L343 - Implementation: Method `MLXPlatform.get_device_communicator_cls` returns `'vllm_mlx.distributed.MLXCommunicator'`. Return the communicator class for distributed. - Inputs: none - Return annotation: `str` - Decorators: classmethod - Return expressions: 'vllm_mlx.distributed.MLXCommunicator' ## `vllm_mlx.vllm_platform.MLXPlatform.get_punica_wrapper` - Kind: method - Signature: `def get_punica_wrapper(cls) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/vllm_platform.py#L346-L348 - Implementation: Method `MLXPlatform.get_punica_wrapper` calls `NotImplementedError`; can raise `NotImplementedError`. Return LoRA wrapper (not yet implemented for MLX). - Inputs: none - Return annotation: `str` - Decorators: classmethod - Calls: NotImplementedError - Raises directly: NotImplementedError ## `vllm_mlx.vllm_platform.MLXPlatform.__repr__` - Kind: method - Signature: `def __repr__(self) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/vllm_platform.py#L350-L351 - Implementation: Method `MLXPlatform.__repr__` returns `f''`. Method `MLXPlatform.__repr__` returns `f''`. - Inputs: none - Return annotation: `str` - State reads: self.device_name - Return expressions: f'' # Module `vllm_mlx.worker` MLX Worker for vLLM. This module implements a vLLM worker that uses Apple's MLX framework for model execution on Apple Silicon. Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/worker.py#L1-L278 ## `vllm_mlx.worker.MLXWorker` - Kind: class - Signature: `class MLXWorker` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/worker.py#L23-L278 - Implementation: Class `MLXWorker` declares 21 direct member(s). Worker implementation for MLX-based inference on Apple Silicon. This worker uses mlx-lm for model loading and inference, providing native Apple Silicon GPU acceleration through Metal. Unlike CUDA workers that use PyTorch with CUDA, this worker: - Uses MLX arrays instead of PyTorch tensors for model weights - Leverages unified memory (no CPU<->GPU transfers needed) - Uses Metal-optimized kernels for attention and other operations - Inputs: - `vllm_config` ('VllmConfig'; required): Complete vLLM configuration - `local_rank` (int; required): Local device index (usually 0 for single GPU) - `rank` (int; required): Global rank in distributed setup - `distributed_init_method` (str; required): Distributed initialization method - `is_driver_worker` (bool; optional; default `False`): Whether this worker handles driver responsibilities - Constructs: `vllm_mlx.worker.MLXWorker` ## `vllm_mlx.worker.MLXWorker.__init__` - Kind: method - Signature: `def __init__(self, vllm_config: 'VllmConfig', local_rank: int, rank: int, distributed_init_method: str, is_driver_worker: bool=False) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/worker.py#L36-L75 - Implementation: Method `MLXWorker.__init__` updates `self.vllm_config`, `self.model_config`, `self.cache_config`, `self.parallel_config`; calls `torch.device`, `logger.info`. Initialize MLX worker. Args: vllm_config: Complete vLLM configuration local_rank: Local device index (usually 0 for single GPU) rank: Global rank in distributed setup distributed_init_method: Distributed initialization method is_driver_worker: Whether this worker handles driver responsibilities - Inputs: - `vllm_config` ('VllmConfig'; required): Complete vLLM configuration - `local_rank` (int; required): Local device index (usually 0 for single GPU) - `rank` (int; required): Global rank in distributed setup - `distributed_init_method` (str; required): Distributed initialization method - `is_driver_worker` (bool; optional; default `False`): Whether this worker handles driver responsibilities - Return annotation: `None` - Calls: torch.device, logger.info - State writes: self.vllm_config, self.model_config, self.cache_config, self.parallel_config, self.scheduler_config, self.device_config, self.load_config, self.local_rank, self.rank, self.distributed_init_method, self.is_driver_worker, self.model, self.tokenizer, self.model_runner, self.device ## `vllm_mlx.worker.MLXWorker.init_device` - Kind: method - Signature: `def init_device(self) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/worker.py#L77-L103 - Implementation: Method `MLXWorker.init_device` updates `self.model_runner`; calls `mx.default_device`, `logger.info`, `get_mlx_device_info`, `MLXModelRunner`; can raise `ImportError`. Initialize MLX device and verify it's working. - Inputs: none - Return annotation: `None` - Calls: mx.default_device, logger.info, get_mlx_device_info, MLXModelRunner, ImportError - State reads: self.vllm_config - State writes: self.model_runner - Raises directly: ImportError ## `vllm_mlx.worker.MLXWorker.load_model` - Kind: method - Signature: `def load_model(self) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/worker.py#L105-L111 - Implementation: Method `MLXWorker.load_model` calls `RuntimeError`, `self.model_runner.load_model`, `logger.info`; can raise `RuntimeError`. Load model using mlx-lm. - Inputs: none - Return annotation: `None` - Calls: RuntimeError, self.model_runner.load_model, logger.info - State reads: self.model_runner, self.model_runner.load_model, self.model_config.model, self.model_config - Raises directly: RuntimeError ## `vllm_mlx.worker.MLXWorker.determine_available_memory` - Kind: method - Signature: `def determine_available_memory(self) -> int` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/worker.py#L113-L143 - Implementation: Method `MLXWorker.determine_available_memory` calls `subprocess.run`, `int`, `result.stdout.strip`, `logger.info`; has 2 explicit return paths. Determine available memory for KV cache. On Apple Silicon with unified memory, we use a portion of system RAM. - Inputs: none - Return annotation: `int` - Calls: subprocess.run, int, result.stdout.strip, logger.info, logger.warning - State reads: self.cache_config.gpu_memory_utilization, self.cache_config - Return expressions: available; 4 * 1024 * 1024 * 1024 ## `vllm_mlx.worker.MLXWorker.initialize_cache` - Kind: method - Signature: `def initialize_cache(self, num_gpu_blocks: int, num_cpu_blocks: int) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/worker.py#L145-L153 - Implementation: Method `MLXWorker.initialize_cache` updates `self.cache_config.num_gpu_blocks`, `self.cache_config.num_cpu_blocks`; calls `self.model_runner.initialize_cache`, `logger.info`. Initialize KV cache with the given size. - Inputs: - `num_gpu_blocks` (int; required): Required positional or keyword input. - `num_cpu_blocks` (int; required): Required positional or keyword input. - Return annotation: `None` - Calls: self.model_runner.initialize_cache, logger.info - State reads: self.cache_config, self.model_runner, self.model_runner.initialize_cache - State writes: self.cache_config.num_gpu_blocks, self.cache_config.num_cpu_blocks ## `vllm_mlx.worker.MLXWorker.get_kv_cache_spec` - Kind: method - Signature: `def get_kv_cache_spec(self) -> dict` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/worker.py#L155-L159 - Implementation: Method `MLXWorker.get_kv_cache_spec` calls `self.model_runner.get_kv_cache_spec`; has 2 explicit return paths. Get KV cache specification. - Inputs: none - Return annotation: `dict` - Calls: self.model_runner.get_kv_cache_spec - State reads: self.model_runner, self.model_runner.get_kv_cache_spec - Return expressions: self.model_runner.get_kv_cache_spec(); {} ## `vllm_mlx.worker.MLXWorker.compile_or_warm_up_model` - Kind: method - Signature: `def compile_or_warm_up_model(self) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/worker.py#L161-L165 - Implementation: Method `MLXWorker.compile_or_warm_up_model` calls `self.model_runner.warm_up`, `logger.info`. Warm up model for inference. - Inputs: none - Return annotation: `None` - Calls: self.model_runner.warm_up, logger.info - State reads: self.model_runner, self.model_runner.warm_up ## `vllm_mlx.worker.MLXWorker.execute_model` - Kind: method - Signature: `def execute_model(self, scheduler_output: 'SchedulerOutput') -> 'ModelRunnerOutput | None'` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/worker.py#L167-L183 - Implementation: Method `MLXWorker.execute_model` calls `RuntimeError`, `self.model_runner.execute_model`; can raise `RuntimeError`; returns `self.model_runner.execute_model(scheduler_output)`. Execute model inference for the given scheduler output. Args: scheduler_output: Contains requests to process Returns: ModelRunnerOutput with generation results - Inputs: - `scheduler_output` ('SchedulerOutput'; required): Contains requests to process - Return annotation: `'ModelRunnerOutput | None'` - Calls: RuntimeError, self.model_runner.execute_model - State reads: self.model_runner, self.model_runner.execute_model - Raises directly: RuntimeError - Return expressions: self.model_runner.execute_model(scheduler_output) ## `vllm_mlx.worker.MLXWorker.get_model` - Kind: method - Signature: `def get_model(self)` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/worker.py#L185-L189 - Implementation: Method `MLXWorker.get_model` has 2 explicit return paths. Get the underlying model. - Inputs: none - Return annotation: `not annotated` - State reads: self.model_runner, self.model_runner.model - Return expressions: self.model_runner.model; None ## `vllm_mlx.worker.MLXWorker.check_health` - Kind: method - Signature: `def check_health(self) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/worker.py#L191-L200 - Implementation: Method `MLXWorker.check_health` calls `mx.array`, `mx.sum(test).item`, `mx.sum`, `RuntimeError`; can raise `RuntimeError`. Check worker health. - Inputs: none - Return annotation: `None` - Calls: mx.array, mx.sum(test).item, mx.sum, RuntimeError - Raises directly: RuntimeError ## `vllm_mlx.worker.MLXWorker.shutdown` - Kind: method - Signature: `def shutdown(self) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/worker.py#L202-L219 - Implementation: Method `MLXWorker.shutdown` updates `self.model`, `self.tokenizer`, `self.model_runner`; calls `logger.info`, `mx.clear_cache`, `gc.collect`. Clean up resources. - Inputs: none - Return annotation: `None` - Calls: logger.info, mx.clear_cache, gc.collect - State writes: self.model, self.tokenizer, self.model_runner ## `vllm_mlx.worker.MLXWorker.add_lora` - Kind: method - Signature: `def add_lora(self, lora_request) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/worker.py#L222-L226 - Implementation: Method `MLXWorker.add_lora` calls `logger.warning`; returns `False`. Report that dynamically adding a LoRA adapter is unsupported. - Inputs: - `lora_request` (not annotated; required): Required positional or keyword input. - Return annotation: `bool` - Calls: logger.warning - Return expressions: False ## `vllm_mlx.worker.MLXWorker.remove_lora` - Kind: method - Signature: `def remove_lora(self, lora_id: int) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/worker.py#L228-L231 - Implementation: Method `MLXWorker.remove_lora` returns `False`. Report that dynamically removing a LoRA adapter is unsupported. - Inputs: - `lora_id` (int; required): Required positional or keyword input. - Return annotation: `bool` - Return expressions: False ## `vllm_mlx.worker.MLXWorker.pin_lora` - Kind: method - Signature: `def pin_lora(self, lora_id: int) -> bool` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/worker.py#L233-L236 - Implementation: Method `MLXWorker.pin_lora` returns `False`. Report that pinning a LoRA adapter is unsupported. - Inputs: - `lora_id` (int; required): Required positional or keyword input. - Return annotation: `bool` - Return expressions: False ## `vllm_mlx.worker.MLXWorker.list_loras` - Kind: method - Signature: `def list_loras(self) -> set[int]` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/worker.py#L238-L241 - Implementation: Method `MLXWorker.list_loras` calls `set`; returns `set()`. Return the empty set because runtime LoRA adapters are unsupported. - Inputs: none - Return annotation: `set[int]` - Calls: set - Return expressions: set() ## `vllm_mlx.worker.MLXWorker.sleep` - Kind: method - Signature: `def sleep(self, level: int=1) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/worker.py#L244-L247 - Implementation: Method `MLXWorker.sleep` calls `logger.debug`. Leave the worker active because MLX unified memory has no sleep mode. - Inputs: - `level` (int; optional; default `1`): Optional positional or keyword input; defaults to `1`. - Return annotation: `None` - Calls: logger.debug ## `vllm_mlx.worker.MLXWorker.wake_up` - Kind: method - Signature: `def wake_up(self, tags: list[str] | None=None) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/worker.py#L249-L252 - Implementation: Method `MLXWorker.wake_up` calls `logger.debug`. Perform no work because the MLX worker never enters sleep mode. - Inputs: - `tags` (list[str] | None; optional; default `None`): Optional positional or keyword input; defaults to `None`. - Return annotation: `None` - Calls: logger.debug ## `vllm_mlx.worker.MLXWorker.vocab_size` - Kind: method - Signature: `def vocab_size(self) -> int` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/worker.py#L255-L257 - Implementation: Method `MLXWorker.vocab_size` calls `self.model_config.get_vocab_size`; returns `self.model_config.get_vocab_size()`. Get vocabulary size. - Inputs: none - Return annotation: `int` - Decorators: property - Calls: self.model_config.get_vocab_size - State reads: self.model_config.get_vocab_size, self.model_config - Return expressions: self.model_config.get_vocab_size() ## `vllm_mlx.worker.MLXWorker.get_cache_block_size_bytes` - Kind: method - Signature: `def get_cache_block_size_bytes(self) -> int` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/worker.py#L259-L271 - Implementation: Method `MLXWorker.get_cache_block_size_bytes` calls `self.model_runner.get_cache_block_size_bytes`, `self.model_config.get_head_size`, `self.model_config.get_num_kv_heads`, `self.model_config.get_num_layers`; has 2 explicit return paths. Get size of a cache block in bytes. - Inputs: none - Return annotation: `int` - Calls: self.model_runner.get_cache_block_size_bytes, self.model_config.get_head_size, self.model_config.get_num_kv_heads, self.model_config.get_num_layers - State reads: self.model_runner, self.model_runner.get_cache_block_size_bytes, self.model_config.get_head_size, self.model_config, self.model_config.get_num_kv_heads, self.parallel_config, self.model_config.get_num_layers, self.cache_config.block_size, self.cache_config - Return expressions: self.model_runner.get_cache_block_size_bytes(); 2 * block_size * num_layers * num_heads * head_size * 2 ## `vllm_mlx.worker.MLXWorker.profile` - Kind: method - Signature: `def profile(self, is_start: bool=True) -> None` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/worker.py#L273-L275 - Implementation: Method `MLXWorker.profile` calls `logger.debug`. Profiling (not yet implemented for MLX). - Inputs: - `is_start` (bool; optional; default `True`): Optional positional or keyword input; defaults to `True`. - Return annotation: `None` - Calls: logger.debug ## `vllm_mlx.worker.MLXWorker.__repr__` - Kind: method - Signature: `def __repr__(self) -> str` - Source: https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/worker.py#L277-L278 - Implementation: Method `MLXWorker.__repr__` returns `f''`. Method `MLXWorker.__repr__` returns `f''`. - Inputs: none - Return annotation: `str` - State reads: self.rank, self.local_rank - Return expressions: f'' # Complete CLI option inventory This page is generated from every `add_argument` declaration in the runtime, maintenance scripts, and runnable examples. The hand-written [CLI guide](cli.md) explains supported workflows. ## `examples.audio_separation_example.main` ### Parser `parser` #### `audio` Input audio file (mp3, wav, etc.) - Destination: `audio` - Required: `true` - Default: `argparse default` - Choices: `not restricted` - Action: `store` - Source: [L37-L37](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/audio_separation_example.py#L37-L37) #### `--description`, `-d` What to isolate: speech, music, singing, etc. (default: speech) - Destination: `description` - Required: `false` - Default: `speech` - Choices: `not restricted` - Action: `store` - Source: [L38-L39](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/audio_separation_example.py#L38-L39) #### `--output`, `-o` Output file for isolated audio (default: input_voice.wav) - Destination: `output` - Required: `false` - Default: `None` - Choices: `not restricted` - Action: `store` - Source: [L40-L41](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/audio_separation_example.py#L40-L41) #### `--background`, `-b` Output file for background audio (optional) - Destination: `background` - Required: `false` - Default: `None` - Choices: `not restricted` - Action: `store` - Source: [L42-L43](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/audio_separation_example.py#L42-L43) #### `--model`, `-m` SAM-Audio model to use - Destination: `model` - Required: `false` - Default: `mlx-community/sam-audio-large-fp16` - Choices: `not restricted` - Action: `store` - Source: [L44-L45](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/audio_separation_example.py#L44-L45) #### `--chunk`, `-c` Process in chunks of N seconds (for long audio) - Destination: `chunk` - Required: `false` - Default: `None` - Choices: `not restricted` - Action: `store` - Source: [L46-L47](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/audio_separation_example.py#L46-L47) #### `--play`, `-p` Play result after processing (macOS) - Destination: `play` - Required: `false` - Default: `argparse default` - Choices: `not restricted` - Action: `store_true` - Source: [L48-L49](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/audio_separation_example.py#L48-L49) ## `examples.benchmark_audio.main` ### Parser `parser` #### `--tts` Run TTS benchmarks - Destination: `tts` - Required: `false` - Default: `argparse default` - Choices: `not restricted` - Action: `store_true` - Source: [L302-L302](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/benchmark_audio.py#L302-L302) #### `--stt` Run STT benchmarks - Destination: `stt` - Required: `false` - Default: `argparse default` - Choices: `not restricted` - Action: `store_true` - Source: [L303-L303](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/benchmark_audio.py#L303-L303) #### `--audio` Audio file for STT benchmark - Destination: `audio` - Required: `false` - Default: `argparse default` - Choices: `not restricted` - Action: `store` - Source: [L304-L304](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/benchmark_audio.py#L304-L304) #### `--all` Run all benchmarks - Destination: `all` - Required: `false` - Default: `argparse default` - Choices: `not restricted` - Action: `store_true` - Source: [L305-L305](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/benchmark_audio.py#L305-L305) ## `examples.closed_captions.main` ### Parser `parser` #### `--model`, `-m` No argparse help text is declared. - Destination: `model` - Required: `false` - Default: `whisper-large-v3` - Choices: `not restricted` - Action: `store` - Source: [L150-L150](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/closed_captions.py#L150-L150) #### `--language`, `-l` es, en, etc. - Destination: `language` - Required: `false` - Default: `None` - Choices: `not restricted` - Action: `store` - Source: [L151-L151](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/closed_captions.py#L151-L151) #### `--chunk`, `-c` Chunk size (default: 3.0s) - Destination: `chunk` - Required: `false` - Default: `3.0` - Choices: `not restricted` - Action: `store` - Source: [L152-L152](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/closed_captions.py#L152-L152) ## `examples.mic_live.main` ### Parser `parser` #### `--model`, `-m` Model (whisper-small, whisper-medium, parakeet) - Destination: `model` - Required: `false` - Default: `whisper-small` - Choices: `not restricted` - Action: `store` - Source: [L206-L207](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/mic_live.py#L206-L207) #### `--language`, `-l` Language code (en, es, etc.) - Destination: `language` - Required: `false` - Default: `argparse default` - Choices: `not restricted` - Action: `store` - Source: [L208-L208](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/mic_live.py#L208-L208) #### `--sensitivity`, `-s` Mic sensitivity 0.01-0.05 (default: 0.015) - Destination: `sensitivity` - Required: `false` - Default: `0.015` - Choices: `not restricted` - Action: `store` - Source: [L209-L210](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/mic_live.py#L209-L210) ## `examples.mic_realtime.main` ### Parser `parser` #### `--model`, `-m` Model to use (default: whisper-small) - Destination: `model` - Required: `false` - Default: `whisper-small` - Choices: `not restricted` - Action: `store` - Source: [L186-L187](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/mic_realtime.py#L186-L187) #### `--chunk`, `-c` Chunk duration in seconds (default: 3.0) - Destination: `chunk` - Required: `false` - Default: `3.0` - Choices: `not restricted` - Action: `store` - Source: [L188-L189](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/mic_realtime.py#L188-L189) #### `--language`, `-l` Language code (e.g., en, es) - Destination: `language` - Required: `false` - Default: `argparse default` - Choices: `not restricted` - Action: `store` - Source: [L190-L190](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/mic_realtime.py#L190-L190) #### `--list-models` List available models - Destination: `list_models` - Required: `false` - Default: `argparse default` - Choices: `not restricted` - Action: `store_true` - Source: [L191-L191](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/mic_realtime.py#L191-L191) ## `examples.mic_transcribe.main` ### Parser `parser` #### `--duration`, `-d` Recording duration in seconds - Destination: `duration` - Required: `false` - Default: `argparse default` - Choices: `not restricted` - Action: `store` - Source: [L115-L115](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/mic_transcribe.py#L115-L115) #### `--model`, `-m` Model: whisper-small, whisper-medium, whisper-large-v3, parakeet - Destination: `model` - Required: `false` - Default: `whisper-small` - Choices: `not restricted` - Action: `store` - Source: [L116-L117](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/mic_transcribe.py#L116-L117) #### `--language`, `-l` Language code (e.g., en, es). Auto-detect if not set - Destination: `language` - Required: `false` - Default: `argparse default` - Choices: `not restricted` - Action: `store` - Source: [L118-L118](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/mic_transcribe.py#L118-L118) #### `--continuous`, `-c` Continuous mode: keep recording and transcribing - Destination: `continuous` - Required: `false` - Default: `argparse default` - Choices: `not restricted` - Action: `store_true` - Source: [L119-L120](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/mic_transcribe.py#L119-L120) #### `--save`, `-s` Save recorded audio to this file - Destination: `save` - Required: `false` - Default: `argparse default` - Choices: `not restricted` - Action: `store` - Source: [L121-L121](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/mic_transcribe.py#L121-L121) #### `--list-models` List available models - Destination: `list_models` - Required: `false` - Default: `argparse default` - Choices: `not restricted` - Action: `store_true` - Source: [L122-L122](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/mic_transcribe.py#L122-L122) #### `--list-devices` List audio input devices - Destination: `list_devices` - Required: `false` - Default: `argparse default` - Choices: `not restricted` - Action: `store_true` - Source: [L123-L123](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/mic_transcribe.py#L123-L123) ## `examples.mllm_benchmark.main` ### Parser `parser` #### `--server-url` URL of the vllm-mlx server - Destination: `server_url` - Required: `false` - Default: `http://localhost:8000` - Choices: `not restricted` - Action: `store` - Source: [L390-L395](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/mllm_benchmark.py#L390-L395) #### `--output` Save results to JSON file - Destination: `output` - Required: `false` - Default: `None` - Choices: `not restricted` - Action: `store` - Source: [L396-L401](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/mllm_benchmark.py#L396-L401) #### `--warmup` Number of warmup runs (default: 1) - Destination: `warmup` - Required: `false` - Default: `1` - Choices: `not restricted` - Action: `store` - Source: [L402-L407](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/mllm_benchmark.py#L402-L407) #### `--quick` Run quick benchmark with fewer resolutions - Destination: `quick` - Required: `false` - Default: `argparse default` - Choices: `not restricted` - Action: `store_true` - Source: [L408-L412](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/mllm_benchmark.py#L408-L412) ## `examples.test_batching.main` ### Parser `parser` #### `--model` Model to use - Destination: `model` - Required: `false` - Default: `mlx-community/Llama-3.2-1B-Instruct-4bit` - Choices: `not restricted` - Action: `store` - Source: [L118-L123](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/test_batching.py#L118-L123) #### `--num-requests` Number of concurrent requests - Destination: `num_requests` - Required: `false` - Default: `5` - Choices: `not restricted` - Action: `store` - Source: [L124-L129](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/test_batching.py#L124-L129) #### `--max-tokens` Max tokens per request - Destination: `max_tokens` - Required: `false` - Default: `30` - Choices: `not restricted` - Action: `store` - Source: [L130-L135](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/test_batching.py#L130-L135) #### `--temperature` Sampling temperature - Destination: `temperature` - Required: `false` - Default: `0.7` - Choices: `not restricted` - Action: `store` - Source: [L136-L141](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/test_batching.py#L136-L141) ## `examples.test_openai_compatibility.main` ### Parser `parser` #### `--server-url` URL of the vllm-mlx server (default: http://localhost:8000) - Destination: `server_url` - Required: `false` - Default: `http://localhost:8000` - Choices: `not restricted` - Action: `store` - Source: [L703-L708](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/test_openai_compatibility.py#L703-L708) #### `--no-image` Skip image tests - Destination: `no_image` - Required: `false` - Default: `argparse default` - Choices: `not restricted` - Action: `store_true` - Source: [L709-L713](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/test_openai_compatibility.py#L709-L713) #### `--no-video` Skip video tests - Destination: `no_video` - Required: `false` - Default: `argparse default` - Choices: `not restricted` - Action: `store_true` - Source: [L714-L718](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/test_openai_compatibility.py#L714-L718) ## `examples.test_video.main` ### Parser `parser` #### `--video` Path to video file (will create test video if not provided) - Destination: `video` - Required: `false` - Default: `argparse default` - Choices: `not restricted` - Action: `store` - Source: [L271-L275](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/test_video.py#L271-L275) #### `--video-url` URL to a video file to test URL support - Destination: `video_url` - Required: `false` - Default: `argparse default` - Choices: `not restricted` - Action: `store` - Source: [L276-L280](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/test_video.py#L276-L280) #### `--model` VLM model to use - Destination: `model` - Required: `false` - Default: `mlx-community/Qwen3-VL-4B-Instruct-3bit` - Choices: `not restricted` - Action: `store` - Source: [L281-L286](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/test_video.py#L281-L286) #### `--extract-only` Only test frame extraction (no model loading) - Destination: `extract_only` - Required: `false` - Default: `argparse default` - Choices: `not restricted` - Action: `store_true` - Source: [L287-L291](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/test_video.py#L287-L291) #### `--create-test-video` Create synthetic test video instead of downloading - Destination: `create_test_video` - Required: `false` - Default: `argparse default` - Choices: `not restricted` - Action: `store_true` - Source: [L292-L296](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/test_video.py#L292-L296) #### `--url-only` Only test video URL support (requires --video-url) - Destination: `url_only` - Required: `false` - Default: `argparse default` - Choices: `not restricted` - Action: `store_true` - Source: [L297-L301](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/test_video.py#L297-L301) ## `examples.tts_example.main` ### Parser `parser` #### `text` Text to synthesize - Destination: `text` - Required: `true` - Default: `argparse default` - Choices: `not restricted` - Action: `store` - Source: [L48-L48](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/tts_example.py#L48-L48) #### `--voice`, `-v` Voice ID (default: af_heart) - Destination: `voice` - Required: `false` - Default: `af_heart` - Choices: `not restricted` - Action: `store` - Source: [L49-L49](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/tts_example.py#L49-L49) #### `--lang`, `-l` Language code: a=English, e/es=Spanish, f=French, etc. - Destination: `lang` - Required: `false` - Default: `a` - Choices: `not restricted` - Action: `store` - Source: [L50-L50](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/tts_example.py#L50-L50) #### `--speed`, `-s` Speech speed 0.5-2.0 (default: 1.0) - Destination: `speed` - Required: `false` - Default: `1.0` - Choices: `not restricted` - Action: `store` - Source: [L51-L51](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/tts_example.py#L51-L51) #### `--output`, `-o` Output file (default: output.wav) - Destination: `output` - Required: `false` - Default: `output.wav` - Choices: `not restricted` - Action: `store` - Source: [L52-L52](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/tts_example.py#L52-L52) #### `--model`, `-m` TTS model - Destination: `model` - Required: `false` - Default: `mlx-community/Kokoro-82M-bf16` - Choices: `not restricted` - Action: `store` - Source: [L53-L53](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/tts_example.py#L53-L53) #### `--list-voices` List available voices - Destination: `list_voices` - Required: `false` - Default: `argparse default` - Choices: `not restricted` - Action: `store_true` - Source: [L54-L54](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/tts_example.py#L54-L54) #### `--list-languages` List available languages - Destination: `list_languages` - Required: `false` - Default: `argparse default` - Choices: `not restricted` - Action: `store_true` - Source: [L55-L55](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/tts_example.py#L55-L55) #### `--play`, `-p` Play audio after generation (macOS) - Destination: `play` - Required: `false` - Default: `argparse default` - Choices: `not restricted` - Action: `store_true` - Source: [L56-L56](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/tts_example.py#L56-L56) ## `examples.tts_multilingual.main` ### Parser `parser` #### `text` Text to synthesize - Destination: `text` - Required: `true` - Default: `argparse default` - Choices: `not restricted` - Action: `store` - Source: [L261-L261](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/tts_multilingual.py#L261-L261) #### `--model`, `-m` Model: kokoro, chatterbox, vibevoice, voxcpm, or 'auto' - Destination: `model` - Required: `false` - Default: `auto` - Choices: `not restricted` - Action: `store` - Source: [L262-L263](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/tts_multilingual.py#L262-L263) #### `--lang`, `-l` Language code: en, es, fr, ja, zh, etc. - Destination: `lang` - Required: `false` - Default: `en` - Choices: `not restricted` - Action: `store` - Source: [L264-L265](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/tts_multilingual.py#L264-L265) #### `--voice`, `-v` Voice ID (model-specific) - Destination: `voice` - Required: `false` - Default: `None` - Choices: `not restricted` - Action: `store` - Source: [L266-L267](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/tts_multilingual.py#L266-L267) #### `--speed`, `-s` Speech speed 0.5-2.0 - Destination: `speed` - Required: `false` - Default: `1.0` - Choices: `not restricted` - Action: `store` - Source: [L268-L269](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/tts_multilingual.py#L268-L269) #### `--output`, `-o` Output file - Destination: `output` - Required: `false` - Default: `output.wav` - Choices: `not restricted` - Action: `store` - Source: [L270-L271](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/tts_multilingual.py#L270-L271) #### `--play`, `-p` Play audio after generation (macOS) - Destination: `play` - Required: `false` - Default: `argparse default` - Choices: `not restricted` - Action: `store_true` - Source: [L272-L273](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/tts_multilingual.py#L272-L273) #### `--list-models` List available models - Destination: `list_models` - Required: `false` - Default: `argparse default` - Choices: `not restricted` - Action: `store_true` - Source: [L274-L275](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/tts_multilingual.py#L274-L275) #### `--list-languages` List supported languages - Destination: `list_languages` - Required: `false` - Default: `argparse default` - Choices: `not restricted` - Action: `store_true` - Source: [L276-L277](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/tts_multilingual.py#L276-L277) ## `examples.video_benchmark.main` ### Parser `parser` #### `--model` VLM model to use - Destination: `model` - Required: `false` - Default: `mlx-community/Qwen3-VL-4B-Instruct-3bit` - Choices: `not restricted` - Action: `store` - Source: [L500-L505](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/video_benchmark.py#L500-L505) #### `--video` Path to local video file - Destination: `video` - Required: `false` - Default: `None` - Choices: `not restricted` - Action: `store` - Source: [L506-L511](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/video_benchmark.py#L506-L511) #### `--video-url` URL to download video from - Destination: `video_url` - Required: `false` - Default: `None` - Choices: `not restricted` - Action: `store` - Source: [L512-L517](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/video_benchmark.py#L512-L517) #### `--duration` Duration of synthetic test video (seconds) - Destination: `duration` - Required: `false` - Default: `10.0` - Choices: `not restricted` - Action: `store` - Source: [L518-L523](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/video_benchmark.py#L518-L523) #### `--warmup` Number of warmup runs - Destination: `warmup` - Required: `false` - Default: `1` - Choices: `not restricted` - Action: `store` - Source: [L524-L529](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/video_benchmark.py#L524-L529) #### `--quick` Run quick benchmark with fewer configurations - Destination: `quick` - Required: `false` - Default: `argparse default` - Choices: `not restricted` - Action: `store_true` - Source: [L530-L534](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/video_benchmark.py#L530-L534) #### `--output` Save results to JSON file - Destination: `output` - Required: `false` - Default: `None` - Choices: `not restricted` - Action: `store` - Source: [L535-L540](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/video_benchmark.py#L535-L540) ## `scripts.add_mtp_weights.main` ### Parser `parser` #### `--mlx-model-path` f'Path to MLX model directory (default: {DEFAULT_MLX_MODEL})' - Destination: `mlx_model_path` - Required: `false` - Default: `DEFAULT_MLX_MODEL` - Choices: `not restricted` - Action: `store` - Source: [L243-L248](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/scripts/add_mtp_weights.py#L243-L248) #### `--source-model` f'HuggingFace model to download MTP shard from (default: {DEFAULT_SOURCE_MODEL})' - Destination: `source_model` - Required: `false` - Default: `DEFAULT_SOURCE_MODEL` - Choices: `not restricted` - Action: `store` - Source: [L249-L254](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/scripts/add_mtp_weights.py#L249-L254) #### `--download-dir` Directory to download MTP shard to (default: temp dir) - Destination: `download_dir` - Required: `false` - Default: `None` - Choices: `not restricted` - Action: `store` - Source: [L255-L260](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/scripts/add_mtp_weights.py#L255-L260) #### `--bits` Quantization bits (default: 6, matching 6-bit model) - Destination: `bits` - Required: `false` - Default: `6` - Choices: `not restricted` - Action: `store` - Source: [L261-L266](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/scripts/add_mtp_weights.py#L261-L266) #### `--skip-download` Skip download (use existing shard) - Destination: `skip_download` - Required: `false` - Default: `argparse default` - Choices: `not restricted` - Action: `store_true` - Source: [L267-L271](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/scripts/add_mtp_weights.py#L267-L271) ## `scripts.add_mtp_weights_qwen35.main` ### Parser `parser` #### `--mlx-model-path` Path to MLX model directory (HF cache or direct path) - Destination: `mlx_model_path` - Required: `true` - Default: `argparse default` - Choices: `not restricted` - Action: `store` - Source: [L327-L332](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/scripts/add_mtp_weights_qwen35.py#L327-L332) #### `--source-model` HuggingFace BF16 model to download MTP shards from (e.g., Qwen/Qwen3.5-122B-A10B) - Destination: `source_model` - Required: `true` - Default: `argparse default` - Choices: `not restricted` - Action: `store` - Source: [L333-L338](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/scripts/add_mtp_weights_qwen35.py#L333-L338) #### `--download-dir` Directory to download shards to (default: temp dir) - Destination: `download_dir` - Required: `false` - Default: `None` - Choices: `not restricted` - Action: `store` - Source: [L339-L344](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/scripts/add_mtp_weights_qwen35.py#L339-L344) #### `--skip-download` Skip download (use existing shards in download-dir) - Destination: `skip_download` - Required: `false` - Default: `argparse default` - Choices: `not restricted` - Action: `store_true` - Source: [L345-L349](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/scripts/add_mtp_weights_qwen35.py#L345-L349) #### `--keep-shards` Don't delete downloaded BF16 shards after extraction - Destination: `keep_shards` - Required: `false` - Default: `argparse default` - Choices: `not restricted` - Action: `store_true` - Source: [L350-L354](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/scripts/add_mtp_weights_qwen35.py#L350-L354) #### `--no-quantize` Save MTP weights in BF16 (no quantization). Required for correct MTP predictions. - Destination: `no_quantize` - Required: `false` - Default: `argparse default` - Choices: `not restricted` - Action: `store_true` - Source: [L355-L359](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/scripts/add_mtp_weights_qwen35.py#L355-L359) ## `scripts.gen_api_reference.main` ### Parser `parser` #### `--check` fail instead of writing when generated references are stale - Destination: `check` - Required: `false` - Default: `argparse default` - Choices: `not restricted` - Action: `store_true` - Source: [L93-L97](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/scripts/gen_api_reference.py#L93-L97) ## `vllm_mlx.benchmark.main` ### Parser `parser` #### `--model` Model name (HuggingFace model name or local path) - Destination: `model` - Required: `true` - Default: `argparse default` - Choices: `not restricted` - Action: `store` - Source: [L1468-L1473](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/benchmark.py#L1468-L1473) #### `--prompts` Number of prompts to benchmark for LLM (default: 5) - Destination: `prompts` - Required: `false` - Default: `5` - Choices: `not restricted` - Action: `store` - Source: [L1474-L1479](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/benchmark.py#L1474-L1479) #### `--max-tokens` Maximum tokens to generate per prompt (default: 256) - Destination: `max_tokens` - Required: `false` - Default: `256` - Choices: `not restricted` - Action: `store` - Source: [L1480-L1485](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/benchmark.py#L1480-L1485) #### `--temperature` Sampling temperature (default: 0.7) - Destination: `temperature` - Required: `false` - Default: `0.7` - Choices: `not restricted` - Action: `store` - Source: [L1486-L1491](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/benchmark.py#L1486-L1491) #### `--warmup` Number of warmup runs (default: 1) - Destination: `warmup` - Required: `false` - Default: `1` - Choices: `not restricted` - Action: `store` - Source: [L1492-L1497](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/benchmark.py#L1492-L1497) #### `--output` Output file for JSON results - Destination: `output` - Required: `false` - Default: `None` - Choices: `not restricted` - Action: `store` - Source: [L1498-L1503](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/benchmark.py#L1498-L1503) #### `--mllm` Force MLLM benchmark mode (auto-detected by default) - Destination: `mllm` - Required: `false` - Default: `argparse default` - Choices: `not restricted` - Action: `store_true` - Source: [L1504-L1508](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/benchmark.py#L1504-L1508) #### `--quick` Quick benchmark with fewer configurations - Destination: `quick` - Required: `false` - Default: `argparse default` - Choices: `not restricted` - Action: `store_true` - Source: [L1509-L1513](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/benchmark.py#L1509-L1513) #### `--video` Run video benchmark instead of image benchmark (for MLLM models) - Destination: `video` - Required: `false` - Default: `argparse default` - Choices: `not restricted` - Action: `store_true` - Source: [L1515-L1519](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/benchmark.py#L1515-L1519) #### `--video-url` URL of video to use for benchmark (default: Big Buck Bunny 10s) - Destination: `video_url` - Required: `false` - Default: `None` - Choices: `not restricted` - Action: `store` - Source: [L1520-L1525](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/benchmark.py#L1520-L1525) #### `--video-path` Local path to video file for benchmark - Destination: `video_path` - Required: `false` - Default: `None` - Choices: `not restricted` - Action: `store` - Source: [L1526-L1531](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/benchmark.py#L1526-L1531) ## `vllm_mlx.cli.create_parser` ### Parser `serve_parser` #### `model` Model to serve - Destination: `model` - Required: `true` - Default: `argparse default` - Choices: `not restricted` - Action: `store` - Source: [L1008-L1008](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1008-L1008) #### `--models-config` YAML file describing a registry of models for lazy multi-model serving - Destination: `models_config` - Required: `false` - Default: `None` - Choices: `not restricted` - Action: `store` - Source: [L1009-L1014](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1009-L1014) #### `--served-model-name` The model name used in the API. If not specified, the model argument is used. - Destination: `served_model_name` - Required: `false` - Default: `None` - Choices: `not restricted` - Action: `store` - Source: [L1015-L1020](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1015-L1020) #### `--host` Host to bind (default: localhost; use 0.0.0.0 to expose externally) - Destination: `host` - Required: `false` - Default: `127.0.0.1` - Choices: `not restricted` - Action: `store` - Source: [L1021-L1026](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1021-L1026) #### `--port` Port to bind - Destination: `port` - Required: `false` - Default: `8000` - Choices: `not restricted` - Action: `store` - Source: [L1027-L1027](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1027-L1027) #### `--max-num-seqs` Max concurrent sequences - Destination: `max_num_seqs` - Required: `false` - Default: `256` - Choices: `not restricted` - Action: `store` - Source: [L1028-L1030](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1028-L1030) #### `--prefill-batch-size` Prefill batch size - Destination: `prefill_batch_size` - Required: `false` - Default: `8` - Choices: `not restricted` - Action: `store` - Source: [L1031-L1033](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1031-L1033) #### `--completion-batch-size` Completion batch size - Destination: `completion_batch_size` - Required: `false` - Default: `32` - Choices: `not restricted` - Action: `store` - Source: [L1034-L1036](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1034-L1036) #### `--mllm-prefill-step-size` Override MLLM prefill-step guard (0=use MLLM default: 1024) - Destination: `mllm_prefill_step_size` - Required: `false` - Default: `0` - Choices: `not restricted` - Action: `store` - Source: [L1037-L1042](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1037-L1042) #### `--enable-prefix-cache` Enable prefix caching for repeated prompts (default: enabled) - Destination: `enable_prefix_cache` - Required: `false` - Default: `True` - Choices: `not restricted` - Action: `store_true` - Source: [L1043-L1048](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1043-L1048) #### `--disable-prefix-cache` Disable prefix caching - Destination: `disable_prefix_cache` - Required: `false` - Default: `argparse default` - Choices: `not restricted` - Action: `store_true` - Source: [L1049-L1053](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1049-L1053) #### `--prefix-cache-size` Max entries in prefix cache (default: 100, legacy mode only) - Destination: `prefix_cache_size` - Required: `false` - Default: `100` - Choices: `not restricted` - Action: `store` - Source: [L1054-L1059](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1054-L1059) #### `--cache-memory-mb` Cache memory limit in MB (default: auto-detect ~20%% of RAM) - Destination: `cache_memory_mb` - Required: `false` - Default: `None` - Choices: `not restricted` - Action: `store` - Source: [L1061-L1066](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1061-L1066) #### `--cache-memory-percent` Fraction of available RAM for cache if auto-detecting (default: 0.20) - Destination: `cache_memory_percent` - Required: `false` - Default: `0.2` - Choices: `not restricted` - Action: `store` - Source: [L1067-L1072](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1067-L1072) #### `--no-memory-aware-cache` Disable memory-aware cache, use legacy entry-count based cache - Destination: `no_memory_aware_cache` - Required: `false` - Default: `argparse default` - Choices: `not restricted` - Action: `store_true` - Source: [L1073-L1077](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1073-L1077) #### `--kv-cache-quantization` Quantize stored KV caches to reduce memory (8-bit by default) - Destination: `kv_cache_quantization` - Required: `false` - Default: `argparse default` - Choices: `not restricted` - Action: `store_true` - Source: [L1079-L1083](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1079-L1083) #### `--kv-cache-quantization-bits` Bit width for KV cache quantization (default: 8) - Destination: `kv_cache_quantization_bits` - Required: `false` - Default: `8` - Choices: `[4, 8]` - Action: `store` - Source: [L1084-L1090](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1084-L1090) #### `--kv-cache-quantization-group-size` Group size for KV cache quantization (default: 64) - Destination: `kv_cache_quantization_group_size` - Required: `false` - Default: `64` - Choices: `not restricted` - Action: `store` - Source: [L1091-L1096](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1091-L1096) #### `--kv-cache-min-quantize-tokens` Minimum tokens for quantization to apply (default: 256) - Destination: `kv_cache_min_quantize_tokens` - Required: `false` - Default: `256` - Choices: `not restricted` - Action: `store` - Source: [L1097-L1102](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1097-L1102) #### `--ssd-cache-dir` Directory for SSD KV cache tier (default: disabled) - Destination: `ssd_cache_dir` - Required: `false` - Default: `None` - Choices: `not restricted` - Action: `store` - Source: [L1104-L1109](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1104-L1109) #### `--ssd-cache-max-gb` Maximum SSD cache size in GB (default: 10.0) - Destination: `ssd_cache_max_gb` - Required: `false` - Default: `10.0` - Choices: `not restricted` - Action: `store` - Source: [L1110-L1115](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1110-L1115) #### `--warm-prompts` Path to a JSON file with prompts to pre-run at startup. Populates the prefix cache so the first real request hits warm (cold TTFT drops 1.3-2.3x on agent workloads). File format is a list of message arrays, same shape as /v1/chat/completions messages. Prompts are warmed concurrently — keep the file small (1-3 entries for typical agent deployments) to avoid memory pressure at boot. - Destination: `warm_prompts` - Required: `false` - Default: `None` - Choices: `not restricted` - Action: `store` - Source: [L1117-L1129](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1117-L1129) #### `--stream-interval` Tokens to batch before streaming (1=smooth, higher=throughput) - Destination: `stream_interval` - Required: `false` - Default: `1` - Choices: `not restricted` - Action: `store` - Source: [L1130-L1135](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1130-L1135) #### `--max-kv-size` Maximum KV cache size per sequence. When set, uses RotatingKVCache which bounds memory at the cost of losing early context. Reasoning models (e.g. Qwen3, DeepSeek-R1) should use >= 32768 to avoid evicting the think block mid-generation. - Destination: `max_kv_size` - Required: `false` - Default: `None` - Choices: `not restricted` - Action: `store` - Source: [L1136-L1144](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1136-L1144) #### `--max-tokens` Default max tokens for generation (default: 32768) - Destination: `max_tokens` - Required: `false` - Default: `32768` - Choices: `not restricted` - Action: `store` - Source: [L1145-L1150](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1145-L1150) #### `--max-request-tokens` Maximum max_tokens accepted from API clients (default: 32768) - Destination: `max_request_tokens` - Required: `false` - Default: `32768` - Choices: `not restricted` - Action: `store` - Source: [L1151-L1156](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1151-L1156) #### `--continuous-batching` Enable continuous batching for multiple concurrent users (slower for single user) - Destination: `continuous_batching` - Required: `false` - Default: `argparse default` - Choices: `not restricted` - Action: `store_true` - Source: [L1157-L1161](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1157-L1161) #### `--gpu-memory-utilization` Fraction of device memory for Metal allocation limit and emergency cache clear threshold (0.0-1.0, default: 0.90). Increase to 0.95 for large models (200GB+) that need more memory headroom. - Destination: `gpu_memory_utilization` - Required: `false` - Default: `0.9` - Choices: `not restricted` - Action: `store` - Source: [L1162-L1169](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1162-L1169) #### `--use-paged-cache` Use paged KV cache for memory efficiency (experimental) - Destination: `use_paged_cache` - Required: `false` - Default: `argparse default` - Choices: `not restricted` - Action: `store_true` - Source: [L1171-L1175](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1171-L1175) #### `--paged-cache-block-size` Tokens per cache block (default: 64) - Destination: `paged_cache_block_size` - Required: `false` - Default: `64` - Choices: `not restricted` - Action: `store` - Source: [L1176-L1181](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1176-L1181) #### `--max-cache-blocks` Maximum number of cache blocks (default: 1000) - Destination: `max_cache_blocks` - Required: `false` - Default: `1000` - Choices: `not restricted` - Action: `store` - Source: [L1182-L1187](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1182-L1187) #### `--chunked-prefill-tokens` Max prefill tokens per scheduler step (0=disabled). Prevents starvation of active requests during long prefills. - Destination: `chunked_prefill_tokens` - Required: `false` - Default: `0` - Choices: `not restricted` - Action: `store` - Source: [L1189-L1195](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1189-L1195) #### `--enable-mtp` Enable MTP (Multi-Token Prediction) for models with built-in MTP heads. Uses cache snapshot/restore for speculative generation. - Destination: `enable_mtp` - Required: `false` - Default: `False` - Choices: `not restricted` - Action: `store_true` - Source: [L1197-L1203](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1197-L1203) #### `--mtp-num-draft-tokens` Number of draft tokens per MTP step (default: 1) - Destination: `mtp_num_draft_tokens` - Required: `false` - Default: `1` - Choices: `not restricted` - Action: `store` - Source: [L1204-L1209](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1204-L1209) #### `--mtp-optimistic` Skip MTP acceptance check for maximum speed. ~5-10%% wrong tokens. Best for chat, not for code. - Destination: `mtp_optimistic` - Required: `false` - Default: `False` - Choices: `not restricted` - Action: `store_true` - Source: [L1210-L1216](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1210-L1216) #### `--prefill-step-size` Chunk size for prompt prefill processing. Larger values use more memory but can improve prefill throughput. (default: 2048) - Destination: `prefill_step_size` - Required: `false` - Default: `2048` - Choices: `not restricted` - Action: `store` - Source: [L1218-L1224](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1218-L1224) #### `--specprefill` Enable SpecPrefill: use a small draft model to score token importance, then sparse-prefill only the important tokens on the target model. Reduces TTFT on long prompts. Requires --specprefill-draft-model. - Destination: `specprefill` - Required: `false` - Default: `False` - Choices: `not restricted` - Action: `store_true` - Source: [L1226-L1233](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1226-L1233) #### `--specprefill-threshold` Minimum suffix tokens to trigger SpecPrefill (default: 8192). Shorter prompts use full prefill (scoring overhead > savings). - Destination: `specprefill_threshold` - Required: `false` - Default: `8192` - Choices: `not restricted` - Action: `store` - Source: [L1234-L1240](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1234-L1240) #### `--specprefill-keep-pct` Fraction of tokens to keep during sparse prefill (default: 0.3). Lower = faster prefill but more quality loss. - Destination: `specprefill_keep_pct` - Required: `false` - Default: `0.3` - Choices: `not restricted` - Action: `store` - Source: [L1241-L1247](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1241-L1247) #### `--specprefill-backbone-pct` Fraction of chunks reserved for evenly spaced sparse-prefill coverage (default: 0.0). - Destination: `specprefill_backbone_pct` - Required: `false` - Default: `0.0` - Choices: `not restricted` - Action: `store` - Source: [L1248-L1254](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1248-L1254) #### `--specprefill-draft-model` Path to small draft model for SpecPrefill importance scoring. Must share the same tokenizer as the target model. - Destination: `specprefill_draft_model` - Required: `false` - Default: `None` - Choices: `not restricted` - Action: `store` - Source: [L1255-L1261](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1255-L1261) #### `--mllm-draft-model` Path to an mlx-vlm MLLM draft/assistant model. For Gemma 4 assistant drafters, use with --mllm-draft-kind mtp. - Destination: `mllm_draft_model` - Required: `false` - Default: `None` - Choices: `not restricted` - Action: `store` - Source: [L1263-L1269](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1263-L1269) #### `--mllm-draft-kind` mlx-vlm draft kind for --mllm-draft-model. - Destination: `mllm_draft_kind` - Required: `false` - Default: `None` - Choices: `['mtp']` - Action: `store` - Source: [L1270-L1276](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1270-L1276) #### `--mllm-draft-block-size` Draft block size passed to mlx-vlm for --mllm-draft-model. - Destination: `mllm_draft_block_size` - Required: `false` - Default: `None` - Choices: `not restricted` - Action: `store` - Source: [L1277-L1282](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1277-L1282) #### `--mcp-config` Path to MCP configuration file (JSON/YAML) for tool integration - Destination: `mcp_config` - Required: `false` - Default: `None` - Choices: `not restricted` - Action: `store` - Source: [L1284-L1289](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1284-L1289) #### `--api-key` API key for authentication (if not set, no auth required) - Destination: `api_key` - Required: `false` - Default: `None` - Choices: `not restricted` - Action: `store` - Source: [L1291-L1296](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1291-L1296) #### `--rate-limit` Rate limit requests per minute per client (0 = disabled) - Destination: `rate_limit` - Required: `false` - Default: `0` - Choices: `not restricted` - Action: `store` - Source: [L1297-L1302](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1297-L1302) #### `--timeout` Default request timeout in seconds (default: 300) - Destination: `timeout` - Required: `false` - Default: `300.0` - Choices: `not restricted` - Action: `store` - Source: [L1303-L1308](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1303-L1308) #### `--enable-metrics` Expose Prometheus metrics on /metrics (disabled by default) - Destination: `enable_metrics` - Required: `false` - Default: `argparse default` - Choices: `not restricted` - Action: `store_true` - Source: [L1309-L1313](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1309-L1313) #### `--auto-unload-idle-seconds` Unload the main model after this many idle seconds (0 = disabled) - Destination: `auto_unload_idle_seconds` - Required: `false` - Default: `0.0` - Choices: `not restricted` - Action: `store` - Source: [L1314-L1319](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1314-L1319) #### `--lazy-load-model` Register the main model at startup but defer loading until first request - Destination: `lazy_load_model` - Required: `false` - Default: `argparse default` - Choices: `not restricted` - Action: `store_true` - Source: [L1320-L1324](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1320-L1324) #### `--max-audio-upload-mb` Maximum size of uploaded audio files in MiB (default: 25) - Destination: `max_audio_upload_mb` - Required: `false` - Default: `25` - Choices: `not restricted` - Action: `store` - Source: [L1325-L1330](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1325-L1330) #### `--max-tts-input-chars` Maximum number of characters accepted by /v1/audio/speech (default: 4096) - Destination: `max_tts_input_chars` - Required: `false` - Default: `4096` - Choices: `not restricted` - Action: `store` - Source: [L1331-L1336](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1331-L1336) #### `--enable-auto-tool-choice` Enable auto tool choice for supported models. Use --tool-call-parser to specify which parser to use. - Destination: `enable_auto_tool_choice` - Required: `false` - Default: `argparse default` - Choices: `not restricted` - Action: `store_true` - Source: [L1338-L1342](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1338-L1342) #### `--tool-call-parser` Select the tool call parser for the model. Options: auto (auto-detect), mistral, qwen, qwen3_coder, llama, hermes, harmony, gpt-oss, deepseek, gemma4, kimi, granite, nemotron, xlam, functionary, glm47, minimax. Required for --enable-auto-tool-choice. - Destination: `tool_call_parser` - Required: `false` - Default: `None` - Choices: `['auto', 'mistral', 'qwen', 'qwen3_coder', 'llama', 'hermes', 'harmony', 'gpt-oss', 'deepseek', 'kimi', 'granite', 'nemotron', 'xlam', 'functionary', 'gemma4', 'glm47', 'minimax']` - Action: `store` - Source: [L1343-L1373](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1343-L1373) #### `--reasoning-parser` f"Enable reasoning content extraction with specified parser. Extracts ... tags into reasoning_content field. Options: {', '.join(reasoning_choices)}." - Destination: `reasoning_parser` - Required: `false` - Default: `None` - Choices: `reasoning_choices` - Action: `store` - Source: [L1378-L1388](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1378-L1388) #### `--mllm` Force load model as multimodal (vision) even if name doesn't match auto-detection patterns - Destination: `mllm` - Required: `false` - Default: `argparse default` - Choices: `not restricted` - Action: `store_true` - Source: [L1390-L1394](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1390-L1394) #### `--trust-remote-code` Allow HuggingFace remote code execution during model/tokenizer loading - Destination: `trust_remote_code` - Required: `false` - Default: `argparse default` - Choices: `not restricted` - Action: `store_true` - Source: [L1395-L1399](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1395-L1399) #### `--default-temperature` Override default temperature for all requests (default: use model default) - Destination: `default_temperature` - Required: `false` - Default: `None` - Choices: `not restricted` - Action: `store` - Source: [L1401-L1406](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1401-L1406) #### `--default-top-p` Override default top_p for all requests (default: use model default) - Destination: `default_top_p` - Required: `false` - Default: `None` - Choices: `not restricted` - Action: `store` - Source: [L1407-L1412](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1407-L1412) #### `--default-thinking-token-budget` Default thinking token budget for reasoning models. Caps reasoning tokens by forcing the end-think sequence when the budget is exhausted. Per-request thinking_token_budget overrides this. (default: None = unlimited) - Destination: `default_thinking_token_budget` - Required: `false` - Default: `None` - Choices: `not restricted` - Action: `store` - Source: [L1413-L1422](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1413-L1422) #### `--default-chat-template-kwargs` Default chat template kwargs to apply to all requests when request chat_template_kwargs is omitted or empty; empty request kwargs use existing server defaults (JSON object, e.g. {"enable_thinking": true}) - Destination: `default_chat_template_kwargs` - Required: `false` - Default: `None` - Choices: `not restricted` - Action: `store` - Source: [L1423-L1432](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1423-L1432) #### `--default-top-k` Override default top_k for all requests (default: use model default) - Destination: `default_top_k` - Required: `false` - Default: `None` - Choices: `not restricted` - Action: `store` - Source: [L1433-L1438](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1433-L1438) #### `--default-min-p` Override default min_p for all requests (default: use model default) - Destination: `default_min_p` - Required: `false` - Default: `None` - Choices: `not restricted` - Action: `store` - Source: [L1439-L1444](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1439-L1444) #### `--default-presence-penalty` Override default presence_penalty for all requests (default: use model default) - Destination: `default_presence_penalty` - Required: `false` - Default: `None` - Choices: `not restricted` - Action: `store` - Source: [L1445-L1453](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1445-L1453) #### `--default-repetition-penalty` Override default repetition_penalty for all requests (default: use model default) - Destination: `default_repetition_penalty` - Required: `false` - Default: `None` - Choices: `not restricted` - Action: `store` - Source: [L1454-L1462](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1454-L1462) #### `--embedding-model` Pre-load an embedding model at startup (e.g. mlx-community/embeddinggemma-300m-6bit) - Destination: `embedding_model` - Required: `false` - Default: `None` - Choices: `not restricted` - Action: `store` - Source: [L1464-L1469](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1464-L1469) #### `--rerank-model` Pre-load a reranker model at startup (e.g. mlx-community/jina-reranker-v2-base-multilingual) - Destination: `rerank_model` - Required: `false` - Default: `None` - Choices: `not restricted` - Action: `store` - Source: [L1471-L1476](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1471-L1476) #### `--download-timeout` Per-file download timeout in seconds (default: 300) - Destination: `download_timeout` - Required: `false` - Default: `300` - Choices: `not restricted` - Action: `store` - Source: [L1478-L1483](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1478-L1483) #### `--download-retries` Number of download retry attempts (default: 3) - Destination: `download_retries` - Required: `false` - Default: `3` - Choices: `not restricted` - Action: `store` - Source: [L1484-L1489](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1484-L1489) #### `--offline` Offline mode — only use locally cached models - Destination: `offline` - Required: `false` - Default: `argparse default` - Choices: `not restricted` - Action: `store_true` - Source: [L1490-L1494](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1490-L1494) ### Parser `bench_parser` #### `model` Model to benchmark - Destination: `model` - Required: `true` - Default: `argparse default` - Choices: `not restricted` - Action: `store` - Source: [L1497-L1497](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1497-L1497) #### `--num-prompts` Number of prompts - Destination: `num_prompts` - Required: `false` - Default: `10` - Choices: `not restricted` - Action: `store` - Source: [L1498-L1500](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1498-L1500) #### `--max-tokens` Max tokens per prompt - Destination: `max_tokens` - Required: `false` - Default: `100` - Choices: `not restricted` - Action: `store` - Source: [L1501-L1503](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1501-L1503) #### `--max-num-seqs` Max concurrent sequences - Destination: `max_num_seqs` - Required: `false` - Default: `32` - Choices: `not restricted` - Action: `store` - Source: [L1504-L1506](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1504-L1506) #### `--prefill-batch-size` Prefill batch size - Destination: `prefill_batch_size` - Required: `false` - Default: `8` - Choices: `not restricted` - Action: `store` - Source: [L1507-L1509](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1507-L1509) #### `--completion-batch-size` Completion batch size - Destination: `completion_batch_size` - Required: `false` - Default: `16` - Choices: `not restricted` - Action: `store` - Source: [L1510-L1512](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1510-L1512) #### `--enable-prefix-cache` Enable prefix caching (default: enabled) - Destination: `enable_prefix_cache` - Required: `false` - Default: `True` - Choices: `not restricted` - Action: `store_true` - Source: [L1513-L1518](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1513-L1518) #### `--disable-prefix-cache` Disable prefix caching - Destination: `disable_prefix_cache` - Required: `false` - Default: `argparse default` - Choices: `not restricted` - Action: `store_true` - Source: [L1519-L1523](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1519-L1523) #### `--prefix-cache-size` Max entries in prefix cache (default: 100, legacy mode only) - Destination: `prefix_cache_size` - Required: `false` - Default: `100` - Choices: `not restricted` - Action: `store` - Source: [L1524-L1529](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1524-L1529) #### `--cache-memory-mb` Cache memory limit in MB (default: auto-detect ~20%% of RAM) - Destination: `cache_memory_mb` - Required: `false` - Default: `None` - Choices: `not restricted` - Action: `store` - Source: [L1531-L1536](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1531-L1536) #### `--cache-memory-percent` Fraction of available RAM for cache if auto-detecting (default: 0.20) - Destination: `cache_memory_percent` - Required: `false` - Default: `0.2` - Choices: `not restricted` - Action: `store` - Source: [L1537-L1542](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1537-L1542) #### `--no-memory-aware-cache` Disable memory-aware cache, use legacy entry-count based cache - Destination: `no_memory_aware_cache` - Required: `false` - Default: `argparse default` - Choices: `not restricted` - Action: `store_true` - Source: [L1543-L1547](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1543-L1547) #### `--kv-cache-quantization` Quantize stored KV caches to reduce memory (8-bit by default) - Destination: `kv_cache_quantization` - Required: `false` - Default: `argparse default` - Choices: `not restricted` - Action: `store_true` - Source: [L1549-L1553](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1549-L1553) #### `--kv-cache-quantization-bits` Bit width for KV cache quantization (default: 8) - Destination: `kv_cache_quantization_bits` - Required: `false` - Default: `8` - Choices: `[4, 8]` - Action: `store` - Source: [L1554-L1560](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1554-L1560) #### `--kv-cache-quantization-group-size` Group size for KV cache quantization (default: 64) - Destination: `kv_cache_quantization_group_size` - Required: `false` - Default: `64` - Choices: `not restricted` - Action: `store` - Source: [L1561-L1566](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1561-L1566) #### `--kv-cache-min-quantize-tokens` Minimum tokens for quantization to apply (default: 256) - Destination: `kv_cache_min_quantize_tokens` - Required: `false` - Default: `256` - Choices: `not restricted` - Action: `store` - Source: [L1567-L1572](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1567-L1572) #### `--use-paged-cache` Use paged KV cache for memory efficiency (experimental) - Destination: `use_paged_cache` - Required: `false` - Default: `argparse default` - Choices: `not restricted` - Action: `store_true` - Source: [L1574-L1578](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1574-L1578) #### `--paged-cache-block-size` Tokens per cache block (default: 64) - Destination: `paged_cache_block_size` - Required: `false` - Default: `64` - Choices: `not restricted` - Action: `store` - Source: [L1579-L1584](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1579-L1584) #### `--max-cache-blocks` Maximum number of cache blocks (default: 1000) - Destination: `max_cache_blocks` - Required: `false` - Default: `1000` - Choices: `not restricted` - Action: `store` - Source: [L1585-L1590](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1585-L1590) ### Parser `detok_parser` #### `model` Model to use for tokenizer (default: mlx-community/Qwen3-0.6B-8bit) - Destination: `model` - Required: `true` - Default: `mlx-community/Qwen3-0.6B-8bit` - Choices: `not restricted` - Action: `store` - Source: [L1596-L1602](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1596-L1602) #### `--iterations` Benchmark iterations (default: 5) - Destination: `iterations` - Required: `false` - Default: `5` - Choices: `not restricted` - Action: `store` - Source: [L1603-L1605](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1603-L1605) ### Parser `kv_cache_parser` #### `--layers` Number of layers (default: 32) - Destination: `layers` - Required: `false` - Default: `32` - Choices: `not restricted` - Action: `store` - Source: [L1611-L1613](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1611-L1613) #### `--seq-len` Sequence length (default: 512) - Destination: `seq_len` - Required: `false` - Default: `512` - Choices: `not restricted` - Action: `store` - Source: [L1614-L1616](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1614-L1616) #### `--heads` Number of attention heads (default: 32) - Destination: `heads` - Required: `false` - Default: `32` - Choices: `not restricted` - Action: `store` - Source: [L1617-L1619](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1617-L1619) #### `--head-dim` Head dimension (default: 128) - Destination: `head_dim` - Required: `false` - Default: `128` - Choices: `not restricted` - Action: `store` - Source: [L1620-L1622](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1620-L1622) #### `--group-size` Quantization group size (default: 64) - Destination: `group_size` - Required: `false` - Default: `64` - Choices: `not restricted` - Action: `store` - Source: [L1623-L1628](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1623-L1628) ### Parser `download_parser` #### `model` Model to download - Destination: `model` - Required: `true` - Default: `argparse default` - Choices: `not restricted` - Action: `store` - Source: [L1634-L1634](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1634-L1634) #### `--timeout` Per-file download timeout in seconds (default: 300) - Destination: `timeout` - Required: `false` - Default: `300` - Choices: `not restricted` - Action: `store` - Source: [L1635-L1640](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1635-L1640) #### `--retries` Number of retry attempts (default: 3) - Destination: `retries` - Required: `false` - Default: `3` - Choices: `not restricted` - Action: `store` - Source: [L1641-L1646](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1641-L1646) #### `--mllm` Download as multimodal model (broader file patterns) - Destination: `mllm` - Required: `false` - Default: `argparse default` - Choices: `not restricted` - Action: `store_true` - Source: [L1647-L1651](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1647-L1651) ### Parser `model_inspect_parser` #### `model` Local model path or Hugging Face model id - Destination: `model` - Required: `true` - Default: `argparse default` - Choices: `not restricted` - Action: `store` - Source: [L1666-L1670](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1666-L1670) #### `--revision` Hugging Face revision to inspect - Destination: `revision` - Required: `false` - Default: `None` - Choices: `not restricted` - Action: `store` - Source: [L1671-L1676](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1671-L1676) #### `--local-files-only` Use only local Hugging Face cache files - Destination: `local_files_only` - Required: `false` - Default: `argparse default` - Choices: `not restricted` - Action: `store_true` - Source: [L1677-L1681](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1677-L1681) ### Parser `model_acquire_parser` #### `model` Hugging Face model id - Destination: `model` - Required: `true` - Default: `argparse default` - Choices: `not restricted` - Action: `store` - Source: [L1687-L1687](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1687-L1687) #### `--revision` Hugging Face revision to download - Destination: `revision` - Required: `false` - Default: `None` - Choices: `not restricted` - Action: `store` - Source: [L1688-L1693](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1688-L1693) #### `--target-dir` Final local directory. Defaults to Hugging Face cache. - Destination: `target_dir` - Required: `false` - Default: `None` - Choices: `not restricted` - Action: `store` - Source: [L1694-L1699](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1694-L1699) #### `--staging-dir` Directory for temporary staged downloads before finalizing target-dir - Destination: `staging_dir` - Required: `false` - Default: `None` - Choices: `not restricted` - Action: `store` - Source: [L1700-L1705](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1700-L1705) #### `--mllm` Acquire multimodal model files using broader allow patterns - Destination: `mllm` - Required: `false` - Default: `argparse default` - Choices: `not restricted` - Action: `store_true` - Source: [L1706-L1710](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1706-L1710) #### `--no-fast-transfer` Do not set HF_HUB_ENABLE_HF_TRANSFER=1 during download - Destination: `no_fast_transfer` - Required: `false` - Default: `argparse default` - Choices: `not restricted` - Action: `store_true` - Source: [L1711-L1715](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1711-L1715) #### `--local-files-only` Use only local Hugging Face cache files - Destination: `local_files_only` - Required: `false` - Default: `argparse default` - Choices: `not restricted` - Action: `store_true` - Source: [L1716-L1720](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1716-L1720) ### Parser `model_convert_parser` #### `source` Hugging Face model id or local source path - Destination: `source` - Required: `true` - Default: `argparse default` - Choices: `not restricted` - Action: `store` - Source: [L1726-L1730](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1726-L1730) #### `--output` Output directory for the converted MLX model - Destination: `output` - Required: `true` - Default: `argparse default` - Choices: `not restricted` - Action: `store` - Source: [L1731-L1736](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1731-L1736) #### `--quantize` Generate a quantized MLX model - Destination: `quantize` - Required: `false` - Default: `argparse default` - Choices: `not restricted` - Action: `store_true` - Source: [L1737-L1741](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1737-L1741) #### `--q-bits` Quantization bit width (e.g. 3, 4, 8) - Destination: `q_bits` - Required: `false` - Default: `None` - Choices: `not restricted` - Action: `store` - Source: [L1742-L1747](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1742-L1747) #### `--q-group-size` Quantization group size (default: mlx-lm default) - Destination: `q_group_size` - Required: `false` - Default: `None` - Choices: `not restricted` - Action: `store` - Source: [L1748-L1753](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1748-L1753) #### `--q-mode` No argparse help text is declared. - Destination: `q_mode` - Required: `false` - Default: `None` - Choices: `['affine', 'mxfp4', 'nvfp4', 'mxfp8']` - Action: `store` - Source: [L1754-L1758](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1754-L1758) #### `--quant-predicate` mlx-lm mixed-bit quantization recipe - Destination: `quant_predicate` - Required: `false` - Default: `None` - Choices: `['mixed_2_6', 'mixed_3_4', 'mixed_3_6', 'mixed_4_6']` - Action: `store` - Source: [L1759-L1764](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1759-L1764) #### `--dtype` Non-quantized parameter dtype - Destination: `dtype` - Required: `false` - Default: `None` - Choices: `['float16', 'bfloat16', 'float32']` - Action: `store` - Source: [L1765-L1770](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1765-L1770) #### `--trust-remote-code` Allow Hugging Face remote code during mlx-lm conversion - Destination: `trust_remote_code` - Required: `false` - Default: `argparse default` - Choices: `not restricted` - Action: `store_true` - Source: [L1771-L1775](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1771-L1775) #### `--dry-run` Print the conversion command and manifest without executing - Destination: `dry_run` - Required: `false` - Default: `argparse default` - Choices: `not restricted` - Action: `store_true` - Source: [L1776-L1780](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1776-L1780) ### Parser `model_register_parser` #### `artifact` Finalized local model artifact directory - Destination: `artifact` - Required: `true` - Default: `argparse default` - Choices: `not restricted` - Action: `store` - Source: [L1786-L1790](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1786-L1790) #### `--model-id` Override model ID (default: directory name of artifact) - Destination: `model_id` - Required: `false` - Default: `None` - Choices: `not restricted` - Action: `store` - Source: [L1791-L1796](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1791-L1796) #### `--served-model-name` Model name exposed by the API (default: model-id) - Destination: `served_model_name` - Required: `false` - Default: `None` - Choices: `not restricted` - Action: `store` - Source: [L1797-L1802](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1797-L1802) #### `--preset-alias` Optional alias for preset lookup in registry - Destination: `preset_alias` - Required: `false` - Default: `None` - Choices: `not restricted` - Action: `store` - Source: [L1803-L1808](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1803-L1808) #### `--output` Manifest path. Defaults to artifact/vllm_mlx_registration_manifest.json - Destination: `output` - Required: `false` - Default: `None` - Choices: `not restricted` - Action: `store` - Source: [L1809-L1814](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1809-L1814) ### Parser `mllm_group` #### `--mllm` Mark the artifact as an MLLM serving candidate - Destination: `mllm` - Required: `false` - Default: `None` - Choices: `not restricted` - Action: `store_true` - Source: [L1816-L1821](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1816-L1821) #### `--no-mllm` Explicitly mark the artifact as text-only - Destination: `mllm` - Required: `false` - Default: `argparse default` - Choices: `not restricted` - Action: `store_false` - Source: [L1822-L1827](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1822-L1827) ### Parser `model_register_parser` #### `--tool-call-parser` Tool call parser name for the model (e.g. qwen3_coder, mistral) - Destination: `tool_call_parser` - Required: `false` - Default: `None` - Choices: `not restricted` - Action: `store` - Source: [L1828-L1833](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1828-L1833) #### `--reasoning-parser` Reasoning parser name for thinking models (e.g. qwen3) - Destination: `reasoning_parser` - Required: `false` - Default: `None` - Choices: `not restricted` - Action: `store` - Source: [L1834-L1839](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1834-L1839) #### `--default-temperature` Default temperature for all requests - Destination: `default_temperature` - Required: `false` - Default: `None` - Choices: `not restricted` - Action: `store` - Source: [L1840-L1845](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1840-L1845) #### `--default-top-p` Default top_p for all requests - Destination: `default_top_p` - Required: `false` - Default: `None` - Choices: `not restricted` - Action: `store` - Source: [L1846-L1851](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1846-L1851) #### `--default-top-k` Default top_k for all requests - Destination: `default_top_k` - Required: `false` - Default: `None` - Choices: `not restricted` - Action: `store` - Source: [L1852-L1857](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1852-L1857) #### `--default-min-p` Default min_p for all requests - Destination: `default_min_p` - Required: `false` - Default: `None` - Choices: `not restricted` - Action: `store` - Source: [L1858-L1863](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1858-L1863) #### `--default-presence-penalty` Default presence_penalty for all requests - Destination: `default_presence_penalty` - Required: `false` - Default: `None` - Choices: `not restricted` - Action: `store` - Source: [L1864-L1869](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1864-L1869) #### `--default-repetition-penalty` Default repetition_penalty for all requests - Destination: `default_repetition_penalty` - Required: `false` - Default: `None` - Choices: `not restricted` - Action: `store` - Source: [L1870-L1875](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1870-L1875) #### `--default-chat-template-kwargs` Default chat template kwargs as JSON, e.g. {"enable_thinking": true} - Destination: `default_chat_template_kwargs` - Required: `false` - Default: `None` - Choices: `not restricted` - Action: `store` - Source: [L1876-L1881](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1876-L1881) #### `--feature-flag` Feature flag to record in the registration manifest. Repeatable. - Destination: `feature_flag` - Required: `false` - Default: `[]` - Choices: `not restricted` - Action: `append` - Source: [L1882-L1887](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1882-L1887) ### Parser `model_qualify_parser` #### `model_id` Model ID to qualify against the running server - Destination: `model_id` - Required: `true` - Default: `argparse default` - Choices: `not restricted` - Action: `store` - Source: [L1893-L1897](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1893-L1897) #### `--url` Running server URL for bench-serve - Destination: `url` - Required: `false` - Default: `http://127.0.0.1:8080` - Choices: `not restricted` - Action: `store` - Source: [L1898-L1903](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1898-L1903) #### `--workload` bench-serve workload contract path - Destination: `workload` - Required: `false` - Default: `None` - Choices: `not restricted` - Action: `store` - Source: [L1904-L1909](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1904-L1909) #### `--output` Qualification request manifest path - Destination: `output` - Required: `false` - Default: `None` - Choices: `not restricted` - Action: `store` - Source: [L1910-L1915](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1910-L1915) #### `--result-output` Result output path passed to bench-serve - Destination: `result_output` - Required: `false` - Default: `None` - Choices: `not restricted` - Action: `store` - Source: [L1916-L1921](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1916-L1921) #### `--repetitions` Number of repetitions per benchmark sweep configuration - Destination: `repetitions` - Required: `false` - Default: `None` - Choices: `not restricted` - Action: `store` - Source: [L1922-L1927](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1922-L1927) #### `--dry-run` Write or print the qualification command without running it - Destination: `dry_run` - Required: `false` - Default: `argparse default` - Choices: `not restricted` - Action: `store_true` - Source: [L1928-L1932](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1928-L1932) #### `--extra-arg` Extra argument passed through to bench-serve. Repeatable. - Destination: `extra_arg` - Required: `false` - Default: `[]` - Choices: `not restricted` - Action: `append` - Source: [L1933-L1938](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1933-L1938) ### Parser `bench_serve_parser` #### `--url` Base URL of the running server (default: http://127.0.0.1:8080) - Destination: `url` - Required: `false` - Default: `http://127.0.0.1:8080` - Choices: `not restricted` - Action: `store` - Source: [L1944-L1949](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1944-L1949) #### `--model` Model ID to benchmark (default: auto-detected from server) - Destination: `model` - Required: `false` - Default: `None` - Choices: `not restricted` - Action: `store` - Source: [L1950-L1955](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1950-L1955) #### `--workload` Path to a declarative workload JSON file. When set, bench-serve runs contract-style cases with per-case quality checks and comparison-only policy timeouts instead of the prompt sweep. - Destination: `workload` - Required: `false` - Default: `None` - Choices: `not restricted` - Action: `store` - Source: [L1956-L1965](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1956-L1965) #### `--prompts` Comma-separated prompt set names or paths (default: short,medium,long) - Destination: `prompts` - Required: `false` - Default: `short,medium,long` - Choices: `not restricted` - Action: `store` - Source: [L1966-L1971](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1966-L1971) #### `--prompt-file` Path to an additional prompt file (JSON list of message dicts) - Destination: `prompt_file` - Required: `false` - Default: `None` - Choices: `not restricted` - Action: `store` - Source: [L1972-L1977](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1972-L1977) #### `--system-prompt-file` Path to a text file whose contents are prepended as a system message to every prompt. Use this together with --warm-prompts to benchmark the warm-cache path (the warmup populates the prefix cache with this same system, so every request in the bench hits the cache). - Destination: `system_prompt_file` - Required: `false` - Default: `None` - Choices: `not restricted` - Action: `store` - Source: [L1978-L1989](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1978-L1989) #### `--skip-preflight-token-count` Skip the pre-flight max_tokens=1 request that counts prompt tokens per prompt set. That request populates the prefix cache with the full prompt, which defeats cold-vs-warm comparisons. Auto-enabled when --system-prompt-file is set; pass this flag explicitly to force-enable regardless. - Destination: `skip_preflight_token_count` - Required: `false` - Default: `argparse default` - Choices: `not restricted` - Action: `store_true` - Source: [L1990-L2000](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L1990-L2000) #### `--concurrency` Comma-separated concurrency levels to sweep (default: 1,4) - Destination: `concurrency` - Required: `false` - Default: `1,4` - Choices: `not restricted` - Action: `store` - Source: [L2001-L2006](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L2001-L2006) #### `--max-tokens` Maximum tokens to generate per request (default: 256) - Destination: `max_tokens` - Required: `false` - Default: `256` - Choices: `not restricted` - Action: `store` - Source: [L2007-L2012](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L2007-L2012) #### `--repetitions` Number of repetitions per sweep configuration or workload case (default: 3) - Destination: `repetitions` - Required: `false` - Default: `3` - Choices: `not restricted` - Action: `store` - Source: [L2013-L2018](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L2013-L2018) #### `--warmup` Warmup rounds before the first measured repetition (default: 1) - Destination: `warmup` - Required: `false` - Default: `1` - Choices: `not restricted` - Action: `store` - Source: [L2019-L2024](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L2019-L2024) #### `--enable-thinking` Enable thinking mode: "true", "false", or "true,false" to sweep both - Destination: `enable_thinking` - Required: `false` - Default: `None` - Choices: `not restricted` - Action: `store` - Source: [L2025-L2030](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L2025-L2030) #### `--extra-body` Comma-separated JSON dicts to pass as extra body parameters - Destination: `extra_body` - Required: `false` - Default: `None` - Choices: `not restricted` - Action: `store` - Source: [L2031-L2036](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L2031-L2036) #### `--output` File path to write results to (default: stdout) - Destination: `output` - Required: `false` - Default: `None` - Choices: `not restricted` - Action: `store` - Source: [L2037-L2042](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L2037-L2042) #### `--format` Output format (auto = table for prompt sweeps, json for workloads; sqlite requires --output) - Destination: `format` - Required: `false` - Default: `auto` - Choices: `['auto', 'table', 'json', 'csv', 'sql', 'sqlite']` - Action: `store` - Source: [L2043-L2052](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L2043-L2052) #### `--validate` Validate responses (default: true) - Destination: `validate` - Required: `false` - Default: `true` - Choices: `['true', 'false']` - Action: `store` - Source: [L2053-L2059](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L2053-L2059) #### `--scrape-metrics` Scrape /metrics before and after each run (default: true) - Destination: `scrape_metrics` - Required: `false` - Default: `true` - Choices: `['true', 'false']` - Action: `store` - Source: [L2060-L2066](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L2060-L2066) #### `--include-content` Include full generated content in workload JSON output - Destination: `include_content` - Required: `false` - Default: `argparse default` - Choices: `not restricted` - Action: `store_true` - Source: [L2067-L2071](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L2067-L2071) #### `--request-timeout-s` HTTP transport timeout for workload mode in seconds (default: 300). Use 0 to disable; product policy timeouts belong in the workload. - Destination: `request_timeout_s` - Required: `false` - Default: `300.0` - Choices: `not restricted` - Action: `store` - Source: [L2072-L2080](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L2072-L2080) #### `--cache-policy` Workload cache handling (default: workload defaults or preserve). Use before-case for cold, uncontaminated per-case qualification. Workload JSON may also spell these with underscores. - Destination: `cache_policy` - Required: `false` - Default: `None` - Choices: `['preserve', 'before-run', 'before-case']` - Action: `store` - Source: [L2081-L2091](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L2081-L2091) #### `--tag` Optional tag string stored in every result row - Destination: `tag` - Required: `false` - Default: `None` - Choices: `not restricted` - Action: `store` - Source: [L2092-L2097](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L2092-L2097) #### `--override-field` Override result fields as key=value pairs (e.g. chip=M4Pro) - Destination: `override_field` - Required: `false` - Default: `[]` - Choices: `not restricted` - Action: `store` - Source: [L2098-L2103](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/cli.py#L2098-L2103) ## `vllm_mlx.gradio_app.main` ### Parser `parser` #### `--server-url` URL of the vllm-mlx server (default: http://localhost:8000) - Destination: `server_url` - Required: `false` - Default: `http://localhost:8000` - Choices: `not restricted` - Action: `store` - Source: [L280-L285](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/gradio_app.py#L280-L285) #### `--port` Port for Gradio interface (default: 7860) - Destination: `port` - Required: `false` - Default: `7860` - Choices: `not restricted` - Action: `store` - Source: [L286-L291](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/gradio_app.py#L286-L291) #### `--share` Create a public share link - Destination: `share` - Required: `false` - Default: `argparse default` - Choices: `not restricted` - Action: `store_true` - Source: [L292-L296](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/gradio_app.py#L292-L296) #### `--max-tokens` Maximum tokens to generate (default: 2048) - Destination: `max_tokens` - Required: `false` - Default: `2048` - Choices: `not restricted` - Action: `store` - Source: [L297-L302](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/gradio_app.py#L297-L302) #### `--temperature` Sampling temperature (default: 0.7) - Destination: `temperature` - Required: `false` - Default: `0.7` - Choices: `not restricted` - Action: `store` - Source: [L303-L308](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/gradio_app.py#L303-L308) #### `--served-model-name` Model name to send in /v1/chat/completions requests (default: default) - Destination: `served_model_name` - Required: `false` - Default: `default` - Choices: `not restricted` - Action: `store` - Source: [L309-L316](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/gradio_app.py#L309-L316) #### `--text-only` Use text-only mode (no image/video support, faster for LLM-only models) - Destination: `text_only` - Required: `false` - Default: `argparse default` - Choices: `not restricted` - Action: `store_true` - Source: [L317-L321](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/gradio_app.py#L317-L321) ## `vllm_mlx.gradio_text_app.main` ### Parser `parser` #### `--server-url` URL of the vllm-mlx server (default: http://localhost:8000) - Destination: `server_url` - Required: `false` - Default: `http://localhost:8000` - Choices: `not restricted` - Action: `store` - Source: [L136-L141](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/gradio_text_app.py#L136-L141) #### `--port` Port for Gradio interface (default: 7861) - Destination: `port` - Required: `false` - Default: `7861` - Choices: `not restricted` - Action: `store` - Source: [L142-L147](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/gradio_text_app.py#L142-L147) #### `--share` Create a public share link - Destination: `share` - Required: `false` - Default: `argparse default` - Choices: `not restricted` - Action: `store_true` - Source: [L148-L152](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/gradio_text_app.py#L148-L152) #### `--max-tokens` Maximum tokens to generate (default: 512) - Destination: `max_tokens` - Required: `false` - Default: `512` - Choices: `not restricted` - Action: `store` - Source: [L153-L158](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/gradio_text_app.py#L153-L158) #### `--temperature` Sampling temperature (default: 0.7) - Destination: `temperature` - Required: `false` - Default: `0.7` - Choices: `not restricted` - Action: `store` - Source: [L159-L164](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/gradio_text_app.py#L159-L164) #### `--served-model-name` Model name to send in /v1/chat/completions requests (default: default) - Destination: `served_model_name` - Required: `false` - Default: `default` - Choices: `not restricted` - Action: `store` - Source: [L165-L172](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/gradio_text_app.py#L165-L172) ## `vllm_mlx.server.create_parser` ### Parser `parser` #### `--model` Model to load (HuggingFace model name or local path) - Destination: `model` - Required: `false` - Default: `mlx-community/Llama-3.2-3B-Instruct-4bit` - Choices: `not restricted` - Action: `store` - Source: [L6728-L6733](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L6728-L6733) #### `--host` Host to bind to (default: localhost; use 0.0.0.0 to expose externally) - Destination: `host` - Required: `false` - Default: `127.0.0.1` - Choices: `not restricted` - Action: `store` - Source: [L6734-L6739](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L6734-L6739) #### `--port` Port to bind to - Destination: `port` - Required: `false` - Default: `8000` - Choices: `not restricted` - Action: `store` - Source: [L6740-L6745](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L6740-L6745) #### `--mllm` Force loading as MLLM (multimodal language model) - Destination: `mllm` - Required: `false` - Default: `argparse default` - Choices: `not restricted` - Action: `store_true` - Source: [L6746-L6750](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L6746-L6750) #### `--trust-remote-code` Allow HuggingFace remote code execution during model/tokenizer loading - Destination: `trust_remote_code` - Required: `false` - Default: `argparse default` - Choices: `not restricted` - Action: `store_true` - Source: [L6751-L6755](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L6751-L6755) #### `--continuous-batching` Enable continuous batching for multiple concurrent users - Destination: `continuous_batching` - Required: `false` - Default: `argparse default` - Choices: `not restricted` - Action: `store_true` - Source: [L6756-L6760](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L6756-L6760) #### `--mllm-draft-model` Path to an mlx-vlm MLLM draft/assistant model. - Destination: `mllm_draft_model` - Required: `false` - Default: `None` - Choices: `not restricted` - Action: `store` - Source: [L6761-L6766](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L6761-L6766) #### `--mllm-draft-kind` mlx-vlm draft kind for --mllm-draft-model. - Destination: `mllm_draft_kind` - Required: `false` - Default: `None` - Choices: `['mtp']` - Action: `store` - Source: [L6767-L6773](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L6767-L6773) #### `--mllm-draft-block-size` Draft block size passed to mlx-vlm for --mllm-draft-model. - Destination: `mllm_draft_block_size` - Required: `false` - Default: `None` - Choices: `not restricted` - Action: `store` - Source: [L6774-L6779](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L6774-L6779) #### `--mcp-config` Path to MCP configuration file (JSON/YAML) - Destination: `mcp_config` - Required: `false` - Default: `None` - Choices: `not restricted` - Action: `store` - Source: [L6780-L6785](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L6780-L6785) #### `--max-tokens` Default max tokens for generation - Destination: `max_tokens` - Required: `false` - Default: `32768` - Choices: `not restricted` - Action: `store` - Source: [L6786-L6791](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L6786-L6791) #### `--max-request-tokens` Maximum max_tokens accepted from API clients (default: 32768) - Destination: `max_request_tokens` - Required: `false` - Default: `32768` - Choices: `not restricted` - Action: `store` - Source: [L6792-L6797](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L6792-L6797) #### `--api-key` API key for authentication (if not set, no auth required) - Destination: `api_key` - Required: `false` - Default: `None` - Choices: `not restricted` - Action: `store` - Source: [L6798-L6803](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L6798-L6803) #### `--timeout` Default request timeout in seconds (default: 300) - Destination: `timeout` - Required: `false` - Default: `300.0` - Choices: `not restricted` - Action: `store` - Source: [L6804-L6809](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L6804-L6809) #### `--enable-metrics` Expose Prometheus metrics on /metrics (disabled by default) - Destination: `enable_metrics` - Required: `false` - Default: `argparse default` - Choices: `not restricted` - Action: `store_true` - Source: [L6810-L6814](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L6810-L6814) #### `--auto-unload-idle-seconds` Unload the main model after this many idle seconds (0 = disabled) - Destination: `auto_unload_idle_seconds` - Required: `false` - Default: `0.0` - Choices: `not restricted` - Action: `store` - Source: [L6815-L6820](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L6815-L6820) #### `--lazy-load-model` Register the main model at startup but defer loading until first request - Destination: `lazy_load_model` - Required: `false` - Default: `argparse default` - Choices: `not restricted` - Action: `store_true` - Source: [L6821-L6825](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L6821-L6825) #### `--rate-limit` Rate limit requests per minute per client (0 = disabled) - Destination: `rate_limit` - Required: `false` - Default: `0` - Choices: `not restricted` - Action: `store` - Source: [L6826-L6831](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L6826-L6831) #### `--reasoning-parser` f"Enable reasoning content extraction with specified parser. Options: {', '.join(reasoning_choices)}." - Destination: `reasoning_parser` - Required: `false` - Default: `None` - Choices: `reasoning_choices` - Action: `store` - Source: [L6836-L6845](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L6836-L6845) #### `--embedding-model` Pre-load an embedding model at startup (e.g. mlx-community/all-MiniLM-L6-v2-4bit) - Destination: `embedding_model` - Required: `false` - Default: `None` - Choices: `not restricted` - Action: `store` - Source: [L6846-L6851](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L6846-L6851) #### `--default-temperature` Default temperature for generation when not specified in request - Destination: `default_temperature` - Required: `false` - Default: `None` - Choices: `not restricted` - Action: `store` - Source: [L6852-L6857](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L6852-L6857) #### `--default-top-p` Default top_p for generation when not specified in request - Destination: `default_top_p` - Required: `false` - Default: `None` - Choices: `not restricted` - Action: `store` - Source: [L6858-L6863](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L6858-L6863) #### `--default-chat-template-kwargs` Default chat template kwargs to apply to all requests when request chat_template_kwargs is omitted or empty; empty request kwargs use existing server defaults (JSON object, e.g. {"enable_thinking": false}) - Destination: `default_chat_template_kwargs` - Required: `false` - Default: `None` - Choices: `not restricted` - Action: `store` - Source: [L6864-L6873](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L6864-L6873) #### `--default-top-k` Default top_k for generation when not specified in request - Destination: `default_top_k` - Required: `false` - Default: `None` - Choices: `not restricted` - Action: `store` - Source: [L6874-L6879](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L6874-L6879) #### `--default-min-p` Default min_p for generation when not specified in request - Destination: `default_min_p` - Required: `false` - Default: `None` - Choices: `not restricted` - Action: `store` - Source: [L6880-L6885](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L6880-L6885) #### `--default-presence-penalty` Default presence_penalty for generation when not specified in request - Destination: `default_presence_penalty` - Required: `false` - Default: `None` - Choices: `not restricted` - Action: `store` - Source: [L6886-L6891](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L6886-L6891) #### `--default-repetition-penalty` Default repetition_penalty for generation when not specified in request - Destination: `default_repetition_penalty` - Required: `false` - Default: `None` - Choices: `not restricted` - Action: `store` - Source: [L6892-L6899](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L6892-L6899) #### `--max-audio-upload-mb` Maximum size of uploaded audio files in MiB (default: 25) - Destination: `max_audio_upload_mb` - Required: `false` - Default: `DEFAULT_MAX_AUDIO_UPLOAD_MB` - Choices: `not restricted` - Action: `store` - Source: [L6900-L6905](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L6900-L6905) #### `--max-tts-input-chars` Maximum number of characters accepted by /v1/audio/speech (default: 4096) - Destination: `max_tts_input_chars` - Required: `false` - Default: `DEFAULT_MAX_TTS_INPUT_CHARS` - Choices: `not restricted` - Action: `store` - Source: [L6906-L6911](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/server.py#L6906-L6911)