Complete CLI option inventory¶
This page is generated from every add_argument declaration in the runtime, maintenance scripts, and runnable examples. The hand-written CLI guide explains supported workflows.
examples.audio_separation_example.main¶
Parser parser¶
audio¶
Input audio file (mp3, wav, etc.)
- Destination:
audio - Required:
true - Default:
argparse default - Choices:
not restricted - Action:
store - Source: L37-L37
--description, -d¶
What to isolate: speech, music, singing, etc. (default: speech)
- Destination:
description - Required:
false - Default:
speech - Choices:
not restricted - Action:
store - Source: L38-L39
--output, -o¶
Output file for isolated audio (default: input_voice.wav)
- Destination:
output - Required:
false - Default:
None - Choices:
not restricted - Action:
store - Source: L40-L41
--background, -b¶
Output file for background audio (optional)
- Destination:
background - Required:
false - Default:
None - Choices:
not restricted - Action:
store - Source: L42-L43
--model, -m¶
SAM-Audio model to use
- Destination:
model - Required:
false - Default:
mlx-community/sam-audio-large-fp16 - Choices:
not restricted - Action:
store - Source: L44-L45
--chunk, -c¶
Process in chunks of N seconds (for long audio)
- Destination:
chunk - Required:
false - Default:
None - Choices:
not restricted - Action:
store - Source: L46-L47
--play, -p¶
Play result after processing (macOS)
- Destination:
play - Required:
false - Default:
argparse default - Choices:
not restricted - Action:
store_true - Source: L48-L49
examples.benchmark_audio.main¶
Parser parser¶
--tts¶
Run TTS benchmarks
- Destination:
tts - Required:
false - Default:
argparse default - Choices:
not restricted - Action:
store_true - Source: L302-L302
--stt¶
Run STT benchmarks
- Destination:
stt - Required:
false - Default:
argparse default - Choices:
not restricted - Action:
store_true - Source: L303-L303
--audio¶
Audio file for STT benchmark
- Destination:
audio - Required:
false - Default:
argparse default - Choices:
not restricted - Action:
store - Source: L304-L304
--all¶
Run all benchmarks
- Destination:
all - Required:
false - Default:
argparse default - Choices:
not restricted - Action:
store_true - Source: L305-L305
examples.closed_captions.main¶
Parser parser¶
--model, -m¶
No argparse help text is declared.
- Destination:
model - Required:
false - Default:
whisper-large-v3 - Choices:
not restricted - Action:
store - Source: L150-L150
--language, -l¶
es, en, etc.
- Destination:
language - Required:
false - Default:
None - Choices:
not restricted - Action:
store - Source: L151-L151
--chunk, -c¶
Chunk size (default: 3.0s)
- Destination:
chunk - Required:
false - Default:
3.0 - Choices:
not restricted - Action:
store - Source: L152-L152
examples.mic_live.main¶
Parser parser¶
--model, -m¶
Model (whisper-small, whisper-medium, parakeet)
- Destination:
model - Required:
false - Default:
whisper-small - Choices:
not restricted - Action:
store - Source: L206-L207
--language, -l¶
Language code (en, es, etc.)
- Destination:
language - Required:
false - Default:
argparse default - Choices:
not restricted - Action:
store - Source: L208-L208
--sensitivity, -s¶
Mic sensitivity 0.01-0.05 (default: 0.015)
- Destination:
sensitivity - Required:
false - Default:
0.015 - Choices:
not restricted - Action:
store - Source: L209-L210
examples.mic_realtime.main¶
Parser parser¶
--model, -m¶
Model to use (default: whisper-small)
- Destination:
model - Required:
false - Default:
whisper-small - Choices:
not restricted - Action:
store - Source: L186-L187
--chunk, -c¶
Chunk duration in seconds (default: 3.0)
- Destination:
chunk - Required:
false - Default:
3.0 - Choices:
not restricted - Action:
store - Source: L188-L189
--language, -l¶
Language code (e.g., en, es)
- Destination:
language - Required:
false - Default:
argparse default - Choices:
not restricted - Action:
store - Source: L190-L190
--list-models¶
List available models
- Destination:
list_models - Required:
false - Default:
argparse default - Choices:
not restricted - Action:
store_true - Source: L191-L191
examples.mic_transcribe.main¶
Parser parser¶
--duration, -d¶
Recording duration in seconds
- Destination:
duration - Required:
false - Default:
argparse default - Choices:
not restricted - Action:
store - Source: L115-L115
--model, -m¶
Model: whisper-small, whisper-medium, whisper-large-v3, parakeet
- Destination:
model - Required:
false - Default:
whisper-small - Choices:
not restricted - Action:
store - Source: L116-L117
--language, -l¶
Language code (e.g., en, es). Auto-detect if not set
- Destination:
language - Required:
false - Default:
argparse default - Choices:
not restricted - Action:
store - Source: L118-L118
--continuous, -c¶
Continuous mode: keep recording and transcribing
- Destination:
continuous - Required:
false - Default:
argparse default - Choices:
not restricted - Action:
store_true - Source: L119-L120
--save, -s¶
Save recorded audio to this file
- Destination:
save - Required:
false - Default:
argparse default - Choices:
not restricted - Action:
store - Source: L121-L121
--list-models¶
List available models
- Destination:
list_models - Required:
false - Default:
argparse default - Choices:
not restricted - Action:
store_true - Source: L122-L122
--list-devices¶
List audio input devices
- Destination:
list_devices - Required:
false - Default:
argparse default - Choices:
not restricted - Action:
store_true - Source: L123-L123
examples.mllm_benchmark.main¶
Parser parser¶
--server-url¶
URL of the vllm-mlx server
- Destination:
server_url - Required:
false - Default:
http://localhost:8000 - Choices:
not restricted - Action:
store - Source: L390-L395
--output¶
Save results to JSON file
- Destination:
output - Required:
false - Default:
None - Choices:
not restricted - Action:
store - Source: L396-L401
--warmup¶
Number of warmup runs (default: 1)
- Destination:
warmup - Required:
false - Default:
1 - Choices:
not restricted - Action:
store - Source: L402-L407
--quick¶
Run quick benchmark with fewer resolutions
- Destination:
quick - Required:
false - Default:
argparse default - Choices:
not restricted - Action:
store_true - Source: L408-L412
examples.test_batching.main¶
Parser parser¶
--model¶
Model to use
- Destination:
model - Required:
false - Default:
mlx-community/Llama-3.2-1B-Instruct-4bit - Choices:
not restricted - Action:
store - Source: L118-L123
--num-requests¶
Number of concurrent requests
- Destination:
num_requests - Required:
false - Default:
5 - Choices:
not restricted - Action:
store - Source: L124-L129
--max-tokens¶
Max tokens per request
- Destination:
max_tokens - Required:
false - Default:
30 - Choices:
not restricted - Action:
store - Source: L130-L135
--temperature¶
Sampling temperature
- Destination:
temperature - Required:
false - Default:
0.7 - Choices:
not restricted - Action:
store - Source: L136-L141
examples.test_openai_compatibility.main¶
Parser parser¶
--server-url¶
URL of the vllm-mlx server (default: http://localhost:8000)
- Destination:
server_url - Required:
false - Default:
http://localhost:8000 - Choices:
not restricted - Action:
store - Source: L703-L708
--no-image¶
Skip image tests
- Destination:
no_image - Required:
false - Default:
argparse default - Choices:
not restricted - Action:
store_true - Source: L709-L713
--no-video¶
Skip video tests
- Destination:
no_video - Required:
false - Default:
argparse default - Choices:
not restricted - Action:
store_true - Source: L714-L718
examples.test_video.main¶
Parser parser¶
--video¶
Path to video file (will create test video if not provided)
- Destination:
video - Required:
false - Default:
argparse default - Choices:
not restricted - Action:
store - Source: L271-L275
--video-url¶
URL to a video file to test URL support
- Destination:
video_url - Required:
false - Default:
argparse default - Choices:
not restricted - Action:
store - Source: L276-L280
--model¶
VLM model to use
- Destination:
model - Required:
false - Default:
mlx-community/Qwen3-VL-4B-Instruct-3bit - Choices:
not restricted - Action:
store - Source: L281-L286
--extract-only¶
Only test frame extraction (no model loading)
- Destination:
extract_only - Required:
false - Default:
argparse default - Choices:
not restricted - Action:
store_true - Source: L287-L291
--create-test-video¶
Create synthetic test video instead of downloading
- Destination:
create_test_video - Required:
false - Default:
argparse default - Choices:
not restricted - Action:
store_true - Source: L292-L296
--url-only¶
Only test video URL support (requires --video-url)
- Destination:
url_only - Required:
false - Default:
argparse default - Choices:
not restricted - Action:
store_true - Source: L297-L301
examples.tts_example.main¶
Parser parser¶
text¶
Text to synthesize
- Destination:
text - Required:
true - Default:
argparse default - Choices:
not restricted - Action:
store - Source: L48-L48
--voice, -v¶
Voice ID (default: af_heart)
- Destination:
voice - Required:
false - Default:
af_heart - Choices:
not restricted - Action:
store - Source: L49-L49
--lang, -l¶
Language code: a=English, e/es=Spanish, f=French, etc.
- Destination:
lang - Required:
false - Default:
a - Choices:
not restricted - Action:
store - Source: L50-L50
--speed, -s¶
Speech speed 0.5-2.0 (default: 1.0)
- Destination:
speed - Required:
false - Default:
1.0 - Choices:
not restricted - Action:
store - Source: L51-L51
--output, -o¶
Output file (default: output.wav)
- Destination:
output - Required:
false - Default:
output.wav - Choices:
not restricted - Action:
store - Source: L52-L52
--model, -m¶
TTS model
- Destination:
model - Required:
false - Default:
mlx-community/Kokoro-82M-bf16 - Choices:
not restricted - Action:
store - Source: L53-L53
--list-voices¶
List available voices
- Destination:
list_voices - Required:
false - Default:
argparse default - Choices:
not restricted - Action:
store_true - Source: L54-L54
--list-languages¶
List available languages
- Destination:
list_languages - Required:
false - Default:
argparse default - Choices:
not restricted - Action:
store_true - Source: L55-L55
--play, -p¶
Play audio after generation (macOS)
- Destination:
play - Required:
false - Default:
argparse default - Choices:
not restricted - Action:
store_true - Source: L56-L56
examples.tts_multilingual.main¶
Parser parser¶
text¶
Text to synthesize
- Destination:
text - Required:
true - Default:
argparse default - Choices:
not restricted - Action:
store - Source: L261-L261
--model, -m¶
Model: kokoro, chatterbox, vibevoice, voxcpm, or 'auto'
- Destination:
model - Required:
false - Default:
auto - Choices:
not restricted - Action:
store - Source: L262-L263
--lang, -l¶
Language code: en, es, fr, ja, zh, etc.
- Destination:
lang - Required:
false - Default:
en - Choices:
not restricted - Action:
store - Source: L264-L265
--voice, -v¶
Voice ID (model-specific)
- Destination:
voice - Required:
false - Default:
None - Choices:
not restricted - Action:
store - Source: L266-L267
--speed, -s¶
Speech speed 0.5-2.0
- Destination:
speed - Required:
false - Default:
1.0 - Choices:
not restricted - Action:
store - Source: L268-L269
--output, -o¶
Output file
- Destination:
output - Required:
false - Default:
output.wav - Choices:
not restricted - Action:
store - Source: L270-L271
--play, -p¶
Play audio after generation (macOS)
- Destination:
play - Required:
false - Default:
argparse default - Choices:
not restricted - Action:
store_true - Source: L272-L273
--list-models¶
List available models
- Destination:
list_models - Required:
false - Default:
argparse default - Choices:
not restricted - Action:
store_true - Source: L274-L275
--list-languages¶
List supported languages
- Destination:
list_languages - Required:
false - Default:
argparse default - Choices:
not restricted - Action:
store_true - Source: L276-L277
examples.video_benchmark.main¶
Parser parser¶
--model¶
VLM model to use
- Destination:
model - Required:
false - Default:
mlx-community/Qwen3-VL-4B-Instruct-3bit - Choices:
not restricted - Action:
store - Source: L500-L505
--video¶
Path to local video file
- Destination:
video - Required:
false - Default:
None - Choices:
not restricted - Action:
store - Source: L506-L511
--video-url¶
URL to download video from
- Destination:
video_url - Required:
false - Default:
None - Choices:
not restricted - Action:
store - Source: L512-L517
--duration¶
Duration of synthetic test video (seconds)
- Destination:
duration - Required:
false - Default:
10.0 - Choices:
not restricted - Action:
store - Source: L518-L523
--warmup¶
Number of warmup runs
- Destination:
warmup - Required:
false - Default:
1 - Choices:
not restricted - Action:
store - Source: L524-L529
--quick¶
Run quick benchmark with fewer configurations
- Destination:
quick - Required:
false - Default:
argparse default - Choices:
not restricted - Action:
store_true - Source: L530-L534
--output¶
Save results to JSON file
- Destination:
output - Required:
false - Default:
None - Choices:
not restricted - Action:
store - Source: L535-L540
scripts.add_mtp_weights.main¶
Parser parser¶
--mlx-model-path¶
f'Path to MLX model directory (default: {DEFAULT_MLX_MODEL})'
- Destination:
mlx_model_path - Required:
false - Default:
DEFAULT_MLX_MODEL - Choices:
not restricted - Action:
store - Source: L243-L248
--source-model¶
f'HuggingFace model to download MTP shard from (default: {DEFAULT_SOURCE_MODEL})'
- Destination:
source_model - Required:
false - Default:
DEFAULT_SOURCE_MODEL - Choices:
not restricted - Action:
store - Source: L249-L254
--download-dir¶
Directory to download MTP shard to (default: temp dir)
- Destination:
download_dir - Required:
false - Default:
None - Choices:
not restricted - Action:
store - Source: L255-L260
--bits¶
Quantization bits (default: 6, matching 6-bit model)
- Destination:
bits - Required:
false - Default:
6 - Choices:
not restricted - Action:
store - Source: L261-L266
--skip-download¶
Skip download (use existing shard)
- Destination:
skip_download - Required:
false - Default:
argparse default - Choices:
not restricted - Action:
store_true - Source: L267-L271
scripts.add_mtp_weights_qwen35.main¶
Parser parser¶
--mlx-model-path¶
Path to MLX model directory (HF cache or direct path)
- Destination:
mlx_model_path - Required:
true - Default:
argparse default - Choices:
not restricted - Action:
store - Source: L327-L332
--source-model¶
HuggingFace BF16 model to download MTP shards from (e.g., Qwen/Qwen3.5-122B-A10B)
- Destination:
source_model - Required:
true - Default:
argparse default - Choices:
not restricted - Action:
store - Source: L333-L338
--download-dir¶
Directory to download shards to (default: temp dir)
- Destination:
download_dir - Required:
false - Default:
None - Choices:
not restricted - Action:
store - Source: L339-L344
--skip-download¶
Skip download (use existing shards in download-dir)
- Destination:
skip_download - Required:
false - Default:
argparse default - Choices:
not restricted - Action:
store_true - Source: L345-L349
--keep-shards¶
Don't delete downloaded BF16 shards after extraction
- Destination:
keep_shards - Required:
false - Default:
argparse default - Choices:
not restricted - Action:
store_true - Source: L350-L354
--no-quantize¶
Save MTP weights in BF16 (no quantization). Required for correct MTP predictions.
- Destination:
no_quantize - Required:
false - Default:
argparse default - Choices:
not restricted - Action:
store_true - Source: L355-L359
scripts.gen_api_reference.main¶
Parser parser¶
--check¶
fail instead of writing when generated references are stale
- Destination:
check - Required:
false - Default:
argparse default - Choices:
not restricted - Action:
store_true - Source: L93-L97
vllm_mlx.benchmark.main¶
Parser parser¶
--model¶
Model name (HuggingFace model name or local path)
- Destination:
model - Required:
true - Default:
argparse default - Choices:
not restricted - Action:
store - Source: L1468-L1473
--prompts¶
Number of prompts to benchmark for LLM (default: 5)
- Destination:
prompts - Required:
false - Default:
5 - Choices:
not restricted - Action:
store - Source: L1474-L1479
--max-tokens¶
Maximum tokens to generate per prompt (default: 256)
- Destination:
max_tokens - Required:
false - Default:
256 - Choices:
not restricted - Action:
store - Source: L1480-L1485
--temperature¶
Sampling temperature (default: 0.7)
- Destination:
temperature - Required:
false - Default:
0.7 - Choices:
not restricted - Action:
store - Source: L1486-L1491
--warmup¶
Number of warmup runs (default: 1)
- Destination:
warmup - Required:
false - Default:
1 - Choices:
not restricted - Action:
store - Source: L1492-L1497
--output¶
Output file for JSON results
- Destination:
output - Required:
false - Default:
None - Choices:
not restricted - Action:
store - Source: L1498-L1503
--mllm¶
Force MLLM benchmark mode (auto-detected by default)
- Destination:
mllm - Required:
false - Default:
argparse default - Choices:
not restricted - Action:
store_true - Source: L1504-L1508
--quick¶
Quick benchmark with fewer configurations
- Destination:
quick - Required:
false - Default:
argparse default - Choices:
not restricted - Action:
store_true - Source: L1509-L1513
--video¶
Run video benchmark instead of image benchmark (for MLLM models)
- Destination:
video - Required:
false - Default:
argparse default - Choices:
not restricted - Action:
store_true - Source: L1515-L1519
--video-url¶
URL of video to use for benchmark (default: Big Buck Bunny 10s)
- Destination:
video_url - Required:
false - Default:
None - Choices:
not restricted - Action:
store - Source: L1520-L1525
--video-path¶
Local path to video file for benchmark
- Destination:
video_path - Required:
false - Default:
None - Choices:
not restricted - Action:
store - Source: L1526-L1531
vllm_mlx.cli.create_parser¶
Parser serve_parser¶
model¶
Model to serve
- Destination:
model - Required:
true - Default:
argparse default - Choices:
not restricted - Action:
store - Source: L1008-L1008
--models-config¶
YAML file describing a registry of models for lazy multi-model serving
- Destination:
models_config - Required:
false - Default:
None - Choices:
not restricted - Action:
store - Source: L1009-L1014
--served-model-name¶
The model name used in the API. If not specified, the model argument is used.
- Destination:
served_model_name - Required:
false - Default:
None - Choices:
not restricted - Action:
store - Source: L1015-L1020
--host¶
Host to bind (default: localhost; use 0.0.0.0 to expose externally)
- Destination:
host - Required:
false - Default:
127.0.0.1 - Choices:
not restricted - Action:
store - Source: L1021-L1026
--port¶
Port to bind
- Destination:
port - Required:
false - Default:
8000 - Choices:
not restricted - Action:
store - Source: L1027-L1027
--max-num-seqs¶
Max concurrent sequences
- Destination:
max_num_seqs - Required:
false - Default:
256 - Choices:
not restricted - Action:
store - Source: L1028-L1030
--prefill-batch-size¶
Prefill batch size
- Destination:
prefill_batch_size - Required:
false - Default:
8 - Choices:
not restricted - Action:
store - Source: L1031-L1033
--completion-batch-size¶
Completion batch size
- Destination:
completion_batch_size - Required:
false - Default:
32 - Choices:
not restricted - Action:
store - Source: L1034-L1036
--mllm-prefill-step-size¶
Override MLLM prefill-step guard (0=use MLLM default: 1024)
- Destination:
mllm_prefill_step_size - Required:
false - Default:
0 - Choices:
not restricted - Action:
store - Source: L1037-L1042
--enable-prefix-cache¶
Enable prefix caching for repeated prompts (default: enabled)
- Destination:
enable_prefix_cache - Required:
false - Default:
True - Choices:
not restricted - Action:
store_true - Source: L1043-L1048
--disable-prefix-cache¶
Disable prefix caching
- Destination:
disable_prefix_cache - Required:
false - Default:
argparse default - Choices:
not restricted - Action:
store_true - Source: L1049-L1053
--prefix-cache-size¶
Max entries in prefix cache (default: 100, legacy mode only)
- Destination:
prefix_cache_size - Required:
false - Default:
100 - Choices:
not restricted - Action:
store - Source: L1054-L1059
--cache-memory-mb¶
Cache memory limit in MB (default: auto-detect ~20%% of RAM)
- Destination:
cache_memory_mb - Required:
false - Default:
None - Choices:
not restricted - Action:
store - Source: L1061-L1066
--cache-memory-percent¶
Fraction of available RAM for cache if auto-detecting (default: 0.20)
- Destination:
cache_memory_percent - Required:
false - Default:
0.2 - Choices:
not restricted - Action:
store - Source: L1067-L1072
--no-memory-aware-cache¶
Disable memory-aware cache, use legacy entry-count based cache
- Destination:
no_memory_aware_cache - Required:
false - Default:
argparse default - Choices:
not restricted - Action:
store_true - Source: L1073-L1077
--kv-cache-quantization¶
Quantize stored KV caches to reduce memory (8-bit by default)
- Destination:
kv_cache_quantization - Required:
false - Default:
argparse default - Choices:
not restricted - Action:
store_true - Source: L1079-L1083
--kv-cache-quantization-bits¶
Bit width for KV cache quantization (default: 8)
- Destination:
kv_cache_quantization_bits - Required:
false - Default:
8 - Choices:
[4, 8] - Action:
store - Source: L1084-L1090
--kv-cache-quantization-group-size¶
Group size for KV cache quantization (default: 64)
- Destination:
kv_cache_quantization_group_size - Required:
false - Default:
64 - Choices:
not restricted - Action:
store - Source: L1091-L1096
--kv-cache-min-quantize-tokens¶
Minimum tokens for quantization to apply (default: 256)
- Destination:
kv_cache_min_quantize_tokens - Required:
false - Default:
256 - Choices:
not restricted - Action:
store - Source: L1097-L1102
--ssd-cache-dir¶
Directory for SSD KV cache tier (default: disabled)
- Destination:
ssd_cache_dir - Required:
false - Default:
None - Choices:
not restricted - Action:
store - Source: L1104-L1109
--ssd-cache-max-gb¶
Maximum SSD cache size in GB (default: 10.0)
- Destination:
ssd_cache_max_gb - Required:
false - Default:
10.0 - Choices:
not restricted - Action:
store - Source: L1110-L1115
--warm-prompts¶
Path to a JSON file with prompts to pre-run at startup. Populates the prefix cache so the first real request hits warm (cold TTFT drops 1.3-2.3x on agent workloads). File format is a list of message arrays, same shape as /v1/chat/completions messages. Prompts are warmed concurrently — keep the file small (1-3 entries for typical agent deployments) to avoid memory pressure at boot.
- Destination:
warm_prompts - Required:
false - Default:
None - Choices:
not restricted - Action:
store - Source: L1117-L1129
--stream-interval¶
Tokens to batch before streaming (1=smooth, higher=throughput)
- Destination:
stream_interval - Required:
false - Default:
1 - Choices:
not restricted - Action:
store - Source: L1130-L1135
--max-kv-size¶
Maximum KV cache size per sequence. When set, uses RotatingKVCache which bounds memory at the cost of losing early context. Reasoning models (e.g. Qwen3, DeepSeek-R1) should use >= 32768 to avoid evicting the think block mid-generation.
- Destination:
max_kv_size - Required:
false - Default:
None - Choices:
not restricted - Action:
store - Source: L1136-L1144
--max-tokens¶
Default max tokens for generation (default: 32768)
- Destination:
max_tokens - Required:
false - Default:
32768 - Choices:
not restricted - Action:
store - Source: L1145-L1150
--max-request-tokens¶
Maximum max_tokens accepted from API clients (default: 32768)
- Destination:
max_request_tokens - Required:
false - Default:
32768 - Choices:
not restricted - Action:
store - Source: L1151-L1156
--continuous-batching¶
Enable continuous batching for multiple concurrent users (slower for single user)
- Destination:
continuous_batching - Required:
false - Default:
argparse default - Choices:
not restricted - Action:
store_true - Source: L1157-L1161
--gpu-memory-utilization¶
Fraction of device memory for Metal allocation limit and emergency cache clear threshold (0.0-1.0, default: 0.90). Increase to 0.95 for large models (200GB+) that need more memory headroom.
- Destination:
gpu_memory_utilization - Required:
false - Default:
0.9 - Choices:
not restricted - Action:
store - Source: L1162-L1169
--use-paged-cache¶
Use paged KV cache for memory efficiency (experimental)
- Destination:
use_paged_cache - Required:
false - Default:
argparse default - Choices:
not restricted - Action:
store_true - Source: L1171-L1175
--paged-cache-block-size¶
Tokens per cache block (default: 64)
- Destination:
paged_cache_block_size - Required:
false - Default:
64 - Choices:
not restricted - Action:
store - Source: L1176-L1181
--max-cache-blocks¶
Maximum number of cache blocks (default: 1000)
- Destination:
max_cache_blocks - Required:
false - Default:
1000 - Choices:
not restricted - Action:
store - Source: L1182-L1187
--chunked-prefill-tokens¶
Max prefill tokens per scheduler step (0=disabled). Prevents starvation of active requests during long prefills.
- Destination:
chunked_prefill_tokens - Required:
false - Default:
0 - Choices:
not restricted - Action:
store - Source: L1189-L1195
--enable-mtp¶
Enable MTP (Multi-Token Prediction) for models with built-in MTP heads. Uses cache snapshot/restore for speculative generation.
- Destination:
enable_mtp - Required:
false - Default:
False - Choices:
not restricted - Action:
store_true - Source: L1197-L1203
--mtp-num-draft-tokens¶
Number of draft tokens per MTP step (default: 1)
- Destination:
mtp_num_draft_tokens - Required:
false - Default:
1 - Choices:
not restricted - Action:
store - Source: L1204-L1209
--mtp-optimistic¶
Skip MTP acceptance check for maximum speed. ~5-10%% wrong tokens. Best for chat, not for code.
- Destination:
mtp_optimistic - Required:
false - Default:
False - Choices:
not restricted - Action:
store_true - Source: L1210-L1216
--prefill-step-size¶
Chunk size for prompt prefill processing. Larger values use more memory but can improve prefill throughput. (default: 2048)
- Destination:
prefill_step_size - Required:
false - Default:
2048 - Choices:
not restricted - Action:
store - Source: L1218-L1224
--specprefill¶
Enable SpecPrefill: use a small draft model to score token importance, then sparse-prefill only the important tokens on the target model. Reduces TTFT on long prompts. Requires --specprefill-draft-model.
- Destination:
specprefill - Required:
false - Default:
False - Choices:
not restricted - Action:
store_true - Source: L1226-L1233
--specprefill-threshold¶
Minimum suffix tokens to trigger SpecPrefill (default: 8192). Shorter prompts use full prefill (scoring overhead > savings).
- Destination:
specprefill_threshold - Required:
false - Default:
8192 - Choices:
not restricted - Action:
store - Source: L1234-L1240
--specprefill-keep-pct¶
Fraction of tokens to keep during sparse prefill (default: 0.3). Lower = faster prefill but more quality loss.
- Destination:
specprefill_keep_pct - Required:
false - Default:
0.3 - Choices:
not restricted - Action:
store - Source: L1241-L1247
--specprefill-backbone-pct¶
Fraction of chunks reserved for evenly spaced sparse-prefill coverage (default: 0.0).
- Destination:
specprefill_backbone_pct - Required:
false - Default:
0.0 - Choices:
not restricted - Action:
store - Source: L1248-L1254
--specprefill-draft-model¶
Path to small draft model for SpecPrefill importance scoring. Must share the same tokenizer as the target model.
- Destination:
specprefill_draft_model - Required:
false - Default:
None - Choices:
not restricted - Action:
store - Source: L1255-L1261
--mllm-draft-model¶
Path to an mlx-vlm MLLM draft/assistant model. For Gemma 4 assistant drafters, use with --mllm-draft-kind mtp.
- Destination:
mllm_draft_model - Required:
false - Default:
None - Choices:
not restricted - Action:
store - Source: L1263-L1269
--mllm-draft-kind¶
mlx-vlm draft kind for --mllm-draft-model.
- Destination:
mllm_draft_kind - Required:
false - Default:
None - Choices:
['mtp'] - Action:
store - Source: L1270-L1276
--mllm-draft-block-size¶
Draft block size passed to mlx-vlm for --mllm-draft-model.
- Destination:
mllm_draft_block_size - Required:
false - Default:
None - Choices:
not restricted - Action:
store - Source: L1277-L1282
--mcp-config¶
Path to MCP configuration file (JSON/YAML) for tool integration
- Destination:
mcp_config - Required:
false - Default:
None - Choices:
not restricted - Action:
store - Source: L1284-L1289
--api-key¶
API key for authentication (if not set, no auth required)
- Destination:
api_key - Required:
false - Default:
None - Choices:
not restricted - Action:
store - Source: L1291-L1296
--rate-limit¶
Rate limit requests per minute per client (0 = disabled)
- Destination:
rate_limit - Required:
false - Default:
0 - Choices:
not restricted - Action:
store - Source: L1297-L1302
--timeout¶
Default request timeout in seconds (default: 300)
- Destination:
timeout - Required:
false - Default:
300.0 - Choices:
not restricted - Action:
store - Source: L1303-L1308
--enable-metrics¶
Expose Prometheus metrics on /metrics (disabled by default)
- Destination:
enable_metrics - Required:
false - Default:
argparse default - Choices:
not restricted - Action:
store_true - Source: L1309-L1313
--auto-unload-idle-seconds¶
Unload the main model after this many idle seconds (0 = disabled)
- Destination:
auto_unload_idle_seconds - Required:
false - Default:
0.0 - Choices:
not restricted - Action:
store - Source: L1314-L1319
--lazy-load-model¶
Register the main model at startup but defer loading until first request
- Destination:
lazy_load_model - Required:
false - Default:
argparse default - Choices:
not restricted - Action:
store_true - Source: L1320-L1324
--max-audio-upload-mb¶
Maximum size of uploaded audio files in MiB (default: 25)
- Destination:
max_audio_upload_mb - Required:
false - Default:
25 - Choices:
not restricted - Action:
store - Source: L1325-L1330
--max-tts-input-chars¶
Maximum number of characters accepted by /v1/audio/speech (default: 4096)
- Destination:
max_tts_input_chars - Required:
false - Default:
4096 - Choices:
not restricted - Action:
store - Source: L1331-L1336
--enable-auto-tool-choice¶
Enable auto tool choice for supported models. Use --tool-call-parser to specify which parser to use.
- Destination:
enable_auto_tool_choice - Required:
false - Default:
argparse default - Choices:
not restricted - Action:
store_true - Source: L1338-L1342
--tool-call-parser¶
Select the tool call parser for the model. Options: auto (auto-detect), mistral, qwen, qwen3_coder, llama, hermes, harmony, gpt-oss, deepseek, gemma4, kimi, granite, nemotron, xlam, functionary, glm47, minimax. Required for --enable-auto-tool-choice.
- Destination:
tool_call_parser - Required:
false - Default:
None - Choices:
['auto', 'mistral', 'qwen', 'qwen3_coder', 'llama', 'hermes', 'harmony', 'gpt-oss', 'deepseek', 'kimi', 'granite', 'nemotron', 'xlam', 'functionary', 'gemma4', 'glm47', 'minimax'] - Action:
store - Source: L1343-L1373
--reasoning-parser¶
f"Enable reasoning content extraction with specified parser. Extracts
- Destination:
reasoning_parser - Required:
false - Default:
None - Choices:
reasoning_choices - Action:
store - Source: L1378-L1388
--mllm¶
Force load model as multimodal (vision) even if name doesn't match auto-detection patterns
- Destination:
mllm - Required:
false - Default:
argparse default - Choices:
not restricted - Action:
store_true - Source: L1390-L1394
--trust-remote-code¶
Allow HuggingFace remote code execution during model/tokenizer loading
- Destination:
trust_remote_code - Required:
false - Default:
argparse default - Choices:
not restricted - Action:
store_true - Source: L1395-L1399
--default-temperature¶
Override default temperature for all requests (default: use model default)
- Destination:
default_temperature - Required:
false - Default:
None - Choices:
not restricted - Action:
store - Source: L1401-L1406
--default-top-p¶
Override default top_p for all requests (default: use model default)
- Destination:
default_top_p - Required:
false - Default:
None - Choices:
not restricted - Action:
store - Source: L1407-L1412
--default-thinking-token-budget¶
Default thinking token budget for reasoning models. Caps reasoning tokens by forcing the end-think sequence when the budget is exhausted. Per-request thinking_token_budget overrides this. (default: None = unlimited)
- Destination:
default_thinking_token_budget - Required:
false - Default:
None - Choices:
not restricted - Action:
store - Source: L1413-L1422
--default-chat-template-kwargs¶
Default chat template kwargs to apply to all requests when request chat_template_kwargs is omitted or empty; empty request kwargs use existing server defaults (JSON object, e.g. {"enable_thinking": true})
- Destination:
default_chat_template_kwargs - Required:
false - Default:
None - Choices:
not restricted - Action:
store - Source: L1423-L1432
--default-top-k¶
Override default top_k for all requests (default: use model default)
- Destination:
default_top_k - Required:
false - Default:
None - Choices:
not restricted - Action:
store - Source: L1433-L1438
--default-min-p¶
Override default min_p for all requests (default: use model default)
- Destination:
default_min_p - Required:
false - Default:
None - Choices:
not restricted - Action:
store - Source: L1439-L1444
--default-presence-penalty¶
Override default presence_penalty for all requests (default: use model default)
- Destination:
default_presence_penalty - Required:
false - Default:
None - Choices:
not restricted - Action:
store - Source: L1445-L1453
--default-repetition-penalty¶
Override default repetition_penalty for all requests (default: use model default)
- Destination:
default_repetition_penalty - Required:
false - Default:
None - Choices:
not restricted - Action:
store - Source: L1454-L1462
--embedding-model¶
Pre-load an embedding model at startup (e.g. mlx-community/embeddinggemma-300m-6bit)
- Destination:
embedding_model - Required:
false - Default:
None - Choices:
not restricted - Action:
store - Source: L1464-L1469
--rerank-model¶
Pre-load a reranker model at startup (e.g. mlx-community/jina-reranker-v2-base-multilingual)
- Destination:
rerank_model - Required:
false - Default:
None - Choices:
not restricted - Action:
store - Source: L1471-L1476
--download-timeout¶
Per-file download timeout in seconds (default: 300)
- Destination:
download_timeout - Required:
false - Default:
300 - Choices:
not restricted - Action:
store - Source: L1478-L1483
--download-retries¶
Number of download retry attempts (default: 3)
- Destination:
download_retries - Required:
false - Default:
3 - Choices:
not restricted - Action:
store - Source: L1484-L1489
--offline¶
Offline mode — only use locally cached models
- Destination:
offline - Required:
false - Default:
argparse default - Choices:
not restricted - Action:
store_true - Source: L1490-L1494
Parser bench_parser¶
model¶
Model to benchmark
- Destination:
model - Required:
true - Default:
argparse default - Choices:
not restricted - Action:
store - Source: L1497-L1497
--num-prompts¶
Number of prompts
- Destination:
num_prompts - Required:
false - Default:
10 - Choices:
not restricted - Action:
store - Source: L1498-L1500
--max-tokens¶
Max tokens per prompt
- Destination:
max_tokens - Required:
false - Default:
100 - Choices:
not restricted - Action:
store - Source: L1501-L1503
--max-num-seqs¶
Max concurrent sequences
- Destination:
max_num_seqs - Required:
false - Default:
32 - Choices:
not restricted - Action:
store - Source: L1504-L1506
--prefill-batch-size¶
Prefill batch size
- Destination:
prefill_batch_size - Required:
false - Default:
8 - Choices:
not restricted - Action:
store - Source: L1507-L1509
--completion-batch-size¶
Completion batch size
- Destination:
completion_batch_size - Required:
false - Default:
16 - Choices:
not restricted - Action:
store - Source: L1510-L1512
--enable-prefix-cache¶
Enable prefix caching (default: enabled)
- Destination:
enable_prefix_cache - Required:
false - Default:
True - Choices:
not restricted - Action:
store_true - Source: L1513-L1518
--disable-prefix-cache¶
Disable prefix caching
- Destination:
disable_prefix_cache - Required:
false - Default:
argparse default - Choices:
not restricted - Action:
store_true - Source: L1519-L1523
--prefix-cache-size¶
Max entries in prefix cache (default: 100, legacy mode only)
- Destination:
prefix_cache_size - Required:
false - Default:
100 - Choices:
not restricted - Action:
store - Source: L1524-L1529
--cache-memory-mb¶
Cache memory limit in MB (default: auto-detect ~20%% of RAM)
- Destination:
cache_memory_mb - Required:
false - Default:
None - Choices:
not restricted - Action:
store - Source: L1531-L1536
--cache-memory-percent¶
Fraction of available RAM for cache if auto-detecting (default: 0.20)
- Destination:
cache_memory_percent - Required:
false - Default:
0.2 - Choices:
not restricted - Action:
store - Source: L1537-L1542
--no-memory-aware-cache¶
Disable memory-aware cache, use legacy entry-count based cache
- Destination:
no_memory_aware_cache - Required:
false - Default:
argparse default - Choices:
not restricted - Action:
store_true - Source: L1543-L1547
--kv-cache-quantization¶
Quantize stored KV caches to reduce memory (8-bit by default)
- Destination:
kv_cache_quantization - Required:
false - Default:
argparse default - Choices:
not restricted - Action:
store_true - Source: L1549-L1553
--kv-cache-quantization-bits¶
Bit width for KV cache quantization (default: 8)
- Destination:
kv_cache_quantization_bits - Required:
false - Default:
8 - Choices:
[4, 8] - Action:
store - Source: L1554-L1560
--kv-cache-quantization-group-size¶
Group size for KV cache quantization (default: 64)
- Destination:
kv_cache_quantization_group_size - Required:
false - Default:
64 - Choices:
not restricted - Action:
store - Source: L1561-L1566
--kv-cache-min-quantize-tokens¶
Minimum tokens for quantization to apply (default: 256)
- Destination:
kv_cache_min_quantize_tokens - Required:
false - Default:
256 - Choices:
not restricted - Action:
store - Source: L1567-L1572
--use-paged-cache¶
Use paged KV cache for memory efficiency (experimental)
- Destination:
use_paged_cache - Required:
false - Default:
argparse default - Choices:
not restricted - Action:
store_true - Source: L1574-L1578
--paged-cache-block-size¶
Tokens per cache block (default: 64)
- Destination:
paged_cache_block_size - Required:
false - Default:
64 - Choices:
not restricted - Action:
store - Source: L1579-L1584
--max-cache-blocks¶
Maximum number of cache blocks (default: 1000)
- Destination:
max_cache_blocks - Required:
false - Default:
1000 - Choices:
not restricted - Action:
store - Source: L1585-L1590
Parser detok_parser¶
model¶
Model to use for tokenizer (default: mlx-community/Qwen3-0.6B-8bit)
- Destination:
model - Required:
true - Default:
mlx-community/Qwen3-0.6B-8bit - Choices:
not restricted - Action:
store - Source: L1596-L1602
--iterations¶
Benchmark iterations (default: 5)
- Destination:
iterations - Required:
false - Default:
5 - Choices:
not restricted - Action:
store - Source: L1603-L1605
Parser kv_cache_parser¶
--layers¶
Number of layers (default: 32)
- Destination:
layers - Required:
false - Default:
32 - Choices:
not restricted - Action:
store - Source: L1611-L1613
--seq-len¶
Sequence length (default: 512)
- Destination:
seq_len - Required:
false - Default:
512 - Choices:
not restricted - Action:
store - Source: L1614-L1616
--heads¶
Number of attention heads (default: 32)
- Destination:
heads - Required:
false - Default:
32 - Choices:
not restricted - Action:
store - Source: L1617-L1619
--head-dim¶
Head dimension (default: 128)
- Destination:
head_dim - Required:
false - Default:
128 - Choices:
not restricted - Action:
store - Source: L1620-L1622
--group-size¶
Quantization group size (default: 64)
- Destination:
group_size - Required:
false - Default:
64 - Choices:
not restricted - Action:
store - Source: L1623-L1628
Parser download_parser¶
model¶
Model to download
- Destination:
model - Required:
true - Default:
argparse default - Choices:
not restricted - Action:
store - Source: L1634-L1634
--timeout¶
Per-file download timeout in seconds (default: 300)
- Destination:
timeout - Required:
false - Default:
300 - Choices:
not restricted - Action:
store - Source: L1635-L1640
--retries¶
Number of retry attempts (default: 3)
- Destination:
retries - Required:
false - Default:
3 - Choices:
not restricted - Action:
store - Source: L1641-L1646
--mllm¶
Download as multimodal model (broader file patterns)
- Destination:
mllm - Required:
false - Default:
argparse default - Choices:
not restricted - Action:
store_true - Source: L1647-L1651
Parser model_inspect_parser¶
model¶
Local model path or Hugging Face model id
- Destination:
model - Required:
true - Default:
argparse default - Choices:
not restricted - Action:
store - Source: L1666-L1670
--revision¶
Hugging Face revision to inspect
- Destination:
revision - Required:
false - Default:
None - Choices:
not restricted - Action:
store - Source: L1671-L1676
--local-files-only¶
Use only local Hugging Face cache files
- Destination:
local_files_only - Required:
false - Default:
argparse default - Choices:
not restricted - Action:
store_true - Source: L1677-L1681
Parser model_acquire_parser¶
model¶
Hugging Face model id
- Destination:
model - Required:
true - Default:
argparse default - Choices:
not restricted - Action:
store - Source: L1687-L1687
--revision¶
Hugging Face revision to download
- Destination:
revision - Required:
false - Default:
None - Choices:
not restricted - Action:
store - Source: L1688-L1693
--target-dir¶
Final local directory. Defaults to Hugging Face cache.
- Destination:
target_dir - Required:
false - Default:
None - Choices:
not restricted - Action:
store - Source: L1694-L1699
--staging-dir¶
Directory for temporary staged downloads before finalizing target-dir
- Destination:
staging_dir - Required:
false - Default:
None - Choices:
not restricted - Action:
store - Source: L1700-L1705
--mllm¶
Acquire multimodal model files using broader allow patterns
- Destination:
mllm - Required:
false - Default:
argparse default - Choices:
not restricted - Action:
store_true - Source: L1706-L1710
--no-fast-transfer¶
Do not set HF_HUB_ENABLE_HF_TRANSFER=1 during download
- Destination:
no_fast_transfer - Required:
false - Default:
argparse default - Choices:
not restricted - Action:
store_true - Source: L1711-L1715
--local-files-only¶
Use only local Hugging Face cache files
- Destination:
local_files_only - Required:
false - Default:
argparse default - Choices:
not restricted - Action:
store_true - Source: L1716-L1720
Parser model_convert_parser¶
source¶
Hugging Face model id or local source path
- Destination:
source - Required:
true - Default:
argparse default - Choices:
not restricted - Action:
store - Source: L1726-L1730
--output¶
Output directory for the converted MLX model
- Destination:
output - Required:
true - Default:
argparse default - Choices:
not restricted - Action:
store - Source: L1731-L1736
--quantize¶
Generate a quantized MLX model
- Destination:
quantize - Required:
false - Default:
argparse default - Choices:
not restricted - Action:
store_true - Source: L1737-L1741
--q-bits¶
Quantization bit width (e.g. 3, 4, 8)
- Destination:
q_bits - Required:
false - Default:
None - Choices:
not restricted - Action:
store - Source: L1742-L1747
--q-group-size¶
Quantization group size (default: mlx-lm default)
- Destination:
q_group_size - Required:
false - Default:
None - Choices:
not restricted - Action:
store - Source: L1748-L1753
--q-mode¶
No argparse help text is declared.
- Destination:
q_mode - Required:
false - Default:
None - Choices:
['affine', 'mxfp4', 'nvfp4', 'mxfp8'] - Action:
store - Source: L1754-L1758
--quant-predicate¶
mlx-lm mixed-bit quantization recipe
- Destination:
quant_predicate - Required:
false - Default:
None - Choices:
['mixed_2_6', 'mixed_3_4', 'mixed_3_6', 'mixed_4_6'] - Action:
store - Source: L1759-L1764
--dtype¶
Non-quantized parameter dtype
- Destination:
dtype - Required:
false - Default:
None - Choices:
['float16', 'bfloat16', 'float32'] - Action:
store - Source: L1765-L1770
--trust-remote-code¶
Allow Hugging Face remote code during mlx-lm conversion
- Destination:
trust_remote_code - Required:
false - Default:
argparse default - Choices:
not restricted - Action:
store_true - Source: L1771-L1775
--dry-run¶
Print the conversion command and manifest without executing
- Destination:
dry_run - Required:
false - Default:
argparse default - Choices:
not restricted - Action:
store_true - Source: L1776-L1780
Parser model_register_parser¶
artifact¶
Finalized local model artifact directory
- Destination:
artifact - Required:
true - Default:
argparse default - Choices:
not restricted - Action:
store - Source: L1786-L1790
--model-id¶
Override model ID (default: directory name of artifact)
- Destination:
model_id - Required:
false - Default:
None - Choices:
not restricted - Action:
store - Source: L1791-L1796
--served-model-name¶
Model name exposed by the API (default: model-id)
- Destination:
served_model_name - Required:
false - Default:
None - Choices:
not restricted - Action:
store - Source: L1797-L1802
--preset-alias¶
Optional alias for preset lookup in registry
- Destination:
preset_alias - Required:
false - Default:
None - Choices:
not restricted - Action:
store - Source: L1803-L1808
--output¶
Manifest path. Defaults to artifact/vllm_mlx_registration_manifest.json
- Destination:
output - Required:
false - Default:
None - Choices:
not restricted - Action:
store - Source: L1809-L1814
Parser mllm_group¶
--mllm¶
Mark the artifact as an MLLM serving candidate
- Destination:
mllm - Required:
false - Default:
None - Choices:
not restricted - Action:
store_true - Source: L1816-L1821
--no-mllm¶
Explicitly mark the artifact as text-only
- Destination:
mllm - Required:
false - Default:
argparse default - Choices:
not restricted - Action:
store_false - Source: L1822-L1827
Parser model_register_parser¶
--tool-call-parser¶
Tool call parser name for the model (e.g. qwen3_coder, mistral)
- Destination:
tool_call_parser - Required:
false - Default:
None - Choices:
not restricted - Action:
store - Source: L1828-L1833
--reasoning-parser¶
Reasoning parser name for thinking models (e.g. qwen3)
- Destination:
reasoning_parser - Required:
false - Default:
None - Choices:
not restricted - Action:
store - Source: L1834-L1839
--default-temperature¶
Default temperature for all requests
- Destination:
default_temperature - Required:
false - Default:
None - Choices:
not restricted - Action:
store - Source: L1840-L1845
--default-top-p¶
Default top_p for all requests
- Destination:
default_top_p - Required:
false - Default:
None - Choices:
not restricted - Action:
store - Source: L1846-L1851
--default-top-k¶
Default top_k for all requests
- Destination:
default_top_k - Required:
false - Default:
None - Choices:
not restricted - Action:
store - Source: L1852-L1857
--default-min-p¶
Default min_p for all requests
- Destination:
default_min_p - Required:
false - Default:
None - Choices:
not restricted - Action:
store - Source: L1858-L1863
--default-presence-penalty¶
Default presence_penalty for all requests
- Destination:
default_presence_penalty - Required:
false - Default:
None - Choices:
not restricted - Action:
store - Source: L1864-L1869
--default-repetition-penalty¶
Default repetition_penalty for all requests
- Destination:
default_repetition_penalty - Required:
false - Default:
None - Choices:
not restricted - Action:
store - Source: L1870-L1875
--default-chat-template-kwargs¶
Default chat template kwargs as JSON, e.g. {"enable_thinking": true}
- Destination:
default_chat_template_kwargs - Required:
false - Default:
None - Choices:
not restricted - Action:
store - Source: L1876-L1881
--feature-flag¶
Feature flag to record in the registration manifest. Repeatable.
- Destination:
feature_flag - Required:
false - Default:
[] - Choices:
not restricted - Action:
append - Source: L1882-L1887
Parser model_qualify_parser¶
model_id¶
Model ID to qualify against the running server
- Destination:
model_id - Required:
true - Default:
argparse default - Choices:
not restricted - Action:
store - Source: L1893-L1897
--url¶
Running server URL for bench-serve
- Destination:
url - Required:
false - Default:
http://127.0.0.1:8080 - Choices:
not restricted - Action:
store - Source: L1898-L1903
--workload¶
bench-serve workload contract path
- Destination:
workload - Required:
false - Default:
None - Choices:
not restricted - Action:
store - Source: L1904-L1909
--output¶
Qualification request manifest path
- Destination:
output - Required:
false - Default:
None - Choices:
not restricted - Action:
store - Source: L1910-L1915
--result-output¶
Result output path passed to bench-serve
- Destination:
result_output - Required:
false - Default:
None - Choices:
not restricted - Action:
store - Source: L1916-L1921
--repetitions¶
Number of repetitions per benchmark sweep configuration
- Destination:
repetitions - Required:
false - Default:
None - Choices:
not restricted - Action:
store - Source: L1922-L1927
--dry-run¶
Write or print the qualification command without running it
- Destination:
dry_run - Required:
false - Default:
argparse default - Choices:
not restricted - Action:
store_true - Source: L1928-L1932
--extra-arg¶
Extra argument passed through to bench-serve. Repeatable.
- Destination:
extra_arg - Required:
false - Default:
[] - Choices:
not restricted - Action:
append - Source: L1933-L1938
Parser bench_serve_parser¶
--url¶
Base URL of the running server (default: http://127.0.0.1:8080)
- Destination:
url - Required:
false - Default:
http://127.0.0.1:8080 - Choices:
not restricted - Action:
store - Source: L1944-L1949
--model¶
Model ID to benchmark (default: auto-detected from server)
- Destination:
model - Required:
false - Default:
None - Choices:
not restricted - Action:
store - Source: L1950-L1955
--workload¶
Path to a declarative workload JSON file. When set, bench-serve runs contract-style cases with per-case quality checks and comparison-only policy timeouts instead of the prompt sweep.
- Destination:
workload - Required:
false - Default:
None - Choices:
not restricted - Action:
store - Source: L1956-L1965
--prompts¶
Comma-separated prompt set names or paths (default: short,medium,long)
- Destination:
prompts - Required:
false - Default:
short,medium,long - Choices:
not restricted - Action:
store - Source: L1966-L1971
--prompt-file¶
Path to an additional prompt file (JSON list of message dicts)
- Destination:
prompt_file - Required:
false - Default:
None - Choices:
not restricted - Action:
store - Source: L1972-L1977
--system-prompt-file¶
Path to a text file whose contents are prepended as a system message to every prompt. Use this together with --warm-prompts to benchmark the warm-cache path (the warmup populates the prefix cache with this same system, so every request in the bench hits the cache).
- Destination:
system_prompt_file - Required:
false - Default:
None - Choices:
not restricted - Action:
store - Source: L1978-L1989
--skip-preflight-token-count¶
Skip the pre-flight max_tokens=1 request that counts prompt tokens per prompt set. That request populates the prefix cache with the full prompt, which defeats cold-vs-warm comparisons. Auto-enabled when --system-prompt-file is set; pass this flag explicitly to force-enable regardless.
- Destination:
skip_preflight_token_count - Required:
false - Default:
argparse default - Choices:
not restricted - Action:
store_true - Source: L1990-L2000
--concurrency¶
Comma-separated concurrency levels to sweep (default: 1,4)
- Destination:
concurrency - Required:
false - Default:
1,4 - Choices:
not restricted - Action:
store - Source: L2001-L2006
--max-tokens¶
Maximum tokens to generate per request (default: 256)
- Destination:
max_tokens - Required:
false - Default:
256 - Choices:
not restricted - Action:
store - Source: L2007-L2012
--repetitions¶
Number of repetitions per sweep configuration or workload case (default: 3)
- Destination:
repetitions - Required:
false - Default:
3 - Choices:
not restricted - Action:
store - Source: L2013-L2018
--warmup¶
Warmup rounds before the first measured repetition (default: 1)
- Destination:
warmup - Required:
false - Default:
1 - Choices:
not restricted - Action:
store - Source: L2019-L2024
--enable-thinking¶
Enable thinking mode: "true", "false", or "true,false" to sweep both
- Destination:
enable_thinking - Required:
false - Default:
None - Choices:
not restricted - Action:
store - Source: L2025-L2030
--extra-body¶
Comma-separated JSON dicts to pass as extra body parameters
- Destination:
extra_body - Required:
false - Default:
None - Choices:
not restricted - Action:
store - Source: L2031-L2036
--output¶
File path to write results to (default: stdout)
- Destination:
output - Required:
false - Default:
None - Choices:
not restricted - Action:
store - Source: L2037-L2042
--format¶
Output format (auto = table for prompt sweeps, json for workloads; sqlite requires --output)
- Destination:
format - Required:
false - Default:
auto - Choices:
['auto', 'table', 'json', 'csv', 'sql', 'sqlite'] - Action:
store - Source: L2043-L2052
--validate¶
Validate responses (default: true)
- Destination:
validate - Required:
false - Default:
true - Choices:
['true', 'false'] - Action:
store - Source: L2053-L2059
--scrape-metrics¶
Scrape /metrics before and after each run (default: true)
- Destination:
scrape_metrics - Required:
false - Default:
true - Choices:
['true', 'false'] - Action:
store - Source: L2060-L2066
--include-content¶
Include full generated content in workload JSON output
- Destination:
include_content - Required:
false - Default:
argparse default - Choices:
not restricted - Action:
store_true - Source: L2067-L2071
--request-timeout-s¶
HTTP transport timeout for workload mode in seconds (default: 300). Use 0 to disable; product policy timeouts belong in the workload.
- Destination:
request_timeout_s - Required:
false - Default:
300.0 - Choices:
not restricted - Action:
store - Source: L2072-L2080
--cache-policy¶
Workload cache handling (default: workload defaults or preserve). Use before-case for cold, uncontaminated per-case qualification. Workload JSON may also spell these with underscores.
- Destination:
cache_policy - Required:
false - Default:
None - Choices:
['preserve', 'before-run', 'before-case'] - Action:
store - Source: L2081-L2091
--tag¶
Optional tag string stored in every result row
- Destination:
tag - Required:
false - Default:
None - Choices:
not restricted - Action:
store - Source: L2092-L2097
--override-field¶
Override result fields as key=value pairs (e.g. chip=M4Pro)
- Destination:
override_field - Required:
false - Default:
[] - Choices:
not restricted - Action:
store - Source: L2098-L2103
vllm_mlx.gradio_app.main¶
Parser parser¶
--server-url¶
URL of the vllm-mlx server (default: http://localhost:8000)
- Destination:
server_url - Required:
false - Default:
http://localhost:8000 - Choices:
not restricted - Action:
store - Source: L280-L285
--port¶
Port for Gradio interface (default: 7860)
- Destination:
port - Required:
false - Default:
7860 - Choices:
not restricted - Action:
store - Source: L286-L291
--share¶
Create a public share link
- Destination:
share - Required:
false - Default:
argparse default - Choices:
not restricted - Action:
store_true - Source: L292-L296
--max-tokens¶
Maximum tokens to generate (default: 2048)
- Destination:
max_tokens - Required:
false - Default:
2048 - Choices:
not restricted - Action:
store - Source: L297-L302
--temperature¶
Sampling temperature (default: 0.7)
- Destination:
temperature - Required:
false - Default:
0.7 - Choices:
not restricted - Action:
store - Source: L303-L308
--served-model-name¶
Model name to send in /v1/chat/completions requests (default: default)
- Destination:
served_model_name - Required:
false - Default:
default - Choices:
not restricted - Action:
store - Source: L309-L316
--text-only¶
Use text-only mode (no image/video support, faster for LLM-only models)
- Destination:
text_only - Required:
false - Default:
argparse default - Choices:
not restricted - Action:
store_true - Source: L317-L321
vllm_mlx.gradio_text_app.main¶
Parser parser¶
--server-url¶
URL of the vllm-mlx server (default: http://localhost:8000)
- Destination:
server_url - Required:
false - Default:
http://localhost:8000 - Choices:
not restricted - Action:
store - Source: L136-L141
--port¶
Port for Gradio interface (default: 7861)
- Destination:
port - Required:
false - Default:
7861 - Choices:
not restricted - Action:
store - Source: L142-L147
--share¶
Create a public share link
- Destination:
share - Required:
false - Default:
argparse default - Choices:
not restricted - Action:
store_true - Source: L148-L152
--max-tokens¶
Maximum tokens to generate (default: 512)
- Destination:
max_tokens - Required:
false - Default:
512 - Choices:
not restricted - Action:
store - Source: L153-L158
--temperature¶
Sampling temperature (default: 0.7)
- Destination:
temperature - Required:
false - Default:
0.7 - Choices:
not restricted - Action:
store - Source: L159-L164
--served-model-name¶
Model name to send in /v1/chat/completions requests (default: default)
- Destination:
served_model_name - Required:
false - Default:
default - Choices:
not restricted - Action:
store - Source: L165-L172
vllm_mlx.server.create_parser¶
Parser parser¶
--model¶
Model to load (HuggingFace model name or local path)
- Destination:
model - Required:
false - Default:
mlx-community/Llama-3.2-3B-Instruct-4bit - Choices:
not restricted - Action:
store - Source: L6728-L6733
--host¶
Host to bind to (default: localhost; use 0.0.0.0 to expose externally)
- Destination:
host - Required:
false - Default:
127.0.0.1 - Choices:
not restricted - Action:
store - Source: L6734-L6739
--port¶
Port to bind to
- Destination:
port - Required:
false - Default:
8000 - Choices:
not restricted - Action:
store - Source: L6740-L6745
--mllm¶
Force loading as MLLM (multimodal language model)
- Destination:
mllm - Required:
false - Default:
argparse default - Choices:
not restricted - Action:
store_true - Source: L6746-L6750
--trust-remote-code¶
Allow HuggingFace remote code execution during model/tokenizer loading
- Destination:
trust_remote_code - Required:
false - Default:
argparse default - Choices:
not restricted - Action:
store_true - Source: L6751-L6755
--continuous-batching¶
Enable continuous batching for multiple concurrent users
- Destination:
continuous_batching - Required:
false - Default:
argparse default - Choices:
not restricted - Action:
store_true - Source: L6756-L6760
--mllm-draft-model¶
Path to an mlx-vlm MLLM draft/assistant model.
- Destination:
mllm_draft_model - Required:
false - Default:
None - Choices:
not restricted - Action:
store - Source: L6761-L6766
--mllm-draft-kind¶
mlx-vlm draft kind for --mllm-draft-model.
- Destination:
mllm_draft_kind - Required:
false - Default:
None - Choices:
['mtp'] - Action:
store - Source: L6767-L6773
--mllm-draft-block-size¶
Draft block size passed to mlx-vlm for --mllm-draft-model.
- Destination:
mllm_draft_block_size - Required:
false - Default:
None - Choices:
not restricted - Action:
store - Source: L6774-L6779
--mcp-config¶
Path to MCP configuration file (JSON/YAML)
- Destination:
mcp_config - Required:
false - Default:
None - Choices:
not restricted - Action:
store - Source: L6780-L6785
--max-tokens¶
Default max tokens for generation
- Destination:
max_tokens - Required:
false - Default:
32768 - Choices:
not restricted - Action:
store - Source: L6786-L6791
--max-request-tokens¶
Maximum max_tokens accepted from API clients (default: 32768)
- Destination:
max_request_tokens - Required:
false - Default:
32768 - Choices:
not restricted - Action:
store - Source: L6792-L6797
--api-key¶
API key for authentication (if not set, no auth required)
- Destination:
api_key - Required:
false - Default:
None - Choices:
not restricted - Action:
store - Source: L6798-L6803
--timeout¶
Default request timeout in seconds (default: 300)
- Destination:
timeout - Required:
false - Default:
300.0 - Choices:
not restricted - Action:
store - Source: L6804-L6809
--enable-metrics¶
Expose Prometheus metrics on /metrics (disabled by default)
- Destination:
enable_metrics - Required:
false - Default:
argparse default - Choices:
not restricted - Action:
store_true - Source: L6810-L6814
--auto-unload-idle-seconds¶
Unload the main model after this many idle seconds (0 = disabled)
- Destination:
auto_unload_idle_seconds - Required:
false - Default:
0.0 - Choices:
not restricted - Action:
store - Source: L6815-L6820
--lazy-load-model¶
Register the main model at startup but defer loading until first request
- Destination:
lazy_load_model - Required:
false - Default:
argparse default - Choices:
not restricted - Action:
store_true - Source: L6821-L6825
--rate-limit¶
Rate limit requests per minute per client (0 = disabled)
- Destination:
rate_limit - Required:
false - Default:
0 - Choices:
not restricted - Action:
store - Source: L6826-L6831
--reasoning-parser¶
f"Enable reasoning content extraction with specified parser. Options: {', '.join(reasoning_choices)}."
- Destination:
reasoning_parser - Required:
false - Default:
None - Choices:
reasoning_choices - Action:
store - Source: L6836-L6845
--embedding-model¶
Pre-load an embedding model at startup (e.g. mlx-community/all-MiniLM-L6-v2-4bit)
- Destination:
embedding_model - Required:
false - Default:
None - Choices:
not restricted - Action:
store - Source: L6846-L6851
--default-temperature¶
Default temperature for generation when not specified in request
- Destination:
default_temperature - Required:
false - Default:
None - Choices:
not restricted - Action:
store - Source: L6852-L6857
--default-top-p¶
Default top_p for generation when not specified in request
- Destination:
default_top_p - Required:
false - Default:
None - Choices:
not restricted - Action:
store - Source: L6858-L6863
--default-chat-template-kwargs¶
Default chat template kwargs to apply to all requests when request chat_template_kwargs is omitted or empty; empty request kwargs use existing server defaults (JSON object, e.g. {"enable_thinking": false})
- Destination:
default_chat_template_kwargs - Required:
false - Default:
None - Choices:
not restricted - Action:
store - Source: L6864-L6873
--default-top-k¶
Default top_k for generation when not specified in request
- Destination:
default_top_k - Required:
false - Default:
None - Choices:
not restricted - Action:
store - Source: L6874-L6879
--default-min-p¶
Default min_p for generation when not specified in request
- Destination:
default_min_p - Required:
false - Default:
None - Choices:
not restricted - Action:
store - Source: L6880-L6885
--default-presence-penalty¶
Default presence_penalty for generation when not specified in request
- Destination:
default_presence_penalty - Required:
false - Default:
None - Choices:
not restricted - Action:
store - Source: L6886-L6891
--default-repetition-penalty¶
Default repetition_penalty for generation when not specified in request
- Destination:
default_repetition_penalty - Required:
false - Default:
None - Choices:
not restricted - Action:
store - Source: L6892-L6899
--max-audio-upload-mb¶
Maximum size of uploaded audio files in MiB (default: 25)
- Destination:
max_audio_upload_mb - Required:
false - Default:
DEFAULT_MAX_AUDIO_UPLOAD_MB - Choices:
not restricted - Action:
store - Source: L6900-L6905
--max-tts-input-chars¶
Maximum number of characters accepted by /v1/audio/speech (default: 4096)
- Destination:
max_tts_input_chars - Required:
false - Default:
DEFAULT_MAX_TTS_INPUT_CHARS - Choices:
not restricted - Action:
store - Source: L6906-L6911