vllm_mlx.mllm_scheduler

MLLM Scheduler for multimodal continuous batching.

View the complete module source at #L1-L1242.

API details

Each callable below includes its exact signature, type annotations, inputs, defaults, return contract, documented exceptions, implementation source, and parsed docstring sections when the source provides them.

vllm_mlx.mllm_scheduler

MLLM Scheduler for multimodal continuous batching.

This scheduler handles Multimodal Language Model requests with continuous batching support, following the same architecture as the LLM scheduler.

Key features: - Batch processing of multiple MLLM requests - Vision embedding caching for repeated images - Step-based generation loop (like LLM scheduler) - Support for both streaming and non-streaming generation

Architecture: 1. Requests arrive via add_request() -> waiting queue 2. Scheduler moves requests from waiting to running (via MLLMBatchGenerator) 3. step() method generates one token for ALL running requests 4. Finished requests are removed and outputs returned

vllm_mlx.mllm_scheduler.logger module-attribute

logger = logging.getLogger(__name__)

vllm_mlx.mllm_scheduler.MLLMSchedulerConfig dataclass

MLLMSchedulerConfig(max_num_seqs: int = 16, prefill_batch_size: int = 16, completion_batch_size: int = 16, prefill_step_size: int = 1024, enable_vision_cache: bool = True, vision_cache_size: int = 100, default_max_tokens: int = 256, default_video_fps: float = 2.0, cache_memory_mb: Optional[int] = None, max_video_frames: int = 128, enable_mtp: bool = False, mtp_num_draft_tokens: int = 1, enable_prefix_cache: bool = True, use_memory_aware_cache: bool = True, prefix_cache_memory_mb: Optional[int] = None, kv_cache_quantization: bool = False, kv_cache_quantization_bits: int = 8, kv_cache_quantization_group_size: int = 64, chunked_prefill_tokens: int = 0, max_kv_size: int = 0, ssd_cache_dir: Optional[str] = None, ssd_cache_max_gb: float = 10.0)

Configuration for MLLM scheduler.

vllm_mlx.mllm_scheduler.MLLMSchedulerConfig.max_num_seqs class-attribute instance-attribute

max_num_seqs: int = 16

vllm_mlx.mllm_scheduler.MLLMSchedulerConfig.prefill_batch_size class-attribute instance-attribute

prefill_batch_size: int = 16

vllm_mlx.mllm_scheduler.MLLMSchedulerConfig.completion_batch_size class-attribute instance-attribute

completion_batch_size: int = 16

vllm_mlx.mllm_scheduler.MLLMSchedulerConfig.prefill_step_size class-attribute instance-attribute

prefill_step_size: int = 1024

vllm_mlx.mllm_scheduler.MLLMSchedulerConfig.enable_vision_cache class-attribute instance-attribute

enable_vision_cache: bool = True

vllm_mlx.mllm_scheduler.MLLMSchedulerConfig.vision_cache_size class-attribute instance-attribute

vision_cache_size: int = 100

vllm_mlx.mllm_scheduler.MLLMSchedulerConfig.default_max_tokens class-attribute instance-attribute

default_max_tokens: int = 256

vllm_mlx.mllm_scheduler.MLLMSchedulerConfig.default_video_fps class-attribute instance-attribute

default_video_fps: float = 2.0

vllm_mlx.mllm_scheduler.MLLMSchedulerConfig.cache_memory_mb class-attribute instance-attribute

cache_memory_mb: Optional[int] = None

vllm_mlx.mllm_scheduler.MLLMSchedulerConfig.max_video_frames class-attribute instance-attribute

max_video_frames: int = 128

vllm_mlx.mllm_scheduler.MLLMSchedulerConfig.enable_mtp class-attribute instance-attribute

enable_mtp: bool = False

vllm_mlx.mllm_scheduler.MLLMSchedulerConfig.mtp_num_draft_tokens class-attribute instance-attribute

mtp_num_draft_tokens: int = 1

vllm_mlx.mllm_scheduler.MLLMSchedulerConfig.enable_prefix_cache class-attribute instance-attribute

enable_prefix_cache: bool = True

vllm_mlx.mllm_scheduler.MLLMSchedulerConfig.use_memory_aware_cache class-attribute instance-attribute

use_memory_aware_cache: bool = True

vllm_mlx.mllm_scheduler.MLLMSchedulerConfig.prefix_cache_memory_mb class-attribute instance-attribute

prefix_cache_memory_mb: Optional[int] = None

vllm_mlx.mllm_scheduler.MLLMSchedulerConfig.kv_cache_quantization class-attribute instance-attribute

kv_cache_quantization: bool = False

vllm_mlx.mllm_scheduler.MLLMSchedulerConfig.kv_cache_quantization_bits class-attribute instance-attribute

kv_cache_quantization_bits: int = 8

vllm_mlx.mllm_scheduler.MLLMSchedulerConfig.kv_cache_quantization_group_size class-attribute instance-attribute

kv_cache_quantization_group_size: int = 64

vllm_mlx.mllm_scheduler.MLLMSchedulerConfig.chunked_prefill_tokens class-attribute instance-attribute

chunked_prefill_tokens: int = 0

vllm_mlx.mllm_scheduler.MLLMSchedulerConfig.max_kv_size class-attribute instance-attribute

max_kv_size: int = 0

vllm_mlx.mllm_scheduler.MLLMSchedulerConfig.ssd_cache_dir class-attribute instance-attribute

ssd_cache_dir: Optional[str] = None

vllm_mlx.mllm_scheduler.MLLMSchedulerConfig.ssd_cache_max_gb class-attribute instance-attribute

ssd_cache_max_gb: float = 10.0

vllm_mlx.mllm_scheduler.MLLMRequest dataclass

MLLMRequest(request_id: str, prompt: str, images: Optional[List[str]] = None, videos: Optional[List[str]] = None, audio: Optional[List[str]] = None, sampling_params: SamplingParams = SamplingParams(), arrival_time: float = time(), batch_uid: Optional[int] = None, status: RequestStatus = WAITING, output_text: str = '', output_tokens: List[int] = list(), finish_reason: Optional[str] = None, num_prompt_tokens: int = 0, num_output_tokens: int = 0, mtp_drafts: int = 0, mtp_accepted: int = 0, first_token_time: Optional[float] = None)

Extended request for MLLM processing.

Includes all multimodal data needed for generation.

vllm_mlx.mllm_scheduler.MLLMRequest.request_id instance-attribute

request_id: str

vllm_mlx.mllm_scheduler.MLLMRequest.prompt instance-attribute

prompt: str

vllm_mlx.mllm_scheduler.MLLMRequest.images class-attribute instance-attribute

images: Optional[List[str]] = None

vllm_mlx.mllm_scheduler.MLLMRequest.videos class-attribute instance-attribute

videos: Optional[List[str]] = None

vllm_mlx.mllm_scheduler.MLLMRequest.audio class-attribute instance-attribute

audio: Optional[List[str]] = None

vllm_mlx.mllm_scheduler.MLLMRequest.sampling_params class-attribute instance-attribute

sampling_params: SamplingParams = field(default_factory=SamplingParams)

vllm_mlx.mllm_scheduler.MLLMRequest.arrival_time class-attribute instance-attribute

arrival_time: float = field(default_factory=time.time)

vllm_mlx.mllm_scheduler.MLLMRequest.batch_uid class-attribute instance-attribute

batch_uid: Optional[int] = None

vllm_mlx.mllm_scheduler.MLLMRequest.status class-attribute instance-attribute

status: RequestStatus = RequestStatus.WAITING

vllm_mlx.mllm_scheduler.MLLMRequest.output_text class-attribute instance-attribute

output_text: str = ''

vllm_mlx.mllm_scheduler.MLLMRequest.output_tokens class-attribute instance-attribute

output_tokens: List[int] = field(default_factory=list)

vllm_mlx.mllm_scheduler.MLLMRequest.finish_reason class-attribute instance-attribute

finish_reason: Optional[str] = None

vllm_mlx.mllm_scheduler.MLLMRequest.num_prompt_tokens class-attribute instance-attribute

num_prompt_tokens: int = 0

vllm_mlx.mllm_scheduler.MLLMRequest.num_output_tokens class-attribute instance-attribute

num_output_tokens: int = 0

vllm_mlx.mllm_scheduler.MLLMRequest.mtp_drafts class-attribute instance-attribute

mtp_drafts: int = 0

vllm_mlx.mllm_scheduler.MLLMRequest.mtp_accepted class-attribute instance-attribute

mtp_accepted: int = 0

vllm_mlx.mllm_scheduler.MLLMRequest.first_token_time class-attribute instance-attribute

first_token_time: Optional[float] = None

vllm_mlx.mllm_scheduler.MLLMSchedulerOutput dataclass

MLLMSchedulerOutput(scheduled_request_ids: List[str] = list(), num_scheduled_tokens: int = 0, finished_request_ids: Set[str] = set(), outputs: List[RequestOutput] = list(), has_work: bool = False)

Output from a scheduling step.

Contains information about what was scheduled and results.

vllm_mlx.mllm_scheduler.MLLMSchedulerOutput.scheduled_request_ids class-attribute instance-attribute

scheduled_request_ids: List[str] = field(default_factory=list)

vllm_mlx.mllm_scheduler.MLLMSchedulerOutput.num_scheduled_tokens class-attribute instance-attribute

num_scheduled_tokens: int = 0

vllm_mlx.mllm_scheduler.MLLMSchedulerOutput.finished_request_ids class-attribute instance-attribute

finished_request_ids: Set[str] = field(default_factory=set)

vllm_mlx.mllm_scheduler.MLLMSchedulerOutput.outputs class-attribute instance-attribute

outputs: List[RequestOutput] = field(default_factory=list)

vllm_mlx.mllm_scheduler.MLLMSchedulerOutput.has_work class-attribute instance-attribute

has_work: bool = False

vllm_mlx.mllm_scheduler.MLLMScheduler

MLLMScheduler(model: Any, processor: Any, config: Optional[MLLMSchedulerConfig] = None)

Scheduler for Vision Language Model requests with continuous batching.

This scheduler manages the lifecycle of MLLM requests using the MLLMBatchGenerator for efficient batch processing:

1. Requests arrive and are added to the waiting queue
2. Scheduler moves requests from waiting to running (via batch generator)
3. step() generates one token for ALL running requests simultaneously
4. Finished requests are removed and outputs returned

Example

scheduler = MLLMScheduler(model, processor, config)

Add requests

request_id = scheduler.add_request( ... prompt="What's in this image?", ... images=["photo.jpg"] ... )

Run generation loop

while scheduler.has_requests(): ... output = scheduler.step() ... for req_output in output.outputs: ... if req_output.finished: ... print(f"Finished: {req_output.output_text}")

For async usage with streaming

await scheduler.start() request_id = await scheduler.add_request_async(...) async for output in scheduler.stream_outputs(request_id): ... print(output.new_text, end="")

Initialize MLLM scheduler.

Parameters:

• model (Any) –

  The VLM model

• processor (Any) –

  The VLM processor

• config (Optional[MLLMSchedulerConfig], default: None ) –

  Scheduler configuration

Source code in vllm_mlx/mllm_scheduler.py

def __init__(
    self,
    model: Any,
    processor: Any,
    config: Optional[MLLMSchedulerConfig] = None,
):
    """
    Initialize MLLM scheduler.

    Args:
        model: The VLM model
        processor: The VLM processor
        config: Scheduler configuration
    """
    self.model = model
    self.processor = processor
    self.config = config or MLLMSchedulerConfig()

    # Get model config
    self.model_config = getattr(model, "config", None)

    # Multimodal processor for input preparation
    self.mm_processor = MultimodalProcessor(
        model=model,
        processor=processor,
        config=self.model_config,
    )

    # Get stop tokens from tokenizer
    self.stop_tokens = self._get_stop_tokens()

    # Batch generator (created lazily)
    self.batch_generator: Optional[MLLMBatchGenerator] = None

    # Request management - following vLLM's design
    self.waiting: deque[MLLMRequest] = deque()  # Waiting queue (FCFS)
    self.running: Dict[str, MLLMRequest] = {}  # Running requests by ID
    self.requests: Dict[str, MLLMRequest] = {}  # All requests by ID
    self.finished_req_ids: Set[str] = set()  # Recently finished

    # Mapping between our request IDs and BatchGenerator UIDs
    self.request_id_to_uid: Dict[str, int] = {}
    self.uid_to_request_id: Dict[int, str] = {}

    # Per-request streaming detokenizers for UTF-8-safe incremental decode
    self._detokenizer_pool: Dict[str, Any] = {}

    # Output queues for async streaming
    self.output_queues: Dict[str, asyncio.Queue] = {}

    # Async processing control
    self._running = False
    self._processing_task: Optional[asyncio.Task] = None

    # Memory management: periodic mx.clear_cache() to free Metal buffer pool
    self._step_count = 0
    self._clear_cache_interval = 32

    # Statistics
    self.num_requests_processed = 0
    self.total_prompt_tokens = 0
    self.total_completion_tokens = 0

    # Memory management: periodic mx.clear_cache() to free Metal buffers
    self._step_count = 0
    self._clear_cache_interval = 32

vllm_mlx.mllm_scheduler.MLLMScheduler.model instance-attribute

model = model

vllm_mlx.mllm_scheduler.MLLMScheduler.processor instance-attribute

processor = processor

vllm_mlx.mllm_scheduler.MLLMScheduler.config instance-attribute

config = config or MLLMSchedulerConfig()

vllm_mlx.mllm_scheduler.MLLMScheduler.model_config instance-attribute

model_config = getattr(model, 'config', None)

vllm_mlx.mllm_scheduler.MLLMScheduler.mm_processor instance-attribute

mm_processor = MultimodalProcessor(model=model, processor=processor, config=self.model_config)

vllm_mlx.mllm_scheduler.MLLMScheduler.stop_tokens instance-attribute

stop_tokens = self._get_stop_tokens()

vllm_mlx.mllm_scheduler.MLLMScheduler.batch_generator instance-attribute

batch_generator: Optional[MLLMBatchGenerator] = None

vllm_mlx.mllm_scheduler.MLLMScheduler.waiting instance-attribute

waiting: deque[MLLMRequest] = deque()

vllm_mlx.mllm_scheduler.MLLMScheduler.running instance-attribute

running: Dict[str, MLLMRequest] = {}

vllm_mlx.mllm_scheduler.MLLMScheduler.requests instance-attribute

requests: Dict[str, MLLMRequest] = {}

vllm_mlx.mllm_scheduler.MLLMScheduler.finished_req_ids instance-attribute

finished_req_ids: Set[str] = set()

vllm_mlx.mllm_scheduler.MLLMScheduler.request_id_to_uid instance-attribute

request_id_to_uid: Dict[str, int] = {}

vllm_mlx.mllm_scheduler.MLLMScheduler.uid_to_request_id instance-attribute

uid_to_request_id: Dict[int, str] = {}

vllm_mlx.mllm_scheduler.MLLMScheduler._detokenizer_pool instance-attribute

_detokenizer_pool: Dict[str, Any] = {}

vllm_mlx.mllm_scheduler.MLLMScheduler.output_queues instance-attribute

output_queues: Dict[str, Queue] = {}

vllm_mlx.mllm_scheduler.MLLMScheduler._running instance-attribute

_running = False

vllm_mlx.mllm_scheduler.MLLMScheduler._processing_task instance-attribute

_processing_task: Optional[Task] = None

vllm_mlx.mllm_scheduler.MLLMScheduler.num_requests_processed instance-attribute

num_requests_processed = 0

vllm_mlx.mllm_scheduler.MLLMScheduler.total_prompt_tokens instance-attribute

total_prompt_tokens = 0

vllm_mlx.mllm_scheduler.MLLMScheduler.total_completion_tokens instance-attribute

total_completion_tokens = 0

vllm_mlx.mllm_scheduler.MLLMScheduler._step_count instance-attribute

_step_count = 0

vllm_mlx.mllm_scheduler.MLLMScheduler._clear_cache_interval instance-attribute

_clear_cache_interval = 32

vllm_mlx.mllm_scheduler.MLLMScheduler._get_stop_tokens

_get_stop_tokens() -> Set[int]

Get stop token IDs from tokenizer and generation_config.json.

Source code in vllm_mlx/mllm_scheduler.py

def _get_stop_tokens(self) -> Set[int]:
    """Get stop token IDs from tokenizer and generation_config.json."""
    stop_tokens = set()
    tokenizer = (
        self.processor.tokenizer
        if hasattr(self.processor, "tokenizer")
        else self.processor
    )

    if hasattr(tokenizer, "eos_token_id") and tokenizer.eos_token_id is not None:
        if isinstance(tokenizer.eos_token_id, list):
            stop_tokens.update(tokenizer.eos_token_id)
        else:
            stop_tokens.add(tokenizer.eos_token_id)

    if hasattr(tokenizer, "eos_token_ids") and tokenizer.eos_token_ids is not None:
        if isinstance(tokenizer.eos_token_ids, (list, set, tuple)):
            stop_tokens.update(tokenizer.eos_token_ids)
        else:
            stop_tokens.add(tokenizer.eos_token_ids)

    # Also read generation_config.json which may have additional EOS tokens
    # (e.g., Gemma 4 has <turn|>=106, <|tool_response>=50 as EOS)
    model_path = getattr(tokenizer, "name_or_path", None)
    if model_path:
        import json
        from pathlib import Path

        gc_path = Path(model_path) / "generation_config.json"
        if gc_path.exists():
            try:
                gc = json.loads(gc_path.read_text())
                gc_eos = gc.get("eos_token_id")
                if isinstance(gc_eos, list):
                    stop_tokens.update(gc_eos)
                elif gc_eos is not None:
                    stop_tokens.add(gc_eos)
            except Exception:
                pass

    return stop_tokens

vllm_mlx.mllm_scheduler.MLLMScheduler._ensure_batch_generator

_ensure_batch_generator() -> None

Ensure batch generator exists.

Source code in vllm_mlx/mllm_scheduler.py

def _ensure_batch_generator(self) -> None:
    """Ensure batch generator exists."""
    if self.batch_generator is None:
        from mlx_lm.sample_utils import make_sampler

        from .memory_cache import MemoryCacheConfig

        # Default sampler (can be overridden per-request in future)
        sampler = make_sampler(temp=0.7, top_p=0.9)

        # Configure KV prefix cache for text-only requests
        # KV cache quantization reduces prefix cache memory ~4x (BF16→Q8).
        # Quantization happens on store(), dequantization on fetch() —
        # the model always receives normal KVCache with plain arrays.
        prefix_cache_config = None
        if self.config.enable_prefix_cache and self.config.use_memory_aware_cache:
            prefix_cache_config = MemoryCacheConfig(
                max_memory_mb=self.config.prefix_cache_memory_mb,
                kv_quantize=self.config.kv_cache_quantization,
                kv_bits=self.config.kv_cache_quantization_bits,
                kv_group_size=self.config.kv_cache_quantization_group_size,
            )

        self.batch_generator = MLLMBatchGenerator(
            model=self.model,
            processor=self.processor,
            mm_processor=self.mm_processor,
            max_tokens=self.config.default_max_tokens,
            stop_tokens=self.stop_tokens,
            sampler=sampler,
            prefill_batch_size=self.config.prefill_batch_size,
            completion_batch_size=self.config.completion_batch_size,
            prefill_step_size=self.config.prefill_step_size,
            prefix_cache_config=prefix_cache_config,
            max_kv_size=self.config.max_kv_size,
        )

        # Wire the SSD cold tier onto the MLLM prefix cache, mirroring the
        # standard Scheduler path (see scheduler.py ~1226).  Without this
        # --ssd-cache-dir is a silent no-op for MLLM models (Qwen3.5 et al.)
        # because the SSD tier was only ever attached to the standard
        # Scheduler's MemoryAwarePrefixCache.  No-op when the flag is unset.
        self._ssd_tier = None
        prefix_cache = getattr(self.batch_generator, "prefix_cache", None)
        if self.config.ssd_cache_dir is not None and prefix_cache is not None:
            from .ssd_cache import SSDCacheConfig, SSDCacheTier

            ssd_config = SSDCacheConfig(
                cache_dir=self.config.ssd_cache_dir,
                max_size_gb=self.config.ssd_cache_max_gb,
            )
            self._ssd_tier = SSDCacheTier(ssd_config)
            self._ssd_tier.start_writer()
            self._ssd_tier.reconcile()
            prefix_cache.set_ssd_tier(self._ssd_tier)
            logger.info(
                "[mllm] SSD cache tier enabled on MLLM prefix cache: "
                "dir=%s, max=%sGB",
                self.config.ssd_cache_dir,
                self.config.ssd_cache_max_gb,
            )

        # Install chunked prefill BEFORE MTP (MTP wraps _next,
        # chunked replaces it — MTP then wraps the chunked version)
        if self.config.chunked_prefill_tokens > 0:
            from .mllm_batch_generator import install_chunked_prefill_mllm

            install_chunked_prefill_mllm(
                self.batch_generator,
                budget=self.config.chunked_prefill_tokens,
            )

        # Install MTP if enabled and language model supports it
        if self.config.enable_mtp:
            lm = self.batch_generator.language_model
            if hasattr(lm, "mtp") and lm.mtp is not None:
                from .mllm_batch_generator import install_mtp_mllm

                install_mtp_mllm(
                    self.batch_generator,
                    lm,
                    num_draft_tokens=self.config.mtp_num_draft_tokens,
                )

vllm_mlx.mllm_scheduler.MLLMScheduler.add_request

add_request(prompt: str, images: Optional[List[str]] = None, videos: Optional[List[str]] = None, audio: Optional[List[str]] = None, max_tokens: int = 256, temperature: float = 0.7, top_p: float = 0.9, request_id: Optional[str] = None, **kwargs) -> str

Add a multimodal request to the scheduler (sync version).

Parameters:

• prompt (str) –

  Text prompt (should be formatted with chat template)

• images (Optional[List[str]], default: None ) –

  List of image inputs (paths, URLs, base64)

• videos (Optional[List[str]], default: None ) –

  List of video inputs

• audio (Optional[List[str]], default: None ) –

  List of audio inputs

• max_tokens (int, default: 256 ) –

  Maximum tokens to generate

• temperature (float, default: 0.7 ) –

  Sampling temperature

• top_p (float, default: 0.9 ) –

  Top-p sampling

• request_id (Optional[str], default: None ) –

  Optional custom request ID

• **kwargs –

  Additional generation parameters. logits_processors — list of callables (tokens, logits) -> logits applied during sampling (e.g. constrained JSON decoding).

Returns:

• str –

  Request ID for tracking

Source code in vllm_mlx/mllm_scheduler.py

def add_request(
    self,
    prompt: str,
    images: Optional[List[str]] = None,
    videos: Optional[List[str]] = None,
    audio: Optional[List[str]] = None,
    max_tokens: int = 256,
    temperature: float = 0.7,
    top_p: float = 0.9,
    request_id: Optional[str] = None,
    **kwargs,
) -> str:
    """
    Add a multimodal request to the scheduler (sync version).

    Args:
        prompt: Text prompt (should be formatted with chat template)
        images: List of image inputs (paths, URLs, base64)
        videos: List of video inputs
        audio: List of audio inputs
        max_tokens: Maximum tokens to generate
        temperature: Sampling temperature
        top_p: Top-p sampling
        request_id: Optional custom request ID
        **kwargs: Additional generation parameters.  ``logits_processors``
            — list of callables ``(tokens, logits) -> logits`` applied
            during sampling (e.g. constrained JSON decoding).

    Returns:
        Request ID for tracking
    """
    if request_id is None:
        request_id = str(uuid.uuid4())

    sampling_params = SamplingParams(
        max_tokens=max_tokens,
        temperature=temperature,
        top_p=top_p,
        top_k=kwargs.pop("top_k", 0),
        min_p=kwargs.pop("min_p", 0.0),
        presence_penalty=kwargs.pop("presence_penalty", 0.0),
        repetition_penalty=kwargs.pop("repetition_penalty", 1.0),
        logits_processors=kwargs.pop("logits_processors", None),
    )

    request = MLLMRequest(
        request_id=request_id,
        prompt=prompt,
        images=images,
        videos=videos,
        audio=audio,
        sampling_params=sampling_params,
    )

    # Estimate prompt token count for monitoring (text tokens only;
    # vision tokens are added during prefill but this gives a useful
    # approximation for the status endpoint).
    tokenizer = (
        self.processor.tokenizer
        if hasattr(self.processor, "tokenizer")
        else self.processor
    )
    try:
        request.num_prompt_tokens = len(tokenizer.encode(prompt))
    except Exception:
        pass

    self.requests[request_id] = request
    self.waiting.append(request)

    logger.debug(
        f"Added MLLM request {request_id}: "
        f"{len(images or [])} images, {len(videos or [])} videos"
    )

    return request_id

vllm_mlx.mllm_scheduler.MLLMScheduler.abort_request

abort_request(request_id: str) -> bool

Abort a request.

Parameters:

• request_id (str) –

  The request ID to abort

Returns:

• bool –

  True if request was found and aborted

Source code in vllm_mlx/mllm_scheduler.py

def abort_request(self, request_id: str) -> bool:
    """
    Abort a request.

    Args:
        request_id: The request ID to abort

    Returns:
        True if request was found and aborted
    """
    request = self.requests.get(request_id)
    if request is None:
        return False

    # Signal batch generator to abort any in-progress prefill for this
    # request.  The prefill loop checks _aborted_request_ids between
    # chunks and raises PrefillAbortedError to exit early.
    if self.batch_generator is not None:
        self.batch_generator.abort_prefill(request_id)

    # Remove from waiting queue
    if request.status == RequestStatus.WAITING:
        try:
            self.waiting.remove(request)
        except ValueError:
            pass

    # Remove from batch generator.
    #
    # IMPORTANT: `abort_request` may be called from the asyncio event
    # loop (e.g. in `stream_outputs`' `finally` block on client
    # disconnect) while `scheduler.step()` — and therefore the
    # batch generator's forward pass — is running on a separate
    # executor thread (see engine_core.py: loop.run_in_executor).
    #
    # Calling `batch_generator.remove([uid])` eagerly here would
    # trigger `active_batch.filter(...)`, which creates an
    # `mx.array` and submits Metal work.  If the scheduler thread
    # has an open Metal encoder mid-forward-pass, two threads
    # submit to the same stream concurrently and Metal asserts
    # with ``encodeSignalEvent:value: with uncommitted encoder``,
    # aborting the process.
    #
    # Instead we defer the removal to the scheduler thread: it
    # will drain the queue at the next safe boundary (start of
    # step(), before any forward pass).
    if request_id in self.request_id_to_uid:
        uid = self.request_id_to_uid[request_id]
        if self.batch_generator is not None:
            self.batch_generator.schedule_removal([uid])
        del self.uid_to_request_id[uid]
        del self.request_id_to_uid[request_id]

    if request_id in self.running:
        del self.running[request_id]

    # Credit in-flight tokens so dashboard metrics stay accurate
    # (without this, aborted requests' tokens vanish from /v1/status).
    if request.num_output_tokens > 0:
        self.total_completion_tokens += request.num_output_tokens
        self.total_prompt_tokens += request.num_prompt_tokens

    # Mark as aborted
    request.status = RequestStatus.FINISHED_ABORTED
    self.finished_req_ids.add(request_id)
    self.requests.pop(request_id, None)

    self._detokenizer_pool.pop(request_id, None)

    # Signal output queue
    if request_id in self.output_queues:
        try:
            self.output_queues[request_id].put_nowait(None)
        except asyncio.QueueFull:
            pass

    logger.debug(f"Aborted request {request_id}")
    return True

vllm_mlx.mllm_scheduler.MLLMScheduler.has_requests

has_requests() -> bool

Check if there are any pending or running requests.

Source code in vllm_mlx/mllm_scheduler.py

def has_requests(self) -> bool:
    """Check if there are any pending or running requests."""
    return bool(self.waiting or self.running)

vllm_mlx.mllm_scheduler.MLLMScheduler.get_num_waiting

get_num_waiting() -> int

Get number of waiting requests.

Source code in vllm_mlx/mllm_scheduler.py

def get_num_waiting(self) -> int:
    """Get number of waiting requests."""
    return len(self.waiting)

vllm_mlx.mllm_scheduler.MLLMScheduler.get_num_running

get_num_running() -> int

Get number of running requests.

Source code in vllm_mlx/mllm_scheduler.py

def get_num_running(self) -> int:
    """Get number of running requests."""
    return len(self.running)

vllm_mlx.mllm_scheduler.MLLMScheduler._schedule_waiting

_schedule_waiting() -> List[MLLMRequest]

Move requests from waiting queue to running.

Returns:

• List[MLLMRequest] –

  List of requests that were scheduled

Source code in vllm_mlx/mllm_scheduler.py

def _schedule_waiting(self) -> List[MLLMRequest]:
    """
    Move requests from waiting queue to running.

    Returns:
        List of requests that were scheduled
    """
    self._ensure_batch_generator()

    scheduled = []
    batch_requests = []

    while self.waiting and len(self.running) < self.config.max_num_seqs:
        request = self.waiting.popleft()

        # Create batch request
        batch_req = MLLMBatchRequest(
            uid=-1,  # Will be assigned by batch generator
            request_id=request.request_id,
            prompt=request.prompt,
            images=request.images,
            videos=request.videos,
            audio=request.audio,
            max_tokens=request.sampling_params.max_tokens,
            temperature=request.sampling_params.temperature,
            top_p=request.sampling_params.top_p,
            top_k=request.sampling_params.top_k,
            min_p=request.sampling_params.min_p,
            presence_penalty=request.sampling_params.presence_penalty,
            repetition_penalty=request.sampling_params.repetition_penalty,
            logits_processors=request.sampling_params.logits_processors,
        )
        batch_requests.append(batch_req)

        request.status = RequestStatus.RUNNING
        self.running[request.request_id] = request
        scheduled.append(request)

        self.total_prompt_tokens += request.num_prompt_tokens

    # Insert into batch generator
    if batch_requests and self.batch_generator is not None:
        uids = self.batch_generator.insert(batch_requests)

        for uid, request in zip(uids, scheduled):
            self.request_id_to_uid[request.request_id] = uid
            self.uid_to_request_id[uid] = request.request_id
            request.batch_uid = uid

            logger.debug(f"Scheduled request {request.request_id} (uid={uid})")

    return scheduled

vllm_mlx.mllm_scheduler.MLLMScheduler._process_batch_responses

_process_batch_responses(responses: List[MLLMBatchResponse]) -> Tuple[List[RequestOutput], Set[str]]

Process responses from batch generator.

Parameters:

• responses (List[MLLMBatchResponse]) –

  List of MLLMBatchResponse objects

Returns:

• Tuple[List[RequestOutput], Set[str]] –

  Tuple of (outputs, finished_request_ids)

Source code in vllm_mlx/mllm_scheduler.py

def _process_batch_responses(
    self, responses: List[MLLMBatchResponse]
) -> Tuple[List[RequestOutput], Set[str]]:
    """
    Process responses from batch generator.

    Args:
        responses: List of MLLMBatchResponse objects

    Returns:
        Tuple of (outputs, finished_request_ids)
    """
    outputs = []
    finished_ids = set()

    tokenizer = (
        self.processor.tokenizer
        if hasattr(self.processor, "tokenizer")
        else self.processor
    )

    for response in responses:
        request_id = self.uid_to_request_id.get(response.uid)
        if request_id is None:
            continue

        request = self.running.get(request_id)
        if request is None:
            continue

        # Handle error responses from failed preprocessing
        if response.finish_reason == "error":
            output = RequestOutput(
                request_id=request_id,
                new_token_ids=[],
                new_text="",
                output_token_ids=[],
                prompt_tokens=0,
                completion_tokens=0,
                finished=True,
                finish_reason="error",
            )
            request.status = RequestStatus.FINISHED_ABORTED
            request.output_text = ""
            request.finish_reason = "error"
            finished_ids.add(request_id)
            self.num_requests_processed += 1
            logger.warning(f"Request {request_id} failed during preprocessing")
            outputs.append(output)
            continue

        # Append token to request
        request.output_tokens.append(response.token)
        request.num_output_tokens = len(request.output_tokens)
        if response.mtp_attempted:
            request.mtp_drafts += response.mtp_attempted_count
        if response.from_draft:
            request.mtp_accepted += 1

        if request.first_token_time is None and request.num_output_tokens > 0:
            request.first_token_time = time.time()

        # Decode the new token using streaming detokenizer (UTF-8 safe).
        # Skip stop tokens — they are not content.
        if response.finish_reason == "stop":
            new_text = ""
        else:
            if request_id not in self._detokenizer_pool:
                detok = NaiveStreamingDetokenizer(tokenizer)
                self._detokenizer_pool[request_id] = detok
            detok = self._detokenizer_pool[request_id]
            detok.add_token(response.token)
            new_text = detok.last_segment

        # Create output
        output = RequestOutput(
            request_id=request_id,
            new_token_ids=[response.token],
            new_text=new_text,
            output_token_ids=request.output_tokens,
            prompt_tokens=request.num_prompt_tokens,
            completion_tokens=request.num_output_tokens,
            mtp_drafts=request.mtp_drafts,
            mtp_accepted=request.mtp_accepted,
        )

        # Check if finished
        if response.finish_reason is not None:
            if response.finish_reason == "stop":
                request.status = RequestStatus.FINISHED_STOPPED
            elif response.finish_reason == "length":
                request.status = RequestStatus.FINISHED_LENGTH_CAPPED

            output.finished = True
            output.finish_reason = response.finish_reason
            finished_ids.add(request_id)

            # Finalize streaming detokenizer and get full output
            detok = self._detokenizer_pool.pop(request_id, None)
            if detok is not None:
                detok.finalize()
                output.output_text = detok.text
            else:
                output.output_text = tokenizer.decode(request.output_tokens)
            request.output_text = output.output_text
            request.finish_reason = response.finish_reason

            self.total_completion_tokens += request.num_output_tokens
            self.num_requests_processed += 1

            logger.debug(
                f"Request {request_id} finished: {response.finish_reason}, "
                f"{request.num_output_tokens} tokens"
            )

        outputs.append(output)

    return outputs, finished_ids

vllm_mlx.mllm_scheduler.MLLMScheduler._cleanup_finished

_cleanup_finished(finished_ids: Set[str]) -> None

Clean up finished requests.

Source code in vllm_mlx/mllm_scheduler.py

def _cleanup_finished(self, finished_ids: Set[str]) -> None:
    """Clean up finished requests."""
    for request_id in finished_ids:
        # Remove from running
        if request_id in self.running:
            del self.running[request_id]

        # Drain from requests dict to prevent linear memory growth
        self.requests.pop(request_id, None)

        # Remove UID mappings
        if request_id in self.request_id_to_uid:
            uid = self.request_id_to_uid[request_id]
            if uid in self.uid_to_request_id:
                del self.uid_to_request_id[uid]
            del self.request_id_to_uid[request_id]

        # Clean up detokenizer pool (handles abort/timeout cases)
        self._detokenizer_pool.pop(request_id, None)

        # Track as finished
        self.finished_req_ids.add(request_id)
        self.requests.pop(request_id, None)

    # Clear Metal buffer pool after cleanup to release memory
    if finished_ids:
        mx.clear_cache()

vllm_mlx.mllm_scheduler.MLLMScheduler.step

step() -> MLLMSchedulerOutput

Execute one scheduling step.

This method: 1. Schedules waiting requests into the batch 2. Runs one generation step via MLLMBatchGenerator 3. Processes outputs and handles finished requests

Returns:

• MLLMSchedulerOutput –

  MLLMSchedulerOutput with results of this step

Source code in vllm_mlx/mllm_scheduler.py

def step(self) -> MLLMSchedulerOutput:
    """
    Execute one scheduling step.

    This method:
    1. Schedules waiting requests into the batch
    2. Runs one generation step via MLLMBatchGenerator
    3. Processes outputs and handles finished requests

    Returns:
        MLLMSchedulerOutput with results of this step
    """
    output = MLLMSchedulerOutput()

    # Drain any deferred removals queued from other threads (e.g.
    # the asyncio event loop during client-disconnect aborts).
    # This MUST run before any forward pass to avoid the Metal
    # ``encodeSignalEvent: uncommitted encoder`` race.  See
    # `abort_request` and `MLLMBatchGenerator.schedule_removal`.
    if self.batch_generator is not None:
        self.batch_generator.process_pending_removals()

    # Schedule waiting requests
    scheduled = self._schedule_waiting()
    output.scheduled_request_ids = [r.request_id for r in scheduled]
    output.num_scheduled_tokens = sum(r.num_prompt_tokens for r in scheduled)

    # Run generation step if we have running requests
    if self.batch_generator is not None and self.running:
        responses = self.batch_generator.next()
        output.has_work = True

        if responses:
            outputs, finished_ids = self._process_batch_responses(responses)
            output.outputs = outputs
            output.finished_request_ids = finished_ids

            # Push to async queues
            for req_output in outputs:
                queue = self.output_queues.get(req_output.request_id)
                if queue is not None:
                    try:
                        queue.put_nowait(req_output)
                        if req_output.finished:
                            queue.put_nowait(None)  # Signal end
                    except asyncio.QueueFull:
                        pass

            self._cleanup_finished(finished_ids)
            if finished_ids:
                mx.clear_cache()

    # Adaptive periodic cache clear: scale inversely with concurrency
    # to prevent Metal buffer pool growth during long generations
    active_seqs = len(self.running)
    min_interval = max(4, self._clear_cache_interval // 4)
    effective_interval = max(
        min_interval, self._clear_cache_interval // max(1, active_seqs // 8)
    )

    self._step_count += 1
    if self._step_count % effective_interval == 0:
        mx.clear_cache()

    # Clear finished tracking for next step
    self.finished_req_ids = set()

    return output

vllm_mlx.mllm_scheduler.MLLMScheduler.get_request

get_request(request_id: str) -> Optional[MLLMRequest]

Get a request by ID.

Source code in vllm_mlx/mllm_scheduler.py

def get_request(self, request_id: str) -> Optional[MLLMRequest]:
    """Get a request by ID."""
    return self.requests.get(request_id)

vllm_mlx.mllm_scheduler.MLLMScheduler.remove_finished_request

remove_finished_request(request_id: str) -> Optional[MLLMRequest]

Remove a finished request from tracking.

Source code in vllm_mlx/mllm_scheduler.py

def remove_finished_request(self, request_id: str) -> Optional[MLLMRequest]:
    """Remove a finished request from tracking."""
    return self.requests.pop(request_id, None)

vllm_mlx.mllm_scheduler.MLLMScheduler.start async

start() -> None

Start the async scheduler processing loop.

Source code in vllm_mlx/mllm_scheduler.py

async def start(self) -> None:
    """Start the async scheduler processing loop."""
    if self._running:
        return

    self._running = True
    self._processing_task = asyncio.create_task(self._process_loop())
    logger.info(
        f"MLLM Scheduler started with max_num_seqs={self.config.max_num_seqs}"
    )

vllm_mlx.mllm_scheduler.MLLMScheduler.stop async

stop() -> None

Stop the scheduler.

Source code in vllm_mlx/mllm_scheduler.py

async def stop(self) -> None:
    """Stop the scheduler."""
    self._running = False
    if self._processing_task:
        self._processing_task.cancel()
        try:
            await self._processing_task
        except asyncio.CancelledError:
            pass

    if self.batch_generator is not None:
        self.batch_generator.close()
        self.batch_generator = None

    logger.info("MLLM Scheduler stopped")

vllm_mlx.mllm_scheduler.MLLMScheduler._process_loop async

_process_loop() -> None

Main async processing loop.

MLLM models are loaded on the server/event-loop thread, so their MLX arrays and cache state must be consumed on that same thread. Unlike the text-only EngineCore path, moving MLLM prefill to a worker crosses MLX stream ownership and can fail with "no Stream in current thread".

Text-only preprocessing (Jinja2 template rendering + tokenization) is run BEFORE step() with await asyncio.sleep(0) yields between each request. This prevents long preprocessing (10-30+ s for 40K+ token conversations) from blocking health checks and new connections.

Source code in vllm_mlx/mllm_scheduler.py

async def _process_loop(self) -> None:
    """Main async processing loop.

    MLLM models are loaded on the server/event-loop thread, so their MLX
    arrays and cache state must be consumed on that same thread.  Unlike
    the text-only EngineCore path, moving MLLM prefill to a worker crosses
    MLX stream ownership and can fail with "no Stream in current thread".

    Text-only preprocessing (Jinja2 template rendering + tokenization) is
    run BEFORE ``step()`` with ``await asyncio.sleep(0)`` yields between
    each request.  This prevents long preprocessing (10-30+ s for 40K+
    token conversations) from blocking health checks and new connections.
    """
    streams_bound = False

    def _ensure_streams_bound() -> None:
        nonlocal streams_bound
        if not streams_bound:
            bind_generation_streams()
            streams_bound = True

    loop = asyncio.get_running_loop()

    while self._running:
        try:
            # --- Early preprocessing phase ---
            # Run text-only preprocessing (Jinja2 template rendering +
            # tokenization) in a thread-pool executor so the event loop
            # stays responsive for health checks, new connections, and
            # active streaming requests.  Preprocessing is CPU-bound
            # (no MLX GPU work) and HuggingFace tokenizers are
            # thread-safe, so this is safe to offload.
            bg = self.batch_generator
            if bg is not None:
                for req in list(getattr(bg, "unprocessed_requests", ())):
                    if (
                        req.input_ids is None
                        and not req.images
                        and not req.videos
                        and not req.audio
                    ):
                        try:
                            tic = time.perf_counter()
                            await loop.run_in_executor(
                                None, bg._preprocess_request, req
                            )
                            elapsed = time.perf_counter() - tic
                            if elapsed > 1.0:
                                n_tok = (
                                    req.input_ids.size
                                    if req.input_ids is not None
                                    else 0
                                )
                                logger.info(
                                    f"Preprocessing {req.request_id[:12]}"
                                    f": {n_tok} tokens in {elapsed:.2f}s"
                                )
                        except Exception as e:
                            logger.error(
                                f"Early preprocessing failed for "
                                f"{req.request_id}: {e}"
                            )

            # --- Step phase ---
            if self.has_requests():
                _ensure_streams_bound()
                tic = time.perf_counter()
                self.step()
                elapsed = time.perf_counter() - tic
                if elapsed > 2.0:
                    logger.warning(
                        f"Slow MLLM step: {elapsed:.2f}s "
                        f"(waiting={len(self.waiting)}, "
                        f"running={len(self.running)})"
                    )
                # Yield multiple event-loop cycles so that pending
                # HTTP health checks can complete.  A single
                # asyncio.sleep() gives only ONE _run_once() cycle,
                # but an HTTP request needs ~3 cycles minimum:
                #   1. accept TCP connection
                #   2. read HTTP request / parse headers
                #   3. run handler / write response
                # Using repeated asyncio.sleep(0) gives many cycles
                # with negligible wall-clock overhead (<1ms total).
                n_yields = 10 if elapsed > 1.0 else 5
                for _ in range(n_yields):
                    await asyncio.sleep(0)
            else:
                # No work, wait a bit
                await asyncio.sleep(0.01)

        except asyncio.CancelledError:
            raise
        except Exception as e:
            logger.error(f"Error in MLLM process loop: {e}", exc_info=True)
            await asyncio.sleep(0.1)

vllm_mlx.mllm_scheduler.MLLMScheduler.add_request_async async

add_request_async(prompt: str, images: Optional[List[str]] = None, videos: Optional[List[str]] = None, audio: Optional[List[str]] = None, max_tokens: int = 256, temperature: float = 0.7, top_p: float = 0.9, **kwargs) -> str

Add a multimodal request (async version with output queue).

Parameters:

• prompt (str) –

  Text prompt

• images (Optional[List[str]], default: None ) –

  List of image inputs

• videos (Optional[List[str]], default: None ) –

  List of video inputs

• audio (Optional[List[str]], default: None ) –

  List of audio inputs

• max_tokens (int, default: 256 ) –

  Maximum tokens to generate

• temperature (float, default: 0.7 ) –

  Sampling temperature

• top_p (float, default: 0.9 ) –

  Top-p sampling

• **kwargs –

  Additional parameters

Returns:

• str –

  Request ID for tracking

Source code in vllm_mlx/mllm_scheduler.py

async def add_request_async(
    self,
    prompt: str,
    images: Optional[List[str]] = None,
    videos: Optional[List[str]] = None,
    audio: Optional[List[str]] = None,
    max_tokens: int = 256,
    temperature: float = 0.7,
    top_p: float = 0.9,
    **kwargs,
) -> str:
    """
    Add a multimodal request (async version with output queue).

    Args:
        prompt: Text prompt
        images: List of image inputs
        videos: List of video inputs
        audio: List of audio inputs
        max_tokens: Maximum tokens to generate
        temperature: Sampling temperature
        top_p: Top-p sampling
        **kwargs: Additional parameters

    Returns:
        Request ID for tracking
    """
    request_id = self.add_request(
        prompt=prompt,
        images=images,
        videos=videos,
        audio=audio,
        max_tokens=max_tokens,
        temperature=temperature,
        top_p=top_p,
        **kwargs,
    )

    # Create output queue for streaming
    self.output_queues[request_id] = asyncio.Queue()

    return request_id

vllm_mlx.mllm_scheduler.MLLMScheduler.stream_outputs async

stream_outputs(request_id: str) -> AsyncIterator[RequestOutput]

Stream outputs for a request.

Parameters:

• request_id (str) –

  The request ID to stream

Yields:

• AsyncIterator[RequestOutput] –

  RequestOutput objects as tokens are generated

Source code in vllm_mlx/mllm_scheduler.py

async def stream_outputs(
    self,
    request_id: str,
) -> AsyncIterator[RequestOutput]:
    """
    Stream outputs for a request.

    Args:
        request_id: The request ID to stream

    Yields:
        RequestOutput objects as tokens are generated
    """
    output_queue = self.output_queues.get(request_id)
    if output_queue is None:
        return

    finished_normally = False
    try:
        while True:
            output = await output_queue.get()
            if output is None:
                finished_normally = True
                break
            if output.finished:
                finished_normally = True
                yield output
                break
            yield output
    finally:
        if not finished_normally:
            logger.info(f"Aborting orphaned MLLM request {request_id}")
            self.abort_request(request_id)
        # Cleanup queue
        if request_id in self.output_queues:
            del self.output_queues[request_id]

vllm_mlx.mllm_scheduler.MLLMScheduler.generate async

generate(prompt: str, images: Optional[List[str]] = None, videos: Optional[List[str]] = None, audio: Optional[List[str]] = None, **kwargs) -> RequestOutput

Generate complete output for a request (non-streaming).

Parameters:

• prompt (str) –

  Text prompt

• images (Optional[List[str]], default: None ) –

  Image inputs

• videos (Optional[List[str]], default: None ) –

  Video inputs

• audio (Optional[List[str]], default: None ) –

  Audio inputs

• **kwargs –

  Generation parameters

Returns:

• RequestOutput –

  Final RequestOutput

Source code in vllm_mlx/mllm_scheduler.py

async def generate(
    self,
    prompt: str,
    images: Optional[List[str]] = None,
    videos: Optional[List[str]] = None,
    audio: Optional[List[str]] = None,
    **kwargs,
) -> RequestOutput:
    """
    Generate complete output for a request (non-streaming).

    Args:
        prompt: Text prompt
        images: Image inputs
        videos: Video inputs
        audio: Audio inputs
        **kwargs: Generation parameters

    Returns:
        Final RequestOutput
    """
    request_id = await self.add_request_async(
        prompt=prompt,
        images=images,
        videos=videos,
        audio=audio,
        **kwargs,
    )

    # Collect all outputs
    final_output = None
    async for output in self.stream_outputs(request_id):
        final_output = output
        if output.finished:
            break

    if final_output is None:
        # Create empty output on error
        final_output = RequestOutput(
            request_id=request_id,
            output_text="",
            finished=True,
            finish_reason="error",
        )

    # Cleanup
    if request_id in self.requests:
        del self.requests[request_id]

    return final_output

vllm_mlx.mllm_scheduler.MLLMScheduler.get_running_requests_info

get_running_requests_info() -> List[Dict[str, Any]]

Per-request details for status endpoint.

Source code in vllm_mlx/mllm_scheduler.py

def get_running_requests_info(self) -> List[Dict[str, Any]]:
    """Per-request details for status endpoint."""
    now = time.time()
    result = []

    # Waiting requests
    for req in self.waiting:
        result.append(
            {
                "request_id": req.request_id,
                "status": "waiting",
                "phase": "queued",
                "elapsed_s": round(now - req.arrival_time, 2),
                "prompt_tokens": req.num_prompt_tokens,
                "completion_tokens": 0,
                "max_tokens": req.sampling_params.max_tokens,
                "progress": 0.0,
                "tokens_per_second": None,
                "ttft_s": None,
                "cache_hit_type": None,
                "cached_tokens": 0,
            }
        )

    # Running requests
    for req in self.running.values():
        n_out = req.num_output_tokens
        elapsed = now - req.arrival_time

        if n_out == 0:
            phase = "prefill"
        else:
            phase = "generation"

        tok_s = None
        ttft = None
        if req.first_token_time is not None:
            ttft = round(req.first_token_time - req.arrival_time, 3)
            gen_elapsed = now - req.first_token_time
            if gen_elapsed > 0 and n_out > 0:
                tok_s = round(n_out / gen_elapsed, 1)

        max_tokens = req.sampling_params.max_tokens
        if phase == "prefill" and self.batch_generator is not None:
            pp = self.batch_generator.get_prefill_progress(req.request_id)
            if pp is not None:
                progress = round(pp[0] / pp[1], 3) if pp[1] > 0 else 0.0
            else:
                progress = 0.0
        else:
            progress = round(n_out / max_tokens, 3) if max_tokens > 0 else 0.0

        result.append(
            {
                "request_id": req.request_id,
                "status": "running",
                "phase": phase,
                "elapsed_s": round(elapsed, 2),
                "prompt_tokens": req.num_prompt_tokens,
                "completion_tokens": n_out,
                "max_tokens": max_tokens,
                "progress": min(progress, 1.0),
                "tokens_per_second": tok_s,
                "ttft_s": ttft,
                "cache_hit_type": None,
                "cached_tokens": 0,
            }
        )

    return result

vllm_mlx.mllm_scheduler.MLLMScheduler.get_stats

get_stats() -> Dict[str, Any]

Get scheduler statistics.

Source code in vllm_mlx/mllm_scheduler.py

def get_stats(self) -> Dict[str, Any]:
    """Get scheduler statistics."""
    stats = {
        "num_waiting": len(self.waiting),
        "num_running": len(self.running),
        "num_finished": len(self.finished_req_ids),
        "num_requests_processed": self.num_requests_processed,
        "total_prompt_tokens": self.total_prompt_tokens,
        "total_completion_tokens": self.total_completion_tokens,
        "requests": self.get_running_requests_info(),
    }

    if self.batch_generator is not None:
        batch_stats = self.batch_generator.stats()
        stats["batch_generator"] = batch_stats.to_dict()
        # Vision embedding cache stats from batch generator
        vec_stats = self.batch_generator.get_vision_cache_stats()
        stats["vision_embedding_cache"] = vec_stats
        if hasattr(self.batch_generator, "get_mtp_stats"):
            stats["mtp"] = self.batch_generator.get_mtp_stats()

    # Include Metal memory stats
    try:
        if mx.metal.is_available():
            active_gb = round(mx.get_active_memory() / 1e9, 2)
            peak_gb = round(mx.get_peak_memory() / 1e9, 2)
            cache_gb = round(mx.get_cache_memory() / 1e9, 2)
            stats["metal_active_memory_gb"] = active_gb
            stats["metal_peak_memory_gb"] = peak_gb
            stats["metal_cache_memory_gb"] = cache_gb
    except Exception:
        active_gb = 0
        cache_gb = 0

    # KV prefix cache stats for /v1/status and monitoring UI.
    if self.batch_generator is not None:
        prefix_stats = self.batch_generator.get_prefix_cache_stats()
    else:
        prefix_stats = {
            "hits": 0,
            "misses": 0,
            "hit_rate": 0.0,
            "evictions": 0,
            "tokens_saved": 0,
            "current_memory_mb": 0.0,
            "max_memory_mb": 0.0,
            "memory_utilization": 0.0,
            "entry_count": 0,
        }
    stats["memory_aware_cache"] = prefix_stats

    return stats

vllm_mlx.mllm_scheduler.MLLMScheduler.clear_runtime_caches

clear_runtime_caches() -> Dict[str, bool]

Clear runtime caches without resetting scheduler/request state.

Source code in vllm_mlx/mllm_scheduler.py

def clear_runtime_caches(self) -> Dict[str, bool]:
    """Clear runtime caches without resetting scheduler/request state."""
    cleared = {
        "vision_cache": False,
        "prefix_cache": False,
    }
    if self.vision_cache:
        self.vision_cache.clear()
        cleared["vision_cache"] = True
    if (
        self.batch_generator is not None
        and self.batch_generator.prefix_cache is not None
    ):
        self.batch_generator.prefix_cache.clear()
        cleared["prefix_cache"] = True
    return cleared

vllm_mlx.mllm_scheduler.MLLMScheduler.reset

reset() -> None

Reset the scheduler state.

Source code in vllm_mlx/mllm_scheduler.py

def reset(self) -> None:
    """Reset the scheduler state."""
    # Abort all requests
    for request_id in list(self.requests.keys()):
        self.abort_request(request_id)

    self.waiting.clear()
    self.running.clear()
    self.requests.clear()
    self.finished_req_ids.clear()
    self.request_id_to_uid.clear()
    self.uid_to_request_id.clear()
    self._detokenizer_pool.clear()

    if self.batch_generator is not None:
        self.batch_generator.close()
        self.batch_generator = None

    if self.vision_cache:
        self.vision_cache.clear()

Complete contract reference

Expand any definition for its exact inputs, annotations, defaults, return contract, directly raised exceptions, source-grounded behavior, and immutable line link. This section includes private and nested definitions that ordinary API generators omit.

vllm_mlx.mllm_scheduler.MLLMSchedulerConfig · class

vllm_mlx.mllm_scheduler.MLLMSchedulerConfig(max_num_seqs: int = 16, prefill_batch_size: int = 16, completion_batch_size: int = 16, prefill_step_size: int = 1024, enable_vision_cache: bool = True, vision_cache_size: int = 100, default_max_tokens: int = 256, default_video_fps: float = 2.0, cache_memory_mb: Optional[int] = None, max_video_frames: int = 128, enable_mtp: bool = False, mtp_num_draft_tokens: int = 1, enable_prefix_cache: bool = True, use_memory_aware_cache: bool = True, prefix_cache_memory_mb: Optional[int] = None, kv_cache_quantization: bool = False, kv_cache_quantization_bits: int = 8, kv_cache_quantization_group_size: int = 64, chunked_prefill_tokens: int = 0, max_kv_size: int = 0, ssd_cache_dir: Optional[str] = None, ssd_cache_max_gb: float = 10.0)

Configuration for MLLM scheduler.

Parameters

Name	Type	Required	Default	Description
max_num_seqs	int	no	16	Optional constructor field; defaults to 16.
prefill_batch_size	int	no	16	Optional constructor field; defaults to 16.
completion_batch_size	int	no	16	Optional constructor field; defaults to 16.
prefill_step_size	int	no	1024	Optional constructor field; defaults to 1024.
enable_vision_cache	bool	no	True	Optional constructor field; defaults to True.
vision_cache_size	int	no	100	Optional constructor field; defaults to 100.
default_max_tokens	int	no	256	Optional constructor field; defaults to 256.
default_video_fps	float	no	2.0	Optional constructor field; defaults to 2.0.
cache_memory_mb	Optional[int]	no	None	Optional constructor field; defaults to None.
max_video_frames	int	no	128	Optional constructor field; defaults to 128.
enable_mtp	bool	no	False	Optional constructor field; defaults to False.
mtp_num_draft_tokens	int	no	1	Optional constructor field; defaults to 1.
enable_prefix_cache	bool	no	True	Optional constructor field; defaults to True.
use_memory_aware_cache	bool	no	True	Optional constructor field; defaults to True.
prefix_cache_memory_mb	Optional[int]	no	None	Optional constructor field; defaults to None.
kv_cache_quantization	bool	no	False	Optional constructor field; defaults to False.
kv_cache_quantization_bits	int	no	8	Optional constructor field; defaults to 8.
kv_cache_quantization_group_size	int	no	64	Optional constructor field; defaults to 64.
chunked_prefill_tokens	int	no	0	Optional constructor field; defaults to 0.
max_kv_size	int	no	0	Optional constructor field; defaults to 0.
ssd_cache_dir	Optional[str]	no	None	Optional constructor field; defaults to None.
ssd_cache_max_gb	float	no	10.0	Optional constructor field; defaults to 10.0.

Returns

• Constructs: vllm_mlx.mllm_scheduler.MLLMSchedulerConfig

Exceptions and behavior

Class MLLMSchedulerConfig declares 0 direct member(s). No direct raise statement appears in this definition.

View source #L46-L92.

vllm_mlx.mllm_scheduler.MLLMRequest · class

vllm_mlx.mllm_scheduler.MLLMRequest(request_id: str, prompt: str, images: Optional[List[str]] = None, videos: Optional[List[str]] = None, audio: Optional[List[str]] = None, sampling_params: SamplingParams = field(default_factory=SamplingParams), arrival_time: float = field(default_factory=time.time), batch_uid: Optional[int] = None, status: RequestStatus = RequestStatus.WAITING, output_text: str = '', output_tokens: List[int] = field(default_factory=list), finish_reason: Optional[str] = None, num_prompt_tokens: int = 0, num_output_tokens: int = 0, mtp_drafts: int = 0, mtp_accepted: int = 0, first_token_time: Optional[float] = None)

Extended request for MLLM processing.

Parameters

Name	Type	Required	Default	Description
request_id	str	yes	none	Required constructor field.
prompt	str	yes	none	Required constructor field.
images	Optional[List[str]]	no	None	Optional constructor field; defaults to None.
videos	Optional[List[str]]	no	None	Optional constructor field; defaults to None.
audio	Optional[List[str]]	no	None	Optional constructor field; defaults to None.
sampling_params	SamplingParams	no	field(default_factory=SamplingParams)	Optional constructor field; defaults to field(default_factory=SamplingParams).
arrival_time	float	no	field(default_factory=time.time)	Optional constructor field; defaults to field(default_factory=time.time).
batch_uid	Optional[int]	no	None	Optional constructor field; defaults to None.
status	RequestStatus	no	RequestStatus.WAITING	Optional constructor field; defaults to RequestStatus.WAITING.
output_text	str	no	''	Optional constructor field; defaults to ''.
output_tokens	List[int]	no	field(default_factory=list)	Optional constructor field; defaults to field(default_factory=list).
finish_reason	Optional[str]	no	None	Optional constructor field; defaults to None.
num_prompt_tokens	int	no	0	Optional constructor field; defaults to 0.
num_output_tokens	int	no	0	Optional constructor field; defaults to 0.
mtp_drafts	int	no	0	Optional constructor field; defaults to 0.
mtp_accepted	int	no	0	Optional constructor field; defaults to 0.
first_token_time	Optional[float]	no	None	Optional constructor field; defaults to None.

Returns

• Constructs: vllm_mlx.mllm_scheduler.MLLMRequest

Exceptions and behavior

Class MLLMRequest declares 0 direct member(s). No direct raise statement appears in this definition.

View source #L96-L127.

vllm_mlx.mllm_scheduler.MLLMSchedulerOutput · class

vllm_mlx.mllm_scheduler.MLLMSchedulerOutput(scheduled_request_ids: List[str] = field(default_factory=list), num_scheduled_tokens: int = 0, finished_request_ids: Set[str] = field(default_factory=set), outputs: List[RequestOutput] = field(default_factory=list), has_work: bool = False)

Output from a scheduling step.

Parameters

Name	Type	Required	Default	Description
scheduled_request_ids	List[str]	no	field(default_factory=list)	Optional constructor field; defaults to field(default_factory=list).
num_scheduled_tokens	int	no	0	Optional constructor field; defaults to 0.
finished_request_ids	Set[str]	no	field(default_factory=set)	Optional constructor field; defaults to field(default_factory=set).
outputs	List[RequestOutput]	no	field(default_factory=list)	Optional constructor field; defaults to field(default_factory=list).
has_work	bool	no	False	Optional constructor field; defaults to False.

Returns

• Constructs: vllm_mlx.mllm_scheduler.MLLMSchedulerOutput

Exceptions and behavior

Class MLLMSchedulerOutput declares 0 direct member(s). No direct raise statement appears in this definition.

View source #L131-L147.

vllm_mlx.mllm_scheduler.MLLMScheduler · class

vllm_mlx.mllm_scheduler.MLLMScheduler(model: Any, processor: Any, config: Optional[MLLMSchedulerConfig] = None)

Scheduler for Vision Language Model requests with continuous batching.

Parameters

Name	Type	Required	Default	Description
model	Any	yes	none	The VLM model
processor	Any	yes	none	The VLM processor
config	Optional[MLLMSchedulerConfig]	no	None	Scheduler configuration

Returns

• Constructs: vllm_mlx.mllm_scheduler.MLLMScheduler

Exceptions and behavior

Class MLLMScheduler declares 24 direct member(s). No direct raise statement appears in this definition.

View source #L150-L1242.

vllm_mlx.mllm_scheduler.MLLMScheduler.__init__ · method

vllm_mlx.mllm_scheduler.MLLMScheduler.__init__(model: Any, processor: Any, config: Optional[MLLMSchedulerConfig] = None) -> not annotated

Initialize MLLM scheduler.

Parameters

Name	Type	Required	Default	Description
model	Any	yes	none	The VLM model
processor	Any	yes	none	The VLM processor
config	Optional[MLLMSchedulerConfig]	no	None	Scheduler configuration

Returns

• Type: not annotated

Exceptions and behavior

Method MLLMScheduler.__init__ updates self.model, self.processor, self.config, self.model_config; calls MLLMSchedulerConfig, getattr, MultimodalProcessor, self._get_stop_tokens. No direct raise statement appears in this definition.

View source #L183-L248.

vllm_mlx.mllm_scheduler.MLLMScheduler._get_stop_tokens · method

vllm_mlx.mllm_scheduler.MLLMScheduler._get_stop_tokens() -> Set[int]

Get stop token IDs from tokenizer and generation_config.json.

Parameters

This callable has no explicit inputs.

Returns

• Type: Set[int]
• Direct return expressions: stop_tokens

Exceptions and behavior

Method MLLMScheduler._get_stop_tokens calls set, hasattr, isinstance, stop_tokens.update; returns stop_tokens. No direct raise statement appears in this definition.

View source #L250-L290.

vllm_mlx.mllm_scheduler.MLLMScheduler._ensure_batch_generator · method

vllm_mlx.mllm_scheduler.MLLMScheduler._ensure_batch_generator() -> None

Ensure batch generator exists.

Parameters

This callable has no explicit inputs.

Returns

• Type: None

Exceptions and behavior

Method MLLMScheduler._ensure_batch_generator updates self.batch_generator, self._ssd_tier; calls make_sampler, MemoryCacheConfig, MLLMBatchGenerator, getattr. No direct raise statement appears in this definition.

View source #L292-L374.

vllm_mlx.mllm_scheduler.MLLMScheduler.add_request · method

vllm_mlx.mllm_scheduler.MLLMScheduler.add_request(prompt: str, images: Optional[List[str]] = None, videos: Optional[List[str]] = None, audio: Optional[List[str]] = None, max_tokens: int = 256, temperature: float = 0.7, top_p: float = 0.9, request_id: Optional[str] = None, **kwargs) -> str

Add a multimodal request to the scheduler (sync version).

Parameters

Name	Type	Required	Default	Description
prompt	str	yes	none	Text prompt (should be formatted with chat template)
images	Optional[List[str]]	no	None	List of image inputs (paths, URLs, base64)
videos	Optional[List[str]]	no	None	List of video inputs
audio	Optional[List[str]]	no	None	List of audio inputs
max_tokens	int	no	256	Maximum tokens to generate
temperature	float	no	0.7	Sampling temperature
top_p	float	no	0.9	Top-p sampling
request_id	Optional[str]	no	None	Optional custom request ID
**kwargs	not annotated	no	none	Additional generation parameters. logits_processors — list of callables (tokens, logits) -> logits applied during sampling (e.g. constrained JSON decoding).

Returns

• Type: str
• Direct return expressions: request_id

Exceptions and behavior

Method MLLMScheduler.add_request calls str, uuid.uuid4, SamplingParams, kwargs.pop; returns request_id. No direct raise statement appears in this definition.

View source #L378-L453.

vllm_mlx.mllm_scheduler.MLLMScheduler.abort_request · method

vllm_mlx.mllm_scheduler.MLLMScheduler.abort_request(request_id: str) -> bool

Abort a request.

Parameters

Name	Type	Required	Default	Description
request_id	str	yes	none	The request ID to abort

Returns

• Type: bool
• Direct return expressions: False; True

Exceptions and behavior

Method MLLMScheduler.abort_request updates self.total_completion_tokens, self.total_prompt_tokens; calls self.requests.get, self.batch_generator.abort_prefill, self.waiting.remove, self.batch_generator.schedule_removal; has 2 explicit return paths. No direct raise statement appears in this definition.

View source #L455-L532.

vllm_mlx.mllm_scheduler.MLLMScheduler.has_requests · method

vllm_mlx.mllm_scheduler.MLLMScheduler.has_requests() -> bool

Check if there are any pending or running requests.

Parameters

This callable has no explicit inputs.

Returns

• Type: bool
• Direct return expressions: bool(self.waiting or self.running)

Exceptions and behavior

Method MLLMScheduler.has_requests calls bool; returns bool(self.waiting or self.running). No direct raise statement appears in this definition.

View source #L534-L536.

vllm_mlx.mllm_scheduler.MLLMScheduler.get_num_waiting · method

vllm_mlx.mllm_scheduler.MLLMScheduler.get_num_waiting() -> int

Get number of waiting requests.

Parameters

This callable has no explicit inputs.

Returns

• Type: int
• Direct return expressions: len(self.waiting)

Exceptions and behavior

Method MLLMScheduler.get_num_waiting calls len; returns len(self.waiting). No direct raise statement appears in this definition.

View source #L538-L540.

vllm_mlx.mllm_scheduler.MLLMScheduler.get_num_running · method

vllm_mlx.mllm_scheduler.MLLMScheduler.get_num_running() -> int

Get number of running requests.

Parameters

This callable has no explicit inputs.

Returns

• Type: int
• Direct return expressions: len(self.running)

Exceptions and behavior

Method MLLMScheduler.get_num_running calls len; returns len(self.running). No direct raise statement appears in this definition.

View source #L542-L544.

vllm_mlx.mllm_scheduler.MLLMScheduler._schedule_waiting · method

vllm_mlx.mllm_scheduler.MLLMScheduler._schedule_waiting() -> List[MLLMRequest]

Move requests from waiting queue to running.

Parameters

This callable has no explicit inputs.

Returns

• Type: List[MLLMRequest]
• Direct return expressions: scheduled

Exceptions and behavior

Method MLLMScheduler._schedule_waiting updates self.total_prompt_tokens; calls self._ensure_batch_generator, len, self.waiting.popleft, MLLMBatchRequest; returns scheduled. No direct raise statement appears in this definition.

View source #L546-L597.

vllm_mlx.mllm_scheduler.MLLMScheduler._process_batch_responses · method

vllm_mlx.mllm_scheduler.MLLMScheduler._process_batch_responses(responses: List[MLLMBatchResponse]) -> Tuple[List[RequestOutput], Set[str]]

Process responses from batch generator.

Parameters

Name	Type	Required	Default	Description
responses	List[MLLMBatchResponse]	yes	none	List of MLLMBatchResponse objects

Returns

• Type: Tuple[List[RequestOutput], Set[str]]
• Direct return expressions: (outputs, finished_ids)

Exceptions and behavior

Method MLLMScheduler._process_batch_responses updates self.num_requests_processed, self.total_completion_tokens; calls set, hasattr, self.uid_to_request_id.get, self.running.get; returns (outputs, finished_ids). No direct raise statement appears in this definition.

View source #L599-L716.

vllm_mlx.mllm_scheduler.MLLMScheduler._cleanup_finished · method

vllm_mlx.mllm_scheduler.MLLMScheduler._cleanup_finished(finished_ids: Set[str]) -> None

Clean up finished requests.

Parameters

Name	Type	Required	Default	Description
finished_ids	Set[str]	yes	none	Required positional or keyword input.

Returns

• Type: None

Exceptions and behavior

Method MLLMScheduler._cleanup_finished calls self.requests.pop, self._detokenizer_pool.pop, self.finished_req_ids.add, mx.clear_cache. No direct raise statement appears in this definition.

View source #L718-L744.

vllm_mlx.mllm_scheduler.MLLMScheduler.step · method

vllm_mlx.mllm_scheduler.MLLMScheduler.step() -> MLLMSchedulerOutput

Execute one scheduling step.

Parameters

This callable has no explicit inputs.

Returns

• Type: MLLMSchedulerOutput
• Direct return expressions: output

Exceptions and behavior

Method MLLMScheduler.step updates self._step_count, self.finished_req_ids; calls MLLMSchedulerOutput, self.batch_generator.process_pending_removals, self._schedule_waiting, sum; returns output. No direct raise statement appears in this definition.

View source #L746-L813.

vllm_mlx.mllm_scheduler.MLLMScheduler.get_request · method

vllm_mlx.mllm_scheduler.MLLMScheduler.get_request(request_id: str) -> Optional[MLLMRequest]

Get a request by ID.

Parameters

Name	Type	Required	Default	Description
request_id	str	yes	none	Required positional or keyword input.

Returns

• Type: Optional[MLLMRequest]
• Direct return expressions: self.requests.get(request_id)

Exceptions and behavior

Method MLLMScheduler.get_request calls self.requests.get; returns self.requests.get(request_id). No direct raise statement appears in this definition.

View source #L815-L817.

vllm_mlx.mllm_scheduler.MLLMScheduler.remove_finished_request · method

vllm_mlx.mllm_scheduler.MLLMScheduler.remove_finished_request(request_id: str) -> Optional[MLLMRequest]

Remove a finished request from tracking.

Parameters

Name	Type	Required	Default	Description
request_id	str	yes	none	Required positional or keyword input.

Returns

• Type: Optional[MLLMRequest]
• Direct return expressions: self.requests.pop(request_id, None)

Exceptions and behavior

Method MLLMScheduler.remove_finished_request calls self.requests.pop; returns self.requests.pop(request_id, None). No direct raise statement appears in this definition.

View source #L819-L821.

vllm_mlx.mllm_scheduler.MLLMScheduler.start · method

async vllm_mlx.mllm_scheduler.MLLMScheduler.start() -> None

Start the async scheduler processing loop.

Parameters

This callable has no explicit inputs.

Returns

• Type: None
• Direct return expressions: None

Exceptions and behavior

Method MLLMScheduler.start updates self._running, self._processing_task; calls asyncio.create_task, self._process_loop, logger.info; returns None. No direct raise statement appears in this definition.

View source #L825-L834.

vllm_mlx.mllm_scheduler.MLLMScheduler.stop · method

async vllm_mlx.mllm_scheduler.MLLMScheduler.stop() -> None

Stop the scheduler.

Parameters

This callable has no explicit inputs.

Returns

• Type: None

Exceptions and behavior

Method MLLMScheduler.stop updates self._running, self.batch_generator; calls self._processing_task.cancel, self.batch_generator.close, logger.info; awaits asynchronous work. No direct raise statement appears in this definition.

View source #L836-L850.

vllm_mlx.mllm_scheduler.MLLMScheduler._process_loop · method

async vllm_mlx.mllm_scheduler.MLLMScheduler._process_loop() -> None

Main async processing loop.

Parameters

This callable has no explicit inputs.

Returns

• Type: None

Exceptions and behavior

Method MLLMScheduler._process_loop calls asyncio.get_running_loop, list, getattr, time.perf_counter; awaits asynchronous work. No direct raise statement appears in this definition.

View source #L852-L947.

vllm_mlx.mllm_scheduler.MLLMScheduler._process_loop._ensure_streams_bound · nested function

vllm_mlx.mllm_scheduler.MLLMScheduler._process_loop._ensure_streams_bound() -> None

Nested Function MLLMScheduler._process_loop._ensure_streams_bound calls bind_generation_streams.

Parameters

This callable has no explicit inputs.

Returns

• Type: None

Exceptions and behavior

Nested Function MLLMScheduler._process_loop._ensure_streams_bound calls bind_generation_streams. No direct raise statement appears in this definition.

View source #L867-L871.

vllm_mlx.mllm_scheduler.MLLMScheduler.add_request_async · method

async vllm_mlx.mllm_scheduler.MLLMScheduler.add_request_async(prompt: str, images: Optional[List[str]] = None, videos: Optional[List[str]] = None, audio: Optional[List[str]] = None, max_tokens: int = 256, temperature: float = 0.7, top_p: float = 0.9, **kwargs) -> str

Add a multimodal request (async version with output queue).

Parameters

Name	Type	Required	Default	Description
prompt	str	yes	none	Text prompt
images	Optional[List[str]]	no	None	List of image inputs
videos	Optional[List[str]]	no	None	List of video inputs
audio	Optional[List[str]]	no	None	List of audio inputs
max_tokens	int	no	256	Maximum tokens to generate
temperature	float	no	0.7	Sampling temperature
top_p	float	no	0.9	Top-p sampling
**kwargs	not annotated	no	none	Additional parameters

Returns

• Type: str
• Direct return expressions: request_id

Exceptions and behavior

Method MLLMScheduler.add_request_async calls self.add_request, asyncio.Queue; returns request_id. No direct raise statement appears in this definition.

View source #L949-L990.

vllm_mlx.mllm_scheduler.MLLMScheduler.stream_outputs · method

async vllm_mlx.mllm_scheduler.MLLMScheduler.stream_outputs(request_id: str) -> AsyncIterator[RequestOutput]

Stream outputs for a request.

Parameters

Name	Type	Required	Default	Description
request_id	str	yes	none	The request ID to stream

Returns

• Type: AsyncIterator[RequestOutput]
• Direct return expressions: None
• Yields values incrementally.

Exceptions and behavior

Method MLLMScheduler.stream_outputs calls self.output_queues.get, output_queue.get, logger.info, self.abort_request; awaits asynchronous work; yields values incrementally; returns None. No direct raise statement appears in this definition.

View source #L992-L1027.

vllm_mlx.mllm_scheduler.MLLMScheduler.generate · method

async vllm_mlx.mllm_scheduler.MLLMScheduler.generate(prompt: str, images: Optional[List[str]] = None, videos: Optional[List[str]] = None, audio: Optional[List[str]] = None, **kwargs) -> RequestOutput

Generate complete output for a request (non-streaming).

Parameters

Name	Type	Required	Default	Description
prompt	str	yes	none	Text prompt
images	Optional[List[str]]	no	None	Image inputs
videos	Optional[List[str]]	no	None	Video inputs
audio	Optional[List[str]]	no	None	Audio inputs
**kwargs	not annotated	no	none	Generation parameters

Returns

• Type: RequestOutput
• Direct return expressions: final_output

Exceptions and behavior

Method MLLMScheduler.generate calls self.add_request_async, self.stream_outputs, RequestOutput; awaits asynchronous work; returns final_output. No direct raise statement appears in this definition.

View source #L1029-L1078.

vllm_mlx.mllm_scheduler.MLLMScheduler.get_running_requests_info · method

vllm_mlx.mllm_scheduler.MLLMScheduler.get_running_requests_info() -> List[Dict[str, Any]]

Per-request details for status endpoint.

Parameters

This callable has no explicit inputs.

Returns

• Type: List[Dict[str, Any]]
• Direct return expressions: result

Exceptions and behavior

Method MLLMScheduler.get_running_requests_info calls time.time, result.append, round, self.running.values; returns result. No direct raise statement appears in this definition.

View source #L1082-L1151.

vllm_mlx.mllm_scheduler.MLLMScheduler.get_stats · method

vllm_mlx.mllm_scheduler.MLLMScheduler.get_stats() -> Dict[str, Any]

Get scheduler statistics.

Parameters

This callable has no explicit inputs.

Returns

• Type: Dict[str, Any]
• Direct return expressions: stats

Exceptions and behavior

Method MLLMScheduler.get_stats calls len, self.get_running_requests_info, self.batch_generator.stats, batch_stats.to_dict; returns stats. No direct raise statement appears in this definition.

View source #L1153-L1204.

vllm_mlx.mllm_scheduler.MLLMScheduler.clear_runtime_caches · method

vllm_mlx.mllm_scheduler.MLLMScheduler.clear_runtime_caches() -> Dict[str, bool]

Clear runtime caches without resetting scheduler/request state.

Parameters

This callable has no explicit inputs.

Returns

• Type: Dict[str, bool]
• Direct return expressions: cleared

Exceptions and behavior

Method MLLMScheduler.clear_runtime_caches calls self.vision_cache.clear, self.batch_generator.prefix_cache.clear; returns cleared. No direct raise statement appears in this definition.

View source #L1206-L1221.

vllm_mlx.mllm_scheduler.MLLMScheduler.reset · method

vllm_mlx.mllm_scheduler.MLLMScheduler.reset() -> None

Reset the scheduler state.

Parameters

This callable has no explicit inputs.

Returns

• Type: None

Exceptions and behavior

Method MLLMScheduler.reset updates self.batch_generator; calls list, self.requests.keys, self.abort_request, self.waiting.clear. No direct raise statement appears in this definition.

View source #L1223-L1242.

Complete symbol map

This map also includes private definitions and nested helpers. The signature column exposes every explicit input even when an internal helper has no dedicated parameter prose.

Symbol	Kind	Signature and inputs	What it does	Source
MLLMSchedulerConfig	class	MLLMSchedulerConfig(max_num_seqs: int = 16, prefill_batch_size: int = 16, completion_batch_size: int = 16, prefill_step_size: int = 1024, enable_vision_cache: bool = True, vision_cache_size: int = 100, default_max_tokens: int = 256, default_video_fps: float = 2.0, cache_memory_mb: Optional[int] = None, max_video_frames: int = 128, enable_mtp: bool = False, mtp_num_draft_tokens: int = 1, enable_prefix_cache: bool = True, use_memory_aware_cache: bool = True, prefix_cache_memory_mb: Optional[int] = None, kv_cache_quantization: bool = False, kv_cache_quantization_bits: int = 8, kv_cache_quantization_group_size: int = 64, chunked_prefill_tokens: int = 0, max_kv_size: int = 0, ssd_cache_dir: Optional[str] = None, ssd_cache_max_gb: float = 10.0)	Configuration for MLLM scheduler.	#L46-L92
MLLMRequest	class	MLLMRequest(request_id: str, prompt: str, images: Optional[List[str]] = None, videos: Optional[List[str]] = None, audio: Optional[List[str]] = None, sampling_params: SamplingParams = field(default_factory=SamplingParams), arrival_time: float = field(default_factory=time.time), batch_uid: Optional[int] = None, status: RequestStatus = RequestStatus.WAITING, output_text: str = '', output_tokens: List[int] = field(default_factory=list), finish_reason: Optional[str] = None, num_prompt_tokens: int = 0, num_output_tokens: int = 0, mtp_drafts: int = 0, mtp_accepted: int = 0, first_token_time: Optional[float] = None)	Extended request for MLLM processing.	#L96-L127
MLLMSchedulerOutput	class	MLLMSchedulerOutput(scheduled_request_ids: List[str] = field(default_factory=list), num_scheduled_tokens: int = 0, finished_request_ids: Set[str] = field(default_factory=set), outputs: List[RequestOutput] = field(default_factory=list), has_work: bool = False)	Output from a scheduling step.	#L131-L147
MLLMScheduler	class	MLLMScheduler(model: Any, processor: Any, config: Optional[MLLMSchedulerConfig] = None)	Scheduler for Vision Language Model requests with continuous batching.	#L150-L1242
MLLMScheduler.__init__	method	MLLMScheduler.__init__(model: Any, processor: Any, config: Optional[MLLMSchedulerConfig] = None) -> not annotated	Initialize MLLM scheduler.	#L183-L248
MLLMScheduler._get_stop_tokens	method	MLLMScheduler._get_stop_tokens() -> Set[int]	Get stop token IDs from tokenizer and generation_config.json.	#L250-L290
MLLMScheduler._ensure_batch_generator	method	MLLMScheduler._ensure_batch_generator() -> None	Ensure batch generator exists.	#L292-L374
MLLMScheduler.add_request	method	MLLMScheduler.add_request(prompt: str, images: Optional[List[str]] = None, videos: Optional[List[str]] = None, audio: Optional[List[str]] = None, max_tokens: int = 256, temperature: float = 0.7, top_p: float = 0.9, request_id: Optional[str] = None, **kwargs) -> str	Add a multimodal request to the scheduler (sync version).	#L378-L453
MLLMScheduler.abort_request	method	MLLMScheduler.abort_request(request_id: str) -> bool	Abort a request.	#L455-L532
MLLMScheduler.has_requests	method	MLLMScheduler.has_requests() -> bool	Check if there are any pending or running requests.	#L534-L536
MLLMScheduler.get_num_waiting	method	MLLMScheduler.get_num_waiting() -> int	Get number of waiting requests.	#L538-L540
MLLMScheduler.get_num_running	method	MLLMScheduler.get_num_running() -> int	Get number of running requests.	#L542-L544
MLLMScheduler._schedule_waiting	method	MLLMScheduler._schedule_waiting() -> List[MLLMRequest]	Move requests from waiting queue to running.	#L546-L597
MLLMScheduler._process_batch_responses	method	MLLMScheduler._process_batch_responses(responses: List[MLLMBatchResponse]) -> Tuple[List[RequestOutput], Set[str]]	Process responses from batch generator.	#L599-L716
MLLMScheduler._cleanup_finished	method	MLLMScheduler._cleanup_finished(finished_ids: Set[str]) -> None	Clean up finished requests.	#L718-L744
MLLMScheduler.step	method	MLLMScheduler.step() -> MLLMSchedulerOutput	Execute one scheduling step.	#L746-L813
MLLMScheduler.get_request	method	MLLMScheduler.get_request(request_id: str) -> Optional[MLLMRequest]	Get a request by ID.	#L815-L817
MLLMScheduler.remove_finished_request	method	MLLMScheduler.remove_finished_request(request_id: str) -> Optional[MLLMRequest]	Remove a finished request from tracking.	#L819-L821
MLLMScheduler.start	method	async MLLMScheduler.start() -> None	Start the async scheduler processing loop.	#L825-L834
MLLMScheduler.stop	method	async MLLMScheduler.stop() -> None	Stop the scheduler.	#L836-L850
MLLMScheduler._process_loop	method	async MLLMScheduler._process_loop() -> None	Main async processing loop.	#L852-L947
MLLMScheduler._process_loop._ensure_streams_bound	nested function	MLLMScheduler._process_loop._ensure_streams_bound() -> None	Nested Function MLLMScheduler._process_loop._ensure_streams_bound calls bind_generation_streams.	#L867-L871
MLLMScheduler.add_request_async	method	async MLLMScheduler.add_request_async(prompt: str, images: Optional[List[str]] = None, videos: Optional[List[str]] = None, audio: Optional[List[str]] = None, max_tokens: int = 256, temperature: float = 0.7, top_p: float = 0.9, **kwargs) -> str	Add a multimodal request (async version with output queue).	#L949-L990
MLLMScheduler.stream_outputs	method	async MLLMScheduler.stream_outputs(request_id: str) -> AsyncIterator[RequestOutput]	Stream outputs for a request.	#L992-L1027
MLLMScheduler.generate	method	async MLLMScheduler.generate(prompt: str, images: Optional[List[str]] = None, videos: Optional[List[str]] = None, audio: Optional[List[str]] = None, **kwargs) -> RequestOutput	Generate complete output for a request (non-streaming).	#L1029-L1078
MLLMScheduler.get_running_requests_info	method	MLLMScheduler.get_running_requests_info() -> List[Dict[str, Any]]	Per-request details for status endpoint.	#L1082-L1151
MLLMScheduler.get_stats	method	MLLMScheduler.get_stats() -> Dict[str, Any]	Get scheduler statistics.	#L1153-L1204
MLLMScheduler.clear_runtime_caches	method	MLLMScheduler.clear_runtime_caches() -> Dict[str, bool]	Clear runtime caches without resetting scheduler/request state.	#L1206-L1221
MLLMScheduler.reset	method	MLLMScheduler.reset() -> None	Reset the scheduler state.	#L1223-L1242