vllm_mlx.multimodal_processor

Multimodal processor for VLM continuous batching.

View the complete module source at #L1-L431.

API details

Each callable below includes its exact signature, type annotations, inputs, defaults, return contract, documented exceptions, implementation source, and parsed docstring sections when the source provides them.

vllm_mlx.multimodal_processor

Multimodal processor for VLM continuous batching.

This module handles preprocessing of multimodal inputs (images, videos) for use with the continuous batching scheduler. It extracts processed inputs that can be batched together efficiently.

vllm_mlx.multimodal_processor.logger module-attribute

logger = logging.getLogger(__name__)

vllm_mlx.multimodal_processor.ProcessedMultimodalInput dataclass

ProcessedMultimodalInput(input_ids: array, pixel_values: Optional[array] = None, attention_mask: Optional[array] = None, image_grid_thw: Optional[array] = None, num_images: int = 0, num_tokens: int = 0, extra_kwargs: Dict[str, Any] = dict())

Container for processed multimodal inputs ready for batching.

Attributes:

• input_ids (array) –

  Tokenized text with image/video tokens (mx.array)

• pixel_values (Optional[array]) –

  Processed image tensors (mx.array)

• attention_mask (Optional[array]) –

  Attention mask for the input (mx.array)

• image_grid_thw (Optional[array]) –

  Grid info for Qwen-VL models (mx.array)

• num_images (int) –

  Number of images in this input

• num_tokens (int) –

  Number of tokens in input_ids

• extra_kwargs (Dict[str, Any]) –

  Additional model-specific kwargs

vllm_mlx.multimodal_processor.ProcessedMultimodalInput.input_ids instance-attribute

input_ids: array

vllm_mlx.multimodal_processor.ProcessedMultimodalInput.pixel_values class-attribute instance-attribute

pixel_values: Optional[array] = None

vllm_mlx.multimodal_processor.ProcessedMultimodalInput.attention_mask class-attribute instance-attribute

attention_mask: Optional[array] = None

vllm_mlx.multimodal_processor.ProcessedMultimodalInput.image_grid_thw class-attribute instance-attribute

image_grid_thw: Optional[array] = None

vllm_mlx.multimodal_processor.ProcessedMultimodalInput.num_images class-attribute instance-attribute

num_images: int = 0

vllm_mlx.multimodal_processor.ProcessedMultimodalInput.num_tokens class-attribute instance-attribute

num_tokens: int = 0

vllm_mlx.multimodal_processor.ProcessedMultimodalInput.extra_kwargs class-attribute instance-attribute

extra_kwargs: Dict[str, Any] = field(default_factory=dict)

vllm_mlx.multimodal_processor.MultimodalProcessor

MultimodalProcessor(model: Any, processor: Any, config: Optional[Any] = None)

Processor for preparing multimodal inputs for VLM batching.

This class wraps mlx_vlm's prepare_inputs function and provides a clean interface for the scheduler to preprocess requests.

Example

processor = MultimodalProcessor(model, vlm_processor) processed = processor.process( ... prompt="What's in this image?", ... images=["photo.jpg"] ... )

processed.input_ids, processed.pixel_values ready for batching

Initialize the multimodal processor.

Parameters:

• model (Any) –

  The VLM model (for config access)

• processor (Any) –

  The VLM processor (tokenizer + image processor)

• config (Optional[Any], default: None ) –

  Optional model config

Source code in vllm_mlx/multimodal_processor.py

def __init__(
    self,
    model: Any,
    processor: Any,
    config: Optional[Any] = None,
):
    """
    Initialize the multimodal processor.

    Args:
        model: The VLM model (for config access)
        processor: The VLM processor (tokenizer + image processor)
        config: Optional model config
    """
    self.model = model
    self.processor = processor
    self.config = config or getattr(model, "config", None)

    # Get tokenizer from processor
    self.tokenizer = (
        processor.tokenizer if hasattr(processor, "tokenizer") else processor
    )

    # Get image token index if available
    self.image_token_index = (
        getattr(self.config, "image_token_index", None) if self.config else None
    )

vllm_mlx.multimodal_processor.MultimodalProcessor.model instance-attribute

model = model

vllm_mlx.multimodal_processor.MultimodalProcessor.processor instance-attribute

processor = processor

vllm_mlx.multimodal_processor.MultimodalProcessor.config instance-attribute

config = config or getattr(model, 'config', None)

vllm_mlx.multimodal_processor.MultimodalProcessor.tokenizer instance-attribute

tokenizer = processor.tokenizer if hasattr(processor, 'tokenizer') else processor

vllm_mlx.multimodal_processor.MultimodalProcessor.image_token_index instance-attribute

image_token_index = getattr(self.config, 'image_token_index', None) if self.config else None

vllm_mlx.multimodal_processor.MultimodalProcessor.process

process(prompt: str, images: Optional[List[str]] = None, videos: Optional[List[str]] = None, video_fps: float = DEFAULT_FPS, video_max_frames: int = MAX_FRAMES, add_special_tokens: bool = True, **kwargs) -> ProcessedMultimodalInput

Process multimodal inputs for batching.

Parameters:

• prompt (str) –

  Text prompt (already formatted with chat template)

• images (Optional[List[str]], default: None ) –

  List of image URLs or base64 strings

• videos (Optional[List[str]], default: None ) –

  List of video URLs or base64 inputs

• video_fps (float, default: DEFAULT_FPS ) –

  FPS for video frame extraction

• video_max_frames (int, default: MAX_FRAMES ) –

  Max frames per video

• add_special_tokens (bool, default: True ) –

  Whether to add special tokens

• **kwargs –

  Additional model-specific parameters

Returns:

• ProcessedMultimodalInput –

  ProcessedMultimodalInput with all processed tensors

Source code in vllm_mlx/multimodal_processor.py

def process(
    self,
    prompt: str,
    images: Optional[List[str]] = None,
    videos: Optional[List[str]] = None,
    video_fps: float = DEFAULT_FPS,
    video_max_frames: int = MAX_FRAMES,
    add_special_tokens: bool = True,
    **kwargs,
) -> ProcessedMultimodalInput:
    """
    Process multimodal inputs for batching.

    Args:
        prompt: Text prompt (already formatted with chat template)
        images: List of image URLs or base64 strings
        videos: List of video URLs or base64 inputs
        video_fps: FPS for video frame extraction
        video_max_frames: Max frames per video
        add_special_tokens: Whether to add special tokens
        **kwargs: Additional model-specific parameters

    Returns:
        ProcessedMultimodalInput with all processed tensors
    """
    from mlx_vlm.utils import prepare_inputs

    # Process raw images
    all_images = []
    if images:
        for img in images:
            try:
                path = process_image_input(img)
                all_images.append(path)
            except Exception as e:
                logger.warning(f"Failed to process image: {e}")

    # Extract frames from videos
    if videos:
        for video in videos:
            try:
                video_path = process_video_input(video)
                frames = extract_video_frames_smart(
                    video_path,
                    fps=video_fps,
                    max_frames=video_max_frames,
                )
                frame_paths = save_frames_to_temp(frames)
                all_images.extend(frame_paths)
                logger.debug(f"Extracted {len(frame_paths)} frames from video")
            except Exception as e:
                logger.warning(f"Failed to process video: {e}")

    # Determine add_special_tokens based on model type
    if self.config and self.config.model_type in ["gemma3", "gemma3n"]:
        add_special_tokens = not hasattr(self.processor, "chat_template")

    # Prepare inputs using mlx_vlm
    inputs = prepare_inputs(
        self.processor,
        images=all_images if all_images else None,
        prompts=prompt,
        image_token_index=self.image_token_index,
        add_special_tokens=add_special_tokens,
        **kwargs,
    )

    # Extract processed tensors
    input_ids = inputs.get("input_ids")
    pixel_values = inputs.get("pixel_values")
    attention_mask = inputs.get("attention_mask")

    # Extract model-specific kwargs
    extra_kwargs = {
        k: v
        for k, v in inputs.items()
        if k not in ["input_ids", "pixel_values", "attention_mask"]
    }

    # Get image_grid_thw for Qwen-VL models
    image_grid_thw = extra_kwargs.pop("image_grid_thw", None)

    return ProcessedMultimodalInput(
        input_ids=input_ids,
        pixel_values=pixel_values,
        attention_mask=attention_mask,
        image_grid_thw=image_grid_thw,
        num_images=len(all_images),
        num_tokens=input_ids.size if input_ids is not None else 0,
        extra_kwargs=extra_kwargs,
    )

vllm_mlx.multimodal_processor.MultimodalProcessor.process_for_request

process_for_request(prompt: str, images: Optional[List[str]] = None, videos: Optional[List[str]] = None, **kwargs) -> Dict[str, Any]

Process inputs and return a dict suitable for Request fields.

This is a convenience method that returns the processed data in a format that can be directly assigned to Request fields.

Parameters:

• prompt (str) –

  Text prompt

• images (Optional[List[str]], default: None ) –

  List of image inputs

• videos (Optional[List[str]], default: None ) –

  List of video inputs

• **kwargs –

  Additional parameters

Returns:

• Dict[str, Any] –

  Dict with keys matching Request multimodal fields

Source code in vllm_mlx/multimodal_processor.py

def process_for_request(
    self,
    prompt: str,
    images: Optional[List[str]] = None,
    videos: Optional[List[str]] = None,
    **kwargs,
) -> Dict[str, Any]:
    """
    Process inputs and return a dict suitable for Request fields.

    This is a convenience method that returns the processed data
    in a format that can be directly assigned to Request fields.

    Args:
        prompt: Text prompt
        images: List of image inputs
        videos: List of video inputs
        **kwargs: Additional parameters

    Returns:
        Dict with keys matching Request multimodal fields
    """
    processed = self.process(prompt, images, videos, **kwargs)

    return {
        "prompt_token_ids": (
            processed.input_ids.tolist()
            if processed.input_ids is not None
            else None
        ),
        "num_prompt_tokens": processed.num_tokens,
        "pixel_values": processed.pixel_values,
        "attention_mask": processed.attention_mask,
        "image_grid_thw": processed.image_grid_thw,
        "multimodal_kwargs": processed.extra_kwargs,
        "is_multimodal": processed.num_images > 0,
    }

vllm_mlx.multimodal_processor.MultimodalProcessor.batch_pixel_values

batch_pixel_values(pixel_values_list: List[Optional[array]]) -> Optional[array]

Batch multiple pixel_values tensors together.

For VLM batching, we need to concatenate pixel values from multiple requests. This handles the case where some requests may not have images.

Parameters:

• pixel_values_list (List[Optional[array]]) –

  List of pixel_values from multiple requests

Returns:

• Optional[array] –

  Batched pixel_values or None if no images

Source code in vllm_mlx/multimodal_processor.py

def batch_pixel_values(
    self,
    pixel_values_list: List[Optional[mx.array]],
) -> Optional[mx.array]:
    """
    Batch multiple pixel_values tensors together.

    For VLM batching, we need to concatenate pixel values from
    multiple requests. This handles the case where some requests
    may not have images.

    Args:
        pixel_values_list: List of pixel_values from multiple requests

    Returns:
        Batched pixel_values or None if no images
    """
    # Filter out None values
    valid_pixels = [p for p in pixel_values_list if p is not None]

    if not valid_pixels:
        return None

    # Concatenate along batch dimension
    try:
        return mx.concatenate(valid_pixels, axis=0)
    except Exception as e:
        logger.warning(f"Failed to batch pixel_values: {e}")
        # Fall back to returning first valid
        return valid_pixels[0] if valid_pixels else None

vllm_mlx.multimodal_processor.MultimodalProcessor.batch_image_grid_thw

batch_image_grid_thw(grid_thw_list: List[Optional[array]]) -> Optional[array]

Batch multiple image_grid_thw tensors together.

Parameters:

• grid_thw_list (List[Optional[array]]) –

  List of image_grid_thw from multiple requests

Returns:

• Optional[array] –

  Batched image_grid_thw or None

Source code in vllm_mlx/multimodal_processor.py

def batch_image_grid_thw(
    self,
    grid_thw_list: List[Optional[mx.array]],
) -> Optional[mx.array]:
    """
    Batch multiple image_grid_thw tensors together.

    Args:
        grid_thw_list: List of image_grid_thw from multiple requests

    Returns:
        Batched image_grid_thw or None
    """
    valid_grids = [g for g in grid_thw_list if g is not None]

    if not valid_grids:
        return None

    try:
        return mx.concatenate(valid_grids, axis=0)
    except Exception as e:
        logger.warning(f"Failed to batch image_grid_thw: {e}")
        return valid_grids[0] if valid_grids else None

vllm_mlx.multimodal_processor.MultimodalProcessor.prepare_for_batch

prepare_for_batch(processed_inputs: List[ProcessedMultimodalInput]) -> Tuple[array, Dict[str, Any], List[int]]

Prepare multiple processed inputs for batch generation.

This method takes a list of ProcessedMultimodalInput objects and combines them into batched tensors suitable for the MLLMBatchGenerator.

Parameters:

• processed_inputs (List[ProcessedMultimodalInput]) –

  List of ProcessedMultimodalInput from process()

Returns:

• array –

  Tuple of:

• Dict[str, Any] –
  • input_ids: Left-padded input tokens [batch_size, max_seq_len]
• List[int] –
  • batch_kwargs: Dict with batched pixel_values, attention_mask, etc.
• Tuple[array, Dict[str, Any], List[int]] –
  • padding_amounts: List of padding amounts for each request

Source code in vllm_mlx/multimodal_processor.py

def prepare_for_batch(
    self,
    processed_inputs: List[ProcessedMultimodalInput],
) -> Tuple[mx.array, Dict[str, Any], List[int]]:
    """
    Prepare multiple processed inputs for batch generation.

    This method takes a list of ProcessedMultimodalInput objects and
    combines them into batched tensors suitable for the MLLMBatchGenerator.

    Args:
        processed_inputs: List of ProcessedMultimodalInput from process()

    Returns:
        Tuple of:
        - input_ids: Left-padded input tokens [batch_size, max_seq_len]
        - batch_kwargs: Dict with batched pixel_values, attention_mask, etc.
        - padding_amounts: List of padding amounts for each request
    """
    if not processed_inputs:
        return mx.array([]), {}, []

    # Get all input_ids and compute padding
    input_ids_list = [p.input_ids for p in processed_inputs]
    lengths = [ids.size if ids is not None else 0 for ids in input_ids_list]
    max_length = max(lengths) if lengths else 0
    padding_amounts = [max_length - seq_len for seq_len in lengths]

    # Left-pad input_ids
    padded_ids = []
    for ids, pad_amount in zip(input_ids_list, padding_amounts):
        if ids is None:
            padded_ids.append([0] * max_length)
        else:
            tokens = ids.tolist() if hasattr(ids, "tolist") else list(ids)
            padded_ids.append([0] * pad_amount + tokens)

    input_ids = mx.array(padded_ids)

    # Batch pixel values
    pixel_values = self.batch_pixel_values(
        [p.pixel_values for p in processed_inputs]
    )

    # Batch image_grid_thw
    image_grid_thw = self.batch_image_grid_thw(
        [p.image_grid_thw for p in processed_inputs]
    )

    # Batch attention masks
    attention_masks = [p.attention_mask for p in processed_inputs]
    valid_masks = [m for m in attention_masks if m is not None]
    batched_attention_mask = None
    if valid_masks:
        try:
            # Pad attention masks to match input_ids shape
            padded_masks = []
            for mask, pad_amount in zip(attention_masks, padding_amounts):
                if mask is None:
                    padded_masks.append(mx.ones((max_length,)))
                else:
                    mask_flat = mask.reshape(-1)
                    pad = mx.zeros((pad_amount,))
                    padded_masks.append(mx.concatenate([pad, mask_flat]))
            batched_attention_mask = mx.stack(padded_masks)
        except Exception as e:
            logger.warning(f"Failed to batch attention masks: {e}")

    # Merge extra_kwargs (take first non-empty set)
    merged_extra = {}
    for p in processed_inputs:
        if p.extra_kwargs:
            merged_extra.update(p.extra_kwargs)
            break

    batch_kwargs = {
        "pixel_values": pixel_values,
        "attention_mask": batched_attention_mask,
        "image_grid_thw": image_grid_thw,
        **merged_extra,
    }

    # Remove None values
    batch_kwargs = {k: v for k, v in batch_kwargs.items() if v is not None}

    return input_ids, batch_kwargs, padding_amounts

vllm_mlx.multimodal_processor.MultimodalProcessor.extract_vision_embeddings

extract_vision_embeddings(pixel_values: array, image_grid_thw: Optional[array] = None) -> array

Extract vision embeddings from pixel values.

This runs the vision encoder part of the VLM to get embeddings that can be cached and reused.

Parameters:

• pixel_values (array) –

  Processed image tensors

• image_grid_thw (Optional[array], default: None ) –

  Optional grid info for Qwen-VL models

Returns:

• array –

  Vision embeddings tensor

Source code in vllm_mlx/multimodal_processor.py

def extract_vision_embeddings(
    self,
    pixel_values: mx.array,
    image_grid_thw: Optional[mx.array] = None,
) -> mx.array:
    """
    Extract vision embeddings from pixel values.

    This runs the vision encoder part of the VLM to get embeddings
    that can be cached and reused.

    Args:
        pixel_values: Processed image tensors
        image_grid_thw: Optional grid info for Qwen-VL models

    Returns:
        Vision embeddings tensor
    """
    if not hasattr(self.model, "vision_tower") and not hasattr(
        self.model, "vision_model"
    ):
        raise ValueError("Model does not have a vision encoder")

    # Get the vision encoder
    vision_encoder = getattr(self.model, "vision_tower", None)
    if vision_encoder is None:
        vision_encoder = getattr(self.model, "vision_model", None)

    if vision_encoder is None:
        raise ValueError("Could not find vision encoder in model")

    # Run vision encoding
    if image_grid_thw is not None:
        # Qwen-VL style with grid info
        try:
            embeddings = vision_encoder(pixel_values, grid_thw=image_grid_thw)
        except TypeError:
            embeddings = vision_encoder(pixel_values)
    else:
        embeddings = vision_encoder(pixel_values)

    return embeddings

vllm_mlx.multimodal_processor.MultimodalProcessor.compute_vision_hash

compute_vision_hash(pixel_values: array) -> str

Compute a hash for pixel values for caching purposes.

Parameters:

• pixel_values (array) –

  Processed image tensors

Returns:

• str –

  Hash string for the vision inputs

Source code in vllm_mlx/multimodal_processor.py

def compute_vision_hash(
    self,
    pixel_values: mx.array,
) -> str:
    """
    Compute a hash for pixel values for caching purposes.

    Args:
        pixel_values: Processed image tensors

    Returns:
        Hash string for the vision inputs
    """
    import hashlib

    # Use shape and a sample of values for hashing
    shape_str = str(pixel_values.shape)
    sample_values = pixel_values.reshape(-1)[:100].tolist()
    hash_input = f"{shape_str}_{sample_values}"

    return hashlib.sha256(hash_input.encode()).hexdigest()[:16]

Complete contract reference

Expand any definition for its exact inputs, annotations, defaults, return contract, directly raised exceptions, source-grounded behavior, and immutable line link. This section includes private and nested definitions that ordinary API generators omit.

vllm_mlx.multimodal_processor.ProcessedMultimodalInput · class

vllm_mlx.multimodal_processor.ProcessedMultimodalInput(input_ids: mx.array, pixel_values: Optional[mx.array] = None, attention_mask: Optional[mx.array] = None, image_grid_thw: Optional[mx.array] = None, num_images: int = 0, num_tokens: int = 0, extra_kwargs: Dict[str, Any] = field(default_factory=dict))

Container for processed multimodal inputs ready for batching.

Parameters

Name	Type	Required	Default	Description
input_ids	mx.array	yes	none	Required constructor field.
pixel_values	Optional[mx.array]	no	None	Optional constructor field; defaults to None.
attention_mask	Optional[mx.array]	no	None	Optional constructor field; defaults to None.
image_grid_thw	Optional[mx.array]	no	None	Optional constructor field; defaults to None.
num_images	int	no	0	Optional constructor field; defaults to 0.
num_tokens	int	no	0	Optional constructor field; defaults to 0.
extra_kwargs	Dict[str, Any]	no	field(default_factory=dict)	Optional constructor field; defaults to field(default_factory=dict).

Returns

• Constructs: vllm_mlx.multimodal_processor.ProcessedMultimodalInput

Exceptions and behavior

Class ProcessedMultimodalInput declares 0 direct member(s). No direct raise statement appears in this definition.

View source #L29-L49.

vllm_mlx.multimodal_processor.MultimodalProcessor · class

vllm_mlx.multimodal_processor.MultimodalProcessor(model: Any, processor: Any, config: Optional[Any] = None)

Processor for preparing multimodal inputs for VLM batching.

Parameters

Name	Type	Required	Default	Description
model	Any	yes	none	The VLM model (for config access)
processor	Any	yes	none	The VLM processor (tokenizer + image processor)
config	Optional[Any]	no	None	Optional model config

Returns

• Constructs: vllm_mlx.multimodal_processor.MultimodalProcessor

Exceptions and behavior

Class MultimodalProcessor declares 8 direct member(s). No direct raise statement appears in this definition.

View source #L52-L431.

vllm_mlx.multimodal_processor.MultimodalProcessor.__init__ · method

vllm_mlx.multimodal_processor.MultimodalProcessor.__init__(model: Any, processor: Any, config: Optional[Any] = None) -> not annotated

Initialize the multimodal processor.

Parameters

Name	Type	Required	Default	Description
model	Any	yes	none	The VLM model (for config access)
processor	Any	yes	none	The VLM processor (tokenizer + image processor)
config	Optional[Any]	no	None	Optional model config

Returns

• Type: not annotated

Exceptions and behavior

Method MultimodalProcessor.__init__ updates self.model, self.processor, self.config, self.tokenizer; calls getattr, hasattr. No direct raise statement appears in this definition.

View source #L68-L94.

vllm_mlx.multimodal_processor.MultimodalProcessor.process · method

vllm_mlx.multimodal_processor.MultimodalProcessor.process(prompt: str, images: Optional[List[str]] = None, videos: Optional[List[str]] = None, video_fps: float = DEFAULT_FPS, video_max_frames: int = MAX_FRAMES, add_special_tokens: bool = True, **kwargs) -> ProcessedMultimodalInput

Process multimodal inputs for batching.

Parameters

Name	Type	Required	Default	Description
prompt	str	yes	none	Text prompt (already formatted with chat template)
images	Optional[List[str]]	no	None	List of image URLs or base64 strings
videos	Optional[List[str]]	no	None	List of video URLs or base64 inputs
video_fps	float	no	DEFAULT_FPS	FPS for video frame extraction
video_max_frames	int	no	MAX_FRAMES	Max frames per video
add_special_tokens	bool	no	True	Whether to add special tokens
**kwargs	not annotated	no	none	Additional model-specific parameters

Returns

• Type: ProcessedMultimodalInput
• Direct return expressions: ProcessedMultimodalInput(input_ids=input_ids, pixel_values=pixel_values, attention_mask=attention_mask, image_grid_thw=…

Exceptions and behavior

Method MultimodalProcessor.process calls process_image_input, all_images.append, logger.warning, process_video_input; returns ProcessedMultimodalInput(input_ids=input_ids, pixel_values=pixel_values, attention_mask=attention_mask, image_grid_thw=…. No direct raise statement appears in this definition.

View source #L96-L186.

vllm_mlx.multimodal_processor.MultimodalProcessor.process_for_request · method

vllm_mlx.multimodal_processor.MultimodalProcessor.process_for_request(prompt: str, images: Optional[List[str]] = None, videos: Optional[List[str]] = None, **kwargs) -> Dict[str, Any]

Process inputs and return a dict suitable for Request fields.

Parameters

Name	Type	Required	Default	Description
prompt	str	yes	none	Text prompt
images	Optional[List[str]]	no	None	List of image inputs
videos	Optional[List[str]]	no	None	List of video inputs
**kwargs	not annotated	no	none	Additional parameters

Returns

• Type: Dict[str, Any]
• Direct return expressions: {'prompt_token_ids': processed.input_ids.tolist() if processed.input_ids is not None else None, 'num_prompt_tokens': pr…

Exceptions and behavior

Method MultimodalProcessor.process_for_request calls self.process, processed.input_ids.tolist; returns {'prompt_token_ids': processed.input_ids.tolist() if processed.input_ids is not None else None, 'num_prompt_tokens': pr…. No direct raise statement appears in this definition.

View source #L188-L224.

vllm_mlx.multimodal_processor.MultimodalProcessor.batch_pixel_values · method

vllm_mlx.multimodal_processor.MultimodalProcessor.batch_pixel_values(pixel_values_list: List[Optional[mx.array]]) -> Optional[mx.array]

Batch multiple pixel_values tensors together.

Parameters

Name	Type	Required	Default	Description
pixel_values_list	List[Optional[mx.array]]	yes	none	List of pixel_values from multiple requests

Returns

• Type: Optional[mx.array]
• Direct return expressions: None; mx.concatenate(valid_pixels, axis=0); valid_pixels[0] if valid_pixels else None

Exceptions and behavior

Method MultimodalProcessor.batch_pixel_values calls mx.concatenate, logger.warning; has 3 explicit return paths. No direct raise statement appears in this definition.

View source #L226-L255.

vllm_mlx.multimodal_processor.MultimodalProcessor.batch_image_grid_thw · method

vllm_mlx.multimodal_processor.MultimodalProcessor.batch_image_grid_thw(grid_thw_list: List[Optional[mx.array]]) -> Optional[mx.array]

Batch multiple image_grid_thw tensors together.

Parameters

Name	Type	Required	Default	Description
grid_thw_list	List[Optional[mx.array]]	yes	none	List of image_grid_thw from multiple requests

Returns

• Type: Optional[mx.array]
• Direct return expressions: None; mx.concatenate(valid_grids, axis=0); valid_grids[0] if valid_grids else None

Exceptions and behavior

Method MultimodalProcessor.batch_image_grid_thw calls mx.concatenate, logger.warning; has 3 explicit return paths. No direct raise statement appears in this definition.

View source #L257-L279.

vllm_mlx.multimodal_processor.MultimodalProcessor.prepare_for_batch · method

vllm_mlx.multimodal_processor.MultimodalProcessor.prepare_for_batch(processed_inputs: List[ProcessedMultimodalInput]) -> Tuple[mx.array, Dict[str, Any], List[int]]

Prepare multiple processed inputs for batch generation.

Parameters

Name	Type	Required	Default	Description
processed_inputs	List[ProcessedMultimodalInput]	yes	none	List of ProcessedMultimodalInput from process()

Returns

• Type: Tuple[mx.array, Dict[str, Any], List[int]]
• Direct return expressions: (mx.array([]), {}, []); (input_ids, batch_kwargs, padding_amounts)

Exceptions and behavior

Method MultimodalProcessor.prepare_for_batch calls mx.array, max, zip, padded_ids.append; has 2 explicit return paths. No direct raise statement appears in this definition.

View source #L281-L366.

vllm_mlx.multimodal_processor.MultimodalProcessor.extract_vision_embeddings · method

vllm_mlx.multimodal_processor.MultimodalProcessor.extract_vision_embeddings(pixel_values: mx.array, image_grid_thw: Optional[mx.array] = None) -> mx.array

Extract vision embeddings from pixel values.

Parameters

Name	Type	Required	Default	Description
pixel_values	mx.array	yes	none	Processed image tensors
image_grid_thw	Optional[mx.array]	no	None	Optional grid info for Qwen-VL models

Returns

• Type: mx.array
• Direct return expressions: embeddings

Exceptions and behavior

Method MultimodalProcessor.extract_vision_embeddings calls hasattr, ValueError, getattr, vision_encoder; can raise ValueError; returns embeddings. Directly raised exceptions: ValueError.

View source #L368-L409.

vllm_mlx.multimodal_processor.MultimodalProcessor.compute_vision_hash · method

vllm_mlx.multimodal_processor.MultimodalProcessor.compute_vision_hash(pixel_values: mx.array) -> str

Compute a hash for pixel values for caching purposes.

Parameters

Name	Type	Required	Default	Description
pixel_values	mx.array	yes	none	Processed image tensors

Returns

• Type: str
• Direct return expressions: hashlib.sha256(hash_input.encode()).hexdigest()[:16]

Exceptions and behavior

Method MultimodalProcessor.compute_vision_hash calls str, pixel_values.reshape(-1)[:100].tolist, pixel_values.reshape, hashlib.sha256(hash_input.encode()).hexdigest; returns hashlib.sha256(hash_input.encode()).hexdigest()[:16]. No direct raise statement appears in this definition.

View source #L411-L431.

Complete symbol map

This map also includes private definitions and nested helpers. The signature column exposes every explicit input even when an internal helper has no dedicated parameter prose.

Symbol	Kind	Signature and inputs	What it does	Source
ProcessedMultimodalInput	class	ProcessedMultimodalInput(input_ids: mx.array, pixel_values: Optional[mx.array] = None, attention_mask: Optional[mx.array] = None, image_grid_thw: Optional[mx.array] = None, num_images: int = 0, num_tokens: int = 0, extra_kwargs: Dict[str, Any] = field(default_factory=dict))	Container for processed multimodal inputs ready for batching.	#L29-L49
MultimodalProcessor	class	MultimodalProcessor(model: Any, processor: Any, config: Optional[Any] = None)	Processor for preparing multimodal inputs for VLM batching.	#L52-L431
MultimodalProcessor.__init__	method	MultimodalProcessor.__init__(model: Any, processor: Any, config: Optional[Any] = None) -> not annotated	Initialize the multimodal processor.	#L68-L94
MultimodalProcessor.process	method	MultimodalProcessor.process(prompt: str, images: Optional[List[str]] = None, videos: Optional[List[str]] = None, video_fps: float = DEFAULT_FPS, video_max_frames: int = MAX_FRAMES, add_special_tokens: bool = True, **kwargs) -> ProcessedMultimodalInput	Process multimodal inputs for batching.	#L96-L186
MultimodalProcessor.process_for_request	method	MultimodalProcessor.process_for_request(prompt: str, images: Optional[List[str]] = None, videos: Optional[List[str]] = None, **kwargs) -> Dict[str, Any]	Process inputs and return a dict suitable for Request fields.	#L188-L224
MultimodalProcessor.batch_pixel_values	method	MultimodalProcessor.batch_pixel_values(pixel_values_list: List[Optional[mx.array]]) -> Optional[mx.array]	Batch multiple pixel_values tensors together.	#L226-L255
MultimodalProcessor.batch_image_grid_thw	method	MultimodalProcessor.batch_image_grid_thw(grid_thw_list: List[Optional[mx.array]]) -> Optional[mx.array]	Batch multiple image_grid_thw tensors together.	#L257-L279
MultimodalProcessor.prepare_for_batch	method	MultimodalProcessor.prepare_for_batch(processed_inputs: List[ProcessedMultimodalInput]) -> Tuple[mx.array, Dict[str, Any], List[int]]	Prepare multiple processed inputs for batch generation.	#L281-L366
MultimodalProcessor.extract_vision_embeddings	method	MultimodalProcessor.extract_vision_embeddings(pixel_values: mx.array, image_grid_thw: Optional[mx.array] = None) -> mx.array	Extract vision embeddings from pixel values.	#L368-L409
MultimodalProcessor.compute_vision_hash	method	MultimodalProcessor.compute_vision_hash(pixel_values: mx.array) -> str	Compute a hash for pixel values for caching purposes.	#L411-L431