# `vllm_mlx.multimodal_processor`

Multimodal processor for VLM continuous batching.

[View the complete module source at #L1-L431](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/multimodal_processor.py#L1-L431).

## API details

Each callable below includes its exact signature, type annotations, inputs, defaults, return contract, documented exceptions, implementation source, and parsed docstring sections when the source provides them.

::: vllm_mlx.multimodal_processor
    options:
      members:
        - logger
        - ProcessedMultimodalInput
        - MultimodalProcessor
      filters: []
      show_if_no_docstring: true

## Complete contract reference

Expand any definition for its exact inputs, annotations, defaults, return contract, directly raised exceptions, source-grounded behavior, and immutable line link. This section includes private and nested definitions that ordinary API generators omit.

<details class="api-contract" id="contract-vllm_mlx.multimodal_processor.ProcessedMultimodalInput" markdown="1">
<summary><code>vllm_mlx.multimodal_processor.ProcessedMultimodalInput</code> · class</summary>

```python
vllm_mlx.multimodal_processor.ProcessedMultimodalInput(input_ids: mx.array, pixel_values: Optional[mx.array] = None, attention_mask: Optional[mx.array] = None, image_grid_thw: Optional[mx.array] = None, num_images: int = 0, num_tokens: int = 0, extra_kwargs: Dict[str, Any] = field(default_factory=dict))
```

Container for processed multimodal inputs ready for batching.

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `input_ids` | `mx.array` | `yes` | `none` | Required constructor field. |
| `pixel_values` | `Optional[mx.array]` | `no` | `None` | Optional constructor field; defaults to `None`. |
| `attention_mask` | `Optional[mx.array]` | `no` | `None` | Optional constructor field; defaults to `None`. |
| `image_grid_thw` | `Optional[mx.array]` | `no` | `None` | Optional constructor field; defaults to `None`. |
| `num_images` | `int` | `no` | `0` | Optional constructor field; defaults to `0`. |
| `num_tokens` | `int` | `no` | `0` | Optional constructor field; defaults to `0`. |
| `extra_kwargs` | `Dict[str, Any]` | `no` | `field(default_factory=dict)` | Optional constructor field; defaults to `field(default_factory=dict)`. |

**Returns**

- Constructs: `vllm_mlx.multimodal_processor.ProcessedMultimodalInput`

**Exceptions and behavior**

Class `ProcessedMultimodalInput` declares 0 direct member(s).
No direct `raise` statement appears in this definition.

[View source #L29-L49](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/multimodal_processor.py#L29-L49).

</details>

<details class="api-contract" id="contract-vllm_mlx.multimodal_processor.MultimodalProcessor" markdown="1">
<summary><code>vllm_mlx.multimodal_processor.MultimodalProcessor</code> · class</summary>

```python
vllm_mlx.multimodal_processor.MultimodalProcessor(model: Any, processor: Any, config: Optional[Any] = None)
```

Processor for preparing multimodal inputs for VLM batching.

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `model` | `Any` | `yes` | `none` | The VLM model (for config access) |
| `processor` | `Any` | `yes` | `none` | The VLM processor (tokenizer + image processor) |
| `config` | `Optional[Any]` | `no` | `None` | Optional model config |

**Returns**

- Constructs: `vllm_mlx.multimodal_processor.MultimodalProcessor`

**Exceptions and behavior**

Class `MultimodalProcessor` declares 8 direct member(s).
No direct `raise` statement appears in this definition.

[View source #L52-L431](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/multimodal_processor.py#L52-L431).

</details>

<details class="api-contract" id="contract-vllm_mlx.multimodal_processor.MultimodalProcessor.__init__" markdown="1">
<summary><code>vllm_mlx.multimodal_processor.MultimodalProcessor.__init__</code> · method</summary>

```python
vllm_mlx.multimodal_processor.MultimodalProcessor.__init__(model: Any, processor: Any, config: Optional[Any] = None) -> not annotated
```

Initialize the multimodal processor.

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `model` | `Any` | `yes` | `none` | The VLM model (for config access) |
| `processor` | `Any` | `yes` | `none` | The VLM processor (tokenizer + image processor) |
| `config` | `Optional[Any]` | `no` | `None` | Optional model config |

**Returns**

- Type: `not annotated`

**Exceptions and behavior**

Method `MultimodalProcessor.__init__` updates `self.model`, `self.processor`, `self.config`, `self.tokenizer`; calls `getattr`, `hasattr`.
No direct `raise` statement appears in this definition.

[View source #L68-L94](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/multimodal_processor.py#L68-L94).

</details>

<details class="api-contract" id="contract-vllm_mlx.multimodal_processor.MultimodalProcessor.process" markdown="1">
<summary><code>vllm_mlx.multimodal_processor.MultimodalProcessor.process</code> · method</summary>

```python
vllm_mlx.multimodal_processor.MultimodalProcessor.process(prompt: str, images: Optional[List[str]] = None, videos: Optional[List[str]] = None, video_fps: float = DEFAULT_FPS, video_max_frames: int = MAX_FRAMES, add_special_tokens: bool = True, **kwargs) -> ProcessedMultimodalInput
```

Process multimodal inputs for batching.

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `prompt` | `str` | `yes` | `none` | Text prompt (already formatted with chat template) |
| `images` | `Optional[List[str]]` | `no` | `None` | List of image URLs or base64 strings |
| `videos` | `Optional[List[str]]` | `no` | `None` | List of video URLs or base64 inputs |
| `video_fps` | `float` | `no` | `DEFAULT_FPS` | FPS for video frame extraction |
| `video_max_frames` | `int` | `no` | `MAX_FRAMES` | Max frames per video |
| `add_special_tokens` | `bool` | `no` | `True` | Whether to add special tokens |
| `**kwargs` | `not annotated` | `no` | `none` | Additional model-specific parameters |

**Returns**

- Type: `ProcessedMultimodalInput`
- Direct return expressions: `ProcessedMultimodalInput(input_ids=input_ids, pixel_values=pixel_values, attention_mask=attention_mask, image_grid_thw=…`

**Exceptions and behavior**

Method `MultimodalProcessor.process` calls `process_image_input`, `all_images.append`, `logger.warning`, `process_video_input`; returns `ProcessedMultimodalInput(input_ids=input_ids, pixel_values=pixel_values, attention_mask=attention_mask, image_grid_thw=…`.
No direct `raise` statement appears in this definition.

[View source #L96-L186](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/multimodal_processor.py#L96-L186).

</details>

<details class="api-contract" id="contract-vllm_mlx.multimodal_processor.MultimodalProcessor.process_for_request" markdown="1">
<summary><code>vllm_mlx.multimodal_processor.MultimodalProcessor.process_for_request</code> · method</summary>

```python
vllm_mlx.multimodal_processor.MultimodalProcessor.process_for_request(prompt: str, images: Optional[List[str]] = None, videos: Optional[List[str]] = None, **kwargs) -> Dict[str, Any]
```

Process inputs and return a dict suitable for Request fields.

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `prompt` | `str` | `yes` | `none` | Text prompt |
| `images` | `Optional[List[str]]` | `no` | `None` | List of image inputs |
| `videos` | `Optional[List[str]]` | `no` | `None` | List of video inputs |
| `**kwargs` | `not annotated` | `no` | `none` | Additional parameters |

**Returns**

- Type: `Dict[str, Any]`
- Direct return expressions: `{'prompt_token_ids': processed.input_ids.tolist() if processed.input_ids is not None else None, 'num_prompt_tokens': pr…`

**Exceptions and behavior**

Method `MultimodalProcessor.process_for_request` calls `self.process`, `processed.input_ids.tolist`; returns `{'prompt_token_ids': processed.input_ids.tolist() if processed.input_ids is not None else None, 'num_prompt_tokens': pr…`.
No direct `raise` statement appears in this definition.

[View source #L188-L224](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/multimodal_processor.py#L188-L224).

</details>

<details class="api-contract" id="contract-vllm_mlx.multimodal_processor.MultimodalProcessor.batch_pixel_values" markdown="1">
<summary><code>vllm_mlx.multimodal_processor.MultimodalProcessor.batch_pixel_values</code> · method</summary>

```python
vllm_mlx.multimodal_processor.MultimodalProcessor.batch_pixel_values(pixel_values_list: List[Optional[mx.array]]) -> Optional[mx.array]
```

Batch multiple pixel_values tensors together.

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `pixel_values_list` | `List[Optional[mx.array]]` | `yes` | `none` | List of pixel_values from multiple requests |

**Returns**

- Type: `Optional[mx.array]`
- Direct return expressions: `None`; `mx.concatenate(valid_pixels, axis=0)`; `valid_pixels[0] if valid_pixels else None`

**Exceptions and behavior**

Method `MultimodalProcessor.batch_pixel_values` calls `mx.concatenate`, `logger.warning`; has 3 explicit return paths.
No direct `raise` statement appears in this definition.

[View source #L226-L255](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/multimodal_processor.py#L226-L255).

</details>

<details class="api-contract" id="contract-vllm_mlx.multimodal_processor.MultimodalProcessor.batch_image_grid_thw" markdown="1">
<summary><code>vllm_mlx.multimodal_processor.MultimodalProcessor.batch_image_grid_thw</code> · method</summary>

```python
vllm_mlx.multimodal_processor.MultimodalProcessor.batch_image_grid_thw(grid_thw_list: List[Optional[mx.array]]) -> Optional[mx.array]
```

Batch multiple image_grid_thw tensors together.

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `grid_thw_list` | `List[Optional[mx.array]]` | `yes` | `none` | List of image_grid_thw from multiple requests |

**Returns**

- Type: `Optional[mx.array]`
- Direct return expressions: `None`; `mx.concatenate(valid_grids, axis=0)`; `valid_grids[0] if valid_grids else None`

**Exceptions and behavior**

Method `MultimodalProcessor.batch_image_grid_thw` calls `mx.concatenate`, `logger.warning`; has 3 explicit return paths.
No direct `raise` statement appears in this definition.

[View source #L257-L279](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/multimodal_processor.py#L257-L279).

</details>

<details class="api-contract" id="contract-vllm_mlx.multimodal_processor.MultimodalProcessor.prepare_for_batch" markdown="1">
<summary><code>vllm_mlx.multimodal_processor.MultimodalProcessor.prepare_for_batch</code> · method</summary>

```python
vllm_mlx.multimodal_processor.MultimodalProcessor.prepare_for_batch(processed_inputs: List[ProcessedMultimodalInput]) -> Tuple[mx.array, Dict[str, Any], List[int]]
```

Prepare multiple processed inputs for batch generation.

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `processed_inputs` | `List[ProcessedMultimodalInput]` | `yes` | `none` | List of ProcessedMultimodalInput from process() |

**Returns**

- Type: `Tuple[mx.array, Dict[str, Any], List[int]]`
- Direct return expressions: `(mx.array([]), {}, [])`; `(input_ids, batch_kwargs, padding_amounts)`

**Exceptions and behavior**

Method `MultimodalProcessor.prepare_for_batch` calls `mx.array`, `max`, `zip`, `padded_ids.append`; has 2 explicit return paths.
No direct `raise` statement appears in this definition.

[View source #L281-L366](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/multimodal_processor.py#L281-L366).

</details>

<details class="api-contract" id="contract-vllm_mlx.multimodal_processor.MultimodalProcessor.extract_vision_embeddings" markdown="1">
<summary><code>vllm_mlx.multimodal_processor.MultimodalProcessor.extract_vision_embeddings</code> · method</summary>

```python
vllm_mlx.multimodal_processor.MultimodalProcessor.extract_vision_embeddings(pixel_values: mx.array, image_grid_thw: Optional[mx.array] = None) -> mx.array
```

Extract vision embeddings from pixel values.

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `pixel_values` | `mx.array` | `yes` | `none` | Processed image tensors |
| `image_grid_thw` | `Optional[mx.array]` | `no` | `None` | Optional grid info for Qwen-VL models |

**Returns**

- Type: `mx.array`
- Direct return expressions: `embeddings`

**Exceptions and behavior**

Method `MultimodalProcessor.extract_vision_embeddings` calls `hasattr`, `ValueError`, `getattr`, `vision_encoder`; can raise `ValueError`; returns `embeddings`.
Directly raised exceptions: `ValueError`.

[View source #L368-L409](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/multimodal_processor.py#L368-L409).

</details>

<details class="api-contract" id="contract-vllm_mlx.multimodal_processor.MultimodalProcessor.compute_vision_hash" markdown="1">
<summary><code>vllm_mlx.multimodal_processor.MultimodalProcessor.compute_vision_hash</code> · method</summary>

```python
vllm_mlx.multimodal_processor.MultimodalProcessor.compute_vision_hash(pixel_values: mx.array) -> str
```

Compute a hash for pixel values for caching purposes.

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `pixel_values` | `mx.array` | `yes` | `none` | Processed image tensors |

**Returns**

- Type: `str`
- Direct return expressions: `hashlib.sha256(hash_input.encode()).hexdigest()[:16]`

**Exceptions and behavior**

Method `MultimodalProcessor.compute_vision_hash` calls `str`, `pixel_values.reshape(-1)[:100].tolist`, `pixel_values.reshape`, `hashlib.sha256(hash_input.encode()).hexdigest`; returns `hashlib.sha256(hash_input.encode()).hexdigest()[:16]`.
No direct `raise` statement appears in this definition.

[View source #L411-L431](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/multimodal_processor.py#L411-L431).

</details>

## Complete symbol map

This map also includes private definitions and nested helpers. The signature column exposes every explicit input even when an internal helper has no dedicated parameter prose.

| Symbol | Kind | Signature and inputs | What it does | Source |
| --- | --- | --- | --- | --- |
| [`ProcessedMultimodalInput`](#contract-vllm_mlx.multimodal_processor.ProcessedMultimodalInput) | class | `ProcessedMultimodalInput(input_ids: mx.array, pixel_values: Optional[mx.array] = None, attention_mask: Optional[mx.array] = None, image_grid_thw: Optional[mx.array] = None, num_images: int = 0, num_tokens: int = 0, extra_kwargs: Dict[str, Any] = field(default_factory=dict))` | Container for processed multimodal inputs ready for batching. | [#L29-L49](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/multimodal_processor.py#L29-L49) |
| [`MultimodalProcessor`](#contract-vllm_mlx.multimodal_processor.MultimodalProcessor) | class | `MultimodalProcessor(model: Any, processor: Any, config: Optional[Any] = None)` | Processor for preparing multimodal inputs for VLM batching. | [#L52-L431](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/multimodal_processor.py#L52-L431) |
| [`MultimodalProcessor.__init__`](#contract-vllm_mlx.multimodal_processor.MultimodalProcessor.__init__) | method | `MultimodalProcessor.__init__(model: Any, processor: Any, config: Optional[Any] = None) -> not annotated` | Initialize the multimodal processor. | [#L68-L94](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/multimodal_processor.py#L68-L94) |
| [`MultimodalProcessor.process`](#contract-vllm_mlx.multimodal_processor.MultimodalProcessor.process) | method | `MultimodalProcessor.process(prompt: str, images: Optional[List[str]] = None, videos: Optional[List[str]] = None, video_fps: float = DEFAULT_FPS, video_max_frames: int = MAX_FRAMES, add_special_tokens: bool = True, **kwargs) -> ProcessedMultimodalInput` | Process multimodal inputs for batching. | [#L96-L186](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/multimodal_processor.py#L96-L186) |
| [`MultimodalProcessor.process_for_request`](#contract-vllm_mlx.multimodal_processor.MultimodalProcessor.process_for_request) | method | `MultimodalProcessor.process_for_request(prompt: str, images: Optional[List[str]] = None, videos: Optional[List[str]] = None, **kwargs) -> Dict[str, Any]` | Process inputs and return a dict suitable for Request fields. | [#L188-L224](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/multimodal_processor.py#L188-L224) |
| [`MultimodalProcessor.batch_pixel_values`](#contract-vllm_mlx.multimodal_processor.MultimodalProcessor.batch_pixel_values) | method | `MultimodalProcessor.batch_pixel_values(pixel_values_list: List[Optional[mx.array]]) -> Optional[mx.array]` | Batch multiple pixel_values tensors together. | [#L226-L255](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/multimodal_processor.py#L226-L255) |
| [`MultimodalProcessor.batch_image_grid_thw`](#contract-vllm_mlx.multimodal_processor.MultimodalProcessor.batch_image_grid_thw) | method | `MultimodalProcessor.batch_image_grid_thw(grid_thw_list: List[Optional[mx.array]]) -> Optional[mx.array]` | Batch multiple image_grid_thw tensors together. | [#L257-L279](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/multimodal_processor.py#L257-L279) |
| [`MultimodalProcessor.prepare_for_batch`](#contract-vllm_mlx.multimodal_processor.MultimodalProcessor.prepare_for_batch) | method | `MultimodalProcessor.prepare_for_batch(processed_inputs: List[ProcessedMultimodalInput]) -> Tuple[mx.array, Dict[str, Any], List[int]]` | Prepare multiple processed inputs for batch generation. | [#L281-L366](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/multimodal_processor.py#L281-L366) |
| [`MultimodalProcessor.extract_vision_embeddings`](#contract-vllm_mlx.multimodal_processor.MultimodalProcessor.extract_vision_embeddings) | method | `MultimodalProcessor.extract_vision_embeddings(pixel_values: mx.array, image_grid_thw: Optional[mx.array] = None) -> mx.array` | Extract vision embeddings from pixel values. | [#L368-L409](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/multimodal_processor.py#L368-L409) |
| [`MultimodalProcessor.compute_vision_hash`](#contract-vllm_mlx.multimodal_processor.MultimodalProcessor.compute_vision_hash) | method | `MultimodalProcessor.compute_vision_hash(pixel_values: mx.array) -> str` | Compute a hash for pixel values for caching purposes. | [#L411-L431](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/multimodal_processor.py#L411-L431) |