# `vllm_mlx.engine.batched`

Batched engine for continuous batching with multiple concurrent users.

[View the complete module source at #L1-L1231](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/batched.py#L1-L1231).

## API details

Each callable below includes its exact signature, type annotations, inputs, defaults, return contract, documented exceptions, implementation source, and parsed docstring sections when the source provides them.

::: vllm_mlx.engine.batched
    options:
      members:
        - logger
        - _resolve_metal_buffer_cache_limit
        - _normalize_tool_call_arguments_for_template
        - _extract_media_from_messages
        - MLLMModelWrapper
        - BatchedEngine
      filters: []
      show_if_no_docstring: true

## Complete contract reference

Expand any definition for its exact inputs, annotations, defaults, return contract, directly raised exceptions, source-grounded behavior, and immutable line link. This section includes private and nested definitions that ordinary API generators omit.

<details class="api-contract" id="contract-vllm_mlx.engine.batched._resolve_metal_buffer_cache_limit" markdown="1">
<summary><code>vllm_mlx.engine.batched._resolve_metal_buffer_cache_limit</code> · function</summary>

```python
vllm_mlx.engine.batched._resolve_metal_buffer_cache_limit(max_recommended: int, gpu_memory_utilization: float) -> tuple[int, str]
```

Resolve the MLX retained-buffer cache cap for Metal startup.

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `max_recommended` | `int` | `yes` | `none` | Required positional or keyword input. |
| `gpu_memory_utilization` | `float` | `yes` | `none` | Required positional or keyword input. |

**Returns**

- Type: `tuple[int, str]`
- Direct return expressions: `(limit, 'MLX_BUFFER_CACHE_LIMIT')`; `(int(max_recommended * gpu_memory_utilization), 'device-scaled')`

**Exceptions and behavior**

Function `_resolve_metal_buffer_cache_limit` calls `os.environ.get`, `int`, `logger.warning`; has 2 explicit return paths.
No direct `raise` statement appears in this definition.

[View source #L35-L58](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/batched.py#L35-L58).

</details>

<details class="api-contract" id="contract-vllm_mlx.engine.batched._normalize_tool_call_arguments_for_template" markdown="1">
<summary><code>vllm_mlx.engine.batched._normalize_tool_call_arguments_for_template</code> · function</summary>

```python
vllm_mlx.engine.batched._normalize_tool_call_arguments_for_template(messages: list[dict]) -> list[dict]
```

Normalize OpenAI tool-call replay for templates expecting mappings.

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `messages` | `list[dict]` | `yes` | `none` | Required positional or keyword input. |

**Returns**

- Type: `list[dict]`
- Direct return expressions: `normalize_messages_for_chat_template(messages)`

**Exceptions and behavior**

Function `_normalize_tool_call_arguments_for_template` calls `normalize_messages_for_chat_template`; returns `normalize_messages_for_chat_template(messages)`.
No direct `raise` statement appears in this definition.

[View source #L61-L63](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/batched.py#L61-L63).

</details>

<details class="api-contract" id="contract-vllm_mlx.engine.batched._extract_media_from_messages" markdown="1">
<summary><code>vllm_mlx.engine.batched._extract_media_from_messages</code> · function</summary>

```python
vllm_mlx.engine.batched._extract_media_from_messages(messages: list[dict[str, Any]]) -> tuple
```

Extract images, videos, and audio from OpenAI-format messages.

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `messages` | `list[dict[str, Any]]` | `yes` | `none` | Required positional or keyword input. |

**Returns**

- Type: `tuple`
- Direct return expressions: `(has_media, images, videos, audios)`

**Exceptions and behavior**

Function `_extract_media_from_messages` calls `msg.get`, `isinstance`, `hasattr`, `item.model_dump`; returns `(has_media, images, videos, audios)`.
No direct `raise` statement appears in this definition.

[View source #L66-L137](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/batched.py#L66-L137).

</details>

<details class="api-contract" id="contract-vllm_mlx.engine.batched.MLLMModelWrapper" markdown="1">
<summary><code>vllm_mlx.engine.batched.MLLMModelWrapper</code> · class</summary>

```python
vllm_mlx.engine.batched.MLLMModelWrapper(model)
```

Wrapper for MLLM models to make them compatible with BatchGenerator.

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `model` | `not annotated` | `yes` | `none` | Required positional or keyword input. |

**Returns**

- Constructs: `vllm_mlx.engine.batched.MLLMModelWrapper`

**Exceptions and behavior**

Class `MLLMModelWrapper` declares 3 direct member(s).
No direct `raise` statement appears in this definition.

[View source #L140-L175](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/batched.py#L140-L175).

</details>

<details class="api-contract" id="contract-vllm_mlx.engine.batched.MLLMModelWrapper.__init__" markdown="1">
<summary><code>vllm_mlx.engine.batched.MLLMModelWrapper.__init__</code> · method</summary>

```python
vllm_mlx.engine.batched.MLLMModelWrapper.__init__(model) -> not annotated
```

Method `MLLMModelWrapper.__init__` updates `self._model`, `self._is_gemma3`; calls `hasattr`, `str(getattr(model, 'model_type', '')).lower`, `str`, `getattr`.

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `model` | `not annotated` | `yes` | `none` | Required positional or keyword input. |

**Returns**

- Type: `not annotated`

**Exceptions and behavior**

Method `MLLMModelWrapper.__init__` updates `self._model`, `self._is_gemma3`; calls `hasattr`, `str(getattr(model, 'model_type', '')).lower`, `str`, `getattr`.
No direct `raise` statement appears in this definition.

[View source #L152-L158](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/batched.py#L152-L158).

</details>

<details class="api-contract" id="contract-vllm_mlx.engine.batched.MLLMModelWrapper.__call__" markdown="1">
<summary><code>vllm_mlx.engine.batched.MLLMModelWrapper.__call__</code> · method</summary>

```python
vllm_mlx.engine.batched.MLLMModelWrapper.__call__(*args, **kwargs) -> not annotated
```

Call the model and extract logits from LanguageModelOutput.

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `*args` | `not annotated` | `no` | `none` | Additional variadic positional inputs accepted by this callable. |
| `**kwargs` | `not annotated` | `no` | `none` | Additional variadic keyword inputs accepted by this callable. |

**Returns**

- Type: `not annotated`
- Direct return expressions: `output.logits`; `output`

**Exceptions and behavior**

Method `MLLMModelWrapper.__call__` calls `self._model`, `hasattr`; has 2 explicit return paths.
No direct `raise` statement appears in this definition.

[View source #L160-L171](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/batched.py#L160-L171).

</details>

<details class="api-contract" id="contract-vllm_mlx.engine.batched.MLLMModelWrapper.__getattr__" markdown="1">
<summary><code>vllm_mlx.engine.batched.MLLMModelWrapper.__getattr__</code> · method</summary>

```python
vllm_mlx.engine.batched.MLLMModelWrapper.__getattr__(name) -> not annotated
```

Forward all other attributes to the wrapped model.

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `name` | `not annotated` | `yes` | `none` | Required positional or keyword input. |

**Returns**

- Type: `not annotated`
- Direct return expressions: `getattr(self._model, name)`

**Exceptions and behavior**

Method `MLLMModelWrapper.__getattr__` calls `getattr`; returns `getattr(self._model, name)`.
No direct `raise` statement appears in this definition.

[View source #L173-L175](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/batched.py#L173-L175).

</details>

<details class="api-contract" id="contract-vllm_mlx.engine.batched.BatchedEngine" markdown="1">
<summary><code>vllm_mlx.engine.batched.BatchedEngine</code> · class</summary>

```python
vllm_mlx.engine.batched.BatchedEngine(model_name: str, trust_remote_code: bool = False, scheduler_config: Any | None = None, stream_interval: int = 1, force_mllm: bool = False, gpu_memory_utilization: float = 0.9)
```

Batched engine for continuous batching.

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `model_name` | `str` | `yes` | `none` | HuggingFace model name or local path |
| `trust_remote_code` | `bool` | `no` | `False` | Whether to trust remote code |
| `scheduler_config` | `Any \| None` | `no` | `None` | Optional scheduler configuration |
| `stream_interval` | `int` | `no` | `1` | Tokens to batch before streaming (1=every token) |
| `force_mllm` | `bool` | `no` | `False` | Force loading as MLLM even if not auto-detected |
| `gpu_memory_utilization` | `float` | `no` | `0.9` | Fraction of device memory for Metal allocation limit and emergency threshold (0.0-1.0, default 0.90) |

**Returns**

- Constructs: `vllm_mlx.engine.batched.BatchedEngine`

**Exceptions and behavior**

Class `BatchedEngine` derives from `BaseEngine` and declares 28 direct member(s).
No direct `raise` statement appears in this definition.

[View source #L178-L1231](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/batched.py#L178-L1231).

</details>

<details class="api-contract" id="contract-vllm_mlx.engine.batched.BatchedEngine.__init__" markdown="1">
<summary><code>vllm_mlx.engine.batched.BatchedEngine.__init__</code> · method</summary>

```python
vllm_mlx.engine.batched.BatchedEngine.__init__(model_name: str, trust_remote_code: bool = False, scheduler_config: Any | None = None, stream_interval: int = 1, force_mllm: bool = False, gpu_memory_utilization: float = 0.9) -> not annotated
```

Initialize the batched engine.

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `model_name` | `str` | `yes` | `none` | HuggingFace model name or local path |
| `trust_remote_code` | `bool` | `no` | `False` | Whether to trust remote code |
| `scheduler_config` | `Any \| None` | `no` | `None` | Optional scheduler configuration |
| `stream_interval` | `int` | `no` | `1` | Tokens to batch before streaming (1=every token) |
| `force_mllm` | `bool` | `no` | `False` | Force loading as MLLM even if not auto-detected |
| `gpu_memory_utilization` | `float` | `no` | `0.9` | Fraction of device memory for Metal allocation limit and emergency threshold (0.0-1.0, default 0.90) |

**Returns**

- Type: `not annotated`

**Exceptions and behavior**

Method `BatchedEngine.__init__` updates `self._model_name`, `self._created_at`, `self._trust_remote_code`, `self._scheduler_config`; calls `time.time`, `is_mllm_model`.
No direct `raise` statement appears in this definition.

[View source #L189-L224](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/batched.py#L189-L224).

</details>

<details class="api-contract" id="contract-vllm_mlx.engine.batched.BatchedEngine.model_name" markdown="1">
<summary><code>vllm_mlx.engine.batched.BatchedEngine.model_name</code> · method</summary>

```python
vllm_mlx.engine.batched.BatchedEngine.model_name() -> str
```

Get the model name.

**Parameters**

This callable has no explicit inputs.

**Returns**

- Type: `str`
- Direct return expressions: `self._model_name`

**Exceptions and behavior**

Method `BatchedEngine.model_name` returns `self._model_name`.
No direct `raise` statement appears in this definition.

[View source #L227-L229](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/batched.py#L227-L229).

</details>

<details class="api-contract" id="contract-vllm_mlx.engine.batched.BatchedEngine.is_mllm" markdown="1">
<summary><code>vllm_mlx.engine.batched.BatchedEngine.is_mllm</code> · method</summary>

```python
vllm_mlx.engine.batched.BatchedEngine.is_mllm() -> bool
```

Check if this is a multimodal model.

**Parameters**

This callable has no explicit inputs.

**Returns**

- Type: `bool`
- Direct return expressions: `self._is_mllm`

**Exceptions and behavior**

Method `BatchedEngine.is_mllm` returns `self._is_mllm`.
No direct `raise` statement appears in this definition.

[View source #L232-L234](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/batched.py#L232-L234).

</details>

<details class="api-contract" id="contract-vllm_mlx.engine.batched.BatchedEngine.tokenizer" markdown="1">
<summary><code>vllm_mlx.engine.batched.BatchedEngine.tokenizer</code> · method</summary>

```python
vllm_mlx.engine.batched.BatchedEngine.tokenizer() -> Any
```

Get the tokenizer.

**Parameters**

This callable has no explicit inputs.

**Returns**

- Type: `Any`
- Direct return expressions: `getattr(self._processor, 'tokenizer', self._processor)`; `self._tokenizer`

**Exceptions and behavior**

Method `BatchedEngine.tokenizer` calls `getattr`; has 2 explicit return paths.
No direct `raise` statement appears in this definition.

[View source #L237-L241](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/batched.py#L237-L241).

</details>

<details class="api-contract" id="contract-vllm_mlx.engine.batched.BatchedEngine.prepare_for_start" markdown="1">
<summary><code>vllm_mlx.engine.batched.BatchedEngine.prepare_for_start</code> · method</summary>

```python
vllm_mlx.engine.batched.BatchedEngine.prepare_for_start() -> None
```

Load heavyweight model state off the serving event loop.

**Parameters**

This callable has no explicit inputs.

**Returns**

- Type: `None`
- Direct return expressions: `None`

**Exceptions and behavior**

Method `BatchedEngine.prepare_for_start` calls `self._prepare_mllm_model`, `self._prepare_llm_model`; returns `None`.
No direct `raise` statement appears in this definition.

[View source #L243-L251](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/batched.py#L243-L251).

</details>

<details class="api-contract" id="contract-vllm_mlx.engine.batched.BatchedEngine.start" markdown="1">
<summary><code>vllm_mlx.engine.batched.BatchedEngine.start</code> · method</summary>

```python
async vllm_mlx.engine.batched.BatchedEngine.start() -> None
```

Start the engine (load model if not loaded).

**Parameters**

This callable has no explicit inputs.

**Returns**

- Type: `None`
- Direct return expressions: `None`

**Exceptions and behavior**

Method `BatchedEngine.start` updates `self._loaded`; calls `self._uses_default_prepare_for_start`, `self.prepare_for_start`, `run_blocking_startup_work`, `self._start_mllm`; awaits asynchronous work; returns `None`.
No direct `raise` statement appears in this definition.

[View source #L253-L282](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/batched.py#L253-L282).

</details>

<details class="api-contract" id="contract-vllm_mlx.engine.batched.BatchedEngine._uses_default_prepare_for_start" markdown="1">
<summary><code>vllm_mlx.engine.batched.BatchedEngine._uses_default_prepare_for_start</code> · method</summary>

```python
vllm_mlx.engine.batched.BatchedEngine._uses_default_prepare_for_start() -> bool
```

Return True when prepare_for_start is the class implementation.

**Parameters**

This callable has no explicit inputs.

**Returns**

- Type: `bool`
- Direct return expressions: `method is BatchedEngine.prepare_for_start`

**Exceptions and behavior**

Method `BatchedEngine._uses_default_prepare_for_start` calls `getattr`; returns `method is BatchedEngine.prepare_for_start`.
No direct `raise` statement appears in this definition.

[View source #L284-L287](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/batched.py#L284-L287).

</details>

<details class="api-contract" id="contract-vllm_mlx.engine.batched.BatchedEngine._prepare_mllm_model" markdown="1">
<summary><code>vllm_mlx.engine.batched.BatchedEngine._prepare_mllm_model</code> · method</summary>

```python
vllm_mlx.engine.batched.BatchedEngine._prepare_mllm_model() -> None
```

Load the MLLM model before scheduler startup.

**Parameters**

This callable has no explicit inputs.

**Returns**

- Type: `None`

**Exceptions and behavior**

Method `BatchedEngine._prepare_mllm_model` updates `self._mllm_instance`, `self._model`, `self._processor`; calls `getattr`, `MLXMultimodalLM`, `self._mllm_instance.load`, `mx.metal.is_available`.
No direct `raise` statement appears in this definition.

[View source #L289-L334](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/batched.py#L289-L334).

</details>

<details class="api-contract" id="contract-vllm_mlx.engine.batched.BatchedEngine._start_mllm" markdown="1">
<summary><code>vllm_mlx.engine.batched.BatchedEngine._start_mllm</code> · method</summary>

```python
async vllm_mlx.engine.batched.BatchedEngine._start_mllm() -> None
```

Start the MLLM engine with MLLMScheduler (continuous batching).

**Parameters**

This callable has no explicit inputs.

**Returns**

- Type: `None`

**Exceptions and behavior**

Method `BatchedEngine._start_mllm` updates `self._mllm_scheduler`; calls `self._prepare_mllm_model`, `hasattr`, `getattr`, `MLLMSchedulerConfig`; awaits asynchronous work.
No direct `raise` statement appears in this definition.

[View source #L336-L429](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/batched.py#L336-L429).

</details>

<details class="api-contract" id="contract-vllm_mlx.engine.batched.BatchedEngine._inject_mtp_mllm" markdown="1">
<summary><code>vllm_mlx.engine.batched.BatchedEngine._inject_mtp_mllm</code> · method</summary>

```python
vllm_mlx.engine.batched.BatchedEngine._inject_mtp_mllm() -> None
```

Inject MTP weights into the MLLM model's language_model.

**Parameters**

This callable has no explicit inputs.

**Returns**

- Type: `None`
- Direct return expressions: `None`

**Exceptions and behavior**

Method `BatchedEngine._inject_mtp_mllm` calls `Path`, `_download`, `config_path.exists`, `logger.warning`; returns `None`.
No direct `raise` statement appears in this definition.

[View source #L431-L477](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/batched.py#L431-L477).

</details>

<details class="api-contract" id="contract-vllm_mlx.engine.batched.BatchedEngine._prepare_llm_model" markdown="1">
<summary><code>vllm_mlx.engine.batched.BatchedEngine._prepare_llm_model</code> · method</summary>

```python
vllm_mlx.engine.batched.BatchedEngine._prepare_llm_model() -> None
```

Load the LLM model/tokenizer before engine loop startup.

**Parameters**

This callable has no explicit inputs.

**Returns**

- Type: `None`
- Direct return expressions: `None`

**Exceptions and behavior**

Method `BatchedEngine._prepare_llm_model` updates `self._model`, `self._tokenizer`; calls `self._model_name.lower`, `load_model_with_fallback`, `validate_mtp_support`, `validate_35`; returns `None`.
No direct `raise` statement appears in this definition.

[View source #L479-L511](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/batched.py#L479-L511).

</details>

<details class="api-contract" id="contract-vllm_mlx.engine.batched.BatchedEngine._configure_metal_memory_limits" markdown="1">
<summary><code>vllm_mlx.engine.batched.BatchedEngine._configure_metal_memory_limits</code> · method</summary>

```python
vllm_mlx.engine.batched.BatchedEngine._configure_metal_memory_limits() -> None
```

Make MLX allocation failures graceful during startup.

**Parameters**

This callable has no explicit inputs.

**Returns**

- Type: `None`

**Exceptions and behavior**

Method `BatchedEngine._configure_metal_memory_limits` calls `mx.metal.is_available`, `mx.device_info`, `device_info.get`, `int`.
No direct `raise` statement appears in this definition.

[View source #L513-L541](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/batched.py#L513-L541).

</details>

<details class="api-contract" id="contract-vllm_mlx.engine.batched.BatchedEngine._start_llm" markdown="1">
<summary><code>vllm_mlx.engine.batched.BatchedEngine._start_llm</code> · method</summary>

```python
async vllm_mlx.engine.batched.BatchedEngine._start_llm() -> None
```

Start the LLM engine with AsyncEngineCore.

**Parameters**

This callable has no explicit inputs.

**Returns**

- Type: `None`

**Exceptions and behavior**

Method `BatchedEngine._start_llm` updates `self._engine`; calls `self._prepare_llm_model`, `validate_mtp_support`, `logger.info`, `logger.warning`; awaits asynchronous work.
No direct `raise` statement appears in this definition.

[View source #L543-L579](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/batched.py#L543-L579).

</details>

<details class="api-contract" id="contract-vllm_mlx.engine.batched.BatchedEngine.stop" markdown="1">
<summary><code>vllm_mlx.engine.batched.BatchedEngine.stop</code> · method</summary>

```python
async vllm_mlx.engine.batched.BatchedEngine.stop() -> None
```

Stop the engine and cleanup resources.

**Parameters**

This callable has no explicit inputs.

**Returns**

- Type: `None`

**Exceptions and behavior**

Method `BatchedEngine.stop` updates `self._mllm_scheduler`, `self._engine`, `self._model`, `self._tokenizer`; calls `self._mllm_scheduler.stop`, `self._engine.stop`, `self._engine.engine.close`, `logger.info`; awaits asynchronous work.
No direct `raise` statement appears in this definition.

[View source #L581-L597](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/batched.py#L581-L597).

</details>

<details class="api-contract" id="contract-vllm_mlx.engine.batched.BatchedEngine._apply_chat_template" markdown="1">
<summary><code>vllm_mlx.engine.batched.BatchedEngine._apply_chat_template</code> · method</summary>

```python
vllm_mlx.engine.batched.BatchedEngine._apply_chat_template(messages: list[dict[str, Any]], tools: list[dict] | None = None, num_images: int = 0, num_audios: int = 0, chat_template_kwargs: dict[str, Any] | None = None, enable_thinking: bool | None = None) -> str
```

Apply chat template to messages.

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `messages` | `list[dict[str, Any]]` | `yes` | `none` | Required positional or keyword input. |
| `tools` | `list[dict] \| None` | `no` | `None` | Optional positional or keyword input; defaults to `None`. |
| `num_images` | `int` | `no` | `0` | Optional positional or keyword input; defaults to `0`. |
| `num_audios` | `int` | `no` | `0` | Optional positional or keyword input; defaults to `0`. |
| `chat_template_kwargs` | `dict[str, Any] \| None` | `no` | `None` | Optional positional or keyword input; defaults to `None`. |
| `enable_thinking` | `bool \| None` | `no` | `None` | Optional positional or keyword input; defaults to `None`. |

**Returns**

- Type: `str`
- Direct return expressions: `template_applicator.apply_chat_template(messages, **template_kwargs)`; `tokenizer_applicator.apply_chat_template(messages, **template_kwargs)`; `prompt + '\nassistant:'`

**Exceptions and behavior**

Method `BatchedEngine._apply_chat_template` calls `_normalize_tool_call_arguments_for_template`, `hasattr`, `self._prepare_mllm_messages`, `self._model_name.lower`; has 3 explicit return paths.
No direct `raise` statement appears in this definition.

[View source #L599-L687](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/batched.py#L599-L687).

</details>

<details class="api-contract" id="contract-vllm_mlx.engine.batched.BatchedEngine._prepare_mllm_messages" markdown="1">
<summary><code>vllm_mlx.engine.batched.BatchedEngine._prepare_mllm_messages</code> · method</summary>

```python
vllm_mlx.engine.batched.BatchedEngine._prepare_mllm_messages(messages: list[dict[str, Any]]) -> list[dict[str, Any]]
```

Convert OpenAI-style multimodal content to HuggingFace format.

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `messages` | `list[dict[str, Any]]` | `yes` | `none` | List of chat messages in OpenAI format. Each message is a dict with at least ``role`` and ``content`` keys. |

**Returns**

- Type: `list[dict[str, Any]]`
- Direct return expressions: `prepared`

**Exceptions and behavior**

Method `BatchedEngine._prepare_mllm_messages` calls `isinstance`, `msg.get`, `part.get`, `new_content.append`; returns `prepared`.
No direct `raise` statement appears in this definition.

[View source #L690-L726](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/batched.py#L690-L726).

</details>

<details class="api-contract" id="contract-vllm_mlx.engine.batched.BatchedEngine.generate" markdown="1">
<summary><code>vllm_mlx.engine.batched.BatchedEngine.generate</code> · method</summary>

```python
async vllm_mlx.engine.batched.BatchedEngine.generate(prompt: str, max_tokens: int = 256, temperature: float = 0.7, top_p: float = 0.9, stop: list[str] | None = None, images: list[str] | None = None, videos: list[str] | None = None, audio: list[str] | None = None, **kwargs) -> GenerationOutput
```

Generate a complete response (non-streaming).

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `prompt` | `str` | `yes` | `none` | Input text |
| `max_tokens` | `int` | `no` | `256` | Maximum tokens to generate |
| `temperature` | `float` | `no` | `0.7` | Sampling temperature |
| `top_p` | `float` | `no` | `0.9` | Top-p sampling |
| `stop` | `list[str] \| None` | `no` | `None` | Stop sequences |
| `images` | `list[str] \| None` | `no` | `None` | Optional image URLs/paths (for MLLM) |
| `videos` | `list[str] \| None` | `no` | `None` | Optional video URLs/paths (for MLLM) |
| `audio` | `list[str] \| None` | `no` | `None` | Optional audio URLs/paths (for MLLM) |
| `**kwargs` | `not annotated` | `no` | `none` | Additional model-specific parameters |

**Returns**

- Type: `GenerationOutput`
- Direct return expressions: `GenerationOutput(text=clean_output_text(output.output_text), tokens=output.output_token_ids, prompt_tokens=output.promp…`; `GenerationOutput(text=text, tokens=output.output_token_ids, prompt_tokens=output.prompt_tokens, completion_tokens=outpu…`

**Exceptions and behavior**

Method `BatchedEngine.generate` calls `self.start`, `self._mllm_scheduler.generate`, `kwargs.pop`, `GenerationOutput`; awaits asynchronous work; has 2 explicit return paths.
No direct `raise` statement appears in this definition.

[View source #L728-L817](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/batched.py#L728-L817).

</details>

<details class="api-contract" id="contract-vllm_mlx.engine.batched.BatchedEngine.stream_generate" markdown="1">
<summary><code>vllm_mlx.engine.batched.BatchedEngine.stream_generate</code> · method</summary>

```python
async vllm_mlx.engine.batched.BatchedEngine.stream_generate(prompt: str, max_tokens: int = 256, temperature: float = 0.7, top_p: float = 0.9, stop: list[str] | None = None, images: list[str] | None = None, videos: list[str] | None = None, audio: list[str] | None = None, **kwargs) -> AsyncIterator[GenerationOutput]
```

Stream generation token by token.

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `prompt` | `str` | `yes` | `none` | Input text |
| `max_tokens` | `int` | `no` | `256` | Maximum tokens to generate |
| `temperature` | `float` | `no` | `0.7` | Sampling temperature |
| `top_p` | `float` | `no` | `0.9` | Top-p sampling |
| `stop` | `list[str] \| None` | `no` | `None` | Stop sequences |
| `images` | `list[str] \| None` | `no` | `None` | Optional image URLs/paths (for MLLM) |
| `videos` | `list[str] \| None` | `no` | `None` | Optional video URLs/paths (for MLLM) |
| `audio` | `list[str] \| None` | `no` | `None` | Optional audio URLs/paths (for MLLM) |
| `**kwargs` | `not annotated` | `no` | `none` | Additional model-specific parameters |

**Returns**

- Type: `AsyncIterator[GenerationOutput]`
- Direct return expressions: `None`
- Yields values incrementally.

**Exceptions and behavior**

Method `BatchedEngine.stream_generate` calls `self.start`, `self._mllm_scheduler.add_request_async`, `kwargs.pop`, `self._mllm_scheduler.stream_outputs`; awaits asynchronous work; yields values incrementally; returns `None`.
No direct `raise` statement appears in this definition.

[View source #L819-L913](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/batched.py#L819-L913).

</details>

<details class="api-contract" id="contract-vllm_mlx.engine.batched.BatchedEngine.chat" markdown="1">
<summary><code>vllm_mlx.engine.batched.BatchedEngine.chat</code> · method</summary>

```python
async vllm_mlx.engine.batched.BatchedEngine.chat(messages: list[dict[str, Any]], max_tokens: int = 256, temperature: float = 0.7, top_p: float = 0.9, tools: list[dict] | None = None, images: list[str] | None = None, videos: list[str] | None = None, **kwargs) -> GenerationOutput
```

Chat completion (non-streaming).

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `messages` | `list[dict[str, Any]]` | `yes` | `none` | List of chat messages (OpenAI format) |
| `max_tokens` | `int` | `no` | `256` | Maximum tokens to generate |
| `temperature` | `float` | `no` | `0.7` | Sampling temperature |
| `top_p` | `float` | `no` | `0.9` | Top-p sampling |
| `tools` | `list[dict] \| None` | `no` | `None` | Optional tool definitions |
| `images` | `list[str] \| None` | `no` | `None` | Optional image URLs/paths |
| `videos` | `list[str] \| None` | `no` | `None` | Optional video URLs/paths |
| `**kwargs` | `not annotated` | `no` | `none` | Additional model-specific parameters |

**Returns**

- Type: `GenerationOutput`
- Direct return expressions: `await self.generate(prompt=prompt, max_tokens=max_tokens, temperature=temperature, top_p=top_p, images=all_images if al…`

**Exceptions and behavior**

Method `BatchedEngine.chat` calls `self.start`, `extract_multimodal_content`, `convert_tools_for_template`, `dict`; awaits asynchronous work; returns `await self.generate(prompt=prompt, max_tokens=max_tokens, temperature=temperature, top_p=top_p, images=all_images if al…`.
No direct `raise` statement appears in this definition.

[View source #L915-L984](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/batched.py#L915-L984).

</details>

<details class="api-contract" id="contract-vllm_mlx.engine.batched.BatchedEngine._compute_prefix_boundary" markdown="1">
<summary><code>vllm_mlx.engine.batched.BatchedEngine._compute_prefix_boundary</code> · method</summary>

```python
vllm_mlx.engine.batched.BatchedEngine._compute_prefix_boundary(messages: list[dict[str, Any]], tools: list[dict] | None = None, chat_template_kwargs: dict[str, Any] | None = None) -> int
```

Compute token count for the shared prefix across message variations.

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `messages` | `list[dict[str, Any]]` | `yes` | `none` | Required positional or keyword input. |
| `tools` | `list[dict] \| None` | `no` | `None` | Optional positional or keyword input; defaults to `None`. |
| `chat_template_kwargs` | `dict[str, Any] \| None` | `no` | `None` | Optional positional or keyword input; defaults to `None`. |

**Returns**

- Type: `int`
- Direct return expressions: `0`; `lcp`

**Exceptions and behavior**

Method `BatchedEngine._compute_prefix_boundary` calls `range`, `len`, `messages[i].get`, `convert_tools_for_template`; has 2 explicit return paths.
No direct `raise` statement appears in this definition.

[View source #L986-L1046](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/batched.py#L986-L1046).

</details>

<details class="api-contract" id="contract-vllm_mlx.engine.batched.BatchedEngine.stream_chat" markdown="1">
<summary><code>vllm_mlx.engine.batched.BatchedEngine.stream_chat</code> · method</summary>

```python
async vllm_mlx.engine.batched.BatchedEngine.stream_chat(messages: list[dict[str, Any]], max_tokens: int = 256, temperature: float = 0.7, top_p: float = 0.9, tools: list[dict] | None = None, images: list[str] | None = None, videos: list[str] | None = None, **kwargs) -> AsyncIterator[GenerationOutput]
```

Stream chat completion token by token.

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `messages` | `list[dict[str, Any]]` | `yes` | `none` | List of chat messages (OpenAI format) |
| `max_tokens` | `int` | `no` | `256` | Maximum tokens to generate |
| `temperature` | `float` | `no` | `0.7` | Sampling temperature |
| `top_p` | `float` | `no` | `0.9` | Top-p sampling |
| `tools` | `list[dict] \| None` | `no` | `None` | Optional tool definitions |
| `images` | `list[str] \| None` | `no` | `None` | Optional image URLs/paths |
| `videos` | `list[str] \| None` | `no` | `None` | Optional video URLs/paths |
| `**kwargs` | `not annotated` | `no` | `none` | Additional model-specific parameters |

**Returns**

- Type: `AsyncIterator[GenerationOutput]`
- Yields values incrementally.

**Exceptions and behavior**

Method `BatchedEngine.stream_chat` calls `self.start`, `extract_multimodal_content`, `convert_tools_for_template`, `dict`; awaits asynchronous work; yields values incrementally.
No direct `raise` statement appears in this definition.

[View source #L1048-L1127](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/batched.py#L1048-L1127).

</details>

<details class="api-contract" id="contract-vllm_mlx.engine.batched.BatchedEngine.get_stats" markdown="1">
<summary><code>vllm_mlx.engine.batched.BatchedEngine.get_stats</code> · method</summary>

```python
vllm_mlx.engine.batched.BatchedEngine.get_stats() -> dict[str, Any]
```

Get engine statistics.

**Parameters**

This callable has no explicit inputs.

**Returns**

- Type: `dict[str, Any]`
- Direct return expressions: `stats`

**Exceptions and behavior**

Method `BatchedEngine.get_stats` calls `time.time`, `self._mllm_scheduler.get_stats`, `stats.update`, `self._engine.get_stats`; returns `stats`.
No direct `raise` statement appears in this definition.

[View source #L1129-L1169](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/batched.py#L1129-L1169).

</details>

<details class="api-contract" id="contract-vllm_mlx.engine.batched.BatchedEngine.get_cache_stats" markdown="1">
<summary><code>vllm_mlx.engine.batched.BatchedEngine.get_cache_stats</code> · method</summary>

```python
vllm_mlx.engine.batched.BatchedEngine.get_cache_stats() -> dict[str, Any] | None
```

Get cache statistics.

**Parameters**

This callable has no explicit inputs.

**Returns**

- Type: `dict[str, Any] | None`
- Direct return expressions: `{'prefix_cache': self._mllm_scheduler.batch_generator.get_prefix_cache_stats(), 'vision_embedding_cache': self._mllm_sc…`; `self._engine.get_cache_stats()`; `None`

**Exceptions and behavior**

Method `BatchedEngine.get_cache_stats` calls `self._mllm_scheduler.batch_generator.get_prefix_cache_stats`, `self._mllm_scheduler.batch_generator.get_vision_cache_stats`, `self._engine.get_cache_stats`; has 3 explicit return paths.
No direct `raise` statement appears in this definition.

[View source #L1171-L1180](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/batched.py#L1171-L1180).

</details>

<details class="api-contract" id="contract-vllm_mlx.engine.batched.BatchedEngine.clear_runtime_caches" markdown="1">
<summary><code>vllm_mlx.engine.batched.BatchedEngine.clear_runtime_caches</code> · method</summary>

```python
vllm_mlx.engine.batched.BatchedEngine.clear_runtime_caches() -> dict[str, Any] | None
```

Clear engine-managed runtime caches.

**Parameters**

This callable has no explicit inputs.

**Returns**

- Type: `dict[str, Any] | None`
- Direct return expressions: `self._mllm_scheduler.clear_runtime_caches()`; `self._engine.clear_runtime_caches()`; `None`

**Exceptions and behavior**

Method `BatchedEngine.clear_runtime_caches` calls `self._mllm_scheduler.clear_runtime_caches`, `self._engine.clear_runtime_caches`; has 3 explicit return paths.
No direct `raise` statement appears in this definition.

[View source #L1182-L1188](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/batched.py#L1182-L1188).

</details>

<details class="api-contract" id="contract-vllm_mlx.engine.batched.BatchedEngine.abort_request" markdown="1">
<summary><code>vllm_mlx.engine.batched.BatchedEngine.abort_request</code> · method</summary>

```python
async vllm_mlx.engine.batched.BatchedEngine.abort_request(request_id: str) -> bool
```

Abort an active or queued batched request by request ID.

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `request_id` | `str` | `yes` | `none` | Required positional or keyword input. |

**Returns**

- Type: `bool`
- Direct return expressions: `self._mllm_scheduler.abort_request(request_id)`; `await result`; `result`; `False`

**Exceptions and behavior**

Method `BatchedEngine.abort_request` calls `self._mllm_scheduler.abort_request`, `hasattr`, `self._engine.abort_request`, `inspect.isawaitable`; awaits asynchronous work; has 4 explicit return paths.
No direct `raise` statement appears in this definition.

[View source #L1190-L1199](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/batched.py#L1190-L1199).

</details>

<details class="api-contract" id="contract-vllm_mlx.engine.batched.BatchedEngine.save_cache_to_disk" markdown="1">
<summary><code>vllm_mlx.engine.batched.BatchedEngine.save_cache_to_disk</code> · method</summary>

```python
vllm_mlx.engine.batched.BatchedEngine.save_cache_to_disk(cache_dir: str) -> bool
```

Save prefix cache to disk for persistence across restarts.

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `cache_dir` | `str` | `yes` | `none` | Required positional or keyword input. |

**Returns**

- Type: `bool`
- Direct return expressions: `pc.save_to_disk(cache_dir)`; `self._engine.save_cache_to_disk(cache_dir)`; `False`

**Exceptions and behavior**

Method `BatchedEngine.save_cache_to_disk` calls `pc.save_to_disk`, `self._engine.save_cache_to_disk`; has 3 explicit return paths.
No direct `raise` statement appears in this definition.

[View source #L1201-L1209](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/batched.py#L1201-L1209).

</details>

<details class="api-contract" id="contract-vllm_mlx.engine.batched.BatchedEngine.load_cache_from_disk" markdown="1">
<summary><code>vllm_mlx.engine.batched.BatchedEngine.load_cache_from_disk</code> · method</summary>

```python
vllm_mlx.engine.batched.BatchedEngine.load_cache_from_disk(cache_dir: str) -> int
```

Load prefix cache from disk.

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `cache_dir` | `str` | `yes` | `none` | Required positional or keyword input. |

**Returns**

- Type: `int`
- Direct return expressions: `pc.load_from_disk(cache_dir)`; `self._engine.load_cache_from_disk(cache_dir)`; `0`

**Exceptions and behavior**

Method `BatchedEngine.load_cache_from_disk` calls `self._mllm_scheduler._ensure_batch_generator`, `pc.load_from_disk`, `self._engine.load_cache_from_disk`; has 3 explicit return paths.
No direct `raise` statement appears in this definition.

[View source #L1211-L1220](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/batched.py#L1211-L1220).

</details>

<details class="api-contract" id="contract-vllm_mlx.engine.batched.BatchedEngine.clear_prefix_cache" markdown="1">
<summary><code>vllm_mlx.engine.batched.BatchedEngine.clear_prefix_cache</code> · method</summary>

```python
vllm_mlx.engine.batched.BatchedEngine.clear_prefix_cache() -> None
```

Clear the in-memory prefix cache.

**Parameters**

This callable has no explicit inputs.

**Returns**

- Type: `None`
- Direct return expressions: `None`

**Exceptions and behavior**

Method `BatchedEngine.clear_prefix_cache` calls `hasattr`, `pc.clear`, `self._engine.clear_prefix_cache`; returns `None`.
No direct `raise` statement appears in this definition.

[View source #L1222-L1231](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/batched.py#L1222-L1231).

</details>

## Complete symbol map

This map also includes private definitions and nested helpers. The signature column exposes every explicit input even when an internal helper has no dedicated parameter prose.

| Symbol | Kind | Signature and inputs | What it does | Source |
| --- | --- | --- | --- | --- |
| [`_resolve_metal_buffer_cache_limit`](#contract-vllm_mlx.engine.batched._resolve_metal_buffer_cache_limit) | function | `_resolve_metal_buffer_cache_limit(max_recommended: int, gpu_memory_utilization: float) -> tuple[int, str]` | Resolve the MLX retained-buffer cache cap for Metal startup. | [#L35-L58](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/batched.py#L35-L58) |
| [`_normalize_tool_call_arguments_for_template`](#contract-vllm_mlx.engine.batched._normalize_tool_call_arguments_for_template) | function | `_normalize_tool_call_arguments_for_template(messages: list[dict]) -> list[dict]` | Normalize OpenAI tool-call replay for templates expecting mappings. | [#L61-L63](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/batched.py#L61-L63) |
| [`_extract_media_from_messages`](#contract-vllm_mlx.engine.batched._extract_media_from_messages) | function | `_extract_media_from_messages(messages: list[dict[str, Any]]) -> tuple` | Extract images, videos, and audio from OpenAI-format messages. | [#L66-L137](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/batched.py#L66-L137) |
| [`MLLMModelWrapper`](#contract-vllm_mlx.engine.batched.MLLMModelWrapper) | class | `MLLMModelWrapper(model)` | Wrapper for MLLM models to make them compatible with BatchGenerator. | [#L140-L175](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/batched.py#L140-L175) |
| [`MLLMModelWrapper.__init__`](#contract-vllm_mlx.engine.batched.MLLMModelWrapper.__init__) | method | `MLLMModelWrapper.__init__(model) -> not annotated` | Method `MLLMModelWrapper.__init__` updates `self._model`, `self._is_gemma3`; calls `hasattr`, `str(getattr(model, 'model_type', '')).lower`, `str`, `getattr`. | [#L152-L158](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/batched.py#L152-L158) |
| [`MLLMModelWrapper.__call__`](#contract-vllm_mlx.engine.batched.MLLMModelWrapper.__call__) | method | `MLLMModelWrapper.__call__(*args, **kwargs) -> not annotated` | Call the model and extract logits from LanguageModelOutput. | [#L160-L171](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/batched.py#L160-L171) |
| [`MLLMModelWrapper.__getattr__`](#contract-vllm_mlx.engine.batched.MLLMModelWrapper.__getattr__) | method | `MLLMModelWrapper.__getattr__(name) -> not annotated` | Forward all other attributes to the wrapped model. | [#L173-L175](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/batched.py#L173-L175) |
| [`BatchedEngine`](#contract-vllm_mlx.engine.batched.BatchedEngine) | class | `BatchedEngine(model_name: str, trust_remote_code: bool = False, scheduler_config: Any \| None = None, stream_interval: int = 1, force_mllm: bool = False, gpu_memory_utilization: float = 0.9)` | Batched engine for continuous batching. | [#L178-L1231](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/batched.py#L178-L1231) |
| [`BatchedEngine.__init__`](#contract-vllm_mlx.engine.batched.BatchedEngine.__init__) | method | `BatchedEngine.__init__(model_name: str, trust_remote_code: bool = False, scheduler_config: Any \| None = None, stream_interval: int = 1, force_mllm: bool = False, gpu_memory_utilization: float = 0.9) -> not annotated` | Initialize the batched engine. | [#L189-L224](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/batched.py#L189-L224) |
| [`BatchedEngine.model_name`](#contract-vllm_mlx.engine.batched.BatchedEngine.model_name) | method | `BatchedEngine.model_name() -> str` | Get the model name. | [#L227-L229](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/batched.py#L227-L229) |
| [`BatchedEngine.is_mllm`](#contract-vllm_mlx.engine.batched.BatchedEngine.is_mllm) | method | `BatchedEngine.is_mllm() -> bool` | Check if this is a multimodal model. | [#L232-L234](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/batched.py#L232-L234) |
| [`BatchedEngine.tokenizer`](#contract-vllm_mlx.engine.batched.BatchedEngine.tokenizer) | method | `BatchedEngine.tokenizer() -> Any` | Get the tokenizer. | [#L237-L241](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/batched.py#L237-L241) |
| [`BatchedEngine.prepare_for_start`](#contract-vllm_mlx.engine.batched.BatchedEngine.prepare_for_start) | method | `BatchedEngine.prepare_for_start() -> None` | Load heavyweight model state off the serving event loop. | [#L243-L251](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/batched.py#L243-L251) |
| [`BatchedEngine.start`](#contract-vllm_mlx.engine.batched.BatchedEngine.start) | method | `async BatchedEngine.start() -> None` | Start the engine (load model if not loaded). | [#L253-L282](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/batched.py#L253-L282) |
| [`BatchedEngine._uses_default_prepare_for_start`](#contract-vllm_mlx.engine.batched.BatchedEngine._uses_default_prepare_for_start) | method | `BatchedEngine._uses_default_prepare_for_start() -> bool` | Return True when prepare_for_start is the class implementation. | [#L284-L287](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/batched.py#L284-L287) |
| [`BatchedEngine._prepare_mllm_model`](#contract-vllm_mlx.engine.batched.BatchedEngine._prepare_mllm_model) | method | `BatchedEngine._prepare_mllm_model() -> None` | Load the MLLM model before scheduler startup. | [#L289-L334](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/batched.py#L289-L334) |
| [`BatchedEngine._start_mllm`](#contract-vllm_mlx.engine.batched.BatchedEngine._start_mllm) | method | `async BatchedEngine._start_mllm() -> None` | Start the MLLM engine with MLLMScheduler (continuous batching). | [#L336-L429](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/batched.py#L336-L429) |
| [`BatchedEngine._inject_mtp_mllm`](#contract-vllm_mlx.engine.batched.BatchedEngine._inject_mtp_mllm) | method | `BatchedEngine._inject_mtp_mllm() -> None` | Inject MTP weights into the MLLM model's language_model. | [#L431-L477](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/batched.py#L431-L477) |
| [`BatchedEngine._prepare_llm_model`](#contract-vllm_mlx.engine.batched.BatchedEngine._prepare_llm_model) | method | `BatchedEngine._prepare_llm_model() -> None` | Load the LLM model/tokenizer before engine loop startup. | [#L479-L511](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/batched.py#L479-L511) |
| [`BatchedEngine._configure_metal_memory_limits`](#contract-vllm_mlx.engine.batched.BatchedEngine._configure_metal_memory_limits) | method | `BatchedEngine._configure_metal_memory_limits() -> None` | Make MLX allocation failures graceful during startup. | [#L513-L541](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/batched.py#L513-L541) |
| [`BatchedEngine._start_llm`](#contract-vllm_mlx.engine.batched.BatchedEngine._start_llm) | method | `async BatchedEngine._start_llm() -> None` | Start the LLM engine with AsyncEngineCore. | [#L543-L579](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/batched.py#L543-L579) |
| [`BatchedEngine.stop`](#contract-vllm_mlx.engine.batched.BatchedEngine.stop) | method | `async BatchedEngine.stop() -> None` | Stop the engine and cleanup resources. | [#L581-L597](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/batched.py#L581-L597) |
| [`BatchedEngine._apply_chat_template`](#contract-vllm_mlx.engine.batched.BatchedEngine._apply_chat_template) | method | `BatchedEngine._apply_chat_template(messages: list[dict[str, Any]], tools: list[dict] \| None = None, num_images: int = 0, num_audios: int = 0, chat_template_kwargs: dict[str, Any] \| None = None, enable_thinking: bool \| None = None) -> str` | Apply chat template to messages. | [#L599-L687](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/batched.py#L599-L687) |
| [`BatchedEngine._prepare_mllm_messages`](#contract-vllm_mlx.engine.batched.BatchedEngine._prepare_mllm_messages) | method | `BatchedEngine._prepare_mllm_messages(messages: list[dict[str, Any]]) -> list[dict[str, Any]]` | Convert OpenAI-style multimodal content to HuggingFace format. | [#L690-L726](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/batched.py#L690-L726) |
| [`BatchedEngine.generate`](#contract-vllm_mlx.engine.batched.BatchedEngine.generate) | method | `async BatchedEngine.generate(prompt: str, max_tokens: int = 256, temperature: float = 0.7, top_p: float = 0.9, stop: list[str] \| None = None, images: list[str] \| None = None, videos: list[str] \| None = None, audio: list[str] \| None = None, **kwargs) -> GenerationOutput` | Generate a complete response (non-streaming). | [#L728-L817](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/batched.py#L728-L817) |
| [`BatchedEngine.stream_generate`](#contract-vllm_mlx.engine.batched.BatchedEngine.stream_generate) | method | `async BatchedEngine.stream_generate(prompt: str, max_tokens: int = 256, temperature: float = 0.7, top_p: float = 0.9, stop: list[str] \| None = None, images: list[str] \| None = None, videos: list[str] \| None = None, audio: list[str] \| None = None, **kwargs) -> AsyncIterator[GenerationOutput]` | Stream generation token by token. | [#L819-L913](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/batched.py#L819-L913) |
| [`BatchedEngine.chat`](#contract-vllm_mlx.engine.batched.BatchedEngine.chat) | method | `async BatchedEngine.chat(messages: list[dict[str, Any]], max_tokens: int = 256, temperature: float = 0.7, top_p: float = 0.9, tools: list[dict] \| None = None, images: list[str] \| None = None, videos: list[str] \| None = None, **kwargs) -> GenerationOutput` | Chat completion (non-streaming). | [#L915-L984](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/batched.py#L915-L984) |
| [`BatchedEngine._compute_prefix_boundary`](#contract-vllm_mlx.engine.batched.BatchedEngine._compute_prefix_boundary) | method | `BatchedEngine._compute_prefix_boundary(messages: list[dict[str, Any]], tools: list[dict] \| None = None, chat_template_kwargs: dict[str, Any] \| None = None) -> int` | Compute token count for the shared prefix across message variations. | [#L986-L1046](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/batched.py#L986-L1046) |
| [`BatchedEngine.stream_chat`](#contract-vllm_mlx.engine.batched.BatchedEngine.stream_chat) | method | `async BatchedEngine.stream_chat(messages: list[dict[str, Any]], max_tokens: int = 256, temperature: float = 0.7, top_p: float = 0.9, tools: list[dict] \| None = None, images: list[str] \| None = None, videos: list[str] \| None = None, **kwargs) -> AsyncIterator[GenerationOutput]` | Stream chat completion token by token. | [#L1048-L1127](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/batched.py#L1048-L1127) |
| [`BatchedEngine.get_stats`](#contract-vllm_mlx.engine.batched.BatchedEngine.get_stats) | method | `BatchedEngine.get_stats() -> dict[str, Any]` | Get engine statistics. | [#L1129-L1169](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/batched.py#L1129-L1169) |
| [`BatchedEngine.get_cache_stats`](#contract-vllm_mlx.engine.batched.BatchedEngine.get_cache_stats) | method | `BatchedEngine.get_cache_stats() -> dict[str, Any] \| None` | Get cache statistics. | [#L1171-L1180](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/batched.py#L1171-L1180) |
| [`BatchedEngine.clear_runtime_caches`](#contract-vllm_mlx.engine.batched.BatchedEngine.clear_runtime_caches) | method | `BatchedEngine.clear_runtime_caches() -> dict[str, Any] \| None` | Clear engine-managed runtime caches. | [#L1182-L1188](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/batched.py#L1182-L1188) |
| [`BatchedEngine.abort_request`](#contract-vllm_mlx.engine.batched.BatchedEngine.abort_request) | method | `async BatchedEngine.abort_request(request_id: str) -> bool` | Abort an active or queued batched request by request ID. | [#L1190-L1199](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/batched.py#L1190-L1199) |
| [`BatchedEngine.save_cache_to_disk`](#contract-vllm_mlx.engine.batched.BatchedEngine.save_cache_to_disk) | method | `BatchedEngine.save_cache_to_disk(cache_dir: str) -> bool` | Save prefix cache to disk for persistence across restarts. | [#L1201-L1209](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/batched.py#L1201-L1209) |
| [`BatchedEngine.load_cache_from_disk`](#contract-vllm_mlx.engine.batched.BatchedEngine.load_cache_from_disk) | method | `BatchedEngine.load_cache_from_disk(cache_dir: str) -> int` | Load prefix cache from disk. | [#L1211-L1220](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/batched.py#L1211-L1220) |
| [`BatchedEngine.clear_prefix_cache`](#contract-vllm_mlx.engine.batched.BatchedEngine.clear_prefix_cache) | method | `BatchedEngine.clear_prefix_cache() -> None` | Clear the in-memory prefix cache. | [#L1222-L1231](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/engine/batched.py#L1222-L1231) |