# `vllm_mlx.mllm_batch_generator`

MLLM Batch Generator for multimodal continuous batching.

[View the complete module source at #L1-L3073](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L1-L3073).

## API details

Each callable below includes its exact signature, type annotations, inputs, defaults, return contract, documented exceptions, implementation source, and parsed docstring sections when the source provides them.

::: vllm_mlx.mllm_batch_generator
    options:
      members:
        - logger
        - _processors_can_retire
        - _mark_mtp_attempts_on_primary_responses
        - _drop_retired_processors
        - _request_uses_stochastic_sampling
        - _sampling_logprobs
        - _residual_logprobs
        - _accept_sampled_draft
        - PrefillAbortedError
        - _cache_eval_tensors
        - _eval_prompt_cache
        - MLLMBatchRequest
        - MLLMBatchResponse
        - MLLMBatch
        - MLLMBatchStats
        - _left_pad_prompts
        - MLLMBatchGenerator
        - install_mtp_mllm
        - install_chunked_prefill_mllm
      filters: []
      show_if_no_docstring: true

## Complete contract reference

Expand any definition for its exact inputs, annotations, defaults, return contract, directly raised exceptions, source-grounded behavior, and immutable line link. This section includes private and nested definitions that ordinary API generators omit.

<details class="api-contract" id="contract-vllm_mlx.mllm_batch_generator._processors_can_retire" markdown="1">
<summary><code>vllm_mlx.mllm_batch_generator._processors_can_retire</code> · function</summary>

```python
vllm_mlx.mllm_batch_generator._processors_can_retire(processors: Optional[List[Callable]]) -> bool
```

True when any processor advertises a retire-to-content transition.

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `processors` | `Optional[List[Callable]]` | `yes` | `none` | Required positional or keyword input. |

**Returns**

- Type: `bool`
- Direct return expressions: `False`; `bool(processors) and any((isinstance(getattr(p, 'is_retired', None), bool) for p in processors))`

**Exceptions and behavior**

Function `_processors_can_retire` calls `os.getenv`, `bool`, `any`, `isinstance`; has 2 explicit return paths.
No direct `raise` statement appears in this definition.

[View source #L37-L43](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L37-L43).

</details>

<details class="api-contract" id="contract-vllm_mlx.mllm_batch_generator._mark_mtp_attempts_on_primary_responses" markdown="1">
<summary><code>vllm_mlx.mllm_batch_generator._mark_mtp_attempts_on_primary_responses</code> · function</summary>

```python
vllm_mlx.mllm_batch_generator._mark_mtp_attempts_on_primary_responses(responses: List['MLLMBatchResponse'], attempted_drafts_by_uid: Dict[int, int]) -> None
```

Mark only responses from steps that actually attempted MTP drafts.

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `responses` | `List['MLLMBatchResponse']` | `yes` | `none` | Required positional or keyword input. |
| `attempted_drafts_by_uid` | `Dict[int, int]` | `yes` | `none` | Required positional or keyword input. |

**Returns**

- Type: `None`

**Exceptions and behavior**

Function `_mark_mtp_attempts_on_primary_responses` calls `attempted_drafts_by_uid.pop`, `attempted_drafts_by_uid.clear`.
No direct `raise` statement appears in this definition.

[View source #L46-L57](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L46-L57).

</details>

<details class="api-contract" id="contract-vllm_mlx.mllm_batch_generator._drop_retired_processors" markdown="1">
<summary><code>vllm_mlx.mllm_batch_generator._drop_retired_processors</code> · function</summary>

```python
vllm_mlx.mllm_batch_generator._drop_retired_processors(processors: Optional[List[Callable]]) -> tuple[Optional[List[Callable]], int]
```

Drop retire-capable processors that have completed their work.

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `processors` | `Optional[List[Callable]]` | `yes` | `none` | Required positional or keyword input. |

**Returns**

- Type: `tuple[Optional[List[Callable]], int]`
- Direct return expressions: `(processors, 0)`; `(remaining or None, retired_count)`

**Exceptions and behavior**

Function `_drop_retired_processors` calls `getattr`, `remaining.append`; has 2 explicit return paths.
No direct `raise` statement appears in this definition.

[View source #L60-L74](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L60-L74).

</details>

<details class="api-contract" id="contract-vllm_mlx.mllm_batch_generator._request_uses_stochastic_sampling" markdown="1">
<summary><code>vllm_mlx.mllm_batch_generator._request_uses_stochastic_sampling</code> · function</summary>

```python
vllm_mlx.mllm_batch_generator._request_uses_stochastic_sampling(request: Any) -> bool
```

Return whether a request needs sampler-aware speculative verification.

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `request` | `Any` | `yes` | `none` | Required positional or keyword input. |

**Returns**

- Type: `bool`
- Direct return expressions: `False`; `getattr(request, 'top_p', 1.0) < 1.0 or getattr(request, 'top_k', 0) != 0 or getattr(request, 'min_p', 0.0) != 0.0`

**Exceptions and behavior**

Function `_request_uses_stochastic_sampling` calls `getattr`; has 2 explicit return paths.
No direct `raise` statement appears in this definition.

[View source #L77-L92](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L77-L92).

</details>

<details class="api-contract" id="contract-vllm_mlx.mllm_batch_generator._sampling_logprobs" markdown="1">
<summary><code>vllm_mlx.mllm_batch_generator._sampling_logprobs</code> · function</summary>

```python
vllm_mlx.mllm_batch_generator._sampling_logprobs(logits: mx.array, request: Any) -> mx.array
```

Match mlx-lm's request sampler in log-probability space.

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `logits` | `mx.array` | `yes` | `none` | Required positional or keyword input. |
| `request` | `Any` | `yes` | `none` | Required positional or keyword input. |

**Returns**

- Type: `mx.array`
- Direct return expressions: `mx.put_along_axis(result, token[:, None], 0.0, axis=-1)`; `logprobs - mx.logsumexp(logprobs, axis=-1, keepdims=True)`

**Exceptions and behavior**

Function `_sampling_logprobs` calls `getattr`, `mx.logsumexp`, `mx.argmax`, `mx.full`; has 2 explicit return paths.
No direct `raise` statement appears in this definition.

[View source #L95-L123](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L95-L123).

</details>

<details class="api-contract" id="contract-vllm_mlx.mllm_batch_generator._residual_logprobs" markdown="1">
<summary><code>vllm_mlx.mllm_batch_generator._residual_logprobs</code> · function</summary>

```python
vllm_mlx.mllm_batch_generator._residual_logprobs(target_logprobs: mx.array, draft_logprobs: mx.array) -> mx.array
```

Return the normalized residual max(target - draft, 0) distribution.

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `target_logprobs` | `mx.array` | `yes` | `none` | Required positional or keyword input. |
| `draft_logprobs` | `mx.array` | `yes` | `none` | Required positional or keyword input. |

**Returns**

- Type: `mx.array`
- Direct return expressions: `mx.where(mass > 1e-12, normalized, fallback)`

**Exceptions and behavior**

Function `_residual_logprobs` calls `mx.maximum`, `mx.exp`, `mx.sum`, `mx.where`; returns `mx.where(mass > 1e-12, normalized, fallback)`.
No direct `raise` statement appears in this definition.

[View source #L126-L139](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L126-L139).

</details>

<details class="api-contract" id="contract-vllm_mlx.mllm_batch_generator._accept_sampled_draft" markdown="1">
<summary><code>vllm_mlx.mllm_batch_generator._accept_sampled_draft</code> · function</summary>

```python
vllm_mlx.mllm_batch_generator._accept_sampled_draft(target_logprob: float, draft_logprob: float, uniform_draw: float) -> bool
```

Apply the exact min(1, p/q) stochastic speculative acceptance rule.

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `target_logprob` | `float` | `yes` | `none` | Required positional or keyword input. |
| `draft_logprob` | `float` | `yes` | `none` | Required positional or keyword input. |
| `uniform_draw` | `float` | `yes` | `none` | Required positional or keyword input. |

**Returns**

- Type: `bool`
- Direct return expressions: `log_acceptance >= 0.0 or math.log(max(uniform_draw, 1e-35)) < log_acceptance`

**Exceptions and behavior**

Function `_accept_sampled_draft` calls `math.log`, `max`; returns `log_acceptance >= 0.0 or math.log(max(uniform_draw, 1e-35)) < log_acceptance`.
No direct `raise` statement appears in this definition.

[View source #L142-L149](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L142-L149).

</details>

<details class="api-contract" id="contract-vllm_mlx.mllm_batch_generator.PrefillAbortedError" markdown="1">
<summary><code>vllm_mlx.mllm_batch_generator.PrefillAbortedError</code> · class</summary>

```python
vllm_mlx.mllm_batch_generator.PrefillAbortedError(request_id: str)
```

Raised when a prefill is aborted due to client disconnect.

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `request_id` | `str` | `yes` | `none` | Required positional or keyword input. |

**Returns**

- Constructs: `vllm_mlx.mllm_batch_generator.PrefillAbortedError`

**Exceptions and behavior**

Class `PrefillAbortedError` derives from `Exception` and declares 1 direct member(s).
No direct `raise` statement appears in this definition.

[View source #L152-L157](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L152-L157).

</details>

<details class="api-contract" id="contract-vllm_mlx.mllm_batch_generator.PrefillAbortedError.__init__" markdown="1">
<summary><code>vllm_mlx.mllm_batch_generator.PrefillAbortedError.__init__</code> · method</summary>

```python
vllm_mlx.mllm_batch_generator.PrefillAbortedError.__init__(request_id: str) -> not annotated
```

Method `PrefillAbortedError.__init__` updates `self.request_id`; calls `super().__init__`, `super`.

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `request_id` | `str` | `yes` | `none` | Required positional or keyword input. |

**Returns**

- Type: `not annotated`

**Exceptions and behavior**

Method `PrefillAbortedError.__init__` updates `self.request_id`; calls `super().__init__`, `super`.
No direct `raise` statement appears in this definition.

[View source #L155-L157](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L155-L157).

</details>

<details class="api-contract" id="contract-vllm_mlx.mllm_batch_generator._cache_eval_tensors" markdown="1">
<summary><code>vllm_mlx.mllm_batch_generator._cache_eval_tensors</code> · function</summary>

```python
vllm_mlx.mllm_batch_generator._cache_eval_tensors(cache: List[Any]) -> List[Any]
```

Return realized tensors that break lazy cache graphs between chunks.

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `cache` | `List[Any]` | `yes` | `none` | Required positional or keyword input. |

**Returns**

- Type: `List[Any]`
- Direct return expressions: `tensors`

**Exceptions and behavior**

Function `_cache_eval_tensors` calls `getattr`, `tensors.append`, `isinstance`, `tensors.extend`; returns `tensors`.
No direct `raise` statement appears in this definition.

[View source #L160-L183](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L160-L183).

</details>

<details class="api-contract" id="contract-vllm_mlx.mllm_batch_generator._eval_prompt_cache" markdown="1">
<summary><code>vllm_mlx.mllm_batch_generator._eval_prompt_cache</code> · function</summary>

```python
vllm_mlx.mllm_batch_generator._eval_prompt_cache(cache: List[Any]) -> None
```

Evaluate all cache tensors used by hybrid chunked prefill.

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `cache` | `List[Any]` | `yes` | `none` | Required positional or keyword input. |

**Returns**

- Type: `None`

**Exceptions and behavior**

Function `_eval_prompt_cache` calls `_cache_eval_tensors`, `mx.eval`.
No direct `raise` statement appears in this definition.

[View source #L186-L190](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L186-L190).

</details>

<details class="api-contract" id="contract-vllm_mlx.mllm_batch_generator.MLLMBatchRequest" markdown="1">
<summary><code>vllm_mlx.mllm_batch_generator.MLLMBatchRequest</code> · class</summary>

```python
vllm_mlx.mllm_batch_generator.MLLMBatchRequest(uid: int, request_id: str, prompt: str, images: Optional[List[str]] = None, videos: Optional[List[str]] = None, audio: Optional[List[str]] = None, max_tokens: int = 256, temperature: float = 0.7, top_p: float = 0.9, top_k: int = 0, min_p: float = 0.0, presence_penalty: float = 0.0, repetition_penalty: float = 1.0, logits_processors: Optional[List[Callable]] = None, input_ids: Optional[mx.array] = None, pixel_values: Optional[mx.array] = None, attention_mask: Optional[mx.array] = None, image_grid_thw: Optional[mx.array] = None, extra_kwargs: Dict[str, Any] = field(default_factory=dict), is_text_only: bool = False, num_tokens: int = 0, output_tokens: List[int] = field(default_factory=list), vision_encoded: bool = False, cross_attention_states: Optional[Any] = None, encoder_outputs: Optional[Any] = None)
```

Request data for MLLM batch processing.

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `uid` | `int` | `yes` | `none` | Required constructor field. |
| `request_id` | `str` | `yes` | `none` | Required constructor field. |
| `prompt` | `str` | `yes` | `none` | Required constructor field. |
| `images` | `Optional[List[str]]` | `no` | `None` | Optional constructor field; defaults to `None`. |
| `videos` | `Optional[List[str]]` | `no` | `None` | Optional constructor field; defaults to `None`. |
| `audio` | `Optional[List[str]]` | `no` | `None` | Optional constructor field; defaults to `None`. |
| `max_tokens` | `int` | `no` | `256` | Optional constructor field; defaults to `256`. |
| `temperature` | `float` | `no` | `0.7` | Optional constructor field; defaults to `0.7`. |
| `top_p` | `float` | `no` | `0.9` | Optional constructor field; defaults to `0.9`. |
| `top_k` | `int` | `no` | `0` | Optional constructor field; defaults to `0`. |
| `min_p` | `float` | `no` | `0.0` | Optional constructor field; defaults to `0.0`. |
| `presence_penalty` | `float` | `no` | `0.0` | Optional constructor field; defaults to `0.0`. |
| `repetition_penalty` | `float` | `no` | `1.0` | Optional constructor field; defaults to `1.0`. |
| `logits_processors` | `Optional[List[Callable]]` | `no` | `None` | Optional constructor field; defaults to `None`. |
| `input_ids` | `Optional[mx.array]` | `no` | `None` | Optional constructor field; defaults to `None`. |
| `pixel_values` | `Optional[mx.array]` | `no` | `None` | Optional constructor field; defaults to `None`. |
| `attention_mask` | `Optional[mx.array]` | `no` | `None` | Optional constructor field; defaults to `None`. |
| `image_grid_thw` | `Optional[mx.array]` | `no` | `None` | Optional constructor field; defaults to `None`. |
| `extra_kwargs` | `Dict[str, Any]` | `no` | `field(default_factory=dict)` | Optional constructor field; defaults to `field(default_factory=dict)`. |
| `is_text_only` | `bool` | `no` | `False` | Optional constructor field; defaults to `False`. |
| `num_tokens` | `int` | `no` | `0` | Optional constructor field; defaults to `0`. |
| `output_tokens` | `List[int]` | `no` | `field(default_factory=list)` | Optional constructor field; defaults to `field(default_factory=list)`. |
| `vision_encoded` | `bool` | `no` | `False` | Optional constructor field; defaults to `False`. |
| `cross_attention_states` | `Optional[Any]` | `no` | `None` | Optional constructor field; defaults to `None`. |
| `encoder_outputs` | `Optional[Any]` | `no` | `None` | Optional constructor field; defaults to `None`. |

**Returns**

- Constructs: `vllm_mlx.mllm_batch_generator.MLLMBatchRequest`

**Exceptions and behavior**

Class `MLLMBatchRequest` declares 0 direct member(s).
No direct `raise` statement appears in this definition.

[View source #L194-L237](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L194-L237).

</details>

<details class="api-contract" id="contract-vllm_mlx.mllm_batch_generator.MLLMBatchResponse" markdown="1">
<summary><code>vllm_mlx.mllm_batch_generator.MLLMBatchResponse</code> · class</summary>

```python
vllm_mlx.mllm_batch_generator.MLLMBatchResponse(uid: int, request_id: str, token: int, logprobs: mx.array, finish_reason: Optional[str] = None, prompt_cache: Optional[Callable[[], List[Any]]] = None, from_draft: bool = False, mtp_attempted: bool = False, mtp_attempted_count: int = 0)
```

Response from a batch generation step.

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `uid` | `int` | `yes` | `none` | Required constructor field. |
| `request_id` | `str` | `yes` | `none` | Required constructor field. |
| `token` | `int` | `yes` | `none` | Required constructor field. |
| `logprobs` | `mx.array` | `yes` | `none` | Required constructor field. |
| `finish_reason` | `Optional[str]` | `no` | `None` | Optional constructor field; defaults to `None`. |
| `prompt_cache` | `Optional[Callable[[], List[Any]]]` | `no` | `None` | Optional constructor field; defaults to `None`. |
| `from_draft` | `bool` | `no` | `False` | Optional constructor field; defaults to `False`. |
| `mtp_attempted` | `bool` | `no` | `False` | Optional constructor field; defaults to `False`. |
| `mtp_attempted_count` | `int` | `no` | `0` | Optional constructor field; defaults to `0`. |

**Returns**

- Constructs: `vllm_mlx.mllm_batch_generator.MLLMBatchResponse`

**Exceptions and behavior**

Class `MLLMBatchResponse` declares 0 direct member(s).
No direct `raise` statement appears in this definition.

[View source #L241-L256](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L241-L256).

</details>

<details class="api-contract" id="contract-vllm_mlx.mllm_batch_generator.MLLMBatch" markdown="1">
<summary><code>vllm_mlx.mllm_batch_generator.MLLMBatch</code> · class</summary>

```python
vllm_mlx.mllm_batch_generator.MLLMBatch(uids: List[int], request_ids: List[str], y: mx.array, logprobs: List[mx.array], max_tokens: List[int], num_tokens: List[int], cache: List[Any], requests: List[MLLMBatchRequest], logits_processors: Optional[List[Optional[List[Callable]]]] = None, samplers: Optional[List[Optional[Callable]]] = None)
```

Represents an active batch of MLLM requests.

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `uids` | `List[int]` | `yes` | `none` | Required constructor field. |
| `request_ids` | `List[str]` | `yes` | `none` | Required constructor field. |
| `y` | `mx.array` | `yes` | `none` | Required constructor field. |
| `logprobs` | `List[mx.array]` | `yes` | `none` | Required constructor field. |
| `max_tokens` | `List[int]` | `yes` | `none` | Required constructor field. |
| `num_tokens` | `List[int]` | `yes` | `none` | Required constructor field. |
| `cache` | `List[Any]` | `yes` | `none` | Required constructor field. |
| `requests` | `List[MLLMBatchRequest]` | `yes` | `none` | Required constructor field. |
| `logits_processors` | `Optional[List[Optional[List[Callable]]]]` | `no` | `None` | Optional constructor field; defaults to `None`. |
| `samplers` | `Optional[List[Optional[Callable]]]` | `no` | `None` | Optional constructor field; defaults to `None`. |

**Returns**

- Constructs: `vllm_mlx.mllm_batch_generator.MLLMBatch`

**Exceptions and behavior**

Class `MLLMBatch` declares 4 direct member(s).
No direct `raise` statement appears in this definition.

[View source #L260-L392](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L260-L392).

</details>

<details class="api-contract" id="contract-vllm_mlx.mllm_batch_generator.MLLMBatch.__len__" markdown="1">
<summary><code>vllm_mlx.mllm_batch_generator.MLLMBatch.__len__</code> · method</summary>

```python
vllm_mlx.mllm_batch_generator.MLLMBatch.__len__() -> int
```

Method `MLLMBatch.__len__` calls `len`; returns `len(self.uids)`.

**Parameters**

This callable has no explicit inputs.

**Returns**

- Type: `int`
- Direct return expressions: `len(self.uids)`

**Exceptions and behavior**

Method `MLLMBatch.__len__` calls `len`; returns `len(self.uids)`.
No direct `raise` statement appears in this definition.

[View source #L279-L280](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L279-L280).

</details>

<details class="api-contract" id="contract-vllm_mlx.mllm_batch_generator.MLLMBatch.filter" markdown="1">
<summary><code>vllm_mlx.mllm_batch_generator.MLLMBatch.filter</code> · method</summary>

```python
vllm_mlx.mllm_batch_generator.MLLMBatch.filter(keep_idx: List[int]) -> None
```

Filter batch to keep only requests at specified indices.

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `keep_idx` | `List[int]` | `yes` | `none` | Indices of requests to keep |

**Returns**

- Type: `None`

**Exceptions and behavior**

Method `MLLMBatch.filter` updates `self.uids`, `self.request_ids`, `self.logprobs`, `self.max_tokens`; calls `mx.array`, `hasattr`, `c.filter`.
No direct `raise` statement appears in this definition.

[View source #L282-L306](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L282-L306).

</details>

<details class="api-contract" id="contract-vllm_mlx.mllm_batch_generator.MLLMBatch.extend" markdown="1">
<summary><code>vllm_mlx.mllm_batch_generator.MLLMBatch.extend</code> · method</summary>

```python
vllm_mlx.mllm_batch_generator.MLLMBatch.extend(other: 'MLLMBatch') -> None
```

Extend this batch with another batch.

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `other` | `'MLLMBatch'` | `yes` | `none` | Batch to merge into this one |

**Returns**

- Type: `None`

**Exceptions and behavior**

Method `MLLMBatch.extend` updates `self.y`, `self.logits_processors`, `self.samplers`; calls `self.uids.extend`, `self.request_ids.extend`, `mx.concatenate`, `self.logprobs.extend`.
No direct `raise` statement appears in this definition.

[View source #L308-L351](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L308-L351).

</details>

<details class="api-contract" id="contract-vllm_mlx.mllm_batch_generator.MLLMBatch.extract_cache" markdown="1">
<summary><code>vllm_mlx.mllm_batch_generator.MLLMBatch.extract_cache</code> · method</summary>

```python
vllm_mlx.mllm_batch_generator.MLLMBatch.extract_cache(idx: int) -> List[Any]
```

Extract cache for a single request (for prefix caching).

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `idx` | `int` | `yes` | `none` | Required positional or keyword input. |

**Returns**

- Type: `List[Any]`
- Direct return expressions: `result`

**Exceptions and behavior**

Method `MLLMBatch.extract_cache` calls `hasattr`, `result.append`, `isinstance`, `RotatingKVCache`; returns `result`.
No direct `raise` statement appears in this definition.

[View source #L353-L392](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L353-L392).

</details>

<details class="api-contract" id="contract-vllm_mlx.mllm_batch_generator.MLLMBatchStats" markdown="1">
<summary><code>vllm_mlx.mllm_batch_generator.MLLMBatchStats</code> · class</summary>

```python
vllm_mlx.mllm_batch_generator.MLLMBatchStats()
```

Statistics for MLLM batch generation.

**Parameters**

This callable has no explicit inputs.

**Returns**

- Constructs: `vllm_mlx.mllm_batch_generator.MLLMBatchStats`

**Exceptions and behavior**

Class `MLLMBatchStats` declares 4 direct member(s).
No direct `raise` statement appears in this definition.

[View source #L395-L436](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L395-L436).

</details>

<details class="api-contract" id="contract-vllm_mlx.mllm_batch_generator.MLLMBatchStats.__init__" markdown="1">
<summary><code>vllm_mlx.mllm_batch_generator.MLLMBatchStats.__init__</code> · method</summary>

```python
vllm_mlx.mllm_batch_generator.MLLMBatchStats.__init__() -> not annotated
```

Method `MLLMBatchStats.__init__` updates `self.prompt_tokens`, `self.prompt_time`, `self.generation_tokens`, `self.generation_time`.

**Parameters**

This callable has no explicit inputs.

**Returns**

- Type: `not annotated`

**Exceptions and behavior**

Method `MLLMBatchStats.__init__` updates `self.prompt_tokens`, `self.prompt_time`, `self.generation_tokens`, `self.generation_time`.
No direct `raise` statement appears in this definition.

[View source #L398-L405](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L398-L405).

</details>

<details class="api-contract" id="contract-vllm_mlx.mllm_batch_generator.MLLMBatchStats.prompt_tps" markdown="1">
<summary><code>vllm_mlx.mllm_batch_generator.MLLMBatchStats.prompt_tps</code> · method</summary>

```python
vllm_mlx.mllm_batch_generator.MLLMBatchStats.prompt_tps() -> float
```

Return measured multimodal prompt throughput in tokens per second.

**Parameters**

This callable has no explicit inputs.

**Returns**

- Type: `float`
- Direct return expressions: `0`; `self.prompt_tokens / self.prompt_time`

**Exceptions and behavior**

Method `MLLMBatchStats.prompt_tps` has 2 explicit return paths.
No direct `raise` statement appears in this definition.

[View source #L408-L413](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L408-L413).

</details>

<details class="api-contract" id="contract-vllm_mlx.mllm_batch_generator.MLLMBatchStats.generation_tps" markdown="1">
<summary><code>vllm_mlx.mllm_batch_generator.MLLMBatchStats.generation_tps</code> · method</summary>

```python
vllm_mlx.mllm_batch_generator.MLLMBatchStats.generation_tps() -> float
```

Return measured decode throughput in tokens per second.

**Parameters**

This callable has no explicit inputs.

**Returns**

- Type: `float`
- Direct return expressions: `0`; `self.generation_tokens / self.generation_time`

**Exceptions and behavior**

Method `MLLMBatchStats.generation_tps` has 2 explicit return paths.
No direct `raise` statement appears in this definition.

[View source #L416-L421](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L416-L421).

</details>

<details class="api-contract" id="contract-vllm_mlx.mllm_batch_generator.MLLMBatchStats.to_dict" markdown="1">
<summary><code>vllm_mlx.mllm_batch_generator.MLLMBatchStats.to_dict</code> · method</summary>

```python
vllm_mlx.mllm_batch_generator.MLLMBatchStats.to_dict() -> Dict[str, Any]
```

Return token, timing, vision, and peak-memory statistics.

**Parameters**

This callable has no explicit inputs.

**Returns**

- Type: `Dict[str, Any]`
- Direct return expressions: `{'prompt_tokens': self.prompt_tokens, 'prompt_time': self.prompt_time, 'prompt_tps': self.prompt_tps, 'generation_token…`

**Exceptions and behavior**

Method `MLLMBatchStats.to_dict` returns `{'prompt_tokens': self.prompt_tokens, 'prompt_time': self.prompt_time, 'prompt_tps': self.prompt_tps, 'generation_token…`.
No direct `raise` statement appears in this definition.

[View source #L423-L436](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L423-L436).

</details>

<details class="api-contract" id="contract-vllm_mlx.mllm_batch_generator._left_pad_prompts" markdown="1">
<summary><code>vllm_mlx.mllm_batch_generator._left_pad_prompts</code> · function</summary>

```python
vllm_mlx.mllm_batch_generator._left_pad_prompts(prompts: List[List[int]], max_length: Optional[int] = None) -> mx.array
```

Left-pad prompts to uniform length.

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `prompts` | `List[List[int]]` | `yes` | `none` | List of token lists |
| `max_length` | `Optional[int]` | `no` | `None` | Target length (computed if not provided) |

**Returns**

- Type: `mx.array`
- Direct return expressions: `mx.array([[0] * (max_length - len(p)) + list(p) for p in prompts])`

**Exceptions and behavior**

Function `_left_pad_prompts` calls `max`, `len`, `mx.array`, `list`; returns `mx.array([[0] * (max_length - len(p)) + list(p) for p in prompts])`.
No direct `raise` statement appears in this definition.

[View source #L439-L454](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L439-L454).

</details>

<details class="api-contract" id="contract-vllm_mlx.mllm_batch_generator.MLLMBatchGenerator" markdown="1">
<summary><code>vllm_mlx.mllm_batch_generator.MLLMBatchGenerator</code> · class</summary>

```python
vllm_mlx.mllm_batch_generator.MLLMBatchGenerator(model: nn.Module, processor: Any, mm_processor: Optional[MultimodalProcessor] = None, max_tokens: int = 256, stop_tokens: Optional[set] = None, sampler: Optional[Callable[[mx.array], mx.array]] = None, prefill_batch_size: int = 4, completion_batch_size: int = 16, prefill_step_size: int = 1024, enable_vision_cache: bool = True, vision_cache_size: int = 100, prefix_cache_config: Optional[MemoryCacheConfig] = None, max_kv_size: int = 0)
```

Batch generator for Vision Language Models.

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `model` | `nn.Module` | `yes` | `none` | The VLM model (must have model.language_model) |
| `processor` | `Any` | `yes` | `none` | The VLM processor for tokenization and image processing |
| `mm_processor` | `Optional[MultimodalProcessor]` | `no` | `None` | Optional MultimodalProcessor for input preparation |
| `max_tokens` | `int` | `no` | `256` | Default max tokens per request |
| `stop_tokens` | `Optional[set]` | `no` | `None` | Set of stop token IDs |
| `sampler` | `Optional[Callable[[mx.array], mx.array]]` | `no` | `None` | Sampling function (default: argmax) |
| `prefill_batch_size` | `int` | `no` | `4` | Max requests to prefill together |
| `completion_batch_size` | `int` | `no` | `16` | Max requests for completion batching |
| `prefill_step_size` | `int` | `no` | `1024` | Tokens to process per prefill step |
| `enable_vision_cache` | `bool` | `no` | `True` | Enable vision embedding caching |
| `vision_cache_size` | `int` | `no` | `100` | Max entries in vision cache |
| `prefix_cache_config` | `Optional[MemoryCacheConfig]` | `no` | `None` | Config for KV prefix cache (text-only requests) |
| `max_kv_size` | `int` | `no` | `0` | Maximum KV cache size per sequence (0 = unbounded) |

**Returns**

- Constructs: `vllm_mlx.mllm_batch_generator.MLLMBatchGenerator`

**Exceptions and behavior**

Class `MLLMBatchGenerator` declares 26 direct member(s).
No direct `raise` statement appears in this definition.

[View source #L457-L2042](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L457-L2042).

</details>

<details class="api-contract" id="contract-vllm_mlx.mllm_batch_generator.MLLMBatchGenerator.__init__" markdown="1">
<summary><code>vllm_mlx.mllm_batch_generator.MLLMBatchGenerator.__init__</code> · method</summary>

```python
vllm_mlx.mllm_batch_generator.MLLMBatchGenerator.__init__(model: nn.Module, processor: Any, mm_processor: Optional[MultimodalProcessor] = None, max_tokens: int = 256, stop_tokens: Optional[set] = None, sampler: Optional[Callable[[mx.array], mx.array]] = None, prefill_batch_size: int = 4, completion_batch_size: int = 16, prefill_step_size: int = 1024, enable_vision_cache: bool = True, vision_cache_size: int = 100, prefix_cache_config: Optional[MemoryCacheConfig] = None, max_kv_size: int = 0) -> not annotated
```

Initialize MLLM batch generator.

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `model` | `nn.Module` | `yes` | `none` | The VLM model (must have model.language_model) |
| `processor` | `Any` | `yes` | `none` | The VLM processor for tokenization and image processing |
| `mm_processor` | `Optional[MultimodalProcessor]` | `no` | `None` | Optional MultimodalProcessor for input preparation |
| `max_tokens` | `int` | `no` | `256` | Default max tokens per request |
| `stop_tokens` | `Optional[set]` | `no` | `None` | Set of stop token IDs |
| `sampler` | `Optional[Callable[[mx.array], mx.array]]` | `no` | `None` | Sampling function (default: argmax) |
| `prefill_batch_size` | `int` | `no` | `4` | Max requests to prefill together |
| `completion_batch_size` | `int` | `no` | `16` | Max requests for completion batching |
| `prefill_step_size` | `int` | `no` | `1024` | Tokens to process per prefill step |
| `enable_vision_cache` | `bool` | `no` | `True` | Enable vision embedding caching |
| `vision_cache_size` | `int` | `no` | `100` | Max entries in vision cache |
| `prefix_cache_config` | `Optional[MemoryCacheConfig]` | `no` | `None` | Config for KV prefix cache (text-only requests) |
| `max_kv_size` | `int` | `no` | `0` | Maximum KV cache size per sequence (0 = unbounded) |

**Returns**

- Type: `not annotated`

**Exceptions and behavior**

Method `MLLMBatchGenerator.__init__` updates `self.model`, `self.processor`, `self.mm_processor`, `self.max_kv_size`; calls `getattr`, `hasattr`, `logger.info`, `logger.warning`.
No direct `raise` statement appears in this definition.

[View source #L484-L632](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L484-L632).

</details>

<details class="api-contract" id="contract-vllm_mlx.mllm_batch_generator.MLLMBatchGenerator._normalize_chat_template_for_prefix_cache" markdown="1">
<summary><code>vllm_mlx.mllm_batch_generator.MLLMBatchGenerator._normalize_chat_template_for_prefix_cache</code> · method</summary>

```python
vllm_mlx.mllm_batch_generator.MLLMBatchGenerator._normalize_chat_template_for_prefix_cache() -> None
```

Patch chat template so historical assistant turns are prefix-stable.

**Parameters**

This callable has no explicit inputs.

**Returns**

- Type: `None`
- Direct return expressions: `None`

**Exceptions and behavior**

Method `MLLMBatchGenerator._normalize_chat_template_for_prefix_cache` updates `self.processor.chat_template`; calls `getattr`, `re.sub`, `hasattr`, `logger.info`; returns `None`.
No direct `raise` statement appears in this definition.

[View source #L634-L697](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L634-L697).

</details>

<details class="api-contract" id="contract-vllm_mlx.mllm_batch_generator.MLLMBatchGenerator._compute_think_suffix_len" markdown="1">
<summary><code>vllm_mlx.mllm_batch_generator.MLLMBatchGenerator._compute_think_suffix_len</code> · method</summary>

```python
vllm_mlx.mllm_batch_generator.MLLMBatchGenerator._compute_think_suffix_len() -> int
```

Compute how many extra tokens enable_thinking=True adds at the END.

**Parameters**

This callable has no explicit inputs.

**Returns**

- Type: `int`
- Direct return expressions: `0`; `max(0, suffix_len)`

**Exceptions and behavior**

Method `MLLMBatchGenerator._compute_think_suffix_len` calls `getattr`, `hasattr`, `applicator.apply_chat_template`, `text_with.endswith`; has 2 explicit return paths.
No direct `raise` statement appears in this definition.

[View source #L699-L758](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L699-L758).

</details>

<details class="api-contract" id="contract-vllm_mlx.mllm_batch_generator.MLLMBatchGenerator.close" markdown="1">
<summary><code>vllm_mlx.mllm_batch_generator.MLLMBatchGenerator.close</code> · method</summary>

```python
vllm_mlx.mllm_batch_generator.MLLMBatchGenerator.close() -> None
```

Release resources and reset wired limit.

**Parameters**

This callable has no explicit inputs.

**Returns**

- Type: `None`

**Exceptions and behavior**

Method `MLLMBatchGenerator.close` updates `self._old_wired_limit`; calls `mx.synchronize`, `mx.set_wired_limit`.
No direct `raise` statement appears in this definition.

[View source #L760-L765](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L760-L765).

</details>

<details class="api-contract" id="contract-vllm_mlx.mllm_batch_generator.MLLMBatchGenerator.abort_prefill" markdown="1">
<summary><code>vllm_mlx.mllm_batch_generator.MLLMBatchGenerator.abort_prefill</code> · method</summary>

```python
vllm_mlx.mllm_batch_generator.MLLMBatchGenerator.abort_prefill(request_id: str) -> None
```

Signal that a request's prefill should be aborted.

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `request_id` | `str` | `yes` | `none` | Required positional or keyword input. |

**Returns**

- Type: `None`

**Exceptions and behavior**

Method `MLLMBatchGenerator.abort_prefill` calls `self._aborted_request_ids.add`, `logger.info`.
No direct `raise` statement appears in this definition.

[View source #L767-L775](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L767-L775).

</details>

<details class="api-contract" id="contract-vllm_mlx.mllm_batch_generator.MLLMBatchGenerator.schedule_removal" markdown="1">
<summary><code>vllm_mlx.mllm_batch_generator.MLLMBatchGenerator.schedule_removal</code> · method</summary>

```python
vllm_mlx.mllm_batch_generator.MLLMBatchGenerator.schedule_removal(uids: List[int]) -> None
```

Thread-safe deferred removal of UIDs from the batch.

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `uids` | `List[int]` | `yes` | `none` | Required positional or keyword input. |

**Returns**

- Type: `None`

**Exceptions and behavior**

Method `MLLMBatchGenerator.schedule_removal` calls `self._pending_removal_uids.update`.
No direct `raise` statement appears in this definition.

[View source #L777-L789](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L777-L789).

</details>

<details class="api-contract" id="contract-vllm_mlx.mllm_batch_generator.MLLMBatchGenerator.process_pending_removals" markdown="1">
<summary><code>vllm_mlx.mllm_batch_generator.MLLMBatchGenerator.process_pending_removals</code> · method</summary>

```python
vllm_mlx.mllm_batch_generator.MLLMBatchGenerator.process_pending_removals() -> None
```

Remove any UIDs enqueued via :meth:`schedule_removal`.

**Parameters**

This callable has no explicit inputs.

**Returns**

- Type: `None`
- Direct return expressions: `None`

**Exceptions and behavior**

Method `MLLMBatchGenerator.process_pending_removals` updates `self._pending_removal_uids`; calls `set`, `list`, `self.remove`; returns `None`.
No direct `raise` statement appears in this definition.

[View source #L791-L808](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L791-L808).

</details>

<details class="api-contract" id="contract-vllm_mlx.mllm_batch_generator.MLLMBatchGenerator.__del__" markdown="1">
<summary><code>vllm_mlx.mllm_batch_generator.MLLMBatchGenerator.__del__</code> · method</summary>

```python
vllm_mlx.mllm_batch_generator.MLLMBatchGenerator.__del__() -> not annotated
```

Method `MLLMBatchGenerator.__del__` calls `self.close`.

**Parameters**

This callable has no explicit inputs.

**Returns**

- Type: `not annotated`

**Exceptions and behavior**

Method `MLLMBatchGenerator.__del__` calls `self.close`.
No direct `raise` statement appears in this definition.

[View source #L810-L814](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L810-L814).

</details>

<details class="api-contract" id="contract-vllm_mlx.mllm_batch_generator.MLLMBatchGenerator.insert" markdown="1">
<summary><code>vllm_mlx.mllm_batch_generator.MLLMBatchGenerator.insert</code> · method</summary>

```python
vllm_mlx.mllm_batch_generator.MLLMBatchGenerator.insert(requests: List[MLLMBatchRequest]) -> List[int]
```

Insert requests for batch processing.

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `requests` | `List[MLLMBatchRequest]` | `yes` | `none` | List of MLLMBatchRequest to process |

**Returns**

- Type: `List[int]`
- Direct return expressions: `uids`

**Exceptions and behavior**

Method `MLLMBatchGenerator.insert` updates `self.uid_counter`, `self.unprocessed_requests`; calls `self.unprocessed_requests.append`, `uids.append`, `sorted`, `logger.debug`; returns `uids`.
No direct `raise` statement appears in this definition.

[View source #L816-L846](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L816-L846).

</details>

<details class="api-contract" id="contract-vllm_mlx.mllm_batch_generator.MLLMBatchGenerator.remove" markdown="1">
<summary><code>vllm_mlx.mllm_batch_generator.MLLMBatchGenerator.remove</code> · method</summary>

```python
vllm_mlx.mllm_batch_generator.MLLMBatchGenerator.remove(uids: List[int]) -> None
```

Remove requests from processing.

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `uids` | `List[int]` | `yes` | `none` | List of UIDs to remove |

**Returns**

- Type: `None`

**Exceptions and behavior**

Method `MLLMBatchGenerator.remove` updates `self.active_batch`, `self.unprocessed_requests`; calls `set`, `enumerate`, `self.active_batch.filter`.
No direct `raise` statement appears in this definition.

[View source #L848-L870](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L848-L870).

</details>

<details class="api-contract" id="contract-vllm_mlx.mllm_batch_generator.MLLMBatchGenerator._preprocess_request" markdown="1">
<summary><code>vllm_mlx.mllm_batch_generator.MLLMBatchGenerator._preprocess_request</code> · method</summary>

```python
vllm_mlx.mllm_batch_generator.MLLMBatchGenerator._preprocess_request(request: MLLMBatchRequest) -> None
```

Preprocess a single MLLM request (vision encoding).

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `request` | `MLLMBatchRequest` | `yes` | `none` | Request to preprocess |

**Returns**

- Type: `None`
- Direct return expressions: `None`

**Exceptions and behavior**

Method `MLLMBatchGenerator._preprocess_request` updates `self._stats.num_images_processed`, `self._stats.vision_encoding_time`; calls `time.perf_counter`, `process_image_input`, `all_images.append`, `logger.warning`; returns `None`.
No direct `raise` statement appears in this definition.

[View source #L872-L1023](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L872-L1023).

</details>

<details class="api-contract" id="contract-vllm_mlx.mllm_batch_generator.MLLMBatchGenerator._copy_prefix_cache" markdown="1">
<summary><code>vllm_mlx.mllm_batch_generator.MLLMBatchGenerator._copy_prefix_cache</code> · method</summary>

```python
vllm_mlx.mllm_batch_generator.MLLMBatchGenerator._copy_prefix_cache(cache_list) -> not annotated
```

Create shallow copies of cache objects to prevent mutation of stored prefix cache.

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `cache_list` | `not annotated` | `yes` | `none` | Required positional or keyword input. |

**Returns**

- Type: `not annotated`
- Direct return expressions: `copies`

**Exceptions and behavior**

Method `MLLMBatchGenerator._copy_prefix_cache` calls `isinstance`, `RotatingKVCache`, `copies.append`, `KVCache`; returns `copies`.
No direct `raise` statement appears in this definition.

[View source #L1026-L1054](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L1026-L1054).

</details>

<details class="api-contract" id="contract-vllm_mlx.mllm_batch_generator.MLLMBatchGenerator._has_empty_rotating_cache" markdown="1">
<summary><code>vllm_mlx.mllm_batch_generator.MLLMBatchGenerator._has_empty_rotating_cache</code> · method</summary>

```python
vllm_mlx.mllm_batch_generator.MLLMBatchGenerator._has_empty_rotating_cache(cache_list) -> not annotated
```

Check if any RotatingKVCache layer has no data (keys=None).

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `cache_list` | `not annotated` | `yes` | `none` | Required positional or keyword input. |

**Returns**

- Type: `not annotated`
- Direct return expressions: `True`; `False`

**Exceptions and behavior**

Method `MLLMBatchGenerator._has_empty_rotating_cache` calls `isinstance`; has 2 explicit return paths.
No direct `raise` statement appears in this definition.

[View source #L1057-L1069](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L1057-L1069).

</details>

<details class="api-contract" id="contract-vllm_mlx.mllm_batch_generator.MLLMBatchGenerator._trim_rotating_caches" markdown="1">
<summary><code>vllm_mlx.mllm_batch_generator.MLLMBatchGenerator._trim_rotating_caches</code> · method</summary>

```python
vllm_mlx.mllm_batch_generator.MLLMBatchGenerator._trim_rotating_caches(cache_list) -> not annotated
```

Trim RotatingKVCache buffers restored from prefix cache.

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `cache_list` | `not annotated` | `yes` | `none` | Required positional or keyword input. |

**Returns**

- Type: `not annotated`

**Exceptions and behavior**

Method `MLLMBatchGenerator._trim_rotating_caches` calls `isinstance`, `layer_cache._trim`, `min`, `logger.warning`.
No direct `raise` statement appears in this definition.

[View source #L1072-L1106](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L1072-L1106).

</details>

<details class="api-contract" id="contract-vllm_mlx.mllm_batch_generator.MLLMBatchGenerator._run_chunked_text_prefill" markdown="1">
<summary><code>vllm_mlx.mllm_batch_generator.MLLMBatchGenerator._run_chunked_text_prefill</code> · method</summary>

```python
vllm_mlx.mllm_batch_generator.MLLMBatchGenerator._run_chunked_text_prefill(request: MLLMBatchRequest, cache: List[Any]) -> mx.array
```

Run prefill in chunks for text-only requests, reporting real progress.

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `request` | `MLLMBatchRequest` | `yes` | `none` | Required positional or keyword input. |
| `cache` | `List[Any]` | `yes` | `none` | Required positional or keyword input. |

**Returns**

- Type: `mx.array`
- Direct return expressions: `output.logits`; `output`

**Exceptions and behavior**

Method `MLLMBatchGenerator._run_chunked_text_prefill` calls `self.language_model`, `request.extra_kwargs.clear`, `hasattr`, `logger.info`; can raise `PrefillAbortedError`; has 2 explicit return paths.
Directly raised exceptions: `PrefillAbortedError`.

[View source #L1108-L1206](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L1108-L1206).

</details>

<details class="api-contract" id="contract-vllm_mlx.mllm_batch_generator.MLLMBatchGenerator._run_vision_encoding" markdown="1">
<summary><code>vllm_mlx.mllm_batch_generator.MLLMBatchGenerator._run_vision_encoding</code> · method</summary>

```python
vllm_mlx.mllm_batch_generator.MLLMBatchGenerator._run_vision_encoding(request: MLLMBatchRequest, cache: Optional[List[Any]] = None) -> mx.array
```

Run the initial VLM forward pass to encode vision and get first logits.

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `request` | `MLLMBatchRequest` | `yes` | `none` | Preprocessed request with input_ids and pixel_values |
| `cache` | `Optional[List[Any]]` | `no` | `None` | KV cache list for the language model. If provided, the language model writes its KV state directly into this cache during the forward pass. |

**Returns**

- Type: `mx.array`
- Direct return expressions: `output.logits`; `output`

**Exceptions and behavior**

Method `MLLMBatchGenerator._run_vision_encoding` calls `dict`, `self.model`, `request.extra_kwargs.clear`, `hasattr`; has 2 explicit return paths.
No direct `raise` statement appears in this definition.

[View source #L1208-L1258](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L1208-L1258).

</details>

<details class="api-contract" id="contract-vllm_mlx.mllm_batch_generator.MLLMBatchGenerator._process_prompts" markdown="1">
<summary><code>vllm_mlx.mllm_batch_generator.MLLMBatchGenerator._process_prompts</code> · method</summary>

```python
vllm_mlx.mllm_batch_generator.MLLMBatchGenerator._process_prompts(requests: List[MLLMBatchRequest]) -> MLLMBatch
```

Process a batch of requests through vision encoding and initial prefill.

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `requests` | `List[MLLMBatchRequest]` | `yes` | `none` | Requests to process |

**Returns**

- Type: `MLLMBatch`
- Direct return expressions: `None`; `MLLMBatch(uids=[req.uid for req in requests], request_ids=[req.request_id for req in requests], y=y, logprobs=all_logpr…`

**Exceptions and behavior**

Method `MLLMBatchGenerator._process_prompts` updates `self._stats.prompt_tokens`, `self._stats.prompt_time`; calls `time.perf_counter`, `self._preprocess_request`, `logger.error`, `type`; can raise `PrefillAbortedError`; has 2 explicit return paths.
Directly raised exceptions: `PrefillAbortedError`.

[View source #L1260-L1682](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L1260-L1682).

</details>

<details class="api-contract" id="contract-vllm_mlx.mllm_batch_generator.MLLMBatchGenerator._process_prompts._sample_first_token" markdown="1">
<summary><code>vllm_mlx.mllm_batch_generator.MLLMBatchGenerator._process_prompts._sample_first_token</code> · nested function</summary>

```python
vllm_mlx.mllm_batch_generator.MLLMBatchGenerator._process_prompts._sample_first_token(req: MLLMBatchRequest, logits: mx.array) -> not annotated
```

Nested Function `MLLMBatchGenerator._process_prompts._sample_first_token` calls `logits_processors_by_request.get`, `mx.array`, `processor`, `mx.logsumexp`; returns `(sampled, logprobs)`.

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `req` | `MLLMBatchRequest` | `yes` | `none` | Required positional or keyword input. |
| `logits` | `mx.array` | `yes` | `none` | Required positional or keyword input. |

**Returns**

- Type: `not annotated`
- Direct return expressions: `(sampled, logprobs)`

**Exceptions and behavior**

Nested Function `MLLMBatchGenerator._process_prompts._sample_first_token` calls `logits_processors_by_request.get`, `mx.array`, `processor`, `mx.logsumexp`; returns `(sampled, logprobs)`.
No direct `raise` statement appears in this definition.

[View source #L1348-L1362](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L1348-L1362).

</details>

<details class="api-contract" id="contract-vllm_mlx.mllm_batch_generator.MLLMBatchGenerator._step" markdown="1">
<summary><code>vllm_mlx.mllm_batch_generator.MLLMBatchGenerator._step</code> · method</summary>

```python
vllm_mlx.mllm_batch_generator.MLLMBatchGenerator._step(input_tokens: mx.array, cache: List[Any], logits_processors: Optional[List[Optional[List[Callable]]]] = None, output_tokens: Optional[List[List[int]]] = None, samplers: Optional[List[Optional[Callable]]] = None) -> Tuple[mx.array, List[mx.array]]
```

Run one generation step through the language model.

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `input_tokens` | `mx.array` | `yes` | `none` | Input tokens [batch_size, 1] or [batch_size] |
| `cache` | `List[Any]` | `yes` | `none` | BatchKVCache for the language model |
| `logits_processors` | `Optional[List[Optional[List[Callable]]]]` | `no` | `None` | Per-request logits processors (e.g. repetition penalty) |
| `output_tokens` | `Optional[List[List[int]]]` | `no` | `None` | Per-request generated tokens so far (needed by processors) |
| `samplers` | `Optional[List[Optional[Callable]]]` | `no` | `None` | Per-request sampler functions (for top_k/min_p) |

**Returns**

- Type: `Tuple[mx.array, List[mx.array]]`
- Direct return expressions: `(sampled, list(logprobs))`

**Exceptions and behavior**

Method `MLLMBatchGenerator._step` calls `self.language_model`, `hasattr`, `any`, `range`; returns `(sampled, list(logprobs))`.
No direct `raise` statement appears in this definition.

[View source #L1684-L1746](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L1684-L1746).

</details>

<details class="api-contract" id="contract-vllm_mlx.mllm_batch_generator.MLLMBatchGenerator._next" markdown="1">
<summary><code>vllm_mlx.mllm_batch_generator.MLLMBatchGenerator._next</code> · method</summary>

```python
vllm_mlx.mllm_batch_generator.MLLMBatchGenerator._next() -> List[MLLMBatchResponse]
```

Internal next() implementation.

**Parameters**

This callable has no explicit inputs.

**Returns**

- Type: `List[MLLMBatchResponse]`
- Direct return expressions: `[]`; `error_responses`; `error_responses + responses`

**Exceptions and behavior**

Method `MLLMBatchGenerator._next` updates `self.active_batch`, `self.unprocessed_requests`, `self._stats.prompt_time`, `self._stats.generation_time`; calls `time.perf_counter`, `len`, `self._process_prompts`, `logger.error`; has 3 explicit return paths.
No direct `raise` statement appears in this definition.

[View source #L1748-L1964](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L1748-L1964).

</details>

<details class="api-contract" id="contract-vllm_mlx.mllm_batch_generator.MLLMBatchGenerator.next" markdown="1">
<summary><code>vllm_mlx.mllm_batch_generator.MLLMBatchGenerator.next</code> · method</summary>

```python
vllm_mlx.mllm_batch_generator.MLLMBatchGenerator.next() -> List[MLLMBatchResponse]
```

Generate next token for all requests in the batch.

**Parameters**

This callable has no explicit inputs.

**Returns**

- Type: `List[MLLMBatchResponse]`
- Direct return expressions: `self._next()`

**Exceptions and behavior**

Method `MLLMBatchGenerator.next` calls `mx.stream`, `self._next`; returns `self._next()`.
No direct `raise` statement appears in this definition.

[View source #L1966-L1974](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L1966-L1974).

</details>

<details class="api-contract" id="contract-vllm_mlx.mllm_batch_generator.MLLMBatchGenerator.stats" markdown="1">
<summary><code>vllm_mlx.mllm_batch_generator.MLLMBatchGenerator.stats</code> · method</summary>

```python
vllm_mlx.mllm_batch_generator.MLLMBatchGenerator.stats() -> MLLMBatchStats
```

Get generation statistics.

**Parameters**

This callable has no explicit inputs.

**Returns**

- Type: `MLLMBatchStats`
- Direct return expressions: `self._stats`

**Exceptions and behavior**

Method `MLLMBatchGenerator.stats` updates `self._stats.peak_memory`; calls `mx.get_peak_memory`; returns `self._stats`.
No direct `raise` statement appears in this definition.

[View source #L1976-L1984](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L1976-L1984).

</details>

<details class="api-contract" id="contract-vllm_mlx.mllm_batch_generator.MLLMBatchGenerator._maybe_store_prefix_cache" markdown="1">
<summary><code>vllm_mlx.mllm_batch_generator.MLLMBatchGenerator._maybe_store_prefix_cache</code> · method</summary>

```python
vllm_mlx.mllm_batch_generator.MLLMBatchGenerator._maybe_store_prefix_cache(batch: MLLMBatch, end_indices: List[int]) -> None
```

Store KV caches for finished text-only requests into prefix cache.

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `batch` | `MLLMBatch` | `yes` | `none` | Required positional or keyword input. |
| `end_indices` | `List[int]` | `yes` | `none` | Required positional or keyword input. |

**Returns**

- Type: `None`
- Direct return expressions: `None`

**Exceptions and behavior**

Method `MLLMBatchGenerator._maybe_store_prefix_cache` calls `batch.extract_cache`, `req.input_ids.reshape(-1).tolist`, `req.input_ids.reshape`, `_trim_cache_offset`; returns `None`.
No direct `raise` statement appears in this definition.

[View source #L1986-L2014](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L1986-L2014).

</details>

<details class="api-contract" id="contract-vllm_mlx.mllm_batch_generator.MLLMBatchGenerator.get_prefill_progress" markdown="1">
<summary><code>vllm_mlx.mllm_batch_generator.MLLMBatchGenerator.get_prefill_progress</code> · method</summary>

```python
vllm_mlx.mllm_batch_generator.MLLMBatchGenerator.get_prefill_progress(request_id: str) -> Optional[Tuple[int, int]]
```

Return (processed_tokens, total_tokens) or None.

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `request_id` | `str` | `yes` | `none` | Required positional or keyword input. |

**Returns**

- Type: `Optional[Tuple[int, int]]`
- Direct return expressions: `self._prefill_progress.get(request_id)`

**Exceptions and behavior**

Method `MLLMBatchGenerator.get_prefill_progress` calls `self._prefill_progress.get`; returns `self._prefill_progress.get(request_id)`.
No direct `raise` statement appears in this definition.

[View source #L2016-L2018](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L2016-L2018).

</details>

<details class="api-contract" id="contract-vllm_mlx.mllm_batch_generator.MLLMBatchGenerator.get_vision_cache_stats" markdown="1">
<summary><code>vllm_mlx.mllm_batch_generator.MLLMBatchGenerator.get_vision_cache_stats</code> · method</summary>

```python
vllm_mlx.mllm_batch_generator.MLLMBatchGenerator.get_vision_cache_stats() -> Dict[str, Any]
```

Get vision cache statistics.

**Parameters**

This callable has no explicit inputs.

**Returns**

- Type: `Dict[str, Any]`
- Direct return expressions: `self.vision_cache.get_stats()`

**Exceptions and behavior**

Method `MLLMBatchGenerator.get_vision_cache_stats` calls `self.vision_cache.get_stats`; returns `self.vision_cache.get_stats()`.
No direct `raise` statement appears in this definition.

[View source #L2020-L2022](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L2020-L2022).

</details>

<details class="api-contract" id="contract-vllm_mlx.mllm_batch_generator.MLLMBatchGenerator.get_prefix_cache_stats" markdown="1">
<summary><code>vllm_mlx.mllm_batch_generator.MLLMBatchGenerator.get_prefix_cache_stats</code> · method</summary>

```python
vllm_mlx.mllm_batch_generator.MLLMBatchGenerator.get_prefix_cache_stats() -> Dict[str, Any]
```

Get KV prefix cache statistics.

**Parameters**

This callable has no explicit inputs.

**Returns**

- Type: `Dict[str, Any]`
- Direct return expressions: `self.prefix_cache.get_stats()`; `{'hits': 0, 'misses': 0, 'hit_rate': 0.0, 'evictions': 0, 'tokens_saved': 0, 'current_memory_mb': 0.0, 'max_memory_mb':…`

**Exceptions and behavior**

Method `MLLMBatchGenerator.get_prefix_cache_stats` calls `self.prefix_cache.get_stats`; has 2 explicit return paths.
No direct `raise` statement appears in this definition.

[View source #L2024-L2038](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L2024-L2038).

</details>

<details class="api-contract" id="contract-vllm_mlx.mllm_batch_generator.MLLMBatchGenerator.has_pending" markdown="1">
<summary><code>vllm_mlx.mllm_batch_generator.MLLMBatchGenerator.has_pending</code> · method</summary>

```python
vllm_mlx.mllm_batch_generator.MLLMBatchGenerator.has_pending() -> bool
```

Check if there are pending or active requests.

**Parameters**

This callable has no explicit inputs.

**Returns**

- Type: `bool`
- Direct return expressions: `bool(self.unprocessed_requests or self.active_batch)`

**Exceptions and behavior**

Method `MLLMBatchGenerator.has_pending` calls `bool`; returns `bool(self.unprocessed_requests or self.active_batch)`.
No direct `raise` statement appears in this definition.

[View source #L2040-L2042](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L2040-L2042).

</details>

<details class="api-contract" id="contract-vllm_mlx.mllm_batch_generator.install_mtp_mllm" markdown="1">
<summary><code>vllm_mlx.mllm_batch_generator.install_mtp_mllm</code> · function</summary>

```python
vllm_mlx.mllm_batch_generator.install_mtp_mllm(batch_gen: 'MLLMBatchGenerator', language_model: Any, num_draft_tokens: int = 1) -> None
```

Install MTP (Multi-Token Prediction) on an MLLMBatchGenerator.

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `batch_gen` | `'MLLMBatchGenerator'` | `yes` | `none` | Required positional or keyword input. |
| `language_model` | `Any` | `yes` | `none` | Required positional or keyword input. |
| `num_draft_tokens` | `int` | `no` | `1` | Optional positional or keyword input; defaults to `1`. |

**Returns**

- Type: `None`

**Exceptions and behavior**

Function `install_mtp_mllm` calls `make_sampler`, `threading.Lock`, `logger.warning`, `logger.info`.
No direct `raise` statement appears in this definition.

[View source #L2045-L2590](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L2045-L2590).

</details>

<details class="api-contract" id="contract-vllm_mlx.mllm_batch_generator.install_mtp_mllm._get_mtp_stats" markdown="1">
<summary><code>vllm_mlx.mllm_batch_generator.install_mtp_mllm._get_mtp_stats</code> · nested function</summary>

```python
vllm_mlx.mllm_batch_generator.install_mtp_mllm._get_mtp_stats() -> Dict[str, Any]
```

Nested Function `install_mtp_mllm._get_mtp_stats` calls `dict`; returns `{'enabled': True, 'requested_draft_tokens': num_draft_tokens, 'effective_draft_tokens': 1, 'mode': 'request_local_sampl…`.

**Parameters**

This callable has no explicit inputs.

**Returns**

- Type: `Dict[str, Any]`
- Direct return expressions: `{'enabled': True, 'requested_draft_tokens': num_draft_tokens, 'effective_draft_tokens': 1, 'mode': 'request_local_sampl…`

**Exceptions and behavior**

Nested Function `install_mtp_mllm._get_mtp_stats` calls `dict`; returns `{'enabled': True, 'requested_draft_tokens': num_draft_tokens, 'effective_draft_tokens': 1, 'mode': 'request_local_sampl…`.
No direct `raise` statement appears in this definition.

[View source #L2089-L2110](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L2089-L2110).

</details>

<details class="api-contract" id="contract-vllm_mlx.mllm_batch_generator.install_mtp_mllm._mtp_step" markdown="1">
<summary><code>vllm_mlx.mllm_batch_generator.install_mtp_mllm._mtp_step</code> · nested function</summary>

```python
vllm_mlx.mllm_batch_generator.install_mtp_mllm._mtp_step(input_tokens: mx.array, cache: List[Any], logits_processors: Optional[List[Optional[List[Callable]]]] = None, output_tokens: Optional[List[List[int]]] = None, samplers: Optional[List[Optional[Callable]]] = None) -> Tuple[mx.array, List[mx.array]]
```

Extended _step with MTP always-advance strategy.

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `input_tokens` | `mx.array` | `yes` | `none` | Required positional or keyword input. |
| `cache` | `List[Any]` | `yes` | `none` | Required positional or keyword input. |
| `logits_processors` | `Optional[List[Optional[List[Callable]]]]` | `no` | `None` | Optional positional or keyword input; defaults to `None`. |
| `output_tokens` | `Optional[List[List[int]]]` | `no` | `None` | Optional positional or keyword input; defaults to `None`. |
| `samplers` | `Optional[List[Optional[Callable]]]` | `no` | `None` | Optional positional or keyword input; defaults to `None`. |

**Returns**

- Type: `Tuple[mx.array, List[mx.array]]`
- Direct return expressions: `_orig_step(input_tokens, cache, logits_processors, output_tokens, samplers)`; `(primary_tokens, list(logprobs))`

**Exceptions and behavior**

Nested Function `install_mtp_mllm._mtp_step` calls `list`, `any`, `_skip_state_by_uid.clear`, `_orig_step`; has 2 explicit return paths.
No direct `raise` statement appears in this definition.

[View source #L2114-L2455](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L2114-L2455).

</details>

<details class="api-contract" id="contract-vllm_mlx.mllm_batch_generator.install_mtp_mllm._mtp_next" markdown="1">
<summary><code>vllm_mlx.mllm_batch_generator.install_mtp_mllm._mtp_next</code> · nested function</summary>

```python
vllm_mlx.mllm_batch_generator.install_mtp_mllm._mtp_next() -> List[MLLMBatchResponse]
```

Wrapper around _next that emits deferred MTP draft tokens.

**Parameters**

This callable has no explicit inputs.

**Returns**

- Type: `List[MLLMBatchResponse]`
- Direct return expressions: `augmented`

**Exceptions and behavior**

Nested Function `install_mtp_mllm._mtp_next` calls `_skip_state_by_uid.clear`, `_deferred_drafts.clear`, `_attempted_drafts_by_uid.clear`, `_deferred_drafts.pop`; returns `augmented`.
No direct `raise` statement appears in this definition.

[View source #L2460-L2576](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L2460-L2576).

</details>

<details class="api-contract" id="contract-vllm_mlx.mllm_batch_generator.install_chunked_prefill_mllm" markdown="1">
<summary><code>vllm_mlx.mllm_batch_generator.install_chunked_prefill_mllm</code> · function</summary>

```python
vllm_mlx.mllm_batch_generator.install_chunked_prefill_mllm(batch_gen: 'MLLMBatchGenerator', budget: int = 1024) -> None
```

Install interleaved prefill/decode on an MLLMBatchGenerator.

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `batch_gen` | `'MLLMBatchGenerator'` | `yes` | `none` | The MLLMBatchGenerator to patch. |
| `budget` | `int` | `no` | `1024` | Max tokens to prefill per step (chunk size). |

**Returns**

- Type: `None`

**Exceptions and behavior**

Function `install_chunked_prefill_mllm` calls `logger.info`.
No direct `raise` statement appears in this definition.

[View source #L2593-L3073](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L2593-L3073).

</details>

<details class="api-contract" id="contract-vllm_mlx.mllm_batch_generator.install_chunked_prefill_mllm._generation_step" markdown="1">
<summary><code>vllm_mlx.mllm_batch_generator.install_chunked_prefill_mllm._generation_step</code> · nested function</summary>

```python
vllm_mlx.mllm_batch_generator.install_chunked_prefill_mllm._generation_step() -> List[MLLMBatchResponse]
```

Run one generation step for the active batch.

**Parameters**

This callable has no explicit inputs.

**Returns**

- Type: `List[MLLMBatchResponse]`
- Direct return expressions: `error_responses`; `error_responses + responses`

**Exceptions and behavior**

Nested Function `install_chunked_prefill_mllm._generation_step` calls `list`, `batch_gen._pending_error_responses.clear`, `time.perf_counter`, `batch_gen._step`; has 2 explicit return paths.
No direct `raise` statement appears in this definition.

[View source #L2623-L2713](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L2623-L2713).

</details>

<details class="api-contract" id="contract-vllm_mlx.mllm_batch_generator.install_chunked_prefill_mllm._chunked_next" markdown="1">
<summary><code>vllm_mlx.mllm_batch_generator.install_chunked_prefill_mllm._chunked_next</code> · nested function</summary>

```python
vllm_mlx.mllm_batch_generator.install_chunked_prefill_mllm._chunked_next() -> List[MLLMBatchResponse]
```

Interleaved prefill/decode: one prefill chunk + one gen step.

**Parameters**

This callable has no explicit inputs.

**Returns**

- Type: `List[MLLMBatchResponse]`
- Direct return expressions: `_generation_step()`; `[]`; `_orig_next()`

**Exceptions and behavior**

Nested Function `install_chunked_prefill_mllm._chunked_next` calls `batch_gen._aborted_request_ids.discard`, `mx.clear_cache`, `batch_gen._prefill_progress.pop`, `batch_gen._pending_error_responses.append`; has 3 explicit return paths.
No direct `raise` statement appears in this definition.

[View source #L2715-L3058](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L2715-L3058).

</details>

<details class="api-contract" id="contract-vllm_mlx.mllm_batch_generator.install_chunked_prefill_mllm._patched_remove" markdown="1">
<summary><code>vllm_mlx.mllm_batch_generator.install_chunked_prefill_mllm._patched_remove</code> · nested function</summary>

```python
vllm_mlx.mllm_batch_generator.install_chunked_prefill_mllm._patched_remove(uids: List[int]) -> None
```

Nested Function `install_chunked_prefill_mllm._patched_remove` calls `set`, `mx.clear_cache`, `_orig_remove`.

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `uids` | `List[int]` | `yes` | `none` | Required positional or keyword input. |

**Returns**

- Type: `None`

**Exceptions and behavior**

Nested Function `install_chunked_prefill_mllm._patched_remove` calls `set`, `mx.clear_cache`, `_orig_remove`.
No direct `raise` statement appears in this definition.

[View source #L3063-L3068](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L3063-L3068).

</details>

## Complete symbol map

This map also includes private definitions and nested helpers. The signature column exposes every explicit input even when an internal helper has no dedicated parameter prose.

| Symbol | Kind | Signature and inputs | What it does | Source |
| --- | --- | --- | --- | --- |
| [`_processors_can_retire`](#contract-vllm_mlx.mllm_batch_generator._processors_can_retire) | function | `_processors_can_retire(processors: Optional[List[Callable]]) -> bool` | True when any processor advertises a retire-to-content transition. | [#L37-L43](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L37-L43) |
| [`_mark_mtp_attempts_on_primary_responses`](#contract-vllm_mlx.mllm_batch_generator._mark_mtp_attempts_on_primary_responses) | function | `_mark_mtp_attempts_on_primary_responses(responses: List['MLLMBatchResponse'], attempted_drafts_by_uid: Dict[int, int]) -> None` | Mark only responses from steps that actually attempted MTP drafts. | [#L46-L57](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L46-L57) |
| [`_drop_retired_processors`](#contract-vllm_mlx.mllm_batch_generator._drop_retired_processors) | function | `_drop_retired_processors(processors: Optional[List[Callable]]) -> tuple[Optional[List[Callable]], int]` | Drop retire-capable processors that have completed their work. | [#L60-L74](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L60-L74) |
| [`_request_uses_stochastic_sampling`](#contract-vllm_mlx.mllm_batch_generator._request_uses_stochastic_sampling) | function | `_request_uses_stochastic_sampling(request: Any) -> bool` | Return whether a request needs sampler-aware speculative verification. | [#L77-L92](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L77-L92) |
| [`_sampling_logprobs`](#contract-vllm_mlx.mllm_batch_generator._sampling_logprobs) | function | `_sampling_logprobs(logits: mx.array, request: Any) -> mx.array` | Match mlx-lm's request sampler in log-probability space. | [#L95-L123](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L95-L123) |
| [`_residual_logprobs`](#contract-vllm_mlx.mllm_batch_generator._residual_logprobs) | function | `_residual_logprobs(target_logprobs: mx.array, draft_logprobs: mx.array) -> mx.array` | Return the normalized residual max(target - draft, 0) distribution. | [#L126-L139](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L126-L139) |
| [`_accept_sampled_draft`](#contract-vllm_mlx.mllm_batch_generator._accept_sampled_draft) | function | `_accept_sampled_draft(target_logprob: float, draft_logprob: float, uniform_draw: float) -> bool` | Apply the exact min(1, p/q) stochastic speculative acceptance rule. | [#L142-L149](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L142-L149) |
| [`PrefillAbortedError`](#contract-vllm_mlx.mllm_batch_generator.PrefillAbortedError) | class | `PrefillAbortedError(request_id: str)` | Raised when a prefill is aborted due to client disconnect. | [#L152-L157](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L152-L157) |
| [`PrefillAbortedError.__init__`](#contract-vllm_mlx.mllm_batch_generator.PrefillAbortedError.__init__) | method | `PrefillAbortedError.__init__(request_id: str) -> not annotated` | Method `PrefillAbortedError.__init__` updates `self.request_id`; calls `super().__init__`, `super`. | [#L155-L157](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L155-L157) |
| [`_cache_eval_tensors`](#contract-vllm_mlx.mllm_batch_generator._cache_eval_tensors) | function | `_cache_eval_tensors(cache: List[Any]) -> List[Any]` | Return realized tensors that break lazy cache graphs between chunks. | [#L160-L183](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L160-L183) |
| [`_eval_prompt_cache`](#contract-vllm_mlx.mllm_batch_generator._eval_prompt_cache) | function | `_eval_prompt_cache(cache: List[Any]) -> None` | Evaluate all cache tensors used by hybrid chunked prefill. | [#L186-L190](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L186-L190) |
| [`MLLMBatchRequest`](#contract-vllm_mlx.mllm_batch_generator.MLLMBatchRequest) | class | `MLLMBatchRequest(uid: int, request_id: str, prompt: str, images: Optional[List[str]] = None, videos: Optional[List[str]] = None, audio: Optional[List[str]] = None, max_tokens: int = 256, temperature: float = 0.7, top_p: float = 0.9, top_k: int = 0, min_p: float = 0.0, presence_penalty: float = 0.0, repetition_penalty: float = 1.0, logits_processors: Optional[List[Callable]] = None, input_ids: Optional[mx.array] = None, pixel_values: Optional[mx.array] = None, attention_mask: Optional[mx.array] = None, image_grid_thw: Optional[mx.array] = None, extra_kwargs: Dict[str, Any] = field(default_factory=dict), is_text_only: bool = False, num_tokens: int = 0, output_tokens: List[int] = field(default_factory=list), vision_encoded: bool = False, cross_attention_states: Optional[Any] = None, encoder_outputs: Optional[Any] = None)` | Request data for MLLM batch processing. | [#L194-L237](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L194-L237) |
| [`MLLMBatchResponse`](#contract-vllm_mlx.mllm_batch_generator.MLLMBatchResponse) | class | `MLLMBatchResponse(uid: int, request_id: str, token: int, logprobs: mx.array, finish_reason: Optional[str] = None, prompt_cache: Optional[Callable[[], List[Any]]] = None, from_draft: bool = False, mtp_attempted: bool = False, mtp_attempted_count: int = 0)` | Response from a batch generation step. | [#L241-L256](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L241-L256) |
| [`MLLMBatch`](#contract-vllm_mlx.mllm_batch_generator.MLLMBatch) | class | `MLLMBatch(uids: List[int], request_ids: List[str], y: mx.array, logprobs: List[mx.array], max_tokens: List[int], num_tokens: List[int], cache: List[Any], requests: List[MLLMBatchRequest], logits_processors: Optional[List[Optional[List[Callable]]]] = None, samplers: Optional[List[Optional[Callable]]] = None)` | Represents an active batch of MLLM requests. | [#L260-L392](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L260-L392) |
| [`MLLMBatch.__len__`](#contract-vllm_mlx.mllm_batch_generator.MLLMBatch.__len__) | method | `MLLMBatch.__len__() -> int` | Method `MLLMBatch.__len__` calls `len`; returns `len(self.uids)`. | [#L279-L280](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L279-L280) |
| [`MLLMBatch.filter`](#contract-vllm_mlx.mllm_batch_generator.MLLMBatch.filter) | method | `MLLMBatch.filter(keep_idx: List[int]) -> None` | Filter batch to keep only requests at specified indices. | [#L282-L306](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L282-L306) |
| [`MLLMBatch.extend`](#contract-vllm_mlx.mllm_batch_generator.MLLMBatch.extend) | method | `MLLMBatch.extend(other: 'MLLMBatch') -> None` | Extend this batch with another batch. | [#L308-L351](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L308-L351) |
| [`MLLMBatch.extract_cache`](#contract-vllm_mlx.mllm_batch_generator.MLLMBatch.extract_cache) | method | `MLLMBatch.extract_cache(idx: int) -> List[Any]` | Extract cache for a single request (for prefix caching). | [#L353-L392](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L353-L392) |
| [`MLLMBatchStats`](#contract-vllm_mlx.mllm_batch_generator.MLLMBatchStats) | class | `MLLMBatchStats()` | Statistics for MLLM batch generation. | [#L395-L436](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L395-L436) |
| [`MLLMBatchStats.__init__`](#contract-vllm_mlx.mllm_batch_generator.MLLMBatchStats.__init__) | method | `MLLMBatchStats.__init__() -> not annotated` | Method `MLLMBatchStats.__init__` updates `self.prompt_tokens`, `self.prompt_time`, `self.generation_tokens`, `self.generation_time`. | [#L398-L405](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L398-L405) |
| [`MLLMBatchStats.prompt_tps`](#contract-vllm_mlx.mllm_batch_generator.MLLMBatchStats.prompt_tps) | method | `MLLMBatchStats.prompt_tps() -> float` | Return measured multimodal prompt throughput in tokens per second. | [#L408-L413](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L408-L413) |
| [`MLLMBatchStats.generation_tps`](#contract-vllm_mlx.mllm_batch_generator.MLLMBatchStats.generation_tps) | method | `MLLMBatchStats.generation_tps() -> float` | Return measured decode throughput in tokens per second. | [#L416-L421](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L416-L421) |
| [`MLLMBatchStats.to_dict`](#contract-vllm_mlx.mllm_batch_generator.MLLMBatchStats.to_dict) | method | `MLLMBatchStats.to_dict() -> Dict[str, Any]` | Return token, timing, vision, and peak-memory statistics. | [#L423-L436](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L423-L436) |
| [`_left_pad_prompts`](#contract-vllm_mlx.mllm_batch_generator._left_pad_prompts) | function | `_left_pad_prompts(prompts: List[List[int]], max_length: Optional[int] = None) -> mx.array` | Left-pad prompts to uniform length. | [#L439-L454](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L439-L454) |
| [`MLLMBatchGenerator`](#contract-vllm_mlx.mllm_batch_generator.MLLMBatchGenerator) | class | `MLLMBatchGenerator(model: nn.Module, processor: Any, mm_processor: Optional[MultimodalProcessor] = None, max_tokens: int = 256, stop_tokens: Optional[set] = None, sampler: Optional[Callable[[mx.array], mx.array]] = None, prefill_batch_size: int = 4, completion_batch_size: int = 16, prefill_step_size: int = 1024, enable_vision_cache: bool = True, vision_cache_size: int = 100, prefix_cache_config: Optional[MemoryCacheConfig] = None, max_kv_size: int = 0)` | Batch generator for Vision Language Models. | [#L457-L2042](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L457-L2042) |
| [`MLLMBatchGenerator.__init__`](#contract-vllm_mlx.mllm_batch_generator.MLLMBatchGenerator.__init__) | method | `MLLMBatchGenerator.__init__(model: nn.Module, processor: Any, mm_processor: Optional[MultimodalProcessor] = None, max_tokens: int = 256, stop_tokens: Optional[set] = None, sampler: Optional[Callable[[mx.array], mx.array]] = None, prefill_batch_size: int = 4, completion_batch_size: int = 16, prefill_step_size: int = 1024, enable_vision_cache: bool = True, vision_cache_size: int = 100, prefix_cache_config: Optional[MemoryCacheConfig] = None, max_kv_size: int = 0) -> not annotated` | Initialize MLLM batch generator. | [#L484-L632](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L484-L632) |
| [`MLLMBatchGenerator._normalize_chat_template_for_prefix_cache`](#contract-vllm_mlx.mllm_batch_generator.MLLMBatchGenerator._normalize_chat_template_for_prefix_cache) | method | `MLLMBatchGenerator._normalize_chat_template_for_prefix_cache() -> None` | Patch chat template so historical assistant turns are prefix-stable. | [#L634-L697](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L634-L697) |
| [`MLLMBatchGenerator._compute_think_suffix_len`](#contract-vllm_mlx.mllm_batch_generator.MLLMBatchGenerator._compute_think_suffix_len) | method | `MLLMBatchGenerator._compute_think_suffix_len() -> int` | Compute how many extra tokens enable_thinking=True adds at the END. | [#L699-L758](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L699-L758) |
| [`MLLMBatchGenerator.close`](#contract-vllm_mlx.mllm_batch_generator.MLLMBatchGenerator.close) | method | `MLLMBatchGenerator.close() -> None` | Release resources and reset wired limit. | [#L760-L765](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L760-L765) |
| [`MLLMBatchGenerator.abort_prefill`](#contract-vllm_mlx.mllm_batch_generator.MLLMBatchGenerator.abort_prefill) | method | `MLLMBatchGenerator.abort_prefill(request_id: str) -> None` | Signal that a request's prefill should be aborted. | [#L767-L775](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L767-L775) |
| [`MLLMBatchGenerator.schedule_removal`](#contract-vllm_mlx.mllm_batch_generator.MLLMBatchGenerator.schedule_removal) | method | `MLLMBatchGenerator.schedule_removal(uids: List[int]) -> None` | Thread-safe deferred removal of UIDs from the batch. | [#L777-L789](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L777-L789) |
| [`MLLMBatchGenerator.process_pending_removals`](#contract-vllm_mlx.mllm_batch_generator.MLLMBatchGenerator.process_pending_removals) | method | `MLLMBatchGenerator.process_pending_removals() -> None` | Remove any UIDs enqueued via :meth:`schedule_removal`. | [#L791-L808](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L791-L808) |
| [`MLLMBatchGenerator.__del__`](#contract-vllm_mlx.mllm_batch_generator.MLLMBatchGenerator.__del__) | method | `MLLMBatchGenerator.__del__() -> not annotated` | Method `MLLMBatchGenerator.__del__` calls `self.close`. | [#L810-L814](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L810-L814) |
| [`MLLMBatchGenerator.insert`](#contract-vllm_mlx.mllm_batch_generator.MLLMBatchGenerator.insert) | method | `MLLMBatchGenerator.insert(requests: List[MLLMBatchRequest]) -> List[int]` | Insert requests for batch processing. | [#L816-L846](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L816-L846) |
| [`MLLMBatchGenerator.remove`](#contract-vllm_mlx.mllm_batch_generator.MLLMBatchGenerator.remove) | method | `MLLMBatchGenerator.remove(uids: List[int]) -> None` | Remove requests from processing. | [#L848-L870](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L848-L870) |
| [`MLLMBatchGenerator._preprocess_request`](#contract-vllm_mlx.mllm_batch_generator.MLLMBatchGenerator._preprocess_request) | method | `MLLMBatchGenerator._preprocess_request(request: MLLMBatchRequest) -> None` | Preprocess a single MLLM request (vision encoding). | [#L872-L1023](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L872-L1023) |
| [`MLLMBatchGenerator._copy_prefix_cache`](#contract-vllm_mlx.mllm_batch_generator.MLLMBatchGenerator._copy_prefix_cache) | method | `MLLMBatchGenerator._copy_prefix_cache(cache_list) -> not annotated` | Create shallow copies of cache objects to prevent mutation of stored prefix cache. | [#L1026-L1054](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L1026-L1054) |
| [`MLLMBatchGenerator._has_empty_rotating_cache`](#contract-vllm_mlx.mllm_batch_generator.MLLMBatchGenerator._has_empty_rotating_cache) | method | `MLLMBatchGenerator._has_empty_rotating_cache(cache_list) -> not annotated` | Check if any RotatingKVCache layer has no data (keys=None). | [#L1057-L1069](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L1057-L1069) |
| [`MLLMBatchGenerator._trim_rotating_caches`](#contract-vllm_mlx.mllm_batch_generator.MLLMBatchGenerator._trim_rotating_caches) | method | `MLLMBatchGenerator._trim_rotating_caches(cache_list) -> not annotated` | Trim RotatingKVCache buffers restored from prefix cache. | [#L1072-L1106](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L1072-L1106) |
| [`MLLMBatchGenerator._run_chunked_text_prefill`](#contract-vllm_mlx.mllm_batch_generator.MLLMBatchGenerator._run_chunked_text_prefill) | method | `MLLMBatchGenerator._run_chunked_text_prefill(request: MLLMBatchRequest, cache: List[Any]) -> mx.array` | Run prefill in chunks for text-only requests, reporting real progress. | [#L1108-L1206](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L1108-L1206) |
| [`MLLMBatchGenerator._run_vision_encoding`](#contract-vllm_mlx.mllm_batch_generator.MLLMBatchGenerator._run_vision_encoding) | method | `MLLMBatchGenerator._run_vision_encoding(request: MLLMBatchRequest, cache: Optional[List[Any]] = None) -> mx.array` | Run the initial VLM forward pass to encode vision and get first logits. | [#L1208-L1258](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L1208-L1258) |
| [`MLLMBatchGenerator._process_prompts`](#contract-vllm_mlx.mllm_batch_generator.MLLMBatchGenerator._process_prompts) | method | `MLLMBatchGenerator._process_prompts(requests: List[MLLMBatchRequest]) -> MLLMBatch` | Process a batch of requests through vision encoding and initial prefill. | [#L1260-L1682](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L1260-L1682) |
| [`MLLMBatchGenerator._process_prompts._sample_first_token`](#contract-vllm_mlx.mllm_batch_generator.MLLMBatchGenerator._process_prompts._sample_first_token) | nested function | `MLLMBatchGenerator._process_prompts._sample_first_token(req: MLLMBatchRequest, logits: mx.array) -> not annotated` | Nested Function `MLLMBatchGenerator._process_prompts._sample_first_token` calls `logits_processors_by_request.get`, `mx.array`, `processor`, `mx.logsumexp`; returns `(sampled, logprobs)`. | [#L1348-L1362](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L1348-L1362) |
| [`MLLMBatchGenerator._step`](#contract-vllm_mlx.mllm_batch_generator.MLLMBatchGenerator._step) | method | `MLLMBatchGenerator._step(input_tokens: mx.array, cache: List[Any], logits_processors: Optional[List[Optional[List[Callable]]]] = None, output_tokens: Optional[List[List[int]]] = None, samplers: Optional[List[Optional[Callable]]] = None) -> Tuple[mx.array, List[mx.array]]` | Run one generation step through the language model. | [#L1684-L1746](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L1684-L1746) |
| [`MLLMBatchGenerator._next`](#contract-vllm_mlx.mllm_batch_generator.MLLMBatchGenerator._next) | method | `MLLMBatchGenerator._next() -> List[MLLMBatchResponse]` | Internal next() implementation. | [#L1748-L1964](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L1748-L1964) |
| [`MLLMBatchGenerator.next`](#contract-vllm_mlx.mllm_batch_generator.MLLMBatchGenerator.next) | method | `MLLMBatchGenerator.next() -> List[MLLMBatchResponse]` | Generate next token for all requests in the batch. | [#L1966-L1974](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L1966-L1974) |
| [`MLLMBatchGenerator.stats`](#contract-vllm_mlx.mllm_batch_generator.MLLMBatchGenerator.stats) | method | `MLLMBatchGenerator.stats() -> MLLMBatchStats` | Get generation statistics. | [#L1976-L1984](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L1976-L1984) |
| [`MLLMBatchGenerator._maybe_store_prefix_cache`](#contract-vllm_mlx.mllm_batch_generator.MLLMBatchGenerator._maybe_store_prefix_cache) | method | `MLLMBatchGenerator._maybe_store_prefix_cache(batch: MLLMBatch, end_indices: List[int]) -> None` | Store KV caches for finished text-only requests into prefix cache. | [#L1986-L2014](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L1986-L2014) |
| [`MLLMBatchGenerator.get_prefill_progress`](#contract-vllm_mlx.mllm_batch_generator.MLLMBatchGenerator.get_prefill_progress) | method | `MLLMBatchGenerator.get_prefill_progress(request_id: str) -> Optional[Tuple[int, int]]` | Return (processed_tokens, total_tokens) or None. | [#L2016-L2018](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L2016-L2018) |
| [`MLLMBatchGenerator.get_vision_cache_stats`](#contract-vllm_mlx.mllm_batch_generator.MLLMBatchGenerator.get_vision_cache_stats) | method | `MLLMBatchGenerator.get_vision_cache_stats() -> Dict[str, Any]` | Get vision cache statistics. | [#L2020-L2022](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L2020-L2022) |
| [`MLLMBatchGenerator.get_prefix_cache_stats`](#contract-vllm_mlx.mllm_batch_generator.MLLMBatchGenerator.get_prefix_cache_stats) | method | `MLLMBatchGenerator.get_prefix_cache_stats() -> Dict[str, Any]` | Get KV prefix cache statistics. | [#L2024-L2038](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L2024-L2038) |
| [`MLLMBatchGenerator.has_pending`](#contract-vllm_mlx.mllm_batch_generator.MLLMBatchGenerator.has_pending) | method | `MLLMBatchGenerator.has_pending() -> bool` | Check if there are pending or active requests. | [#L2040-L2042](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L2040-L2042) |
| [`install_mtp_mllm`](#contract-vllm_mlx.mllm_batch_generator.install_mtp_mllm) | function | `install_mtp_mllm(batch_gen: 'MLLMBatchGenerator', language_model: Any, num_draft_tokens: int = 1) -> None` | Install MTP (Multi-Token Prediction) on an MLLMBatchGenerator. | [#L2045-L2590](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L2045-L2590) |
| [`install_mtp_mllm._get_mtp_stats`](#contract-vllm_mlx.mllm_batch_generator.install_mtp_mllm._get_mtp_stats) | nested function | `install_mtp_mllm._get_mtp_stats() -> Dict[str, Any]` | Nested Function `install_mtp_mllm._get_mtp_stats` calls `dict`; returns `{'enabled': True, 'requested_draft_tokens': num_draft_tokens, 'effective_draft_tokens': 1, 'mode': 'request_local_sampl…`. | [#L2089-L2110](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L2089-L2110) |
| [`install_mtp_mllm._mtp_step`](#contract-vllm_mlx.mllm_batch_generator.install_mtp_mllm._mtp_step) | nested function | `install_mtp_mllm._mtp_step(input_tokens: mx.array, cache: List[Any], logits_processors: Optional[List[Optional[List[Callable]]]] = None, output_tokens: Optional[List[List[int]]] = None, samplers: Optional[List[Optional[Callable]]] = None) -> Tuple[mx.array, List[mx.array]]` | Extended _step with MTP always-advance strategy. | [#L2114-L2455](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L2114-L2455) |
| [`install_mtp_mllm._mtp_next`](#contract-vllm_mlx.mllm_batch_generator.install_mtp_mllm._mtp_next) | nested function | `install_mtp_mllm._mtp_next() -> List[MLLMBatchResponse]` | Wrapper around _next that emits deferred MTP draft tokens. | [#L2460-L2576](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L2460-L2576) |
| [`install_chunked_prefill_mllm`](#contract-vllm_mlx.mllm_batch_generator.install_chunked_prefill_mllm) | function | `install_chunked_prefill_mllm(batch_gen: 'MLLMBatchGenerator', budget: int = 1024) -> None` | Install interleaved prefill/decode on an MLLMBatchGenerator. | [#L2593-L3073](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L2593-L3073) |
| [`install_chunked_prefill_mllm._generation_step`](#contract-vllm_mlx.mllm_batch_generator.install_chunked_prefill_mllm._generation_step) | nested function | `install_chunked_prefill_mllm._generation_step() -> List[MLLMBatchResponse]` | Run one generation step for the active batch. | [#L2623-L2713](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L2623-L2713) |
| [`install_chunked_prefill_mllm._chunked_next`](#contract-vllm_mlx.mllm_batch_generator.install_chunked_prefill_mllm._chunked_next) | nested function | `install_chunked_prefill_mllm._chunked_next() -> List[MLLMBatchResponse]` | Interleaved prefill/decode: one prefill chunk + one gen step. | [#L2715-L3058](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L2715-L3058) |
| [`install_chunked_prefill_mllm._patched_remove`](#contract-vllm_mlx.mllm_batch_generator.install_chunked_prefill_mllm._patched_remove) | nested function | `install_chunked_prefill_mllm._patched_remove(uids: List[int]) -> None` | Nested Function `install_chunked_prefill_mllm._patched_remove` calls `set`, `mx.clear_cache`, `_orig_remove`. | [#L3063-L3068](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/mllm_batch_generator.py#L3063-L3068) |