# `vllm_mlx.utils.tokenizer`

Tokenizer utilities with fallback support for non-standard tokenizers.

[View the complete module source at #L1-L280](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/utils/tokenizer.py#L1-L280).

## API details

Each callable below includes its exact signature, type annotations, inputs, defaults, return contract, documented exceptions, implementation source, and parsed docstring sections when the source provides them.

::: vllm_mlx.utils.tokenizer
    options:
      members:
        - logger
        - FALLBACK_MODELS
        - _needs_tokenizer_fallback
        - _needs_strict_false
        - load_model_with_fallback
        - _load_strict_false
        - _try_inject_mtp
        - _try_inject_mtp_post_load
        - _load_with_tokenizer_fallback
      filters: []
      show_if_no_docstring: true

## Complete contract reference

Expand any definition for its exact inputs, annotations, defaults, return contract, directly raised exceptions, source-grounded behavior, and immutable line link. This section includes private and nested definitions that ordinary API generators omit.

<details class="api-contract" id="contract-vllm_mlx.utils.tokenizer._needs_tokenizer_fallback" markdown="1">
<summary><code>vllm_mlx.utils.tokenizer._needs_tokenizer_fallback</code> · function</summary>

```python
vllm_mlx.utils.tokenizer._needs_tokenizer_fallback(model_name: str) -> bool
```

Check if model needs tokenizer fallback.

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `model_name` | `str` | `yes` | `none` | Required positional or keyword input. |

**Returns**

- Type: `bool`
- Direct return expressions: `any((pattern.lower() in model_lower for pattern in FALLBACK_MODELS))`

**Exceptions and behavior**

Function `_needs_tokenizer_fallback` calls `model_name.lower`, `any`, `pattern.lower`; returns `any((pattern.lower() in model_lower for pattern in FALLBACK_MODELS))`.
No direct `raise` statement appears in this definition.

[View source #L25-L28](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/utils/tokenizer.py#L25-L28).

</details>

<details class="api-contract" id="contract-vllm_mlx.utils.tokenizer._needs_strict_false" markdown="1">
<summary><code>vllm_mlx.utils.tokenizer._needs_strict_false</code> · function</summary>

```python
vllm_mlx.utils.tokenizer._needs_strict_false(model_name: str) -> bool
```

Check if model needs strict=False loading (VLM models with extra weights).

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `model_name` | `str` | `yes` | `none` | Required positional or keyword input. |

**Returns**

- Type: `bool`
- Direct return expressions: `False`; `True`

**Exceptions and behavior**

Function `_needs_strict_false` calls `_download`, `load_config`; has 2 explicit return paths.
No direct `raise` statement appears in this definition.

[View source #L31-L49](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/utils/tokenizer.py#L31-L49).

</details>

<details class="api-contract" id="contract-vllm_mlx.utils.tokenizer.load_model_with_fallback" markdown="1">
<summary><code>vllm_mlx.utils.tokenizer.load_model_with_fallback</code> · function</summary>

```python
vllm_mlx.utils.tokenizer.load_model_with_fallback(model_name: str, tokenizer_config: dict = None) -> not annotated
```

Load model and tokenizer with fallback for non-standard tokenizers.

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `model_name` | `str` | `yes` | `none` | HuggingFace model name or local path |
| `tokenizer_config` | `dict` | `no` | `None` | Optional tokenizer configuration |

**Returns**

- Type: `not annotated`
- Direct return expressions: `_load_with_tokenizer_fallback(model_name)`; `_load_strict_false(model_name, tokenizer_config)`; `(model, tokenizer)`

**Exceptions and behavior**

Function `load_model_with_fallback` calls `_needs_tokenizer_fallback`, `logger.info`, `_load_with_tokenizer_fallback`, `_needs_strict_false`; has 3 explicit return paths.
No direct `raise` statement appears in this definition.

[View source #L52-L111](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/utils/tokenizer.py#L52-L111).

</details>

<details class="api-contract" id="contract-vllm_mlx.utils.tokenizer._load_strict_false" markdown="1">
<summary><code>vllm_mlx.utils.tokenizer._load_strict_false</code> · function</summary>

```python
vllm_mlx.utils.tokenizer._load_strict_false(model_name: str, tokenizer_config: dict = None) -> not annotated
```

Load model with strict=False to discard extra weights.

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `model_name` | `str` | `yes` | `none` | Required positional or keyword input. |
| `tokenizer_config` | `dict` | `no` | `None` | Optional positional or keyword input; defaults to `None`. |

**Returns**

- Type: `not annotated`
- Direct return expressions: `(model, tokenizer)`

**Exceptions and behavior**

Function `_load_strict_false` calls `_download`, `load_model`, `tree_flatten`, `model.parameters`; returns `(model, tokenizer)`.
No direct `raise` statement appears in this definition.

[View source #L114-L153](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/utils/tokenizer.py#L114-L153).

</details>

<details class="api-contract" id="contract-vllm_mlx.utils.tokenizer._try_inject_mtp" markdown="1">
<summary><code>vllm_mlx.utils.tokenizer._try_inject_mtp</code> · function</summary>

```python
vllm_mlx.utils.tokenizer._try_inject_mtp(model, model_path, config) -> not annotated
```

Inject MTP support if model has MTP config + weights.

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `model` | `not annotated` | `yes` | `none` | Required positional or keyword input. |
| `model_path` | `not annotated` | `yes` | `none` | Required positional or keyword input. |
| `config` | `not annotated` | `yes` | `none` | Required positional or keyword input. |

**Returns**

- Type: `not annotated`
- Direct return expressions: `None`

**Exceptions and behavior**

Function `_try_inject_mtp` calls `config.get`, `text_config.get`, `inject_mtp_support`; returns `None`.
No direct `raise` statement appears in this definition.

[View source #L156-L176](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/utils/tokenizer.py#L156-L176).

</details>

<details class="api-contract" id="contract-vllm_mlx.utils.tokenizer._try_inject_mtp_post_load" markdown="1">
<summary><code>vllm_mlx.utils.tokenizer._try_inject_mtp_post_load</code> · function</summary>

```python
vllm_mlx.utils.tokenizer._try_inject_mtp_post_load(model, model_name) -> not annotated
```

Check if MTP weights exist but were stripped by sanitize(), and inject.

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `model` | `not annotated` | `yes` | `none` | Required positional or keyword input. |
| `model_name` | `not annotated` | `yes` | `none` | Required positional or keyword input. |

**Returns**

- Type: `not annotated`
- Direct return expressions: `None`

**Exceptions and behavior**

Function `_try_inject_mtp_post_load` calls `_download`, `Path`, `config_path.exists`, `open`; returns `None`.
No direct `raise` statement appears in this definition.

[View source #L179-L215](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/utils/tokenizer.py#L179-L215).

</details>

<details class="api-contract" id="contract-vllm_mlx.utils.tokenizer._load_with_tokenizer_fallback" markdown="1">
<summary><code>vllm_mlx.utils.tokenizer._load_with_tokenizer_fallback</code> · function</summary>

```python
vllm_mlx.utils.tokenizer._load_with_tokenizer_fallback(model_name: str) -> not annotated
```

Load model with fallback tokenizer for non-standard models like Nemotron.

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `model_name` | `str` | `yes` | `none` | Required positional or keyword input. |

**Returns**

- Type: `not annotated`
- Direct return expressions: `(model, tokenizer)`

**Exceptions and behavior**

Function `_load_with_tokenizer_fallback` calls `logger.info`, `ensure_model_downloaded`, `load_model`, `tokenizer_json.exists`; can raise `ValueError`; returns `(model, tokenizer)`.
Directly raised exceptions: `ValueError`.

[View source #L218-L280](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/utils/tokenizer.py#L218-L280).

</details>

## Complete symbol map

This map also includes private definitions and nested helpers. The signature column exposes every explicit input even when an internal helper has no dedicated parameter prose.

| Symbol | Kind | Signature and inputs | What it does | Source |
| --- | --- | --- | --- | --- |
| [`_needs_tokenizer_fallback`](#contract-vllm_mlx.utils.tokenizer._needs_tokenizer_fallback) | function | `_needs_tokenizer_fallback(model_name: str) -> bool` | Check if model needs tokenizer fallback. | [#L25-L28](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/utils/tokenizer.py#L25-L28) |
| [`_needs_strict_false`](#contract-vllm_mlx.utils.tokenizer._needs_strict_false) | function | `_needs_strict_false(model_name: str) -> bool` | Check if model needs strict=False loading (VLM models with extra weights). | [#L31-L49](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/utils/tokenizer.py#L31-L49) |
| [`load_model_with_fallback`](#contract-vllm_mlx.utils.tokenizer.load_model_with_fallback) | function | `load_model_with_fallback(model_name: str, tokenizer_config: dict = None) -> not annotated` | Load model and tokenizer with fallback for non-standard tokenizers. | [#L52-L111](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/utils/tokenizer.py#L52-L111) |
| [`_load_strict_false`](#contract-vllm_mlx.utils.tokenizer._load_strict_false) | function | `_load_strict_false(model_name: str, tokenizer_config: dict = None) -> not annotated` | Load model with strict=False to discard extra weights. | [#L114-L153](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/utils/tokenizer.py#L114-L153) |
| [`_try_inject_mtp`](#contract-vllm_mlx.utils.tokenizer._try_inject_mtp) | function | `_try_inject_mtp(model, model_path, config) -> not annotated` | Inject MTP support if model has MTP config + weights. | [#L156-L176](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/utils/tokenizer.py#L156-L176) |
| [`_try_inject_mtp_post_load`](#contract-vllm_mlx.utils.tokenizer._try_inject_mtp_post_load) | function | `_try_inject_mtp_post_load(model, model_name) -> not annotated` | Check if MTP weights exist but were stripped by sanitize(), and inject. | [#L179-L215](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/utils/tokenizer.py#L179-L215) |
| [`_load_with_tokenizer_fallback`](#contract-vllm_mlx.utils.tokenizer._load_with_tokenizer_fallback) | function | `_load_with_tokenizer_fallback(model_name: str) -> not annotated` | Load model with fallback tokenizer for non-standard models like Nemotron. | [#L218-L280](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/utils/tokenizer.py#L218-L280) |