# `vllm_mlx.memory_cache`

Memory-aware prefix cache for vllm-mlx.

[View the complete module source at #L1-L1463](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/memory_cache.py#L1-L1463).

## API details

Each callable below includes its exact signature, type annotations, inputs, defaults, return contract, documented exceptions, implementation source, and parsed docstring sections when the source provides them.

::: vllm_mlx.memory_cache
    options:
      members:
        - logger
        - _BYTES_PER_MB
        - _DEFAULT_MEMORY_PERCENT
        - _MIN_MEMORY_BYTES
        - _MAX_ENTRIES_FALLBACK
        - _CACHE_PERSIST_VERSION
        - _get_available_memory
        - _array_memory
        - _nested_array_memory
        - estimate_kv_cache_memory
        - MemoryCacheConfig
        - CacheStats
        - _CacheEntry
        - _is_cache_layer_trimmable
        - _trim_cache_offset
        - _needs_kv_trim
        - _trim_to_offset
        - _QuantizedCacheWrapper
        - _quantize_cache
        - _dequantize_cache
        - _compute_model_fingerprint
        - MemoryAwarePrefixCache
      filters: []
      show_if_no_docstring: true

## Complete contract reference

Expand any definition for its exact inputs, annotations, defaults, return contract, directly raised exceptions, source-grounded behavior, and immutable line link. This section includes private and nested definitions that ordinary API generators omit.

<details class="api-contract" id="contract-vllm_mlx.memory_cache._get_available_memory" markdown="1">
<summary><code>vllm_mlx.memory_cache._get_available_memory</code> · function</summary>

```python
vllm_mlx.memory_cache._get_available_memory() -> int
```

Get available system memory in bytes.

**Parameters**

This callable has no explicit inputs.

**Returns**

- Type: `int`
- Direct return expressions: `psutil.virtual_memory().available`; `0`

**Exceptions and behavior**

Function `_get_available_memory` calls `psutil.virtual_memory`, `logger.warning`; has 2 explicit return paths.
No direct `raise` statement appears in this definition.

[View source #L47-L63](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/memory_cache.py#L47-L63).

</details>

<details class="api-contract" id="contract-vllm_mlx.memory_cache._array_memory" markdown="1">
<summary><code>vllm_mlx.memory_cache._array_memory</code> · function</summary>

```python
vllm_mlx.memory_cache._array_memory(arr) -> int
```

Estimate array memory from shape+dtype without triggering lazy eval.

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `arr` | `not annotated` | `yes` | `none` | An MLX array or similar object. |

**Returns**

- Type: `int`
- Direct return expressions: `math.prod(arr.shape) * dtype.size`; `arr.nbytes`; `0`

**Exceptions and behavior**

Function `_array_memory` calls `hasattr`, `math.prod`; has 3 explicit return paths.
No direct `raise` statement appears in this definition.

[View source #L66-L88](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/memory_cache.py#L66-L88).

</details>

<details class="api-contract" id="contract-vllm_mlx.memory_cache._nested_array_memory" markdown="1">
<summary><code>vllm_mlx.memory_cache._nested_array_memory</code> · function</summary>

```python
vllm_mlx.memory_cache._nested_array_memory(value: Any) -> int
```

Sum ``_array_memory`` over an arbitrarily nested state structure.

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `value` | `Any` | `yes` | `none` | Required positional or keyword input. |

**Returns**

- Type: `int`
- Direct return expressions: `0`; `sum((_nested_array_memory(v) for v in value))`; `_array_memory(value)`

**Exceptions and behavior**

Function `_nested_array_memory` calls `isinstance`, `sum`, `_nested_array_memory`, `_array_memory`; has 3 explicit return paths.
No direct `raise` statement appears in this definition.

[View source #L91-L105](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/memory_cache.py#L91-L105).

</details>

<details class="api-contract" id="contract-vllm_mlx.memory_cache.estimate_kv_cache_memory" markdown="1">
<summary><code>vllm_mlx.memory_cache.estimate_kv_cache_memory</code> · function</summary>

```python
vllm_mlx.memory_cache.estimate_kv_cache_memory(cache: list[Any]) -> int
```

Estimate memory usage of a KV cache in bytes.

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `cache` | `list[Any]` | `yes` | `none` | List of layer cache objects, each containing keys/values tensors. |

**Returns**

- Type: `int`
- Direct return expressions: `0`; `total_bytes`

**Exceptions and behavior**

Function `estimate_kv_cache_memory` calls `isinstance`, `_array_memory`, `hasattr`, `getattr`; has 2 explicit return paths.
No direct `raise` statement appears in this definition.

[View source #L108-L162](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/memory_cache.py#L108-L162).

</details>

<details class="api-contract" id="contract-vllm_mlx.memory_cache.MemoryCacheConfig" markdown="1">
<summary><code>vllm_mlx.memory_cache.MemoryCacheConfig</code> · class</summary>

```python
vllm_mlx.memory_cache.MemoryCacheConfig(max_memory_mb: int | None = None, max_memory_percent: float = _DEFAULT_MEMORY_PERCENT, max_entries: int = 1000, enable_memory_tracking: bool = True, kv_quantize: bool = False, kv_bits: int = 8, kv_group_size: int = 64, kv_min_quantize_tokens: int = 256, min_prefix_tokens: int = 128)
```

Configuration for memory-aware prefix cache.

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `max_memory_mb` | `int \| None` | `no` | `None` | Optional constructor field; defaults to `None`. |
| `max_memory_percent` | `float` | `no` | `_DEFAULT_MEMORY_PERCENT` | Optional constructor field; defaults to `_DEFAULT_MEMORY_PERCENT`. |
| `max_entries` | `int` | `no` | `1000` | Optional constructor field; defaults to `1000`. |
| `enable_memory_tracking` | `bool` | `no` | `True` | Optional constructor field; defaults to `True`. |
| `kv_quantize` | `bool` | `no` | `False` | Optional constructor field; defaults to `False`. |
| `kv_bits` | `int` | `no` | `8` | Optional constructor field; defaults to `8`. |
| `kv_group_size` | `int` | `no` | `64` | Optional constructor field; defaults to `64`. |
| `kv_min_quantize_tokens` | `int` | `no` | `256` | Optional constructor field; defaults to `256`. |
| `min_prefix_tokens` | `int` | `no` | `128` | Optional constructor field; defaults to `128`. |

**Returns**

- Constructs: `vllm_mlx.memory_cache.MemoryCacheConfig`

**Exceptions and behavior**

Class `MemoryCacheConfig` declares 2 direct member(s).
No direct `raise` statement appears in this definition.

[View source #L166-L225](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/memory_cache.py#L166-L225).

</details>

<details class="api-contract" id="contract-vllm_mlx.memory_cache.MemoryCacheConfig.__post_init__" markdown="1">
<summary><code>vllm_mlx.memory_cache.MemoryCacheConfig.__post_init__</code> · method</summary>

```python
vllm_mlx.memory_cache.MemoryCacheConfig.__post_init__() -> None
```

Method `MemoryCacheConfig.__post_init__` calls `ValueError`; can raise `ValueError`.

**Parameters**

This callable has no explicit inputs.

**Returns**

- Type: `None`

**Exceptions and behavior**

Method `MemoryCacheConfig.__post_init__` calls `ValueError`; can raise `ValueError`.
Directly raised exceptions: `ValueError`.

[View source #L192-L206](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/memory_cache.py#L192-L206).

</details>

<details class="api-contract" id="contract-vllm_mlx.memory_cache.MemoryCacheConfig.compute_memory_limit" markdown="1">
<summary><code>vllm_mlx.memory_cache.MemoryCacheConfig.compute_memory_limit</code> · method</summary>

```python
vllm_mlx.memory_cache.MemoryCacheConfig.compute_memory_limit() -> int
```

Compute the memory limit in bytes.

**Parameters**

This callable has no explicit inputs.

**Returns**

- Type: `int`
- Direct return expressions: `self.max_memory_mb * _BYTES_PER_MB`; `max(limit, _MIN_MEMORY_BYTES)`; `int(fallback_total * self.max_memory_percent)`

**Exceptions and behavior**

Method `MemoryCacheConfig.compute_memory_limit` calls `_get_available_memory`, `int`, `max`; has 3 explicit return paths.
No direct `raise` statement appears in this definition.

[View source #L208-L225](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/memory_cache.py#L208-L225).

</details>

<details class="api-contract" id="contract-vllm_mlx.memory_cache.CacheStats" markdown="1">
<summary><code>vllm_mlx.memory_cache.CacheStats</code> · class</summary>

```python
vllm_mlx.memory_cache.CacheStats(hits: int = 0, misses: int = 0, evictions: int = 0, tokens_saved: int = 0, current_memory_bytes: int = 0, max_memory_bytes: int = 0, entry_count: int = 0)
```

Statistics for cache performance monitoring.

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `hits` | `int` | `no` | `0` | Optional constructor field; defaults to `0`. |
| `misses` | `int` | `no` | `0` | Optional constructor field; defaults to `0`. |
| `evictions` | `int` | `no` | `0` | Optional constructor field; defaults to `0`. |
| `tokens_saved` | `int` | `no` | `0` | Optional constructor field; defaults to `0`. |
| `current_memory_bytes` | `int` | `no` | `0` | Optional constructor field; defaults to `0`. |
| `max_memory_bytes` | `int` | `no` | `0` | Optional constructor field; defaults to `0`. |
| `entry_count` | `int` | `no` | `0` | Optional constructor field; defaults to `0`. |

**Returns**

- Constructs: `vllm_mlx.memory_cache.CacheStats`

**Exceptions and behavior**

Class `CacheStats` declares 3 direct member(s).
No direct `raise` statement appears in this definition.

[View source #L229-L268](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/memory_cache.py#L229-L268).

</details>

<details class="api-contract" id="contract-vllm_mlx.memory_cache.CacheStats.hit_rate" markdown="1">
<summary><code>vllm_mlx.memory_cache.CacheStats.hit_rate</code> · method</summary>

```python
vllm_mlx.memory_cache.CacheStats.hit_rate() -> float
```

Return successful lookups divided by all completed lookups.

**Parameters**

This callable has no explicit inputs.

**Returns**

- Type: `float`
- Direct return expressions: `self.hits / total if total > 0 else 0.0`

**Exceptions and behavior**

Method `CacheStats.hit_rate` returns `self.hits / total if total > 0 else 0.0`.
No direct `raise` statement appears in this definition.

[View source #L241-L245](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/memory_cache.py#L241-L245).

</details>

<details class="api-contract" id="contract-vllm_mlx.memory_cache.CacheStats.memory_utilization" markdown="1">
<summary><code>vllm_mlx.memory_cache.CacheStats.memory_utilization</code> · method</summary>

```python
vllm_mlx.memory_cache.CacheStats.memory_utilization() -> float
```

Return the fraction of the configured memory budget in use.

**Parameters**

This callable has no explicit inputs.

**Returns**

- Type: `float`
- Direct return expressions: `0.0`; `self.current_memory_bytes / self.max_memory_bytes`

**Exceptions and behavior**

Method `CacheStats.memory_utilization` has 2 explicit return paths.
No direct `raise` statement appears in this definition.

[View source #L248-L253](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/memory_cache.py#L248-L253).

</details>

<details class="api-contract" id="contract-vllm_mlx.memory_cache.CacheStats.to_dict" markdown="1">
<summary><code>vllm_mlx.memory_cache.CacheStats.to_dict</code> · method</summary>

```python
vllm_mlx.memory_cache.CacheStats.to_dict() -> dict[str, Any]
```

Return rounded cache counters and memory values for APIs and logs.

**Parameters**

This callable has no explicit inputs.

**Returns**

- Type: `dict[str, Any]`
- Direct return expressions: `{'hits': self.hits, 'misses': self.misses, 'hit_rate': round(self.hit_rate, 4), 'evictions': self.evictions, 'tokens_sa…`

**Exceptions and behavior**

Method `CacheStats.to_dict` calls `round`; returns `{'hits': self.hits, 'misses': self.misses, 'hit_rate': round(self.hit_rate, 4), 'evictions': self.evictions, 'tokens_sa…`.
No direct `raise` statement appears in this definition.

[View source #L255-L268](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/memory_cache.py#L255-L268).

</details>

<details class="api-contract" id="contract-vllm_mlx.memory_cache._CacheEntry" markdown="1">
<summary><code>vllm_mlx.memory_cache._CacheEntry</code> · class</summary>

```python
vllm_mlx.memory_cache._CacheEntry(tokens: tuple[int, ...], cache: list[Any], memory_bytes: int)
```

Internal cache entry with memory tracking.

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `tokens` | `tuple[int, ...]` | `yes` | `none` | Required constructor field. |
| `cache` | `list[Any]` | `yes` | `none` | Required constructor field. |
| `memory_bytes` | `int` | `yes` | `none` | Required constructor field. |

**Returns**

- Constructs: `vllm_mlx.memory_cache._CacheEntry`

**Exceptions and behavior**

Class `_CacheEntry` declares 1 direct member(s).
No direct `raise` statement appears in this definition.

[View source #L272-L287](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/memory_cache.py#L272-L287).

</details>

<details class="api-contract" id="contract-vllm_mlx.memory_cache._CacheEntry.create" markdown="1">
<summary><code>vllm_mlx.memory_cache._CacheEntry.create</code> · method</summary>

```python
vllm_mlx.memory_cache._CacheEntry.create(tokens: list[int], cache: list[Any]) -> _CacheEntry
```

Create a cache entry with memory estimation.

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `tokens` | `list[int]` | `yes` | `none` | Required positional or keyword input. |
| `cache` | `list[Any]` | `yes` | `none` | Required positional or keyword input. |

**Returns**

- Type: `_CacheEntry`
- Direct return expressions: `cls(tokens=tuple(tokens), cache=cache, memory_bytes=memory)`

**Exceptions and behavior**

Method `_CacheEntry.create` calls `estimate_kv_cache_memory`, `cls`, `tuple`; returns `cls(tokens=tuple(tokens), cache=cache, memory_bytes=memory)`.
No direct `raise` statement appears in this definition.

[View source #L280-L287](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/memory_cache.py#L280-L287).

</details>

<details class="api-contract" id="contract-vllm_mlx.memory_cache._is_cache_layer_trimmable" markdown="1">
<summary><code>vllm_mlx.memory_cache._is_cache_layer_trimmable</code> · function</summary>

```python
vllm_mlx.memory_cache._is_cache_layer_trimmable(layer_cache: Any) -> bool
```

Return whether a cache layer can safely be rewound for partial reuse.

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `layer_cache` | `Any` | `yes` | `none` | Required positional or keyword input. |

**Returns**

- Type: `bool`
- Direct return expressions: `False`; `hasattr(layer_cache, 'offset') and hasattr(layer_cache, 'keys')`; `bool(is_trimmable())`

**Exceptions and behavior**

Function `_is_cache_layer_trimmable` calls `isinstance`, `hasattr`, `getattr`, `callable`; has 3 explicit return paths.
No direct `raise` statement appears in this definition.

[View source #L290-L314](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/memory_cache.py#L290-L314).

</details>

<details class="api-contract" id="contract-vllm_mlx.memory_cache._trim_cache_offset" markdown="1">
<summary><code>vllm_mlx.memory_cache._trim_cache_offset</code> · function</summary>

```python
vllm_mlx.memory_cache._trim_cache_offset(cache: list[Any], trim_by: int) -> list[Any]
```

Create copies of cache layers with the last ``trim_by`` positions removed.

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `cache` | `list[Any]` | `yes` | `none` | Required positional or keyword input. |
| `trim_by` | `int` | `yes` | `none` | Required positional or keyword input. |

**Returns**

- Type: `list[Any]`
- Direct return expressions: `trimmed`

**Exceptions and behavior**

Function `_trim_cache_offset` calls `isinstance`, `_QuantizedCacheWrapper.__new__`, `max`, `trimmed.append`; returns `trimmed`.
No direct `raise` statement appears in this definition.

[View source #L317-L481](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/memory_cache.py#L317-L481).

</details>

<details class="api-contract" id="contract-vllm_mlx.memory_cache._needs_kv_trim" markdown="1">
<summary><code>vllm_mlx.memory_cache._needs_kv_trim</code> · function</summary>

```python
vllm_mlx.memory_cache._needs_kv_trim(layer: Any) -> bool
```

Check if a cache layer has oversized KV arrays (duck-typed, no MLX import).

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `layer` | `Any` | `yes` | `none` | Required positional or keyword input. |

**Returns**

- Type: `bool`
- Direct return expressions: `False`; `0 < offset < shape[2]`

**Exceptions and behavior**

Function `_needs_kv_trim` calls `getattr`, `isinstance`, `len`; has 2 explicit return paths.
No direct `raise` statement appears in this definition.

[View source #L484-L495](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/memory_cache.py#L484-L495).

</details>

<details class="api-contract" id="contract-vllm_mlx.memory_cache._trim_to_offset" markdown="1">
<summary><code>vllm_mlx.memory_cache._trim_to_offset</code> · function</summary>

```python
vllm_mlx.memory_cache._trim_to_offset(cache: list[Any]) -> list[Any]
```

Trim KV arrays to their actual used size (offset) before storage.

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `cache` | `list[Any]` | `yes` | `none` | List of cache layer objects (KVCache or other types). |

**Returns**

- Type: `list[Any]`
- Direct return expressions: `cache`; `trimmed`

**Exceptions and behavior**

Function `_trim_to_offset` calls `any`, `_needs_kv_trim`, `isinstance`, `trimmed.append`; has 2 explicit return paths.
No direct `raise` statement appears in this definition.

[View source #L498-L538](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/memory_cache.py#L498-L538).

</details>

<details class="api-contract" id="contract-vllm_mlx.memory_cache._QuantizedCacheWrapper" markdown="1">
<summary><code>vllm_mlx.memory_cache._QuantizedCacheWrapper</code> · class</summary>

```python
vllm_mlx.memory_cache._QuantizedCacheWrapper(layer: Any, bits: int, group_size: int)
```

Lightweight wrapper storing quantized KV arrays + original cache metadata.

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `layer` | `Any` | `yes` | `none` | Required positional or keyword input. |
| `bits` | `int` | `yes` | `none` | Required positional or keyword input. |
| `group_size` | `int` | `yes` | `none` | Required positional or keyword input. |

**Returns**

- Constructs: `vllm_mlx.memory_cache._QuantizedCacheWrapper`

**Exceptions and behavior**

Class `_QuantizedCacheWrapper` declares 1 direct member(s).
No direct `raise` statement appears in this definition.

[View source #L541-L571](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/memory_cache.py#L541-L571).

</details>

<details class="api-contract" id="contract-vllm_mlx.memory_cache._QuantizedCacheWrapper.__init__" markdown="1">
<summary><code>vllm_mlx.memory_cache._QuantizedCacheWrapper.__init__</code> · method</summary>

```python
vllm_mlx.memory_cache._QuantizedCacheWrapper.__init__(layer: Any, bits: int, group_size: int) -> not annotated
```

Method `_QuantizedCacheWrapper.__init__` updates `self.keys`, `self.values`, `self.offset`, `self.bits`; calls `mx.quantize`, `type`, `hasattr`, `getattr`.

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `layer` | `Any` | `yes` | `none` | Required positional or keyword input. |
| `bits` | `int` | `yes` | `none` | Required positional or keyword input. |
| `group_size` | `int` | `yes` | `none` | Required positional or keyword input. |

**Returns**

- Type: `not annotated`

**Exceptions and behavior**

Method `_QuantizedCacheWrapper.__init__` updates `self.keys`, `self.values`, `self.offset`, `self.bits`; calls `mx.quantize`, `type`, `hasattr`, `getattr`.
No direct `raise` statement appears in this definition.

[View source #L558-L571](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/memory_cache.py#L558-L571).

</details>

<details class="api-contract" id="contract-vllm_mlx.memory_cache._quantize_cache" markdown="1">
<summary><code>vllm_mlx.memory_cache._quantize_cache</code> · function</summary>

```python
vllm_mlx.memory_cache._quantize_cache(cache: list[Any], bits: int = 8, group_size: int = 64) -> list[Any]
```

Quantize KV cache layers to reduce memory.

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `cache` | `list[Any]` | `yes` | `none` | Required positional or keyword input. |
| `bits` | `int` | `no` | `8` | Optional positional or keyword input; defaults to `8`. |
| `group_size` | `int` | `no` | `64` | Optional positional or keyword input; defaults to `64`. |

**Returns**

- Type: `list[Any]`
- Direct return expressions: `quantized`

**Exceptions and behavior**

Function `_quantize_cache` calls `type`, `getattr`, `quantized.append`, `_QuantizedCacheWrapper`; returns `quantized`.
No direct `raise` statement appears in this definition.

[View source #L574-L590](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/memory_cache.py#L574-L590).

</details>

<details class="api-contract" id="contract-vllm_mlx.memory_cache._dequantize_cache" markdown="1">
<summary><code>vllm_mlx.memory_cache._dequantize_cache</code> · function</summary>

```python
vllm_mlx.memory_cache._dequantize_cache(cache: list[Any]) -> list[Any]
```

Dequantize _QuantizedCacheWrapper layers and copy non-quantized layers.

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `cache` | `list[Any]` | `yes` | `none` | Required positional or keyword input. |

**Returns**

- Type: `list[Any]`
- Direct return expressions: `result`

**Exceptions and behavior**

Function `_dequantize_cache` calls `isinstance`, `orig_cls.__new__`, `mx.dequantize`, `hasattr`; returns `result`.
No direct `raise` statement appears in this definition.

[View source #L593-L645](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/memory_cache.py#L593-L645).

</details>

<details class="api-contract" id="contract-vllm_mlx.memory_cache._compute_model_fingerprint" markdown="1">
<summary><code>vllm_mlx.memory_cache._compute_model_fingerprint</code> · function</summary>

```python
vllm_mlx.memory_cache._compute_model_fingerprint(model: Any) -> str
```

Compute a fingerprint from model architecture for cache compatibility.

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `model` | `Any` | `yes` | `none` | Required positional or keyword input. |

**Returns**

- Type: `str`
- Direct return expressions: `fingerprint`

**Exceptions and behavior**

Function `_compute_model_fingerprint` calls `getattr`, `parts.append`, `hashlib.sha256('|'.join(parts).encode()).hexdigest`, `hashlib.sha256`; returns `fingerprint`.
No direct `raise` statement appears in this definition.

[View source #L648-L682](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/memory_cache.py#L648-L682).

</details>

<details class="api-contract" id="contract-vllm_mlx.memory_cache.MemoryAwarePrefixCache" markdown="1">
<summary><code>vllm_mlx.memory_cache.MemoryAwarePrefixCache</code> · class</summary>

```python
vllm_mlx.memory_cache.MemoryAwarePrefixCache(model: Any, config: MemoryCacheConfig | None = None)
```

Prefix cache with memory-based eviction.

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `model` | `Any` | `yes` | `none` | The MLX model (used for identification). |
| `config` | `MemoryCacheConfig \| None` | `no` | `None` | Cache configuration. Uses defaults if None. |

**Returns**

- Constructs: `vllm_mlx.memory_cache.MemoryAwarePrefixCache`

**Exceptions and behavior**

Class `MemoryAwarePrefixCache` declares 19 direct member(s).
No direct `raise` statement appears in this definition.

[View source #L685-L1463](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/memory_cache.py#L685-L1463).

</details>

<details class="api-contract" id="contract-vllm_mlx.memory_cache.MemoryAwarePrefixCache.__init__" markdown="1">
<summary><code>vllm_mlx.memory_cache.MemoryAwarePrefixCache.__init__</code> · method</summary>

```python
vllm_mlx.memory_cache.MemoryAwarePrefixCache.__init__(model: Any, config: MemoryCacheConfig | None = None) -> None
```

Initialize the memory-aware prefix cache.

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `model` | `Any` | `yes` | `none` | The MLX model (used for identification). |
| `config` | `MemoryCacheConfig \| None` | `no` | `None` | Cache configuration. Uses defaults if None. |

**Returns**

- Type: `None`

**Exceptions and behavior**

Method `MemoryAwarePrefixCache.__init__` updates `self._model_id`, `self._config`, `self._model_fingerprint`, `self._entries`; calls `id`, `MemoryCacheConfig`, `_compute_model_fingerprint`, `OrderedDict`.
No direct `raise` statement appears in this definition.

[View source #L703-L746](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/memory_cache.py#L703-L746).

</details>

<details class="api-contract" id="contract-vllm_mlx.memory_cache.MemoryAwarePrefixCache.fetch" markdown="1">
<summary><code>vllm_mlx.memory_cache.MemoryAwarePrefixCache.fetch</code> · method</summary>

```python
vllm_mlx.memory_cache.MemoryAwarePrefixCache.fetch(tokens: list[int]) -> tuple[list[Any] | None, list[int]]
```

Find cached KV state for the given tokens.

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `tokens` | `list[int]` | `yes` | `none` | Input token sequence. |

**Returns**

- Type: `tuple[list[Any] | None, list[int]]`
- Direct return expressions: `(None, tokens)`; `(cache_out, [])`; `(trimmed_cache, [])`; `(cache_out, remaining)`; `(trimmed_cache, remaining)`

**Exceptions and behavior**

Method `MemoryAwarePrefixCache.fetch` updates `self._stats.misses`, `self._last_match_type`, `self._stats.hits`, `self._stats.tokens_saved`; calls `len`, `tuple`, `self._entries.move_to_end`, `_dequantize_cache`; has 5 explicit return paths.
No direct `raise` statement appears in this definition.

[View source #L748-L977](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/memory_cache.py#L748-L977).

</details>

<details class="api-contract" id="contract-vllm_mlx.memory_cache.MemoryAwarePrefixCache.store" markdown="1">
<summary><code>vllm_mlx.memory_cache.MemoryAwarePrefixCache.store</code> · method</summary>

```python
vllm_mlx.memory_cache.MemoryAwarePrefixCache.store(tokens: list[int], cache: list[Any], evict_prefixes: bool = True) -> bool
```

Store KV cache for future reuse.

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `tokens` | `list[int]` | `yes` | `none` | Token sequence that was processed. |
| `cache` | `list[Any]` | `yes` | `none` | The computed KV cache to store. |
| `evict_prefixes` | `bool` | `no` | `True` | If True, evict existing entries whose token sequence is a strict prefix of ``tokens``. Set to False when storing prompt+output entries to preserve prompt-only entries created by prompt_cache_save (those are the entries that future requests will actually match). |

**Returns**

- Type: `bool`
- Direct return expressions: `False`; `True`

**Exceptions and behavior**

Method `MemoryAwarePrefixCache.store` updates `self._current_memory`, `self._stats.evictions`, `self._stats.entry_count`, `self._stats.current_memory_bytes`; calls `len`, `logger.debug`, `tuple`, `self._entries.move_to_end`; has 2 explicit return paths.
No direct `raise` statement appears in this definition.

[View source #L979-L1092](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/memory_cache.py#L979-L1092).

</details>

<details class="api-contract" id="contract-vllm_mlx.memory_cache.MemoryAwarePrefixCache._remove_from_sorted" markdown="1">
<summary><code>vllm_mlx.memory_cache.MemoryAwarePrefixCache._remove_from_sorted</code> · method</summary>

```python
vllm_mlx.memory_cache.MemoryAwarePrefixCache._remove_from_sorted(key: tuple[int, ...]) -> None
```

Remove a key from the sorted index using bisect for O(log N).

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `key` | `tuple[int, ...]` | `yes` | `none` | Required positional or keyword input. |

**Returns**

- Type: `None`

**Exceptions and behavior**

Method `MemoryAwarePrefixCache._remove_from_sorted` calls `bisect.bisect_left`, `len`, `self._sorted_keys.pop`.
No direct `raise` statement appears in this definition.

[View source #L1094-L1098](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/memory_cache.py#L1094-L1098).

</details>

<details class="api-contract" id="contract-vllm_mlx.memory_cache.MemoryAwarePrefixCache._evict_lru" markdown="1">
<summary><code>vllm_mlx.memory_cache.MemoryAwarePrefixCache._evict_lru</code> · method</summary>

```python
vllm_mlx.memory_cache.MemoryAwarePrefixCache._evict_lru() -> None
```

Evict the least recently used entry.

**Parameters**

This callable has no explicit inputs.

**Returns**

- Type: `None`
- Direct return expressions: `None`

**Exceptions and behavior**

Method `MemoryAwarePrefixCache._evict_lru` updates `self._current_memory`, `self._stats.evictions`, `self._stats.entry_count`, `self._stats.current_memory_bytes`; calls `self._entries.popitem`, `self._remove_from_sorted`, `len`, `self._ssd_tier.enqueue_spill`; returns `None`.
No direct `raise` statement appears in this definition.

[View source #L1100-L1126](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/memory_cache.py#L1100-L1126).

</details>

<details class="api-contract" id="contract-vllm_mlx.memory_cache.MemoryAwarePrefixCache.remove" markdown="1">
<summary><code>vllm_mlx.memory_cache.MemoryAwarePrefixCache.remove</code> · method</summary>

```python
vllm_mlx.memory_cache.MemoryAwarePrefixCache.remove(tokens: list[int]) -> bool
```

Remove a specific cache entry.

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `tokens` | `list[int]` | `yes` | `none` | Token sequence to remove. |

**Returns**

- Type: `bool`
- Direct return expressions: `True`; `False`

**Exceptions and behavior**

Method `MemoryAwarePrefixCache.remove` updates `self._current_memory`, `self._stats.entry_count`, `self._stats.current_memory_bytes`; calls `tuple`, `self._entries.pop`, `self._remove_from_sorted`, `len`; has 2 explicit return paths.
No direct `raise` statement appears in this definition.

[View source #L1128-L1147](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/memory_cache.py#L1128-L1147).

</details>

<details class="api-contract" id="contract-vllm_mlx.memory_cache.MemoryAwarePrefixCache.clear" markdown="1">
<summary><code>vllm_mlx.memory_cache.MemoryAwarePrefixCache.clear</code> · method</summary>

```python
vllm_mlx.memory_cache.MemoryAwarePrefixCache.clear() -> None
```

Clear all cached entries.

**Parameters**

This callable has no explicit inputs.

**Returns**

- Type: `None`

**Exceptions and behavior**

Method `MemoryAwarePrefixCache.clear` updates `self._current_memory`, `self._stats`; calls `self._entries.clear`, `self._sorted_keys.clear`, `CacheStats`, `logger.debug`.
No direct `raise` statement appears in this definition.

[View source #L1149-L1156](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/memory_cache.py#L1149-L1156).

</details>

<details class="api-contract" id="contract-vllm_mlx.memory_cache.MemoryAwarePrefixCache.get_stats" markdown="1">
<summary><code>vllm_mlx.memory_cache.MemoryAwarePrefixCache.get_stats</code> · method</summary>

```python
vllm_mlx.memory_cache.MemoryAwarePrefixCache.get_stats() -> dict[str, Any]
```

Get cache statistics.

**Parameters**

This callable has no explicit inputs.

**Returns**

- Type: `dict[str, Any]`
- Direct return expressions: `self._stats.to_dict()`

**Exceptions and behavior**

Method `MemoryAwarePrefixCache.get_stats` calls `self._stats.to_dict`; returns `self._stats.to_dict()`.
No direct `raise` statement appears in this definition.

[View source #L1158-L1160](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/memory_cache.py#L1158-L1160).

</details>

<details class="api-contract" id="contract-vllm_mlx.memory_cache.MemoryAwarePrefixCache.reset_stats" markdown="1">
<summary><code>vllm_mlx.memory_cache.MemoryAwarePrefixCache.reset_stats</code> · method</summary>

```python
vllm_mlx.memory_cache.MemoryAwarePrefixCache.reset_stats() -> None
```

Reset statistics while preserving cache contents.

**Parameters**

This callable has no explicit inputs.

**Returns**

- Type: `None`

**Exceptions and behavior**

Method `MemoryAwarePrefixCache.reset_stats` updates `self._stats`; calls `CacheStats`, `len`.
No direct `raise` statement appears in this definition.

[View source #L1162-L1169](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/memory_cache.py#L1162-L1169).

</details>

<details class="api-contract" id="contract-vllm_mlx.memory_cache.MemoryAwarePrefixCache.memory_usage_mb" markdown="1">
<summary><code>vllm_mlx.memory_cache.MemoryAwarePrefixCache.memory_usage_mb</code> · method</summary>

```python
vllm_mlx.memory_cache.MemoryAwarePrefixCache.memory_usage_mb() -> float
```

Current memory usage in MB.

**Parameters**

This callable has no explicit inputs.

**Returns**

- Type: `float`
- Direct return expressions: `self._current_memory / _BYTES_PER_MB`

**Exceptions and behavior**

Method `MemoryAwarePrefixCache.memory_usage_mb` returns `self._current_memory / _BYTES_PER_MB`.
No direct `raise` statement appears in this definition.

[View source #L1172-L1174](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/memory_cache.py#L1172-L1174).

</details>

<details class="api-contract" id="contract-vllm_mlx.memory_cache.MemoryAwarePrefixCache.memory_limit_mb" markdown="1">
<summary><code>vllm_mlx.memory_cache.MemoryAwarePrefixCache.memory_limit_mb</code> · method</summary>

```python
vllm_mlx.memory_cache.MemoryAwarePrefixCache.memory_limit_mb() -> float
```

Memory limit in MB.

**Parameters**

This callable has no explicit inputs.

**Returns**

- Type: `float`
- Direct return expressions: `self._max_memory / _BYTES_PER_MB`

**Exceptions and behavior**

Method `MemoryAwarePrefixCache.memory_limit_mb` returns `self._max_memory / _BYTES_PER_MB`.
No direct `raise` statement appears in this definition.

[View source #L1177-L1179](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/memory_cache.py#L1177-L1179).

</details>

<details class="api-contract" id="contract-vllm_mlx.memory_cache.MemoryAwarePrefixCache.try_reserve_memory" markdown="1">
<summary><code>vllm_mlx.memory_cache.MemoryAwarePrefixCache.try_reserve_memory</code> · method</summary>

```python
vllm_mlx.memory_cache.MemoryAwarePrefixCache.try_reserve_memory(nbytes: int) -> bool
```

Tentatively reserve cache memory for an upcoming promotion.

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `nbytes` | `int` | `yes` | `none` | Required positional or keyword input. |

**Returns**

- Type: `bool`
- Direct return expressions: `False`; `True`

**Exceptions and behavior**

Method `MemoryAwarePrefixCache.try_reserve_memory` updates `self._current_memory`, `self._stats.current_memory_bytes`; has 2 explicit return paths.
No direct `raise` statement appears in this definition.

[View source #L1181-L1188](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/memory_cache.py#L1181-L1188).

</details>

<details class="api-contract" id="contract-vllm_mlx.memory_cache.MemoryAwarePrefixCache.release_reserved_memory" markdown="1">
<summary><code>vllm_mlx.memory_cache.MemoryAwarePrefixCache.release_reserved_memory</code> · method</summary>

```python
vllm_mlx.memory_cache.MemoryAwarePrefixCache.release_reserved_memory(nbytes: int) -> None
```

Release memory previously reserved by try_reserve_memory().

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `nbytes` | `int` | `yes` | `none` | Required positional or keyword input. |

**Returns**

- Type: `None`

**Exceptions and behavior**

Method `MemoryAwarePrefixCache.release_reserved_memory` updates `self._current_memory`, `self._stats.current_memory_bytes`; calls `max`.
No direct `raise` statement appears in this definition.

[View source #L1190-L1194](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/memory_cache.py#L1190-L1194).

</details>

<details class="api-contract" id="contract-vllm_mlx.memory_cache.MemoryAwarePrefixCache.__len__" markdown="1">
<summary><code>vllm_mlx.memory_cache.MemoryAwarePrefixCache.__len__</code> · method</summary>

```python
vllm_mlx.memory_cache.MemoryAwarePrefixCache.__len__() -> int
```

Return number of cached entries.

**Parameters**

This callable has no explicit inputs.

**Returns**

- Type: `int`
- Direct return expressions: `len(self._entries)`

**Exceptions and behavior**

Method `MemoryAwarePrefixCache.__len__` calls `len`; returns `len(self._entries)`.
No direct `raise` statement appears in this definition.

[View source #L1196-L1198](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/memory_cache.py#L1196-L1198).

</details>

<details class="api-contract" id="contract-vllm_mlx.memory_cache.MemoryAwarePrefixCache.__contains__" markdown="1">
<summary><code>vllm_mlx.memory_cache.MemoryAwarePrefixCache.__contains__</code> · method</summary>

```python
vllm_mlx.memory_cache.MemoryAwarePrefixCache.__contains__(tokens: list[int]) -> bool
```

Check if tokens are cached.

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `tokens` | `list[int]` | `yes` | `none` | Required positional or keyword input. |

**Returns**

- Type: `bool`
- Direct return expressions: `tuple(tokens) in self._entries`

**Exceptions and behavior**

Method `MemoryAwarePrefixCache.__contains__` calls `tuple`; returns `tuple(tokens) in self._entries`.
No direct `raise` statement appears in this definition.

[View source #L1200-L1202](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/memory_cache.py#L1200-L1202).

</details>

<details class="api-contract" id="contract-vllm_mlx.memory_cache.MemoryAwarePrefixCache.set_ssd_tier" markdown="1">
<summary><code>vllm_mlx.memory_cache.MemoryAwarePrefixCache.set_ssd_tier</code> · method</summary>

```python
vllm_mlx.memory_cache.MemoryAwarePrefixCache.set_ssd_tier(ssd_tier) -> None
```

Attach an SSD cache tier for eviction spilling.

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `ssd_tier` | `not annotated` | `yes` | `none` | An SSDCacheTier instance (or None to disable). |

**Returns**

- Type: `None`

**Exceptions and behavior**

Method `MemoryAwarePrefixCache.set_ssd_tier` updates `self._ssd_tier`; calls `logger.info`.
No direct `raise` statement appears in this definition.

[View source #L1204-L1214](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/memory_cache.py#L1204-L1214).

</details>

<details class="api-contract" id="contract-vllm_mlx.memory_cache.MemoryAwarePrefixCache.check_ssd" markdown="1">
<summary><code>vllm_mlx.memory_cache.MemoryAwarePrefixCache.check_ssd</code> · method</summary>

```python
vllm_mlx.memory_cache.MemoryAwarePrefixCache.check_ssd(tokens: list[int]) -> dict | None
```

Check if tokens have an SSD cache hit (without reading data).

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `tokens` | `list[int]` | `yes` | `none` | Required positional or keyword input. |

**Returns**

- Type: `dict | None`
- Direct return expressions: `None`; `candidate`; `prefix`

**Exceptions and behavior**

Method `MemoryAwarePrefixCache.check_ssd` calls `tuple`, `self._ssd_tier.lookup_ssd`, `len`, `self._ssd_tier.lookup_ssd_prefix`; has 3 explicit return paths.
No direct `raise` statement appears in this definition.

[View source #L1216-L1249](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/memory_cache.py#L1216-L1249).

</details>

<details class="api-contract" id="contract-vllm_mlx.memory_cache.MemoryAwarePrefixCache.save_to_disk" markdown="1">
<summary><code>vllm_mlx.memory_cache.MemoryAwarePrefixCache.save_to_disk</code> · method</summary>

```python
vllm_mlx.memory_cache.MemoryAwarePrefixCache.save_to_disk(cache_dir: str) -> bool
```

Save all cache entries to disk using mlx_lm's safetensors format.

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `cache_dir` | `str` | `yes` | `none` | Required positional or keyword input. |

**Returns**

- Type: `bool`
- Direct return expressions: `False`; `saved > 0`

**Exceptions and behavior**

Method `MemoryAwarePrefixCache.save_to_disk` calls `logger.info`, `_time.monotonic`, `os.makedirs`, `logger.warning`; has 2 explicit return paths.
No direct `raise` statement appears in this definition.

[View source #L1255-L1346](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/memory_cache.py#L1255-L1346).

</details>

<details class="api-contract" id="contract-vllm_mlx.memory_cache.MemoryAwarePrefixCache.load_from_disk" markdown="1">
<summary><code>vllm_mlx.memory_cache.MemoryAwarePrefixCache.load_from_disk</code> · method</summary>

```python
vllm_mlx.memory_cache.MemoryAwarePrefixCache.load_from_disk(cache_dir: str) -> int
```

Load cache entries from disk.

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `cache_dir` | `str` | `yes` | `none` | Required positional or keyword input. |

**Returns**

- Type: `int`
- Direct return expressions: `0`; `loaded`

**Exceptions and behavior**

Method `MemoryAwarePrefixCache.load_from_disk` updates `self._current_memory`, `self._stats.entry_count`, `self._stats.current_memory_bytes`; calls `os.path.join`, `os.path.exists`, `logger.info`, `_time.monotonic`; has 2 explicit return paths.
No direct `raise` statement appears in this definition.

[View source #L1348-L1463](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/memory_cache.py#L1348-L1463).

</details>

## Complete symbol map

This map also includes private definitions and nested helpers. The signature column exposes every explicit input even when an internal helper has no dedicated parameter prose.

| Symbol | Kind | Signature and inputs | What it does | Source |
| --- | --- | --- | --- | --- |
| [`_get_available_memory`](#contract-vllm_mlx.memory_cache._get_available_memory) | function | `_get_available_memory() -> int` | Get available system memory in bytes. | [#L47-L63](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/memory_cache.py#L47-L63) |
| [`_array_memory`](#contract-vllm_mlx.memory_cache._array_memory) | function | `_array_memory(arr) -> int` | Estimate array memory from shape+dtype without triggering lazy eval. | [#L66-L88](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/memory_cache.py#L66-L88) |
| [`_nested_array_memory`](#contract-vllm_mlx.memory_cache._nested_array_memory) | function | `_nested_array_memory(value: Any) -> int` | Sum ``_array_memory`` over an arbitrarily nested state structure. | [#L91-L105](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/memory_cache.py#L91-L105) |
| [`estimate_kv_cache_memory`](#contract-vllm_mlx.memory_cache.estimate_kv_cache_memory) | function | `estimate_kv_cache_memory(cache: list[Any]) -> int` | Estimate memory usage of a KV cache in bytes. | [#L108-L162](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/memory_cache.py#L108-L162) |
| [`MemoryCacheConfig`](#contract-vllm_mlx.memory_cache.MemoryCacheConfig) | class | `MemoryCacheConfig(max_memory_mb: int \| None = None, max_memory_percent: float = _DEFAULT_MEMORY_PERCENT, max_entries: int = 1000, enable_memory_tracking: bool = True, kv_quantize: bool = False, kv_bits: int = 8, kv_group_size: int = 64, kv_min_quantize_tokens: int = 256, min_prefix_tokens: int = 128)` | Configuration for memory-aware prefix cache. | [#L166-L225](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/memory_cache.py#L166-L225) |
| [`MemoryCacheConfig.__post_init__`](#contract-vllm_mlx.memory_cache.MemoryCacheConfig.__post_init__) | method | `MemoryCacheConfig.__post_init__() -> None` | Method `MemoryCacheConfig.__post_init__` calls `ValueError`; can raise `ValueError`. | [#L192-L206](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/memory_cache.py#L192-L206) |
| [`MemoryCacheConfig.compute_memory_limit`](#contract-vllm_mlx.memory_cache.MemoryCacheConfig.compute_memory_limit) | method | `MemoryCacheConfig.compute_memory_limit() -> int` | Compute the memory limit in bytes. | [#L208-L225](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/memory_cache.py#L208-L225) |
| [`CacheStats`](#contract-vllm_mlx.memory_cache.CacheStats) | class | `CacheStats(hits: int = 0, misses: int = 0, evictions: int = 0, tokens_saved: int = 0, current_memory_bytes: int = 0, max_memory_bytes: int = 0, entry_count: int = 0)` | Statistics for cache performance monitoring. | [#L229-L268](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/memory_cache.py#L229-L268) |
| [`CacheStats.hit_rate`](#contract-vllm_mlx.memory_cache.CacheStats.hit_rate) | method | `CacheStats.hit_rate() -> float` | Return successful lookups divided by all completed lookups. | [#L241-L245](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/memory_cache.py#L241-L245) |
| [`CacheStats.memory_utilization`](#contract-vllm_mlx.memory_cache.CacheStats.memory_utilization) | method | `CacheStats.memory_utilization() -> float` | Return the fraction of the configured memory budget in use. | [#L248-L253](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/memory_cache.py#L248-L253) |
| [`CacheStats.to_dict`](#contract-vllm_mlx.memory_cache.CacheStats.to_dict) | method | `CacheStats.to_dict() -> dict[str, Any]` | Return rounded cache counters and memory values for APIs and logs. | [#L255-L268](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/memory_cache.py#L255-L268) |
| [`_CacheEntry`](#contract-vllm_mlx.memory_cache._CacheEntry) | class | `_CacheEntry(tokens: tuple[int, ...], cache: list[Any], memory_bytes: int)` | Internal cache entry with memory tracking. | [#L272-L287](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/memory_cache.py#L272-L287) |
| [`_CacheEntry.create`](#contract-vllm_mlx.memory_cache._CacheEntry.create) | method | `_CacheEntry.create(tokens: list[int], cache: list[Any]) -> _CacheEntry` | Create a cache entry with memory estimation. | [#L280-L287](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/memory_cache.py#L280-L287) |
| [`_is_cache_layer_trimmable`](#contract-vllm_mlx.memory_cache._is_cache_layer_trimmable) | function | `_is_cache_layer_trimmable(layer_cache: Any) -> bool` | Return whether a cache layer can safely be rewound for partial reuse. | [#L290-L314](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/memory_cache.py#L290-L314) |
| [`_trim_cache_offset`](#contract-vllm_mlx.memory_cache._trim_cache_offset) | function | `_trim_cache_offset(cache: list[Any], trim_by: int) -> list[Any]` | Create copies of cache layers with the last ``trim_by`` positions removed. | [#L317-L481](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/memory_cache.py#L317-L481) |
| [`_needs_kv_trim`](#contract-vllm_mlx.memory_cache._needs_kv_trim) | function | `_needs_kv_trim(layer: Any) -> bool` | Check if a cache layer has oversized KV arrays (duck-typed, no MLX import). | [#L484-L495](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/memory_cache.py#L484-L495) |
| [`_trim_to_offset`](#contract-vllm_mlx.memory_cache._trim_to_offset) | function | `_trim_to_offset(cache: list[Any]) -> list[Any]` | Trim KV arrays to their actual used size (offset) before storage. | [#L498-L538](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/memory_cache.py#L498-L538) |
| [`_QuantizedCacheWrapper`](#contract-vllm_mlx.memory_cache._QuantizedCacheWrapper) | class | `_QuantizedCacheWrapper(layer: Any, bits: int, group_size: int)` | Lightweight wrapper storing quantized KV arrays + original cache metadata. | [#L541-L571](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/memory_cache.py#L541-L571) |
| [`_QuantizedCacheWrapper.__init__`](#contract-vllm_mlx.memory_cache._QuantizedCacheWrapper.__init__) | method | `_QuantizedCacheWrapper.__init__(layer: Any, bits: int, group_size: int) -> not annotated` | Method `_QuantizedCacheWrapper.__init__` updates `self.keys`, `self.values`, `self.offset`, `self.bits`; calls `mx.quantize`, `type`, `hasattr`, `getattr`. | [#L558-L571](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/memory_cache.py#L558-L571) |
| [`_quantize_cache`](#contract-vllm_mlx.memory_cache._quantize_cache) | function | `_quantize_cache(cache: list[Any], bits: int = 8, group_size: int = 64) -> list[Any]` | Quantize KV cache layers to reduce memory. | [#L574-L590](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/memory_cache.py#L574-L590) |
| [`_dequantize_cache`](#contract-vllm_mlx.memory_cache._dequantize_cache) | function | `_dequantize_cache(cache: list[Any]) -> list[Any]` | Dequantize _QuantizedCacheWrapper layers and copy non-quantized layers. | [#L593-L645](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/memory_cache.py#L593-L645) |
| [`_compute_model_fingerprint`](#contract-vllm_mlx.memory_cache._compute_model_fingerprint) | function | `_compute_model_fingerprint(model: Any) -> str` | Compute a fingerprint from model architecture for cache compatibility. | [#L648-L682](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/memory_cache.py#L648-L682) |
| [`MemoryAwarePrefixCache`](#contract-vllm_mlx.memory_cache.MemoryAwarePrefixCache) | class | `MemoryAwarePrefixCache(model: Any, config: MemoryCacheConfig \| None = None)` | Prefix cache with memory-based eviction. | [#L685-L1463](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/memory_cache.py#L685-L1463) |
| [`MemoryAwarePrefixCache.__init__`](#contract-vllm_mlx.memory_cache.MemoryAwarePrefixCache.__init__) | method | `MemoryAwarePrefixCache.__init__(model: Any, config: MemoryCacheConfig \| None = None) -> None` | Initialize the memory-aware prefix cache. | [#L703-L746](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/memory_cache.py#L703-L746) |
| [`MemoryAwarePrefixCache.fetch`](#contract-vllm_mlx.memory_cache.MemoryAwarePrefixCache.fetch) | method | `MemoryAwarePrefixCache.fetch(tokens: list[int]) -> tuple[list[Any] \| None, list[int]]` | Find cached KV state for the given tokens. | [#L748-L977](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/memory_cache.py#L748-L977) |
| [`MemoryAwarePrefixCache.store`](#contract-vllm_mlx.memory_cache.MemoryAwarePrefixCache.store) | method | `MemoryAwarePrefixCache.store(tokens: list[int], cache: list[Any], evict_prefixes: bool = True) -> bool` | Store KV cache for future reuse. | [#L979-L1092](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/memory_cache.py#L979-L1092) |
| [`MemoryAwarePrefixCache._remove_from_sorted`](#contract-vllm_mlx.memory_cache.MemoryAwarePrefixCache._remove_from_sorted) | method | `MemoryAwarePrefixCache._remove_from_sorted(key: tuple[int, ...]) -> None` | Remove a key from the sorted index using bisect for O(log N). | [#L1094-L1098](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/memory_cache.py#L1094-L1098) |
| [`MemoryAwarePrefixCache._evict_lru`](#contract-vllm_mlx.memory_cache.MemoryAwarePrefixCache._evict_lru) | method | `MemoryAwarePrefixCache._evict_lru() -> None` | Evict the least recently used entry. | [#L1100-L1126](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/memory_cache.py#L1100-L1126) |
| [`MemoryAwarePrefixCache.remove`](#contract-vllm_mlx.memory_cache.MemoryAwarePrefixCache.remove) | method | `MemoryAwarePrefixCache.remove(tokens: list[int]) -> bool` | Remove a specific cache entry. | [#L1128-L1147](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/memory_cache.py#L1128-L1147) |
| [`MemoryAwarePrefixCache.clear`](#contract-vllm_mlx.memory_cache.MemoryAwarePrefixCache.clear) | method | `MemoryAwarePrefixCache.clear() -> None` | Clear all cached entries. | [#L1149-L1156](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/memory_cache.py#L1149-L1156) |
| [`MemoryAwarePrefixCache.get_stats`](#contract-vllm_mlx.memory_cache.MemoryAwarePrefixCache.get_stats) | method | `MemoryAwarePrefixCache.get_stats() -> dict[str, Any]` | Get cache statistics. | [#L1158-L1160](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/memory_cache.py#L1158-L1160) |
| [`MemoryAwarePrefixCache.reset_stats`](#contract-vllm_mlx.memory_cache.MemoryAwarePrefixCache.reset_stats) | method | `MemoryAwarePrefixCache.reset_stats() -> None` | Reset statistics while preserving cache contents. | [#L1162-L1169](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/memory_cache.py#L1162-L1169) |
| [`MemoryAwarePrefixCache.memory_usage_mb`](#contract-vllm_mlx.memory_cache.MemoryAwarePrefixCache.memory_usage_mb) | method | `MemoryAwarePrefixCache.memory_usage_mb() -> float` | Current memory usage in MB. | [#L1172-L1174](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/memory_cache.py#L1172-L1174) |
| [`MemoryAwarePrefixCache.memory_limit_mb`](#contract-vllm_mlx.memory_cache.MemoryAwarePrefixCache.memory_limit_mb) | method | `MemoryAwarePrefixCache.memory_limit_mb() -> float` | Memory limit in MB. | [#L1177-L1179](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/memory_cache.py#L1177-L1179) |
| [`MemoryAwarePrefixCache.try_reserve_memory`](#contract-vllm_mlx.memory_cache.MemoryAwarePrefixCache.try_reserve_memory) | method | `MemoryAwarePrefixCache.try_reserve_memory(nbytes: int) -> bool` | Tentatively reserve cache memory for an upcoming promotion. | [#L1181-L1188](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/memory_cache.py#L1181-L1188) |
| [`MemoryAwarePrefixCache.release_reserved_memory`](#contract-vllm_mlx.memory_cache.MemoryAwarePrefixCache.release_reserved_memory) | method | `MemoryAwarePrefixCache.release_reserved_memory(nbytes: int) -> None` | Release memory previously reserved by try_reserve_memory(). | [#L1190-L1194](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/memory_cache.py#L1190-L1194) |
| [`MemoryAwarePrefixCache.__len__`](#contract-vllm_mlx.memory_cache.MemoryAwarePrefixCache.__len__) | method | `MemoryAwarePrefixCache.__len__() -> int` | Return number of cached entries. | [#L1196-L1198](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/memory_cache.py#L1196-L1198) |
| [`MemoryAwarePrefixCache.__contains__`](#contract-vllm_mlx.memory_cache.MemoryAwarePrefixCache.__contains__) | method | `MemoryAwarePrefixCache.__contains__(tokens: list[int]) -> bool` | Check if tokens are cached. | [#L1200-L1202](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/memory_cache.py#L1200-L1202) |
| [`MemoryAwarePrefixCache.set_ssd_tier`](#contract-vllm_mlx.memory_cache.MemoryAwarePrefixCache.set_ssd_tier) | method | `MemoryAwarePrefixCache.set_ssd_tier(ssd_tier) -> None` | Attach an SSD cache tier for eviction spilling. | [#L1204-L1214](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/memory_cache.py#L1204-L1214) |
| [`MemoryAwarePrefixCache.check_ssd`](#contract-vllm_mlx.memory_cache.MemoryAwarePrefixCache.check_ssd) | method | `MemoryAwarePrefixCache.check_ssd(tokens: list[int]) -> dict \| None` | Check if tokens have an SSD cache hit (without reading data). | [#L1216-L1249](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/memory_cache.py#L1216-L1249) |
| [`MemoryAwarePrefixCache.save_to_disk`](#contract-vllm_mlx.memory_cache.MemoryAwarePrefixCache.save_to_disk) | method | `MemoryAwarePrefixCache.save_to_disk(cache_dir: str) -> bool` | Save all cache entries to disk using mlx_lm's safetensors format. | [#L1255-L1346](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/memory_cache.py#L1255-L1346) |
| [`MemoryAwarePrefixCache.load_from_disk`](#contract-vllm_mlx.memory_cache.MemoryAwarePrefixCache.load_from_disk) | method | `MemoryAwarePrefixCache.load_from_disk(cache_dir: str) -> int` | Load cache entries from disk. | [#L1348-L1463](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/memory_cache.py#L1348-L1463) |