vllm_mlx.prefix_cache

Prefix Cache Manager for vllm-mlx.

View the complete module source at #L1-L1039.

API details

Each callable below includes its exact signature, type annotations, inputs, defaults, return contract, documented exceptions, implementation source, and parsed docstring sections when the source provides them.

vllm_mlx.prefix_cache

Prefix Cache Manager for vllm-mlx.

Wraps mlx-lm's LRUPromptCache to provide prefix caching functionality, allowing reuse of computed KV cache for common prompt prefixes.

This module provides two implementations: - PrefixCacheManager: Original trie-based LRU cache (for backward compatibility) - BlockAwarePrefixCache: Block-based cache with PagedCacheManager integration

vllm_mlx.prefix_cache.logger module-attribute

logger = logging.getLogger(__name__)

vllm_mlx.prefix_cache.CacheEntry dataclass

CacheEntry(prompt_cache: List[Any], count: int)

Entry in the prefix cache.

vllm_mlx.prefix_cache.CacheEntry.prompt_cache instance-attribute

prompt_cache: List[Any]

vllm_mlx.prefix_cache.CacheEntry.count instance-attribute

count: int

vllm_mlx.prefix_cache.PrefixCacheStats dataclass

PrefixCacheStats(hits: int = 0, misses: int = 0, tokens_saved: int = 0, total_queries: int = 0, evictions: int = 0)

Statistics for prefix cache performance.

vllm_mlx.prefix_cache.PrefixCacheStats.hits class-attribute instance-attribute

hits: int = 0

vllm_mlx.prefix_cache.PrefixCacheStats.misses class-attribute instance-attribute

misses: int = 0

vllm_mlx.prefix_cache.PrefixCacheStats.tokens_saved class-attribute instance-attribute

tokens_saved: int = 0

vllm_mlx.prefix_cache.PrefixCacheStats.total_queries class-attribute instance-attribute

total_queries: int = 0

vllm_mlx.prefix_cache.PrefixCacheStats.evictions class-attribute instance-attribute

evictions: int = 0

vllm_mlx.prefix_cache.PrefixCacheStats.hit_rate property

hit_rate: float

Calculate cache hit rate.

vllm_mlx.prefix_cache.PrefixCacheStats.to_dict

to_dict() -> Dict[str, Any]

Convert stats to dictionary.

Source code in vllm_mlx/prefix_cache.py

def to_dict(self) -> Dict[str, Any]:
    """Convert stats to dictionary."""
    return {
        "hits": self.hits,
        "misses": self.misses,
        "hit_rate": self.hit_rate,
        "tokens_saved": self.tokens_saved,
        "total_queries": self.total_queries,
        "evictions": self.evictions,
    }

vllm_mlx.prefix_cache.PrefixCacheManager

PrefixCacheManager(model: Any, max_entries: int = 100)

Manages prefix caching for vllm-mlx using a trie-based LRU cache.

This implementation is inspired by mlx-lm's LRUPromptCache but adapted for vllm-mlx's batching architecture.

The cache stores KV states keyed by token sequences, allowing: - Exact match: Full prompt found in cache - Shorter match: Partial prefix found, process remaining tokens - Longer match: Cached prefix longer than request, trim excess

Example

cache_manager = PrefixCacheManager(model, max_entries=100)

Check for cached prefix

cache, remaining_tokens = cache_manager.fetch_cache(tokens) if cache: # Use cached KV, only process remaining_tokens pass

After generation, store cache for reuse

cache_manager.store_cache(full_tokens, prompt_cache)

Initialize the prefix cache manager.

Parameters:

• model (Any) –

  The MLX model (used for cache key identification)

• max_entries (int, default: 100 ) –

  Maximum number of cached entries before LRU eviction

Source code in vllm_mlx/prefix_cache.py

def __init__(self, model: Any, max_entries: int = 100):
    """
    Initialize the prefix cache manager.

    Args:
        model: The MLX model (used for cache key identification)
        max_entries: Maximum number of cached entries before LRU eviction
    """
    self.model = model
    self.model_key = id(model)
    self.max_size = max_entries

    # Trie-based cache: nested dicts with token keys
    # Structure: {model_key: {token1: {token2: {..., "cache": CacheEntry}}}}
    self._cache: Dict[Any, Dict] = {}

    # LRU tracking: OrderedDict keyed by (model_key, tuple(tokens)), insertion
    # order = least-recently-used first. move_to_end() and popitem() are O(1).
    self._lru: OrderedDict = OrderedDict()

    # Statistics
    self.stats = PrefixCacheStats()

vllm_mlx.prefix_cache.PrefixCacheManager.model instance-attribute

model = model

vllm_mlx.prefix_cache.PrefixCacheManager.model_key instance-attribute

model_key = id(model)

vllm_mlx.prefix_cache.PrefixCacheManager.max_size instance-attribute

max_size = max_entries

vllm_mlx.prefix_cache.PrefixCacheManager._cache instance-attribute

_cache: Dict[Any, Dict] = {}

vllm_mlx.prefix_cache.PrefixCacheManager._lru instance-attribute

_lru: OrderedDict = OrderedDict()

vllm_mlx.prefix_cache.PrefixCacheManager.stats instance-attribute

stats = PrefixCacheStats()

vllm_mlx.prefix_cache.PrefixCacheManager._search

_search(tokens: List[int]) -> Tuple[Optional[List[int]], Optional[List[int]], Optional[List[int]], int]

Search for cached prefix matching tokens.

Returns:

• Optional[List[int]] –

  Tuple of (exact, shorter, longer, common_prefix_len)

• Optional[List[int]] –
  • exact: Tokens if exact match found
• Optional[List[int]] –
  • shorter: Tokens of shorter cached prefix
• int –
  • longer: Tokens of longer cached prefix
• Tuple[Optional[List[int]], Optional[List[int]], Optional[List[int]], int] –
  • common_prefix_len: Length of common prefix with longer match

Source code in vllm_mlx/prefix_cache.py

def _search(
    self, tokens: List[int]
) -> Tuple[Optional[List[int]], Optional[List[int]], Optional[List[int]], int]:
    """
    Search for cached prefix matching tokens.

    Returns:
        Tuple of (exact, shorter, longer, common_prefix_len)
        - exact: Tokens if exact match found
        - shorter: Tokens of shorter cached prefix
        - longer: Tokens of longer cached prefix
        - common_prefix_len: Length of common prefix with longer match
    """
    if self.model_key not in self._cache:
        return None, None, None, 0

    current = self._cache[self.model_key]
    path = []

    # Traverse trie following token sequence
    for i, tok in enumerate(tokens):
        if tok not in current:
            # No match for this token
            # Check if we have a shorter prefix with cache
            if "cache" in current:
                return None, list(path), None, 0
            return None, None, None, 0

        path.append(tok)
        current = current[tok]

    # Reached end of tokens
    if "cache" in current:
        # Exact match
        return list(tokens), None, None, 0

    # Check for longer cached prefix
    # DFS to find shortest extension with cache
    stack = [(current, list(path))]
    while stack:
        node, node_path = stack.pop()
        if "cache" in node:
            return None, None, node_path, len(tokens)
        for tok, child in node.items():
            if tok != "cache":
                stack.append((child, node_path + [tok]))

    return None, None, None, 0

vllm_mlx.prefix_cache.PrefixCacheManager.fetch_cache

fetch_cache(tokens: List[int]) -> Tuple[Optional[List[Any]], List[int]]

Find cached prefix for the given tokens.

Parameters:

• tokens (List[int]) –

  Input token sequence

Returns:

• Optional[List[Any]] –

  Tuple of (cache, remaining_tokens)

• List[int] –
  • cache: Cached KV state if found, None otherwise
• Tuple[Optional[List[Any]], List[int]] –
  • remaining_tokens: Tokens that still need processing

Source code in vllm_mlx/prefix_cache.py

def fetch_cache(self, tokens: List[int]) -> Tuple[Optional[List[Any]], List[int]]:
    """
    Find cached prefix for the given tokens.

    Args:
        tokens: Input token sequence

    Returns:
        Tuple of (cache, remaining_tokens)
        - cache: Cached KV state if found, None otherwise
        - remaining_tokens: Tokens that still need processing
    """
    self.stats.total_queries += 1
    tokens_tuple = tuple(tokens)

    exact, shorter, longer, common_len = self._search(tokens)

    if exact:
        # Exact match - return full cache
        cache_entry = self._get_cache_entry(exact)
        if cache_entry:
            self.stats.hits += 1
            self.stats.tokens_saved += len(tokens)
            self._touch_lru(tokens_tuple)
            # No copy needed - MLX arrays are immutable
            return cache_entry.prompt_cache, []

    if shorter:
        # Shorter prefix cached - return cache and remaining tokens
        cache_entry = self._get_cache_entry(shorter)
        if cache_entry:
            self.stats.hits += 1
            self.stats.tokens_saved += len(shorter)
            self._touch_lru(tuple(shorter))
            remaining = tokens[len(shorter) :]
            # No copy needed - MLX arrays are immutable
            return cache_entry.prompt_cache, remaining

    if longer:
        # Longer prefix cached - trim to match and return
        cache_entry = self._get_cache_entry(longer)
        if cache_entry:
            # Check if cache supports trimming
            prompt_cache = cache_entry.prompt_cache
            if self._can_trim_cache(prompt_cache):
                trim_amount = len(longer) - len(tokens)
                trimmed_cache = self._trim_cache(
                    copy.deepcopy(prompt_cache), trim_amount
                )
                self.stats.hits += 1
                self.stats.tokens_saved += len(tokens)
                return trimmed_cache, []

    # No cache hit
    self.stats.misses += 1
    return None, tokens

vllm_mlx.prefix_cache.PrefixCacheManager.store_cache

store_cache(tokens: List[int], prompt_cache: List[Any]) -> None

Store computed cache for future reuse.

Parameters:

• tokens (List[int]) –

  Token sequence that was processed

• prompt_cache (List[Any]) –

  The computed KV cache to store

Source code in vllm_mlx/prefix_cache.py

def store_cache(self, tokens: List[int], prompt_cache: List[Any]) -> None:
    """
    Store computed cache for future reuse.

    Args:
        tokens: Token sequence that was processed
        prompt_cache: The computed KV cache to store
    """
    if not tokens:
        return

    tokens_tuple = tuple(tokens)

    # Build trie path
    if self.model_key not in self._cache:
        self._cache[self.model_key] = {}

    current = self._cache[self.model_key]
    for tok in tokens:
        if tok not in current:
            current[tok] = {}
        current = current[tok]

    # Store or update cache entry
    key = (self.model_key, tokens_tuple)
    if "cache" in current:
        current["cache"].count += 1
        # Move to most-recently-used position — O(1) with OrderedDict
        self._lru.move_to_end(key)
    else:
        current["cache"] = CacheEntry(prompt_cache, 1)
        self._lru[key] = None

    # Evict if over capacity
    while len(self._lru) > self.max_size:
        self._evict_lru()

vllm_mlx.prefix_cache.PrefixCacheManager._get_cache_entry

_get_cache_entry(tokens: List[int]) -> Optional[CacheEntry]

Get cache entry for given tokens.

Source code in vllm_mlx/prefix_cache.py

def _get_cache_entry(self, tokens: List[int]) -> Optional[CacheEntry]:
    """Get cache entry for given tokens."""
    if self.model_key not in self._cache:
        return None

    current = self._cache[self.model_key]
    for tok in tokens:
        if tok not in current:
            return None
        current = current[tok]

    return current.get("cache")

vllm_mlx.prefix_cache.PrefixCacheManager._touch_lru

_touch_lru(tokens_tuple: tuple) -> None

Move entry to most-recently-used position — O(1) with OrderedDict.

Source code in vllm_mlx/prefix_cache.py

def _touch_lru(self, tokens_tuple: tuple) -> None:
    """Move entry to most-recently-used position — O(1) with OrderedDict."""
    key = (self.model_key, tokens_tuple)
    if key in self._lru:
        self._lru.move_to_end(key)
    else:
        self._lru[key] = None

vllm_mlx.prefix_cache.PrefixCacheManager._evict_lru

_evict_lru() -> None

Evict least recently used entry — O(1) popitem from OrderedDict.

Source code in vllm_mlx/prefix_cache.py

def _evict_lru(self) -> None:
    """Evict least recently used entry — O(1) popitem from OrderedDict."""
    if not self._lru:
        return

    (model_key, tokens_tuple), _ = self._lru.popitem(last=False)
    self._delete_cache(model_key, list(tokens_tuple))
    self.stats.evictions += 1

vllm_mlx.prefix_cache.PrefixCacheManager._delete_cache

_delete_cache(model_key: Any, tokens: List[int]) -> None

Delete cache entry and clean up empty trie branches.

Source code in vllm_mlx/prefix_cache.py

def _delete_cache(self, model_key: Any, tokens: List[int]) -> None:
    """Delete cache entry and clean up empty trie branches."""
    if model_key not in self._cache:
        return

    # Navigate to entry
    path = [(self._cache[model_key], None)]
    current = self._cache[model_key]

    for tok in tokens:
        if tok not in current:
            return
        path.append((current[tok], tok))
        current = current[tok]

    # Delete cache entry
    if "cache" in current:
        del current["cache"]

    # Clean up empty branches (bottom-up)
    for i in range(len(path) - 1, 0, -1):
        node, tok = path[i]
        parent, _ = path[i - 1]
        if not node:  # Empty dict
            del parent[tok]

vllm_mlx.prefix_cache.PrefixCacheManager._can_trim_cache

_can_trim_cache(prompt_cache: List[Any]) -> bool

Check if cache can be trimmed.

Source code in vllm_mlx/prefix_cache.py

def _can_trim_cache(self, prompt_cache: List[Any]) -> bool:
    """Check if cache can be trimmed."""
    if not prompt_cache:
        return False
    # Check if first cache layer has is_trimmable method
    first_cache = prompt_cache[0]
    if hasattr(first_cache, "is_trimmable"):
        trimmable = first_cache.is_trimmable()
        if not trimmable:
            logger.debug(
                "Prefix cache reuse skipped: cache is not trimmable "
                "(RotatingKVCache does not support trimming)"
            )
        return trimmable
    return hasattr(first_cache, "trim")

vllm_mlx.prefix_cache.PrefixCacheManager._trim_cache

_trim_cache(prompt_cache: List[Any], num_tokens: int) -> List[Any]

Trim cache by removing num_tokens from the end.

Source code in vllm_mlx/prefix_cache.py

def _trim_cache(self, prompt_cache: List[Any], num_tokens: int) -> List[Any]:
    """Trim cache by removing num_tokens from the end."""
    for cache in prompt_cache:
        if hasattr(cache, "trim"):
            cache.trim(num_tokens)
    return prompt_cache

vllm_mlx.prefix_cache.PrefixCacheManager.get_stats

get_stats() -> Dict[str, Any]

Get cache statistics.

Source code in vllm_mlx/prefix_cache.py

def get_stats(self) -> Dict[str, Any]:
    """Get cache statistics."""
    return self.stats.to_dict()

vllm_mlx.prefix_cache.PrefixCacheManager.reset_stats

reset_stats() -> None

Reset statistics.

Source code in vllm_mlx/prefix_cache.py

def reset_stats(self) -> None:
    """Reset statistics."""
    self.stats = PrefixCacheStats()

vllm_mlx.prefix_cache.PrefixCacheManager.clear

clear() -> None

Clear all cached entries.

Source code in vllm_mlx/prefix_cache.py

def clear(self) -> None:
    """Clear all cached entries."""
    self._cache.clear()
    self._lru.clear()
    self.reset_stats()

vllm_mlx.prefix_cache.PrefixCacheManager.__len__

__len__() -> int

Return number of cached entries.

Source code in vllm_mlx/prefix_cache.py

def __len__(self) -> int:
    """Return number of cached entries."""
    return len(self._lru)

vllm_mlx.prefix_cache.BlockCacheEntry dataclass

BlockCacheEntry(block_table: BlockTable, cache_data: List[Any], last_access: float)

Entry mapping a token sequence to cache blocks.

vllm_mlx.prefix_cache.BlockCacheEntry.block_table instance-attribute

block_table: BlockTable

vllm_mlx.prefix_cache.BlockCacheEntry.cache_data instance-attribute

cache_data: List[Any]

vllm_mlx.prefix_cache.BlockCacheEntry.last_access instance-attribute

last_access: float

vllm_mlx.prefix_cache.BlockAwarePrefixCache

BlockAwarePrefixCache(model: Any, paged_cache_manager: PagedCacheManager)

Prefix cache that uses PagedCacheManager for block-based storage.

Features: - Block-level prefix sharing (64 tokens per block) - Copy-on-Write for efficient forking - Hash-based deduplication across requests - Reference counting for memory efficiency

This is the recommended cache for production use when memory efficiency for concurrent requests is important.

Example

paged_manager = PagedCacheManager(block_size=64, max_blocks=1000) cache = BlockAwarePrefixCache(model, paged_manager)

Check for cached prefix

block_table, remaining_tokens = cache.fetch_cache(request_id, tokens)

After generation, store cache

cache.store_cache(request_id, tokens, kv_cache_data)

Clean up when request completes

cache.release_cache(request_id)

Initialize block-aware prefix cache.

Parameters:

• model (Any) –

  The MLX model (used for identification)

• paged_cache_manager (PagedCacheManager) –

  The PagedCacheManager instance for block management

Source code in vllm_mlx/prefix_cache.py

def __init__(
    self,
    model: Any,
    paged_cache_manager: PagedCacheManager,
):
    """
    Initialize block-aware prefix cache.

    Args:
        model: The MLX model (used for identification)
        paged_cache_manager: The PagedCacheManager instance for block management
    """
    self.model = model
    self.model_key = id(model)
    self.paged_cache = paged_cache_manager
    self.block_size = paged_cache_manager.block_size

    # Hash table for quick prefix lookup
    # Maps hash(tokens[:block_size*n]) -> (tokens, block_ids)
    self._prefix_index: Dict[str, Tuple[List[int], List[int]]] = {}

    # Request to block table mapping
    self._request_tables: Dict[str, BlockCacheEntry] = {}

    # Statistics
    self._hits = 0
    self._misses = 0
    self._tokens_saved = 0

vllm_mlx.prefix_cache.BlockAwarePrefixCache.model instance-attribute

model = model

vllm_mlx.prefix_cache.BlockAwarePrefixCache.model_key instance-attribute

model_key = id(model)

vllm_mlx.prefix_cache.BlockAwarePrefixCache.paged_cache instance-attribute

paged_cache = paged_cache_manager

vllm_mlx.prefix_cache.BlockAwarePrefixCache.block_size instance-attribute

block_size = paged_cache_manager.block_size

vllm_mlx.prefix_cache.BlockAwarePrefixCache._prefix_index instance-attribute

_prefix_index: Dict[str, Tuple[List[int], List[int]]] = {}

vllm_mlx.prefix_cache.BlockAwarePrefixCache._request_tables instance-attribute

_request_tables: Dict[str, BlockCacheEntry] = {}

vllm_mlx.prefix_cache.BlockAwarePrefixCache._hits instance-attribute

_hits = 0

vllm_mlx.prefix_cache.BlockAwarePrefixCache._misses instance-attribute

_misses = 0

vllm_mlx.prefix_cache.BlockAwarePrefixCache._tokens_saved instance-attribute

_tokens_saved = 0

vllm_mlx.prefix_cache.BlockAwarePrefixCache.fetch_cache

fetch_cache(request_id: str, tokens: List[int]) -> Tuple[Optional[BlockTable], List[int]]

Find cached prefix blocks for the given tokens.

Parameters:

• request_id (str) –

  Unique request identifier

• tokens (List[int]) –

  Input token sequence

Returns:

• Optional[BlockTable] –

  Tuple of (block_table, remaining_tokens)

• List[int] –
  • block_table: BlockTable if prefix found, None otherwise
• Tuple[Optional[BlockTable], List[int]] –
  • remaining_tokens: Tokens that need processing

Source code in vllm_mlx/prefix_cache.py

def fetch_cache(
    self,
    request_id: str,
    tokens: List[int],
) -> Tuple[Optional[BlockTable], List[int]]:
    """
    Find cached prefix blocks for the given tokens.

    Args:
        request_id: Unique request identifier
        tokens: Input token sequence

    Returns:
        Tuple of (block_table, remaining_tokens)
        - block_table: BlockTable if prefix found, None otherwise
        - remaining_tokens: Tokens that need processing
    """
    if not tokens:
        return None, tokens

    # Try to find shared prefix blocks
    shared_block_ids, remaining = self.paged_cache.find_shared_prefix(tokens)

    if shared_block_ids:
        # Create block table for this request with shared blocks
        block_table = self.paged_cache.create_block_table(request_id)

        for block_id in shared_block_ids:
            # Increment ref count for sharing
            self.paged_cache.increment_ref(block_id)
            block = self.paged_cache.allocated_blocks.get(block_id)
            if block:
                block_table.block_ids.append(block_id)
                block_table.num_tokens += block.token_count

        num_prefix_tokens = len(tokens) - len(remaining)
        self._hits += 1
        self._tokens_saved += num_prefix_tokens

        logger.debug(
            f"Cache hit for {request_id}: "
            f"{len(shared_block_ids)} blocks, {num_prefix_tokens} tokens"
        )

        return block_table, remaining

    # Try prefix index for longer matches
    best_match = self._find_best_prefix_match(tokens)
    if best_match:
        matched_tokens, matched_block_ids = best_match

        # Fork the matched blocks
        block_table = self.paged_cache.create_block_table(request_id)
        for block_id in matched_block_ids:
            self.paged_cache.increment_ref(block_id)
            block = self.paged_cache.allocated_blocks.get(block_id)
            if block:
                block_table.block_ids.append(block_id)
                block_table.num_tokens += block.token_count

        remaining = tokens[len(matched_tokens) :]
        self._hits += 1
        self._tokens_saved += len(matched_tokens)

        logger.debug(
            f"Prefix index hit for {request_id}: "
            f"{len(matched_tokens)} tokens matched"
        )

        return block_table, remaining

    # No cache hit
    self._misses += 1
    logger.debug(f"Cache miss for {request_id}")
    return None, tokens

vllm_mlx.prefix_cache.BlockAwarePrefixCache.store_cache

store_cache(request_id: str, tokens: List[int], cache_data: List[Any]) -> Optional[BlockTable]

Store computed cache for future reuse.

This method stores actual tensor data (not references) when cache_data contains extracted states from mlx-lm's KVCache.state property.

Parameters:

• request_id (str) –

  Unique request identifier

• tokens (List[int]) –

  Token sequence that was processed

• cache_data (List[Any]) –

  The computed KV cache to store. Can be: - List of KVCache objects (legacy, stores references) - List of dicts with 'state': (keys, values) tensors (new, stores slices)

Returns:

• Optional[BlockTable] –

  BlockTable for the stored cache, or None on failure

Source code in vllm_mlx/prefix_cache.py

def store_cache(
    self,
    request_id: str,
    tokens: List[int],
    cache_data: List[Any],
) -> Optional[BlockTable]:
    """
    Store computed cache for future reuse.

    This method stores actual tensor data (not references) when cache_data
    contains extracted states from mlx-lm's KVCache.state property.

    Args:
        request_id: Unique request identifier
        tokens: Token sequence that was processed
        cache_data: The computed KV cache to store. Can be:
            - List of KVCache objects (legacy, stores references)
            - List of dicts with 'state': (keys, values) tensors (new, stores slices)

    Returns:
        BlockTable for the stored cache, or None on failure
    """
    if not tokens:
        return None

    # Check if cache_data contains extracted tensor states
    is_tensor_data = (
        cache_data
        and isinstance(cache_data, list)
        and len(cache_data) > 0
        and isinstance(cache_data[0], dict)
        and "state" in cache_data[0]
    )

    # Get or create block table
    block_table = self.paged_cache.get_block_table(request_id)
    if not block_table:
        block_table = self.paged_cache.create_block_table(request_id)

    # Determine tokens we need to cache (not already in block_table)
    existing_tokens = block_table.num_tokens
    new_tokens = tokens[existing_tokens:]

    if not new_tokens:
        # All tokens already cached
        return block_table

    # Allocate blocks for new tokens
    num_new_blocks = (len(new_tokens) + self.block_size - 1) // self.block_size

    for i in range(num_new_blocks):
        start_idx = i * self.block_size
        end_idx = min(start_idx + self.block_size, len(new_tokens))
        block_tokens = new_tokens[start_idx:end_idx]

        # Token range in the original sequence (accounting for existing tokens)
        global_start = existing_tokens + start_idx
        global_end = existing_tokens + end_idx

        # Check if this block already exists (deduplication)
        if len(block_tokens) == self.block_size:
            existing_block = self.paged_cache.find_cached_block(block_tokens)
            if existing_block:
                # Reuse existing block
                self.paged_cache.increment_ref(existing_block.block_id)
                block_table.block_ids.append(existing_block.block_id)
                block_table.num_tokens += len(block_tokens)
                continue

        # Allocate new block
        block = self.paged_cache.allocate_block()
        if not block:
            # Handle memory pressure
            if not self.paged_cache.handle_memory_pressure(1):
                logger.warning(f"Cannot allocate block for {request_id}")
                break
            block = self.paged_cache.allocate_block()
            if not block:
                break

        # Store block data
        block.token_count = len(block_tokens)
        block_table.block_ids.append(block.block_id)
        block_table.num_tokens += len(block_tokens)

        # Extract and store actual tensor slices for this block
        if is_tensor_data and HAS_MLX:
            block_kv_data = self._extract_block_tensor_slice(
                cache_data, global_start, global_end, len(tokens)
            )
            if block_kv_data:
                block.cache_data = block_kv_data
                logger.debug(
                    f"Stored tensor slice for block {block.block_id}: "
                    f"tokens [{global_start}:{global_end}], {len(block_kv_data)} layers"
                )

        # Register hash for full blocks (for deduplication)
        if len(block_tokens) == self.block_size:
            self.paged_cache.register_block_hash(block, block_tokens)

    # Update prefix index
    self._update_prefix_index(tokens, block_table.block_ids)

    # Store entry for request (for legacy compatibility)
    self._request_tables[request_id] = BlockCacheEntry(
        block_table=block_table,
        cache_data=cache_data,
        last_access=time.time(),
    )

    blocks_with_data = sum(
        1
        for bid in block_table.block_ids
        if self.paged_cache.allocated_blocks.get(bid)
        and self.paged_cache.allocated_blocks[bid].cache_data is not None
    )

    logger.debug(
        f"Stored cache for {request_id}: "
        f"{len(block_table.block_ids)} blocks ({blocks_with_data} with tensor data), "
        f"{block_table.num_tokens} tokens"
    )

    return block_table

vllm_mlx.prefix_cache.BlockAwarePrefixCache._extract_block_tensor_slice

_extract_block_tensor_slice(cache_data: List[Dict[str, Any]], start_idx: int, end_idx: int, total_tokens: int) -> Optional[List[Optional[Dict[str, Any]]]]

Extract per-layer cache data for a single block.

Parameters:

• cache_data (List[Dict[str, Any]]) –

  List of extracted layer states

• start_idx (int) –

  Start token index in the sequence

• end_idx (int) –

  End token index in the sequence

• total_tokens (int) –

  Total number of tokens covered by cache_data

Returns:

• Optional[List[Optional[Dict[str, Any]]]] –

  Per-layer block cache state, or None on failure

Source code in vllm_mlx/prefix_cache.py

def _extract_block_tensor_slice(
    self,
    cache_data: List[Dict[str, Any]],
    start_idx: int,
    end_idx: int,
    total_tokens: int,
) -> Optional[List[Optional[Dict[str, Any]]]]:
    """
    Extract per-layer cache data for a single block.

    Args:
        cache_data: List of extracted layer states
        start_idx: Start token index in the sequence
        end_idx: End token index in the sequence
        total_tokens: Total number of tokens covered by cache_data

    Returns:
        Per-layer block cache state, or None on failure
    """
    if not HAS_MLX or not cache_data:
        return None

    try:
        block_slices: List[Optional[Dict[str, Any]]] = []
        for layer_state in cache_data:
            if "state" not in layer_state:
                block_slices.append(None)
                continue

            state = layer_state["state"]
            meta_state = layer_state.get("meta_state")
            class_ref = layer_state.get("class_ref")
            class_name = layer_state.get("class_name")

            seq_axis = self._cache_state_seq_axis(state)
            if seq_axis is not None:
                state_slice = self._slice_concat_cache_state(
                    state, start_idx, end_idx
                )
                block_slices.append(
                    {
                        "state": state_slice,
                        "meta_state": meta_state,
                        "class_ref": class_ref,
                        "class_name": class_name,
                        "storage": "concat",
                        "seq_axis": seq_axis,
                    }
                )
                continue

            if end_idx == total_tokens:
                block_slices.append(
                    {
                        "state": state,
                        "meta_state": meta_state,
                        "class_ref": class_ref,
                        "class_name": class_name,
                        "storage": "latest",
                    }
                )
            else:
                block_slices.append(None)

        return (
            block_slices
            if any(entry is not None for entry in block_slices)
            else None
        )

    except Exception as e:
        logger.warning(f"Failed to extract block tensor slice: {e}")
        return None

vllm_mlx.prefix_cache.BlockAwarePrefixCache._cache_state_seq_axis

_cache_state_seq_axis(state: Any) -> Optional[int]

Return the sequence axis for cache states that support block concat.

Source code in vllm_mlx/prefix_cache.py

def _cache_state_seq_axis(self, state: Any) -> Optional[int]:
    """Return the sequence axis for cache states that support block concat."""
    if not isinstance(state, (list, tuple)) or not state:
        return None

    ndims = {
        len(tensor.shape)
        for tensor in state
        if tensor is not None and hasattr(tensor, "shape")
    }
    if len(ndims) != 1:
        return None

    ndim = next(iter(ndims))
    if ndim == 4:
        return 2

    # Qwen3.5-style KV caches use (heads, seq, dim).
    if ndim == 3 and len(state) == 2:
        return 1

    return None

vllm_mlx.prefix_cache.BlockAwarePrefixCache._slice_concat_cache_state

_slice_concat_cache_state(state: Tuple[Any, ...] | List[Any], start_idx: int, end_idx: int) -> Tuple[Any, ...] | List[Any]

Slice a sequence-backed cache state across the token axis.

Source code in vllm_mlx/prefix_cache.py

def _slice_concat_cache_state(
    self,
    state: Tuple[Any, ...] | List[Any],
    start_idx: int,
    end_idx: int,
) -> Tuple[Any, ...] | List[Any]:
    """Slice a sequence-backed cache state across the token axis."""
    seq_axis = self._cache_state_seq_axis(state)
    if seq_axis is None:
        raise ValueError("Cache state does not support sequence concatenation")

    seq_len = state[0].shape[seq_axis]
    actual_end = min(end_idx, seq_len)
    if start_idx >= actual_end:
        raise ValueError(
            f"Block slice [{start_idx}:{end_idx}] exceeds seq_len {seq_len}"
        )

    def _slice_tensor(tensor: Any) -> Any:
        slices = [slice(None)] * len(tensor.shape)
        slices[seq_axis] = slice(start_idx, actual_end)
        return tensor[tuple(slices)]

    sliced = [_slice_tensor(tensor) for tensor in state]
    return tuple(sliced) if isinstance(state, tuple) else sliced

vllm_mlx.prefix_cache.BlockAwarePrefixCache._concat_cache_states

_concat_cache_states(states: List[Tuple[Any, ...] | List[Any]], seq_axis: int) -> Optional[Tuple[Any, ...] | List[Any]]

Concatenate state fragments for a sequence-backed cache layer.

Source code in vllm_mlx/prefix_cache.py

def _concat_cache_states(
    self,
    states: List[Tuple[Any, ...] | List[Any]],
    seq_axis: int,
) -> Optional[Tuple[Any, ...] | List[Any]]:
    """Concatenate state fragments for a sequence-backed cache layer."""
    if not states:
        return None
    arity = len(states[0])
    concatenated = []
    for idx in range(arity):
        parts = [state[idx] for state in states]
        if any(part is None for part in parts):
            return None
        concatenated.append(mx.concatenate(parts, axis=seq_axis))
    return tuple(concatenated) if isinstance(states[0], tuple) else concatenated

vllm_mlx.prefix_cache.BlockAwarePrefixCache.get_cache_for_generation

get_cache_for_generation(request_id: str) -> Tuple[Optional[List[Any]], bool]

Get cache data for generation, applying COW if needed.

Parameters:

• request_id (str) –

  Request identifier

Returns:

• Tuple[Optional[List[Any]], bool] –

  Tuple of (cache_data, was_copied)

Source code in vllm_mlx/prefix_cache.py

def get_cache_for_generation(
    self,
    request_id: str,
) -> Tuple[Optional[List[Any]], bool]:
    """
    Get cache data for generation, applying COW if needed.

    Args:
        request_id: Request identifier

    Returns:
        Tuple of (cache_data, was_copied)
    """
    entry = self._request_tables.get(request_id)
    if not entry:
        return None, False

    # Get blocks with COW
    blocks, was_copied = self.paged_cache.get_blocks_for_generation(
        entry.block_table
    )

    if was_copied:
        # Deep copy cache data for modified blocks
        cache_data = copy.deepcopy(entry.cache_data)
    else:
        cache_data = entry.cache_data

    entry.last_access = time.time()
    return cache_data, was_copied

vllm_mlx.prefix_cache.BlockAwarePrefixCache.release_cache

release_cache(request_id: str) -> None

Release cache blocks for a completed request.

Parameters:

• request_id (str) –

  Request identifier

Source code in vllm_mlx/prefix_cache.py

def release_cache(self, request_id: str) -> None:
    """
    Release cache blocks for a completed request.

    Args:
        request_id: Request identifier
    """
    entry = self._request_tables.pop(request_id, None)
    if entry:
        self.paged_cache.delete_block_table(request_id)
        logger.debug(f"Released cache for {request_id}")

vllm_mlx.prefix_cache.BlockAwarePrefixCache.fork_cache

fork_cache(source_request_id: str, new_request_id: str) -> Optional[BlockTable]

Fork cache from one request to another (COW).

Parameters:

• source_request_id (str) –

  Source request ID

• new_request_id (str) –

  New request ID

Returns:

• Optional[BlockTable] –

  Forked BlockTable, or None if source not found

Source code in vllm_mlx/prefix_cache.py

def fork_cache(
    self,
    source_request_id: str,
    new_request_id: str,
) -> Optional[BlockTable]:
    """
    Fork cache from one request to another (COW).

    Args:
        source_request_id: Source request ID
        new_request_id: New request ID

    Returns:
        Forked BlockTable, or None if source not found
    """
    source_entry = self._request_tables.get(source_request_id)
    if not source_entry:
        return None

    # Fork block table (increments ref counts)
    forked_table = self.paged_cache.fork_block_table(
        source_entry.block_table,
        new_request_id,
    )

    # Create new entry with reference to same cache data
    self._request_tables[new_request_id] = BlockCacheEntry(
        block_table=forked_table,
        cache_data=source_entry.cache_data,  # Shared reference
        last_access=time.time(),
    )

    logger.debug(f"Forked cache: {source_request_id} -> {new_request_id}")

    return forked_table

vllm_mlx.prefix_cache.BlockAwarePrefixCache.reconstruct_cache

reconstruct_cache(block_table: BlockTable) -> Optional[List[Any]]

Reconstruct cache objects from stored block tensor data.

Sequence-backed caches are concatenated block-by-block. Recurrent caches such as ArraysCache are restored from the latest sequence boundary snapshot that was actually stored.

Parameters:

• block_table (BlockTable) –

  BlockTable containing block IDs to reconstruct from

Returns:

• Optional[List[Any]] –

  List of reconstructed KVCache objects (one per layer),

• Optional[List[Any]] –

  or None if reconstruction fails

Source code in vllm_mlx/prefix_cache.py

def reconstruct_cache(
    self,
    block_table: BlockTable,
) -> Optional[List[Any]]:
    """
    Reconstruct cache objects from stored block tensor data.

    Sequence-backed caches are concatenated block-by-block. Recurrent
    caches such as ArraysCache are restored from the latest sequence
    boundary snapshot that was actually stored.

    Args:
        block_table: BlockTable containing block IDs to reconstruct from

    Returns:
        List of reconstructed KVCache objects (one per layer),
        or None if reconstruction fails
    """
    if not block_table or not block_table.block_ids:
        return None

    if not HAS_MLX:
        logger.warning("Cannot reconstruct cache: MLX not available")
        return None

    try:
        # Collect cache data from all blocks
        all_block_data = []
        for block_id in block_table.block_ids:
            block = self.paged_cache.allocated_blocks.get(block_id)
            if not block:
                logger.warning(f"Block {block_id} not found in allocated blocks")
                return None

            if block.cache_data is None:
                logger.debug(f"Block {block_id} has no tensor data stored")
                return None

            all_block_data.append(block.cache_data)

        if not all_block_data:
            return None

        # Get number of layers from the richest block
        num_layers = max(len(block_data) for block_data in all_block_data)
        if num_layers == 0:
            return None

        reconstructed_caches = []
        for layer_idx in range(num_layers):
            layer_entries = [
                block_data[layer_idx]
                for block_data in all_block_data
                if layer_idx < len(block_data)
            ]
            layer_entries = [entry for entry in layer_entries if entry is not None]
            if not layer_entries:
                return None

            layer_meta = layer_entries[-1]
            state = layer_meta["state"]
            if layer_meta["storage"] == "concat":
                state = self._concat_cache_states(
                    [entry["state"] for entry in layer_entries],
                    layer_meta["seq_axis"],
                )
            elif layer_meta["storage"] == "latest":
                state = layer_entries[-1]["state"]

            if state is None:
                return None

            cache_cls = layer_meta.get("class_ref")
            meta_state = layer_meta.get("meta_state")

            if cache_cls is not None and hasattr(cache_cls, "from_state"):
                from mlx_lm.models.cache import (
                    BatchKVCache as _BatchKVCache,
                    KVCache as _KVCache,
                )

                if cache_cls is _BatchKVCache:
                    keys, values = state[0], state[1]
                    cache = _KVCache()
                    cache.keys = keys
                    cache.values = values
                    cache.offset = keys.shape[self._cache_state_seq_axis(state)]
                else:
                    cache = cache_cls.from_state(state, meta_state)
            else:
                from mlx_lm.models.cache import KVCache

                if len(state) != 2:
                    return None
                cache = KVCache()
                cache.keys, cache.values = state
                seq_axis = self._cache_state_seq_axis(state)
                if seq_axis is None:
                    return None
                cache.offset = cache.keys.shape[seq_axis]

            reconstructed_caches.append(cache)

        if not reconstructed_caches:
            return None

        logger.debug(
            f"Reconstructed cache: {len(reconstructed_caches)} layers, "
            f"{block_table.num_tokens} tokens from {len(block_table.block_ids)} blocks"
        )

        return reconstructed_caches

    except Exception as e:
        logger.warning(f"Failed to reconstruct cache: {e}")
        import traceback

        logger.debug(traceback.format_exc())
        return None

vllm_mlx.prefix_cache.BlockAwarePrefixCache._find_best_prefix_match

_find_best_prefix_match(tokens: List[int]) -> Optional[Tuple[List[int], List[int]]]

Find best matching prefix in the index.

Source code in vllm_mlx/prefix_cache.py

def _find_best_prefix_match(
    self,
    tokens: List[int],
) -> Optional[Tuple[List[int], List[int]]]:
    """Find best matching prefix in the index."""
    best_match = None
    best_len = 0

    # Try progressively longer prefixes
    for num_blocks in range(1, len(tokens) // self.block_size + 1):
        prefix_len = num_blocks * self.block_size
        if prefix_len > len(tokens):
            break

        prefix_tokens = tokens[:prefix_len]
        prefix_hash = self.paged_cache.compute_block_hash(prefix_tokens)

        if prefix_hash in self._prefix_index:
            cached_tokens, block_ids = self._prefix_index[prefix_hash]
            if cached_tokens == prefix_tokens and len(cached_tokens) > best_len:
                best_match = (cached_tokens, block_ids)
                best_len = len(cached_tokens)

    return best_match

vllm_mlx.prefix_cache.BlockAwarePrefixCache._update_prefix_index

_update_prefix_index(tokens: List[int], block_ids: List[int]) -> None

Update prefix index with new token sequence.

Source code in vllm_mlx/prefix_cache.py

def _update_prefix_index(
    self,
    tokens: List[int],
    block_ids: List[int],
) -> None:
    """Update prefix index with new token sequence."""
    # Index block-aligned prefixes
    for i in range(1, len(block_ids) + 1):
        prefix_len = min(i * self.block_size, len(tokens))
        prefix_tokens = tokens[:prefix_len]
        prefix_hash = self.paged_cache.compute_block_hash(prefix_tokens)
        self._prefix_index[prefix_hash] = (prefix_tokens, block_ids[:i])

vllm_mlx.prefix_cache.BlockAwarePrefixCache.get_stats

get_stats() -> Dict[str, Any]

Get cache statistics.

Source code in vllm_mlx/prefix_cache.py

def get_stats(self) -> Dict[str, Any]:
    """Get cache statistics."""
    paged_stats = self.paged_cache.get_memory_usage()
    return {
        "hits": self._hits,
        "misses": self._misses,
        "hit_rate": (
            self._hits / (self._hits + self._misses)
            if (self._hits + self._misses) > 0
            else 0
        ),
        "tokens_saved": self._tokens_saved,
        "active_requests": len(self._request_tables),
        **paged_stats,
    }

vllm_mlx.prefix_cache.BlockAwarePrefixCache.reset_stats

reset_stats() -> None

Reset statistics.

Source code in vllm_mlx/prefix_cache.py

def reset_stats(self) -> None:
    """Reset statistics."""
    self._hits = 0
    self._misses = 0
    self._tokens_saved = 0
    self.paged_cache.reset_stats()

vllm_mlx.prefix_cache.BlockAwarePrefixCache.clear

clear() -> None

Clear all cached data.

Source code in vllm_mlx/prefix_cache.py

def clear(self) -> None:
    """Clear all cached data."""
    self._request_tables.clear()
    self._prefix_index.clear()
    self.paged_cache.clear()
    self.reset_stats()

vllm_mlx.prefix_cache.BlockAwarePrefixCache.__len__

__len__() -> int

Return number of active request entries.

Source code in vllm_mlx/prefix_cache.py

def __len__(self) -> int:
    """Return number of active request entries."""
    return len(self._request_tables)

Complete contract reference

Expand any definition for its exact inputs, annotations, defaults, return contract, directly raised exceptions, source-grounded behavior, and immutable line link. This section includes private and nested definitions that ordinary API generators omit.

vllm_mlx.prefix_cache.CacheEntry · class

vllm_mlx.prefix_cache.CacheEntry(prompt_cache: List[Any], count: int)

Entry in the prefix cache.

Parameters

Name	Type	Required	Default	Description
prompt_cache	List[Any]	yes	none	Required constructor field.
count	int	yes	none	Required constructor field.

Returns

• Constructs: vllm_mlx.prefix_cache.CacheEntry

Exceptions and behavior

Class CacheEntry declares 0 direct member(s). No direct raise statement appears in this definition.

View source #L33-L37.

vllm_mlx.prefix_cache.PrefixCacheStats · class

vllm_mlx.prefix_cache.PrefixCacheStats(hits: int = 0, misses: int = 0, tokens_saved: int = 0, total_queries: int = 0, evictions: int = 0)

Statistics for prefix cache performance.

Parameters

Name	Type	Required	Default	Description
hits	int	no	0	Optional constructor field; defaults to 0.
misses	int	no	0	Optional constructor field; defaults to 0.
tokens_saved	int	no	0	Optional constructor field; defaults to 0.
total_queries	int	no	0	Optional constructor field; defaults to 0.
evictions	int	no	0	Optional constructor field; defaults to 0.

Returns

• Constructs: vllm_mlx.prefix_cache.PrefixCacheStats

Exceptions and behavior

Class PrefixCacheStats declares 2 direct member(s). No direct raise statement appears in this definition.

View source #L41-L66.

vllm_mlx.prefix_cache.PrefixCacheStats.hit_rate · method

vllm_mlx.prefix_cache.PrefixCacheStats.hit_rate() -> float

Calculate cache hit rate.

Parameters

This callable has no explicit inputs.

Returns

• Type: float
• Direct return expressions: 0.0; self.hits / self.total_queries

Exceptions and behavior

Method PrefixCacheStats.hit_rate has 2 explicit return paths. No direct raise statement appears in this definition.

View source #L51-L55.

vllm_mlx.prefix_cache.PrefixCacheStats.to_dict · method

vllm_mlx.prefix_cache.PrefixCacheStats.to_dict() -> Dict[str, Any]

Convert stats to dictionary.

Parameters

This callable has no explicit inputs.

Returns

• Type: Dict[str, Any]
• Direct return expressions: {'hits': self.hits, 'misses': self.misses, 'hit_rate': self.hit_rate, 'tokens_saved': self.tokens_saved, 'total_queries…

Exceptions and behavior

Method PrefixCacheStats.to_dict returns {'hits': self.hits, 'misses': self.misses, 'hit_rate': self.hit_rate, 'tokens_saved': self.tokens_saved, 'total_queries…. No direct raise statement appears in this definition.

View source #L57-L66.

vllm_mlx.prefix_cache.PrefixCacheManager · class

vllm_mlx.prefix_cache.PrefixCacheManager(model: Any, max_entries: int = 100)

Manages prefix caching for vllm-mlx using a trie-based LRU cache.

Parameters

Name	Type	Required	Default	Description
model	Any	yes	none	The MLX model (used for cache key identification)
max_entries	int	no	100	Maximum number of cached entries before LRU eviction

Returns

• Constructs: vllm_mlx.prefix_cache.PrefixCacheManager

Exceptions and behavior

Class PrefixCacheManager declares 14 direct member(s). No direct raise statement appears in this definition.

View source #L69-L355.

vllm_mlx.prefix_cache.PrefixCacheManager.__init__ · method

vllm_mlx.prefix_cache.PrefixCacheManager.__init__(model: Any, max_entries: int = 100) -> not annotated

Initialize the prefix cache manager.

Parameters

Name	Type	Required	Default	Description
model	Any	yes	none	The MLX model (used for cache key identification)
max_entries	int	no	100	Maximum number of cached entries before LRU eviction

Returns

• Type: not annotated

Exceptions and behavior

Method PrefixCacheManager.__init__ updates self.model, self.model_key, self.max_size, self._cache; calls id, OrderedDict, PrefixCacheStats. No direct raise statement appears in this definition.

View source #L94-L115.

vllm_mlx.prefix_cache.PrefixCacheManager._search · method

vllm_mlx.prefix_cache.PrefixCacheManager._search(tokens: List[int]) -> Tuple[Optional[List[int]], Optional[List[int]], Optional[List[int]], int]

Search for cached prefix matching tokens.

Parameters

Name	Type	Required	Default	Description
tokens	List[int]	yes	none	Required positional or keyword input.

Returns

• Type: Tuple[Optional[List[int]], Optional[List[int]], Optional[List[int]], int]
• Direct return expressions: (None, None, None, 0); (None, list(path), None, 0); (list(tokens), None, None, 0); (None, None, node_path, len(tokens))

Exceptions and behavior

Method PrefixCacheManager._search calls enumerate, list, path.append, stack.pop; has 4 explicit return paths. No direct raise statement appears in this definition.

View source #L117-L164.

vllm_mlx.prefix_cache.PrefixCacheManager.fetch_cache · method

vllm_mlx.prefix_cache.PrefixCacheManager.fetch_cache(tokens: List[int]) -> Tuple[Optional[List[Any]], List[int]]

Find cached prefix for the given tokens.

Parameters

Name	Type	Required	Default	Description
tokens	List[int]	yes	none	Input token sequence

Returns

• Type: Tuple[Optional[List[Any]], List[int]]
• Direct return expressions: (cache_entry.prompt_cache, []); (cache_entry.prompt_cache, remaining); (trimmed_cache, []); (None, tokens)

Exceptions and behavior

Method PrefixCacheManager.fetch_cache updates self.stats.total_queries, self.stats.hits, self.stats.tokens_saved, self.stats.misses; calls tuple, self._search, self._get_cache_entry, len; has 4 explicit return paths. No direct raise statement appears in this definition.

View source #L166-L221.

vllm_mlx.prefix_cache.PrefixCacheManager.store_cache · method

vllm_mlx.prefix_cache.PrefixCacheManager.store_cache(tokens: List[int], prompt_cache: List[Any]) -> None

Store computed cache for future reuse.

Parameters

Name	Type	Required	Default	Description
tokens	List[int]	yes	none	Token sequence that was processed
prompt_cache	List[Any]	yes	none	The computed KV cache to store

Returns

• Type: None
• Direct return expressions: None

Exceptions and behavior

Method PrefixCacheManager.store_cache calls tuple, self._lru.move_to_end, CacheEntry, len; returns None. No direct raise statement appears in this definition.

View source #L223-L258.

vllm_mlx.prefix_cache.PrefixCacheManager._get_cache_entry · method

vllm_mlx.prefix_cache.PrefixCacheManager._get_cache_entry(tokens: List[int]) -> Optional[CacheEntry]

Get cache entry for given tokens.

Parameters

Name	Type	Required	Default	Description
tokens	List[int]	yes	none	Required positional or keyword input.

Returns

• Type: Optional[CacheEntry]
• Direct return expressions: None; current.get('cache')

Exceptions and behavior

Method PrefixCacheManager._get_cache_entry calls current.get; has 2 explicit return paths. No direct raise statement appears in this definition.

View source #L260-L271.

vllm_mlx.prefix_cache.PrefixCacheManager._touch_lru · method

vllm_mlx.prefix_cache.PrefixCacheManager._touch_lru(tokens_tuple: tuple) -> None

Move entry to most-recently-used position — O(1) with OrderedDict.

Parameters

Name	Type	Required	Default	Description
tokens_tuple	tuple	yes	none	Required positional or keyword input.

Returns

• Type: None

Exceptions and behavior

Method PrefixCacheManager._touch_lru calls self._lru.move_to_end. No direct raise statement appears in this definition.

View source #L273-L279.

vllm_mlx.prefix_cache.PrefixCacheManager._evict_lru · method

vllm_mlx.prefix_cache.PrefixCacheManager._evict_lru() -> None

Evict least recently used entry — O(1) popitem from OrderedDict.

Parameters

This callable has no explicit inputs.

Returns

• Type: None
• Direct return expressions: None

Exceptions and behavior

Method PrefixCacheManager._evict_lru updates self.stats.evictions; calls self._lru.popitem, self._delete_cache, list; returns None. No direct raise statement appears in this definition.

View source #L281-L288.

vllm_mlx.prefix_cache.PrefixCacheManager._delete_cache · method

vllm_mlx.prefix_cache.PrefixCacheManager._delete_cache(model_key: Any, tokens: List[int]) -> None

Delete cache entry and clean up empty trie branches.

Parameters

Name	Type	Required	Default	Description
model_key	Any	yes	none	Required positional or keyword input.
tokens	List[int]	yes	none	Required positional or keyword input.

Returns

• Type: None
• Direct return expressions: None

Exceptions and behavior

Method PrefixCacheManager._delete_cache calls path.append, range, len; returns None. No direct raise statement appears in this definition.

View source #L290-L314.

vllm_mlx.prefix_cache.PrefixCacheManager._can_trim_cache · method

vllm_mlx.prefix_cache.PrefixCacheManager._can_trim_cache(prompt_cache: List[Any]) -> bool

Check if cache can be trimmed.

Parameters

Name	Type	Required	Default	Description
prompt_cache	List[Any]	yes	none	Required positional or keyword input.

Returns

• Type: bool
• Direct return expressions: False; trimmable; hasattr(first_cache, 'trim')

Exceptions and behavior

Method PrefixCacheManager._can_trim_cache calls hasattr, first_cache.is_trimmable, logger.debug; has 3 explicit return paths. No direct raise statement appears in this definition.

View source #L316-L330.

vllm_mlx.prefix_cache.PrefixCacheManager._trim_cache · method

vllm_mlx.prefix_cache.PrefixCacheManager._trim_cache(prompt_cache: List[Any], num_tokens: int) -> List[Any]

Trim cache by removing num_tokens from the end.

Parameters

Name	Type	Required	Default	Description
prompt_cache	List[Any]	yes	none	Required positional or keyword input.
num_tokens	int	yes	none	Required positional or keyword input.

Returns

• Type: List[Any]
• Direct return expressions: prompt_cache

Exceptions and behavior

Method PrefixCacheManager._trim_cache calls hasattr, cache.trim; returns prompt_cache. No direct raise statement appears in this definition.

View source #L332-L337.

vllm_mlx.prefix_cache.PrefixCacheManager.get_stats · method

vllm_mlx.prefix_cache.PrefixCacheManager.get_stats() -> Dict[str, Any]

Get cache statistics.

Parameters

This callable has no explicit inputs.

Returns

• Type: Dict[str, Any]
• Direct return expressions: self.stats.to_dict()

Exceptions and behavior

Method PrefixCacheManager.get_stats calls self.stats.to_dict; returns self.stats.to_dict(). No direct raise statement appears in this definition.

View source #L339-L341.

vllm_mlx.prefix_cache.PrefixCacheManager.reset_stats · method

vllm_mlx.prefix_cache.PrefixCacheManager.reset_stats() -> None

Reset statistics.

Parameters

This callable has no explicit inputs.

Returns

• Type: None

Exceptions and behavior

Method PrefixCacheManager.reset_stats updates self.stats; calls PrefixCacheStats. No direct raise statement appears in this definition.

View source #L343-L345.

vllm_mlx.prefix_cache.PrefixCacheManager.clear · method

vllm_mlx.prefix_cache.PrefixCacheManager.clear() -> None

Clear all cached entries.

Parameters

This callable has no explicit inputs.

Returns

• Type: None

Exceptions and behavior

Method PrefixCacheManager.clear calls self._cache.clear, self._lru.clear, self.reset_stats. No direct raise statement appears in this definition.

View source #L347-L351.

vllm_mlx.prefix_cache.PrefixCacheManager.__len__ · method

vllm_mlx.prefix_cache.PrefixCacheManager.__len__() -> int

Return number of cached entries.

Parameters

This callable has no explicit inputs.

Returns

• Type: int
• Direct return expressions: len(self._lru)

Exceptions and behavior

Method PrefixCacheManager.__len__ calls len; returns len(self._lru). No direct raise statement appears in this definition.

View source #L353-L355.

vllm_mlx.prefix_cache.BlockCacheEntry · class

vllm_mlx.prefix_cache.BlockCacheEntry(block_table: BlockTable, cache_data: List[Any], last_access: float)

Entry mapping a token sequence to cache blocks.

Parameters

Name	Type	Required	Default	Description
block_table	BlockTable	yes	none	Required constructor field.
cache_data	List[Any]	yes	none	Required constructor field.
last_access	float	yes	none	Required constructor field.

Returns

• Constructs: vllm_mlx.prefix_cache.BlockCacheEntry

Exceptions and behavior

Class BlockCacheEntry declares 0 direct member(s). No direct raise statement appears in this definition.

View source #L364-L369.

vllm_mlx.prefix_cache.BlockAwarePrefixCache · class

vllm_mlx.prefix_cache.BlockAwarePrefixCache(model: Any, paged_cache_manager: PagedCacheManager)

Prefix cache that uses PagedCacheManager for block-based storage.

Parameters

Name	Type	Required	Default	Description
model	Any	yes	none	The MLX model (used for identification)
paged_cache_manager	PagedCacheManager	yes	none	The PagedCacheManager instance for block management

Returns

• Constructs: vllm_mlx.prefix_cache.BlockAwarePrefixCache

Exceptions and behavior

Class BlockAwarePrefixCache declares 17 direct member(s). No direct raise statement appears in this definition.

View source #L372-L1039.

vllm_mlx.prefix_cache.BlockAwarePrefixCache.__init__ · method

vllm_mlx.prefix_cache.BlockAwarePrefixCache.__init__(model: Any, paged_cache_manager: PagedCacheManager) -> not annotated

Initialize block-aware prefix cache.

Parameters

Name	Type	Required	Default	Description
model	Any	yes	none	The MLX model (used for identification)
paged_cache_manager	PagedCacheManager	yes	none	The PagedCacheManager instance for block management

Returns

• Type: not annotated

Exceptions and behavior

Method BlockAwarePrefixCache.__init__ updates self.model, self.model_key, self.paged_cache, self.block_size; calls id. No direct raise statement appears in this definition.

View source #L399-L426.

vllm_mlx.prefix_cache.BlockAwarePrefixCache.fetch_cache · method

vllm_mlx.prefix_cache.BlockAwarePrefixCache.fetch_cache(request_id: str, tokens: List[int]) -> Tuple[Optional[BlockTable], List[int]]

Find cached prefix blocks for the given tokens.

Parameters

Name	Type	Required	Default	Description
request_id	str	yes	none	Unique request identifier
tokens	List[int]	yes	none	Input token sequence

Returns

• Type: Tuple[Optional[BlockTable], List[int]]
• Direct return expressions: (None, tokens); (block_table, remaining)

Exceptions and behavior

Method BlockAwarePrefixCache.fetch_cache updates self._hits, self._tokens_saved, self._misses; calls self.paged_cache.find_shared_prefix, self.paged_cache.create_block_table, self.paged_cache.increment_ref, self.paged_cache.allocated_blocks.get; has 2 explicit return paths. No direct raise statement appears in this definition.

View source #L428-L502.

vllm_mlx.prefix_cache.BlockAwarePrefixCache.store_cache · method

vllm_mlx.prefix_cache.BlockAwarePrefixCache.store_cache(request_id: str, tokens: List[int], cache_data: List[Any]) -> Optional[BlockTable]

Store computed cache for future reuse.

Parameters

Name	Type	Required	Default	Description
request_id	str	yes	none	Unique request identifier
tokens	List[int]	yes	none	Token sequence that was processed
cache_data	List[Any]	yes	none	The computed KV cache to store. Can be: - List of KVCache objects (legacy, stores references) - List of dicts with 'state': (keys, values) tensors (new, stores slices)

Returns

• Type: Optional[BlockTable]
• Direct return expressions: None; block_table

Exceptions and behavior

Method BlockAwarePrefixCache.store_cache calls isinstance, len, self.paged_cache.get_block_table, self.paged_cache.create_block_table; has 2 explicit return paths. No direct raise statement appears in this definition.

View source #L504-L628.

vllm_mlx.prefix_cache.BlockAwarePrefixCache._extract_block_tensor_slice · method

vllm_mlx.prefix_cache.BlockAwarePrefixCache._extract_block_tensor_slice(cache_data: List[Dict[str, Any]], start_idx: int, end_idx: int, total_tokens: int) -> Optional[List[Optional[Dict[str, Any]]]]

Extract per-layer cache data for a single block.

Parameters

Name	Type	Required	Default	Description
cache_data	List[Dict[str, Any]]	yes	none	List of extracted layer states
start_idx	int	yes	none	Start token index in the sequence
end_idx	int	yes	none	End token index in the sequence
total_tokens	int	yes	none	Total number of tokens covered by cache_data

Returns

• Type: Optional[List[Optional[Dict[str, Any]]]]
• Direct return expressions: None; block_slices if any((entry is not None for entry in block_slices)) else None

Exceptions and behavior

Method BlockAwarePrefixCache._extract_block_tensor_slice calls block_slices.append, layer_state.get, self._cache_state_seq_axis, self._slice_concat_cache_state; has 2 explicit return paths. No direct raise statement appears in this definition.

View source #L630-L702.

vllm_mlx.prefix_cache.BlockAwarePrefixCache._cache_state_seq_axis · method

vllm_mlx.prefix_cache.BlockAwarePrefixCache._cache_state_seq_axis(state: Any) -> Optional[int]

Return the sequence axis for cache states that support block concat.

Parameters

Name	Type	Required	Default	Description
state	Any	yes	none	Required positional or keyword input.

Returns

• Type: Optional[int]
• Direct return expressions: None; 2; 1

Exceptions and behavior

Method BlockAwarePrefixCache._cache_state_seq_axis calls isinstance, len, hasattr, next; has 3 explicit return paths. No direct raise statement appears in this definition.

View source #L704-L725.

vllm_mlx.prefix_cache.BlockAwarePrefixCache._slice_concat_cache_state · method

vllm_mlx.prefix_cache.BlockAwarePrefixCache._slice_concat_cache_state(state: Tuple[Any, ...] | List[Any], start_idx: int, end_idx: int) -> Tuple[Any, ...] | List[Any]

Slice a sequence-backed cache state across the token axis.

Parameters

Name	Type	Required	Default	Description
state	Tuple[Any, ...] \| List[Any]	yes	none	Required positional or keyword input.
start_idx	int	yes	none	Required positional or keyword input.
end_idx	int	yes	none	Required positional or keyword input.

Returns

• Type: Tuple[Any, ...] | List[Any]
• Direct return expressions: tuple(sliced) if isinstance(state, tuple) else sliced

Exceptions and behavior

Method BlockAwarePrefixCache._slice_concat_cache_state calls self._cache_state_seq_axis, ValueError, min, _slice_tensor; can raise ValueError; returns tuple(sliced) if isinstance(state, tuple) else sliced. Directly raised exceptions: ValueError.

View source #L727-L751.

vllm_mlx.prefix_cache.BlockAwarePrefixCache._slice_concat_cache_state._slice_tensor · nested function

vllm_mlx.prefix_cache.BlockAwarePrefixCache._slice_concat_cache_state._slice_tensor(tensor: Any) -> Any

Nested Function BlockAwarePrefixCache._slice_concat_cache_state._slice_tensor calls slice, len, tuple; returns tensor[tuple(slices)].

Parameters

Name	Type	Required	Default	Description
tensor	Any	yes	none	Required positional or keyword input.

Returns

• Type: Any
• Direct return expressions: tensor[tuple(slices)]

Exceptions and behavior

Nested Function BlockAwarePrefixCache._slice_concat_cache_state._slice_tensor calls slice, len, tuple; returns tensor[tuple(slices)]. No direct raise statement appears in this definition.

View source #L745-L748.

vllm_mlx.prefix_cache.BlockAwarePrefixCache._concat_cache_states · method

vllm_mlx.prefix_cache.BlockAwarePrefixCache._concat_cache_states(states: List[Tuple[Any, ...] | List[Any]], seq_axis: int) -> Optional[Tuple[Any, ...] | List[Any]]

Concatenate state fragments for a sequence-backed cache layer.

Parameters

Name	Type	Required	Default	Description
states	List[Tuple[Any, ...] \| List[Any]]	yes	none	Required positional or keyword input.
seq_axis	int	yes	none	Required positional or keyword input.

Returns

• Type: Optional[Tuple[Any, ...] | List[Any]]
• Direct return expressions: None; tuple(concatenated) if isinstance(states[0], tuple) else concatenated

Exceptions and behavior

Method BlockAwarePrefixCache._concat_cache_states calls len, range, any, concatenated.append; has 2 explicit return paths. No direct raise statement appears in this definition.

View source #L753-L768.

vllm_mlx.prefix_cache.BlockAwarePrefixCache.get_cache_for_generation · method

vllm_mlx.prefix_cache.BlockAwarePrefixCache.get_cache_for_generation(request_id: str) -> Tuple[Optional[List[Any]], bool]

Get cache data for generation, applying COW if needed.

Parameters

Name	Type	Required	Default	Description
request_id	str	yes	none	Request identifier

Returns

• Type: Tuple[Optional[List[Any]], bool]
• Direct return expressions: (None, False); (cache_data, was_copied)

Exceptions and behavior

Method BlockAwarePrefixCache.get_cache_for_generation calls self._request_tables.get, self.paged_cache.get_blocks_for_generation, copy.deepcopy, time.time; has 2 explicit return paths. No direct raise statement appears in this definition.

View source #L770-L799.

vllm_mlx.prefix_cache.BlockAwarePrefixCache.release_cache · method

vllm_mlx.prefix_cache.BlockAwarePrefixCache.release_cache(request_id: str) -> None

Release cache blocks for a completed request.

Parameters

Name	Type	Required	Default	Description
request_id	str	yes	none	Request identifier

Returns

• Type: None

Exceptions and behavior

Method BlockAwarePrefixCache.release_cache calls self._request_tables.pop, self.paged_cache.delete_block_table, logger.debug. No direct raise statement appears in this definition.

View source #L801-L811.

vllm_mlx.prefix_cache.BlockAwarePrefixCache.fork_cache · method

vllm_mlx.prefix_cache.BlockAwarePrefixCache.fork_cache(source_request_id: str, new_request_id: str) -> Optional[BlockTable]

Fork cache from one request to another (COW).

Parameters

Name	Type	Required	Default	Description
source_request_id	str	yes	none	Source request ID
new_request_id	str	yes	none	New request ID

Returns

• Type: Optional[BlockTable]
• Direct return expressions: None; forked_table

Exceptions and behavior

Method BlockAwarePrefixCache.fork_cache calls self._request_tables.get, self.paged_cache.fork_block_table, BlockCacheEntry, time.time; has 2 explicit return paths. No direct raise statement appears in this definition.

View source #L813-L847.

vllm_mlx.prefix_cache.BlockAwarePrefixCache.reconstruct_cache · method

vllm_mlx.prefix_cache.BlockAwarePrefixCache.reconstruct_cache(block_table: BlockTable) -> Optional[List[Any]]

Reconstruct cache objects from stored block tensor data.

Parameters

Name	Type	Required	Default	Description
block_table	BlockTable	yes	none	BlockTable containing block IDs to reconstruct from

Returns

• Type: Optional[List[Any]]
• Direct return expressions: None; reconstructed_caches

Exceptions and behavior

Method BlockAwarePrefixCache.reconstruct_cache calls logger.warning, self.paged_cache.allocated_blocks.get, logger.debug, all_block_data.append; has 2 explicit return paths. No direct raise statement appears in this definition.

View source #L849-L967.

vllm_mlx.prefix_cache.BlockAwarePrefixCache._find_best_prefix_match · method

vllm_mlx.prefix_cache.BlockAwarePrefixCache._find_best_prefix_match(tokens: List[int]) -> Optional[Tuple[List[int], List[int]]]

Find best matching prefix in the index.

Parameters

Name	Type	Required	Default	Description
tokens	List[int]	yes	none	Required positional or keyword input.

Returns

• Type: Optional[Tuple[List[int], List[int]]]
• Direct return expressions: best_match

Exceptions and behavior

Method BlockAwarePrefixCache._find_best_prefix_match calls range, len, self.paged_cache.compute_block_hash; returns best_match. No direct raise statement appears in this definition.

View source #L969-L992.

vllm_mlx.prefix_cache.BlockAwarePrefixCache._update_prefix_index · method

vllm_mlx.prefix_cache.BlockAwarePrefixCache._update_prefix_index(tokens: List[int], block_ids: List[int]) -> None

Update prefix index with new token sequence.

Parameters

Name	Type	Required	Default	Description
tokens	List[int]	yes	none	Required positional or keyword input.
block_ids	List[int]	yes	none	Required positional or keyword input.

Returns

• Type: None

Exceptions and behavior

Method BlockAwarePrefixCache._update_prefix_index calls range, len, min, self.paged_cache.compute_block_hash. No direct raise statement appears in this definition.

View source #L994-L1005.

vllm_mlx.prefix_cache.BlockAwarePrefixCache.get_stats · method

vllm_mlx.prefix_cache.BlockAwarePrefixCache.get_stats() -> Dict[str, Any]

Get cache statistics.

Parameters

This callable has no explicit inputs.

Returns

• Type: Dict[str, Any]
• Direct return expressions: {'hits': self._hits, 'misses': self._misses, 'hit_rate': self._hits / (self._hits + self._misses) if self._hits + self.…

Exceptions and behavior

Method BlockAwarePrefixCache.get_stats calls self.paged_cache.get_memory_usage, len; returns {'hits': self._hits, 'misses': self._misses, 'hit_rate': self._hits / (self._hits + self._misses) if self._hits + self.…. No direct raise statement appears in this definition.

View source #L1007-L1021.

vllm_mlx.prefix_cache.BlockAwarePrefixCache.reset_stats · method

vllm_mlx.prefix_cache.BlockAwarePrefixCache.reset_stats() -> None

Reset statistics.

Parameters

This callable has no explicit inputs.

Returns

• Type: None

Exceptions and behavior

Method BlockAwarePrefixCache.reset_stats updates self._hits, self._misses, self._tokens_saved; calls self.paged_cache.reset_stats. No direct raise statement appears in this definition.

View source #L1023-L1028.

vllm_mlx.prefix_cache.BlockAwarePrefixCache.clear · method

vllm_mlx.prefix_cache.BlockAwarePrefixCache.clear() -> None

Clear all cached data.

Parameters

This callable has no explicit inputs.

Returns

• Type: None

Exceptions and behavior

Method BlockAwarePrefixCache.clear calls self._request_tables.clear, self._prefix_index.clear, self.paged_cache.clear, self.reset_stats. No direct raise statement appears in this definition.

View source #L1030-L1035.

vllm_mlx.prefix_cache.BlockAwarePrefixCache.__len__ · method

vllm_mlx.prefix_cache.BlockAwarePrefixCache.__len__() -> int

Return number of active request entries.

Parameters

This callable has no explicit inputs.

Returns

• Type: int
• Direct return expressions: len(self._request_tables)

Exceptions and behavior

Method BlockAwarePrefixCache.__len__ calls len; returns len(self._request_tables). No direct raise statement appears in this definition.

View source #L1037-L1039.

Complete symbol map

This map also includes private definitions and nested helpers. The signature column exposes every explicit input even when an internal helper has no dedicated parameter prose.

Symbol	Kind	Signature and inputs	What it does	Source
CacheEntry	class	CacheEntry(prompt_cache: List[Any], count: int)	Entry in the prefix cache.	#L33-L37
PrefixCacheStats	class	PrefixCacheStats(hits: int = 0, misses: int = 0, tokens_saved: int = 0, total_queries: int = 0, evictions: int = 0)	Statistics for prefix cache performance.	#L41-L66
PrefixCacheStats.hit_rate	method	PrefixCacheStats.hit_rate() -> float	Calculate cache hit rate.	#L51-L55
PrefixCacheStats.to_dict	method	PrefixCacheStats.to_dict() -> Dict[str, Any]	Convert stats to dictionary.	#L57-L66
PrefixCacheManager	class	PrefixCacheManager(model: Any, max_entries: int = 100)	Manages prefix caching for vllm-mlx using a trie-based LRU cache.	#L69-L355
PrefixCacheManager.__init__	method	PrefixCacheManager.__init__(model: Any, max_entries: int = 100) -> not annotated	Initialize the prefix cache manager.	#L94-L115
PrefixCacheManager._search	method	PrefixCacheManager._search(tokens: List[int]) -> Tuple[Optional[List[int]], Optional[List[int]], Optional[List[int]], int]	Search for cached prefix matching tokens.	#L117-L164
PrefixCacheManager.fetch_cache	method	PrefixCacheManager.fetch_cache(tokens: List[int]) -> Tuple[Optional[List[Any]], List[int]]	Find cached prefix for the given tokens.	#L166-L221
PrefixCacheManager.store_cache	method	PrefixCacheManager.store_cache(tokens: List[int], prompt_cache: List[Any]) -> None	Store computed cache for future reuse.	#L223-L258
PrefixCacheManager._get_cache_entry	method	PrefixCacheManager._get_cache_entry(tokens: List[int]) -> Optional[CacheEntry]	Get cache entry for given tokens.	#L260-L271
PrefixCacheManager._touch_lru	method	PrefixCacheManager._touch_lru(tokens_tuple: tuple) -> None	Move entry to most-recently-used position — O(1) with OrderedDict.	#L273-L279
PrefixCacheManager._evict_lru	method	PrefixCacheManager._evict_lru() -> None	Evict least recently used entry — O(1) popitem from OrderedDict.	#L281-L288
PrefixCacheManager._delete_cache	method	PrefixCacheManager._delete_cache(model_key: Any, tokens: List[int]) -> None	Delete cache entry and clean up empty trie branches.	#L290-L314
PrefixCacheManager._can_trim_cache	method	PrefixCacheManager._can_trim_cache(prompt_cache: List[Any]) -> bool	Check if cache can be trimmed.	#L316-L330
PrefixCacheManager._trim_cache	method	PrefixCacheManager._trim_cache(prompt_cache: List[Any], num_tokens: int) -> List[Any]	Trim cache by removing num_tokens from the end.	#L332-L337
PrefixCacheManager.get_stats	method	PrefixCacheManager.get_stats() -> Dict[str, Any]	Get cache statistics.	#L339-L341
PrefixCacheManager.reset_stats	method	PrefixCacheManager.reset_stats() -> None	Reset statistics.	#L343-L345
PrefixCacheManager.clear	method	PrefixCacheManager.clear() -> None	Clear all cached entries.	#L347-L351
PrefixCacheManager.__len__	method	PrefixCacheManager.__len__() -> int	Return number of cached entries.	#L353-L355
BlockCacheEntry	class	BlockCacheEntry(block_table: BlockTable, cache_data: List[Any], last_access: float)	Entry mapping a token sequence to cache blocks.	#L364-L369
BlockAwarePrefixCache	class	BlockAwarePrefixCache(model: Any, paged_cache_manager: PagedCacheManager)	Prefix cache that uses PagedCacheManager for block-based storage.	#L372-L1039
BlockAwarePrefixCache.__init__	method	BlockAwarePrefixCache.__init__(model: Any, paged_cache_manager: PagedCacheManager) -> not annotated	Initialize block-aware prefix cache.	#L399-L426
BlockAwarePrefixCache.fetch_cache	method	BlockAwarePrefixCache.fetch_cache(request_id: str, tokens: List[int]) -> Tuple[Optional[BlockTable], List[int]]	Find cached prefix blocks for the given tokens.	#L428-L502
BlockAwarePrefixCache.store_cache	method	BlockAwarePrefixCache.store_cache(request_id: str, tokens: List[int], cache_data: List[Any]) -> Optional[BlockTable]	Store computed cache for future reuse.	#L504-L628
BlockAwarePrefixCache._extract_block_tensor_slice	method	BlockAwarePrefixCache._extract_block_tensor_slice(cache_data: List[Dict[str, Any]], start_idx: int, end_idx: int, total_tokens: int) -> Optional[List[Optional[Dict[str, Any]]]]	Extract per-layer cache data for a single block.	#L630-L702
BlockAwarePrefixCache._cache_state_seq_axis	method	BlockAwarePrefixCache._cache_state_seq_axis(state: Any) -> Optional[int]	Return the sequence axis for cache states that support block concat.	#L704-L725
BlockAwarePrefixCache._slice_concat_cache_state	method	BlockAwarePrefixCache._slice_concat_cache_state(state: Tuple[Any, ...] \| List[Any], start_idx: int, end_idx: int) -> Tuple[Any, ...] \| List[Any]	Slice a sequence-backed cache state across the token axis.	#L727-L751
BlockAwarePrefixCache._slice_concat_cache_state._slice_tensor	nested function	BlockAwarePrefixCache._slice_concat_cache_state._slice_tensor(tensor: Any) -> Any	Nested Function BlockAwarePrefixCache._slice_concat_cache_state._slice_tensor calls slice, len, tuple; returns tensor[tuple(slices)].	#L745-L748
BlockAwarePrefixCache._concat_cache_states	method	BlockAwarePrefixCache._concat_cache_states(states: List[Tuple[Any, ...] \| List[Any]], seq_axis: int) -> Optional[Tuple[Any, ...] \| List[Any]]	Concatenate state fragments for a sequence-backed cache layer.	#L753-L768
BlockAwarePrefixCache.get_cache_for_generation	method	BlockAwarePrefixCache.get_cache_for_generation(request_id: str) -> Tuple[Optional[List[Any]], bool]	Get cache data for generation, applying COW if needed.	#L770-L799
BlockAwarePrefixCache.release_cache	method	BlockAwarePrefixCache.release_cache(request_id: str) -> None	Release cache blocks for a completed request.	#L801-L811
BlockAwarePrefixCache.fork_cache	method	BlockAwarePrefixCache.fork_cache(source_request_id: str, new_request_id: str) -> Optional[BlockTable]	Fork cache from one request to another (COW).	#L813-L847
BlockAwarePrefixCache.reconstruct_cache	method	BlockAwarePrefixCache.reconstruct_cache(block_table: BlockTable) -> Optional[List[Any]]	Reconstruct cache objects from stored block tensor data.	#L849-L967
BlockAwarePrefixCache._find_best_prefix_match	method	BlockAwarePrefixCache._find_best_prefix_match(tokens: List[int]) -> Optional[Tuple[List[int], List[int]]]	Find best matching prefix in the index.	#L969-L992
BlockAwarePrefixCache._update_prefix_index	method	BlockAwarePrefixCache._update_prefix_index(tokens: List[int], block_ids: List[int]) -> None	Update prefix index with new token sequence.	#L994-L1005
BlockAwarePrefixCache.get_stats	method	BlockAwarePrefixCache.get_stats() -> Dict[str, Any]	Get cache statistics.	#L1007-L1021
BlockAwarePrefixCache.reset_stats	method	BlockAwarePrefixCache.reset_stats() -> None	Reset statistics.	#L1023-L1028
BlockAwarePrefixCache.clear	method	BlockAwarePrefixCache.clear() -> None	Clear all cached data.	#L1030-L1035
BlockAwarePrefixCache.__len__	method	BlockAwarePrefixCache.__len__() -> int	Return number of active request entries.	#L1037-L1039