vllm_mlx.paged_cache

Paged KV Cache Manager for vllm-mlx.

View the complete module source at #L1-L1197.

API details

Each callable below includes its exact signature, type annotations, inputs, defaults, return contract, documented exceptions, implementation source, and parsed docstring sections when the source provides them.

vllm_mlx.paged_cache

Paged KV Cache Manager for vllm-mlx.

This module implements block-based paged KV cache management following vLLM's architecture (vllm/v1/core/block_pool.py), adapted for MLX on Apple Silicon.

Key components: - KVCacheBlock: Metadata for each cache block with doubly linked list pointers - FreeKVCacheBlockQueue: O(1) doubly linked list for LRU block allocation - BlockHashToBlockMap: Hash-to-block cache for prefix caching - PagedCacheManager: Main manager with block allocation, prefix caching, and COW

Features: - Block-based allocation (configurable tokens per block) - Reference counting for shared blocks - Copy-on-Write (COW) for efficient prefix sharing - LRU eviction using doubly linked list (O(1) operations) - Chain hashing for prefix caching (hash depends on parent block)

Reference: vLLM v1 - vllm/v1/core/block_pool.py, vllm/v1/core/kv_cache_utils.py

vllm_mlx.paged_cache.logger module-attribute

logger = logging.getLogger(__name__)

vllm_mlx.paged_cache.BlockHash module-attribute

BlockHash = NewType('BlockHash', bytes)

vllm_mlx.paged_cache.KVCacheBlock module-attribute

KVCacheBlock = CacheBlock

vllm_mlx.paged_cache.CacheBlock dataclass

CacheBlock(block_id: int, ref_count: int = 0, block_hash: Optional[BlockHash] = None, prev_free_block: Optional['CacheBlock'] = None, next_free_block: Optional['CacheBlock'] = None, is_null: bool = False, cache_data: Optional[List[Tuple[Any, Any]]] = None, token_count: int = 0, hash_value: Optional[str] = None, last_access: float = time())

KV cache block metadata following vLLM's design.

Each block represents a fixed number of tokens (block_size) worth of KV cache data. Blocks can be shared across requests via reference counting for prefix caching.

Attributes:

• block_id (int) –

  Physical block index (0 to num_blocks - 1)

• ref_count (int) –

  Reference count for sharing (0 = can be evicted)

• block_hash (Optional[BlockHash]) –

  Content hash for prefix caching (None if not cached)

• prev_free_block (Optional['CacheBlock']) –

  Previous block in free list (doubly linked)

• next_free_block (Optional['CacheBlock']) –

  Next block in free list (doubly linked)

• is_null (bool) –

  True if this is the null/placeholder block

• cache_data (Optional[List[Tuple[Any, Any]]]) –

  Actual KV tensor data stored in this block

• token_count (int) –

  Number of tokens stored in this block

vllm_mlx.paged_cache.CacheBlock.block_id instance-attribute

block_id: int

vllm_mlx.paged_cache.CacheBlock.ref_count class-attribute instance-attribute

ref_count: int = 0

vllm_mlx.paged_cache.CacheBlock.block_hash class-attribute instance-attribute

block_hash: Optional[BlockHash] = None

vllm_mlx.paged_cache.CacheBlock.prev_free_block class-attribute instance-attribute

prev_free_block: Optional['CacheBlock'] = None

vllm_mlx.paged_cache.CacheBlock.next_free_block class-attribute instance-attribute

next_free_block: Optional['CacheBlock'] = None

vllm_mlx.paged_cache.CacheBlock.is_null class-attribute instance-attribute

is_null: bool = False

vllm_mlx.paged_cache.CacheBlock.cache_data class-attribute instance-attribute

cache_data: Optional[List[Tuple[Any, Any]]] = None

vllm_mlx.paged_cache.CacheBlock.token_count class-attribute instance-attribute

token_count: int = 0

vllm_mlx.paged_cache.CacheBlock.hash_value class-attribute instance-attribute

hash_value: Optional[str] = None

vllm_mlx.paged_cache.CacheBlock.last_access class-attribute instance-attribute

last_access: float = field(default_factory=time.time)

vllm_mlx.paged_cache.CacheBlock.is_full

is_full(block_size: int) -> bool

Check if block is at capacity.

Source code in vllm_mlx/paged_cache.py

def is_full(self, block_size: int) -> bool:
    """Check if block is at capacity."""
    return self.token_count >= block_size

vllm_mlx.paged_cache.CacheBlock.is_shared

is_shared() -> bool

Check if block is shared (ref_count > 1).

Source code in vllm_mlx/paged_cache.py

def is_shared(self) -> bool:
    """Check if block is shared (ref_count > 1)."""
    return self.ref_count > 1

vllm_mlx.paged_cache.CacheBlock.reset_hash

reset_hash() -> None

Reset block hash when evicted from cache.

Source code in vllm_mlx/paged_cache.py

def reset_hash(self) -> None:
    """Reset block hash when evicted from cache."""
    self.block_hash = None
    self.hash_value = None

vllm_mlx.paged_cache.CacheBlock.touch

touch() -> None

Update last access time.

Source code in vllm_mlx/paged_cache.py

def touch(self) -> None:
    """Update last access time."""
    self.last_access = time.time()

vllm_mlx.paged_cache.CacheBlock.__repr__

__repr__() -> str

Source code in vllm_mlx/paged_cache.py

def __repr__(self) -> str:
    prev_id = self.prev_free_block.block_id if self.prev_free_block else None
    next_id = self.next_free_block.block_id if self.next_free_block else None
    return (
        f"CacheBlock(id={self.block_id}, ref={self.ref_count}, "
        f"tokens={self.token_count}, prev={prev_id}, next={next_id})"
    )

vllm_mlx.paged_cache.FreeKVCacheBlockQueue

FreeKVCacheBlockQueue(blocks: List[CacheBlock])

Doubly linked list of free blocks following vLLM's design.

Provides O(1) operations for: - popleft(): Allocate block from front (LRU order) - remove(): Remove block from middle (when touched by cache hit) - append(): Return block to end (when freed)

The queue maintains LRU eviction order: - Front = least recently used (evict first) - Back = most recently used (evict last)

Uses fake head/tail sentinels to simplify edge cases.

Initialize queue with all blocks as free.

Parameters:

• blocks (List[CacheBlock]) –

  List of all CacheBlock objects

Source code in vllm_mlx/paged_cache.py

def __init__(self, blocks: List[CacheBlock]) -> None:
    """
    Initialize queue with all blocks as free.

    Args:
        blocks: List of all CacheBlock objects
    """
    self.num_free_blocks = len(blocks)

    # Initialize doubly linked list
    for i in range(len(blocks)):
        if i > 0:
            blocks[i].prev_free_block = blocks[i - 1]
        if i < len(blocks) - 1:
            blocks[i].next_free_block = blocks[i + 1]

    # Create sentinel nodes (never popped)
    self.fake_head = CacheBlock(block_id=-1)
    self.fake_tail = CacheBlock(block_id=-2)

    if blocks:
        self.fake_head.next_free_block = blocks[0]
        blocks[0].prev_free_block = self.fake_head
        self.fake_tail.prev_free_block = blocks[-1]
        blocks[-1].next_free_block = self.fake_tail
    else:
        self.fake_head.next_free_block = self.fake_tail
        self.fake_tail.prev_free_block = self.fake_head

vllm_mlx.paged_cache.FreeKVCacheBlockQueue.num_free_blocks instance-attribute

num_free_blocks = len(blocks)

vllm_mlx.paged_cache.FreeKVCacheBlockQueue.fake_head instance-attribute

fake_head = CacheBlock(block_id=-1)

vllm_mlx.paged_cache.FreeKVCacheBlockQueue.fake_tail instance-attribute

fake_tail = CacheBlock(block_id=-2)

vllm_mlx.paged_cache.FreeKVCacheBlockQueue.popleft

popleft() -> CacheBlock

Pop and return the first (LRU) free block.

Raises:

• ValueError –

  If no free blocks available

Source code in vllm_mlx/paged_cache.py

def popleft(self) -> CacheBlock:
    """
    Pop and return the first (LRU) free block.

    Raises:
        ValueError: If no free blocks available
    """
    if self.fake_head.next_free_block is self.fake_tail:
        raise ValueError("No free blocks available")

    block = self.fake_head.next_free_block
    assert block is not None

    # Remove from list
    self.fake_head.next_free_block = block.next_free_block
    if block.next_free_block:
        block.next_free_block.prev_free_block = self.fake_head

    block.prev_free_block = None
    block.next_free_block = None
    self.num_free_blocks -= 1

    return block

vllm_mlx.paged_cache.FreeKVCacheBlockQueue.popleft_n

popleft_n(n: int) -> List[CacheBlock]

Pop n blocks from the front.

Parameters:

• n (int) –

  Number of blocks to allocate

Returns:

• List[CacheBlock] –

  List of n free blocks

Raises:

• AssertionError –

  If not enough free blocks

Source code in vllm_mlx/paged_cache.py

def popleft_n(self, n: int) -> List[CacheBlock]:
    """
    Pop n blocks from the front.

    Args:
        n: Number of blocks to allocate

    Returns:
        List of n free blocks

    Raises:
        AssertionError: If not enough free blocks
    """
    if n == 0:
        return []

    assert (
        self.num_free_blocks >= n
    ), f"Need {n} blocks, have {self.num_free_blocks}"

    result = []
    curr = self.fake_head.next_free_block

    for _ in range(n):
        assert curr is not None and curr is not self.fake_tail
        result.append(curr)
        last = curr
        curr = curr.next_free_block
        # Clear pointers
        last.prev_free_block = None
        last.next_free_block = None

    # Reconnect list
    self.fake_head.next_free_block = curr
    if curr:
        curr.prev_free_block = self.fake_head

    self.num_free_blocks -= n
    return result

vllm_mlx.paged_cache.FreeKVCacheBlockQueue.remove

remove(block: CacheBlock) -> None

Remove a block from the middle of the queue.

Used when a free block is "touched" (reused by prefix cache hit).

Parameters:

• block (CacheBlock) –

  Block to remove

Raises:

• RuntimeError –

  If block not in queue

Source code in vllm_mlx/paged_cache.py

def remove(self, block: CacheBlock) -> None:
    """
    Remove a block from the middle of the queue.

    Used when a free block is "touched" (reused by prefix cache hit).

    Args:
        block: Block to remove

    Raises:
        RuntimeError: If block not in queue
    """
    if block.prev_free_block is None or block.next_free_block is None:
        raise RuntimeError(f"Block {block.block_id} not in free queue")

    # Unlink
    block.prev_free_block.next_free_block = block.next_free_block
    block.next_free_block.prev_free_block = block.prev_free_block
    block.prev_free_block = None
    block.next_free_block = None

    self.num_free_blocks -= 1

vllm_mlx.paged_cache.FreeKVCacheBlockQueue.append

append(block: CacheBlock) -> None

Append a block to the end (MRU position).

Parameters:

• block (CacheBlock) –

  Block to append

Source code in vllm_mlx/paged_cache.py

def append(self, block: CacheBlock) -> None:
    """
    Append a block to the end (MRU position).

    Args:
        block: Block to append
    """
    last = self.fake_tail.prev_free_block
    assert last is not None

    last.next_free_block = block
    block.prev_free_block = last
    block.next_free_block = self.fake_tail
    self.fake_tail.prev_free_block = block

    self.num_free_blocks += 1

vllm_mlx.paged_cache.FreeKVCacheBlockQueue.append_n

append_n(blocks: List[CacheBlock]) -> None

Append multiple blocks to the end.

Parameters:

• blocks (List[CacheBlock]) –

  Blocks to append (in order)

Source code in vllm_mlx/paged_cache.py

def append_n(self, blocks: List[CacheBlock]) -> None:
    """
    Append multiple blocks to the end.

    Args:
        blocks: Blocks to append (in order)
    """
    if not blocks:
        return

    last = self.fake_tail.prev_free_block
    assert last is not None

    for block in blocks:
        block.prev_free_block = last
        last.next_free_block = block
        last = block

    last.next_free_block = self.fake_tail
    self.fake_tail.prev_free_block = last

    self.num_free_blocks += len(blocks)

vllm_mlx.paged_cache.FreeKVCacheBlockQueue.get_all_free_blocks

get_all_free_blocks() -> List[CacheBlock]

Get all free blocks (for testing).

Source code in vllm_mlx/paged_cache.py

def get_all_free_blocks(self) -> List[CacheBlock]:
    """Get all free blocks (for testing)."""
    result = []
    curr = self.fake_head.next_free_block
    while curr and curr is not self.fake_tail:
        result.append(curr)
        curr = curr.next_free_block
    return result

vllm_mlx.paged_cache.BlockHashToBlockMap

BlockHashToBlockMap()

Cache mapping block hashes to blocks for prefix caching.

Follows vLLM's design where the same hash can map to multiple blocks (for different KV cache groups in hybrid models).

Source code in vllm_mlx/paged_cache.py

def __init__(self) -> None:
    self._cache: Dict[BlockHash, CacheBlock | Dict[int, CacheBlock]] = {}

vllm_mlx.paged_cache.BlockHashToBlockMap._cache instance-attribute

_cache: Dict[BlockHash, CacheBlock | Dict[int, CacheBlock]] = {}

vllm_mlx.paged_cache.BlockHashToBlockMap.get_block

get_block(block_hash: BlockHash) -> Optional[CacheBlock]

Get any block with the given hash.

Source code in vllm_mlx/paged_cache.py

def get_block(self, block_hash: BlockHash) -> Optional[CacheBlock]:
    """Get any block with the given hash."""
    blocks = self._cache.get(block_hash)
    if blocks is None:
        return None
    if isinstance(blocks, CacheBlock):
        return blocks
    if isinstance(blocks, dict):
        return next(iter(blocks.values()))
    return None

vllm_mlx.paged_cache.BlockHashToBlockMap.insert

insert(block_hash: BlockHash, block: CacheBlock) -> None

Insert a block into the cache.

Source code in vllm_mlx/paged_cache.py

def insert(self, block_hash: BlockHash, block: CacheBlock) -> None:
    """Insert a block into the cache."""
    existing = self._cache.get(block_hash)
    if existing is None:
        self._cache[block_hash] = block
    elif isinstance(existing, CacheBlock):
        self._cache[block_hash] = {
            existing.block_id: existing,
            block.block_id: block,
        }
    elif isinstance(existing, dict):
        existing[block.block_id] = block

vllm_mlx.paged_cache.BlockHashToBlockMap.pop

pop(block_hash: BlockHash, block_id: int) -> Optional[CacheBlock]

Remove and return a specific block from the cache.

Source code in vllm_mlx/paged_cache.py

def pop(self, block_hash: BlockHash, block_id: int) -> Optional[CacheBlock]:
    """Remove and return a specific block from the cache."""
    blocks = self._cache.pop(block_hash, None)
    if blocks is None:
        return None

    if isinstance(blocks, CacheBlock):
        if blocks.block_id == block_id:
            return blocks
        # Wrong block ID, put it back
        self._cache[block_hash] = blocks
        return None

    if isinstance(blocks, dict):
        block = blocks.pop(block_id, None)
        if blocks:  # Still has other blocks
            self._cache[block_hash] = blocks
        return block

    return None

vllm_mlx.paged_cache.BlockHashToBlockMap.__len__

__len__() -> int

Source code in vllm_mlx/paged_cache.py

def __len__(self) -> int:
    return len(self._cache)

vllm_mlx.paged_cache.BlockHashToBlockMap.clear

clear() -> None

Remove every block-hash mapping without mutating the blocks.

Source code in vllm_mlx/paged_cache.py

def clear(self) -> None:
    """Remove every block-hash mapping without mutating the blocks."""

    self._cache.clear()

vllm_mlx.paged_cache.BlockTable dataclass

BlockTable(request_id: str, block_ids: List[int] = list(), num_tokens: int = 0)

Per-request block table mapping logical to physical blocks.

Similar to vLLM's block table, this maps a request's token positions to physical cache blocks.

Attributes:

• request_id (str) –

  Unique request identifier

• block_ids (List[int]) –

  List of physical block IDs

• num_tokens (int) –

  Total number of cached tokens

vllm_mlx.paged_cache.BlockTable.request_id instance-attribute

request_id: str

vllm_mlx.paged_cache.BlockTable.block_ids class-attribute instance-attribute

block_ids: List[int] = field(default_factory=list)

vllm_mlx.paged_cache.BlockTable.num_tokens class-attribute instance-attribute

num_tokens: int = 0

vllm_mlx.paged_cache.BlockTable.add_block

add_block(block_id: int, num_tokens: int) -> None

Add a block to the table.

Source code in vllm_mlx/paged_cache.py

def add_block(self, block_id: int, num_tokens: int) -> None:
    """Add a block to the table."""
    self.block_ids.append(block_id)
    self.num_tokens += num_tokens

vllm_mlx.paged_cache.BlockTable.__len__

__len__() -> int

Source code in vllm_mlx/paged_cache.py

def __len__(self) -> int:
    return len(self.block_ids)

vllm_mlx.paged_cache.BlockTable.copy

copy(new_request_id: str) -> 'BlockTable'

Create a copy with new request ID.

Source code in vllm_mlx/paged_cache.py

def copy(self, new_request_id: str) -> "BlockTable":
    """Create a copy with new request ID."""
    return BlockTable(
        request_id=new_request_id,
        block_ids=self.block_ids.copy(),
        num_tokens=self.num_tokens,
    )

vllm_mlx.paged_cache.CacheStats dataclass

CacheStats(total_blocks: int = 0, allocated_blocks: int = 0, free_blocks: int = 0, shared_blocks: int = 0, total_tokens_cached: int = 0, cache_hits: int = 0, cache_misses: int = 0, cow_copies: int = 0, evictions: int = 0)

Statistics for cache monitoring.

vllm_mlx.paged_cache.CacheStats.total_blocks class-attribute instance-attribute

total_blocks: int = 0

vllm_mlx.paged_cache.CacheStats.allocated_blocks class-attribute instance-attribute

allocated_blocks: int = 0

vllm_mlx.paged_cache.CacheStats.free_blocks class-attribute instance-attribute

free_blocks: int = 0

vllm_mlx.paged_cache.CacheStats.shared_blocks class-attribute instance-attribute

shared_blocks: int = 0

vllm_mlx.paged_cache.CacheStats.total_tokens_cached class-attribute instance-attribute

total_tokens_cached: int = 0

vllm_mlx.paged_cache.CacheStats.cache_hits class-attribute instance-attribute

cache_hits: int = 0

vllm_mlx.paged_cache.CacheStats.cache_misses class-attribute instance-attribute

cache_misses: int = 0

vllm_mlx.paged_cache.CacheStats.cow_copies class-attribute instance-attribute

cow_copies: int = 0

vllm_mlx.paged_cache.CacheStats.evictions class-attribute instance-attribute

evictions: int = 0

vllm_mlx.paged_cache.PagedCacheManager

PagedCacheManager(block_size: int = 64, max_blocks: int = 1000, enable_caching: bool = True)

Paged KV cache manager following vLLM's BlockPool architecture.

Features: - Block allocation/deallocation with reference counting - Prefix sharing via chain-based hash deduplication - Copy-on-Write for efficient forking - O(1) LRU eviction using doubly linked list

Parameters:

• block_size (int, default: 64 ) –

  Number of tokens per block (default: 64)

• max_blocks (int, default: 1000 ) –

  Maximum number of blocks to allocate (default: 1000)

• enable_caching (bool, default: True ) –

  Whether to enable prefix caching (default: True)

Source code in vllm_mlx/paged_cache.py

def __init__(
    self,
    block_size: int = 64,
    max_blocks: int = 1000,
    enable_caching: bool = True,
):
    self.block_size = block_size
    self.max_blocks = max_blocks
    self.enable_caching = enable_caching

    # Create all blocks
    self.blocks: List[CacheBlock] = [
        CacheBlock(block_id=i) for i in range(max_blocks)
    ]

    # Free block queue (doubly linked list for O(1) LRU)
    self.free_block_queue = FreeKVCacheBlockQueue(self.blocks)

    # Hash-to-block cache for prefix caching
    self.cached_block_hash_to_block = BlockHashToBlockMap()

    # Legacy hash index for compatibility
    self.hash_to_block: Dict[str, int] = {}

    # Request to block table mapping
    self.request_tables: Dict[str, BlockTable] = {}

    # Allocated blocks (for fast lookup)
    self.allocated_blocks: Dict[int, CacheBlock] = {}

    # Reserve null block (block 0) - never freed
    self.null_block = self.free_block_queue.popleft()
    self.null_block.is_null = True
    self.null_block.ref_count = 1
    self.allocated_blocks[self.null_block.block_id] = self.null_block

    # Statistics
    self.stats = CacheStats(
        total_blocks=max_blocks,
        allocated_blocks=1,  # null block
        free_blocks=max_blocks - 1,
    )

    # Thread safety
    self._lock = threading.RLock()

    logger.info(
        f"PagedCacheManager initialized: block_size={block_size}, "
        f"max_blocks={max_blocks}, max_tokens={block_size * max_blocks}"
    )

vllm_mlx.paged_cache.PagedCacheManager.block_size instance-attribute

block_size = block_size

vllm_mlx.paged_cache.PagedCacheManager.max_blocks instance-attribute

max_blocks = max_blocks

vllm_mlx.paged_cache.PagedCacheManager.enable_caching instance-attribute

enable_caching = enable_caching

vllm_mlx.paged_cache.PagedCacheManager.blocks instance-attribute

blocks: List[CacheBlock] = [CacheBlock(block_id=i) for i in range(max_blocks)]

vllm_mlx.paged_cache.PagedCacheManager.free_block_queue instance-attribute

free_block_queue = FreeKVCacheBlockQueue(self.blocks)

vllm_mlx.paged_cache.PagedCacheManager.cached_block_hash_to_block instance-attribute

cached_block_hash_to_block = BlockHashToBlockMap()

vllm_mlx.paged_cache.PagedCacheManager.hash_to_block instance-attribute

hash_to_block: Dict[str, int] = {}

vllm_mlx.paged_cache.PagedCacheManager.request_tables instance-attribute

request_tables: Dict[str, BlockTable] = {}

vllm_mlx.paged_cache.PagedCacheManager.allocated_blocks instance-attribute

allocated_blocks: Dict[int, CacheBlock] = {}

vllm_mlx.paged_cache.PagedCacheManager.null_block instance-attribute

null_block = self.free_block_queue.popleft()

vllm_mlx.paged_cache.PagedCacheManager.stats instance-attribute

stats = CacheStats(total_blocks=max_blocks, allocated_blocks=1, free_blocks=max_blocks - 1)

vllm_mlx.paged_cache.PagedCacheManager._lock instance-attribute

_lock = threading.RLock()

vllm_mlx.paged_cache.PagedCacheManager.free_blocks property

free_blocks: int

Number of free blocks available.

vllm_mlx.paged_cache.PagedCacheManager.usage property

usage: float

Cache usage ratio (0.0 to 1.0).

vllm_mlx.paged_cache.PagedCacheManager.allocate_block

allocate_block() -> Optional[CacheBlock]

Allocate a new cache block.

Returns:

• Optional[CacheBlock] –

  CacheBlock if available, None if out of memory.

Source code in vllm_mlx/paged_cache.py

def allocate_block(self) -> Optional[CacheBlock]:
    """
    Allocate a new cache block.

    Returns:
        CacheBlock if available, None if out of memory.
    """
    with self._lock:
        if self.free_block_queue.num_free_blocks == 0:
            logger.warning("Out of cache blocks")
            return None

        block = self.free_block_queue.popleft()

        # Evict from hash cache if needed
        if self.enable_caching:
            self._maybe_evict_cached_block(block)

        block.ref_count = 1
        block.touch()
        self.allocated_blocks[block.block_id] = block

        self.stats.allocated_blocks += 1
        self.stats.free_blocks -= 1

        return block

vllm_mlx.paged_cache.PagedCacheManager.get_new_blocks

get_new_blocks(num_blocks: int) -> List[CacheBlock]

Allocate multiple blocks at once (vLLM style).

Parameters:

• num_blocks (int) –

  Number of blocks to allocate

Returns:

• List[CacheBlock] –

  List of allocated blocks

Raises:

• ValueError –

  If not enough free blocks

Source code in vllm_mlx/paged_cache.py

def get_new_blocks(self, num_blocks: int) -> List[CacheBlock]:
    """
    Allocate multiple blocks at once (vLLM style).

    Args:
        num_blocks: Number of blocks to allocate

    Returns:
        List of allocated blocks

    Raises:
        ValueError: If not enough free blocks
    """
    with self._lock:
        if num_blocks > self.free_block_queue.num_free_blocks:
            raise ValueError(
                f"Cannot allocate {num_blocks} blocks, "
                f"only {self.free_block_queue.num_free_blocks} available"
            )

        blocks = self.free_block_queue.popleft_n(num_blocks)

        for block in blocks:
            if self.enable_caching:
                self._maybe_evict_cached_block(block)

            block.ref_count = 1
            block.touch()
            self.allocated_blocks[block.block_id] = block

        self.stats.allocated_blocks += num_blocks
        self.stats.free_blocks -= num_blocks

        return blocks

vllm_mlx.paged_cache.PagedCacheManager._maybe_evict_cached_block

_maybe_evict_cached_block(block: CacheBlock) -> bool

Evict a block from the hash cache if present.

Parameters:

• block (CacheBlock) –

  Block to evict

Returns:

• bool –

  True if block was evicted from cache

Source code in vllm_mlx/paged_cache.py

def _maybe_evict_cached_block(self, block: CacheBlock) -> bool:
    """
    Evict a block from the hash cache if present.

    Args:
        block: Block to evict

    Returns:
        True if block was evicted from cache
    """
    if block.block_hash is None:
        return False

    evicted = self.cached_block_hash_to_block.pop(block.block_hash, block.block_id)

    if evicted:
        # Also remove from legacy hash index
        if block.hash_value and block.hash_value in self.hash_to_block:
            if self.hash_to_block[block.hash_value] == block.block_id:
                del self.hash_to_block[block.hash_value]

        block.reset_hash()
        block.cache_data = None  # Free tensor memory
        self.stats.evictions += 1
        return True

    return False

vllm_mlx.paged_cache.PagedCacheManager.free_block

free_block(block_id: int) -> bool

Free a cache block (decrements ref_count, frees if 0).

Returns:

• bool –

  True if block was freed, False if still referenced.

Source code in vllm_mlx/paged_cache.py

def free_block(self, block_id: int) -> bool:
    """
    Free a cache block (decrements ref_count, frees if 0).

    Returns:
        True if block was freed, False if still referenced.
    """
    with self._lock:
        if block_id not in self.allocated_blocks:
            logger.warning(f"Attempted to free unknown block: {block_id}")
            return False

        block = self.allocated_blocks[block_id]
        if block.is_null:
            return False  # Never free null block

        block.ref_count -= 1

        if block.ref_count <= 0:
            # Remove from allocated
            del self.allocated_blocks[block_id]

            # Add to free queue (back = MRU)
            self.free_block_queue.append(block)

            self.stats.allocated_blocks -= 1
            self.stats.free_blocks += 1
            self.stats.total_tokens_cached -= block.token_count

            return True

        return False

vllm_mlx.paged_cache.PagedCacheManager.touch

touch(blocks: Iterable[CacheBlock]) -> None

Touch blocks to prevent eviction (cache hit, vLLM style).

Increments ref_count and removes from free queue if needed.

Parameters:

• blocks (Iterable[CacheBlock]) –

  Blocks to touch

Source code in vllm_mlx/paged_cache.py

def touch(self, blocks: Iterable[CacheBlock]) -> None:
    """
    Touch blocks to prevent eviction (cache hit, vLLM style).

    Increments ref_count and removes from free queue if needed.

    Args:
        blocks: Blocks to touch
    """
    with self._lock:
        for block in blocks:
            if block.ref_count == 0 and not block.is_null:
                # Block is in free queue, remove it
                try:
                    self.free_block_queue.remove(block)
                    self.stats.free_blocks -= 1
                    self.stats.allocated_blocks += 1
                    self.allocated_blocks[block.block_id] = block
                except RuntimeError:
                    pass  # Block not in queue

            block.ref_count += 1
            block.touch()

vllm_mlx.paged_cache.PagedCacheManager.increment_ref

increment_ref(block_id: int) -> bool

Increment reference count for a block.

Source code in vllm_mlx/paged_cache.py

def increment_ref(self, block_id: int) -> bool:
    """Increment reference count for a block."""
    with self._lock:
        if block_id not in self.allocated_blocks:
            return False

        block = self.allocated_blocks[block_id]
        block.ref_count += 1
        block.touch()

        if block.ref_count == 2:
            self.stats.shared_blocks += 1

        return True

vllm_mlx.paged_cache.PagedCacheManager.decrement_ref

decrement_ref(block_id: int) -> bool

Decrement reference count (alias for free_block).

Source code in vllm_mlx/paged_cache.py

def decrement_ref(self, block_id: int) -> bool:
    """Decrement reference count (alias for free_block)."""
    return self.free_block(block_id)

vllm_mlx.paged_cache.PagedCacheManager.get_cached_block

get_cached_block(block_hash: BlockHash) -> Optional[CacheBlock]

Get a cached block by its hash (vLLM style).

Parameters:

• block_hash (BlockHash) –

  Content hash of the block

Returns:

• Optional[CacheBlock] –

  Cached block if found, None otherwise

Source code in vllm_mlx/paged_cache.py

def get_cached_block(self, block_hash: BlockHash) -> Optional[CacheBlock]:
    """
    Get a cached block by its hash (vLLM style).

    Args:
        block_hash: Content hash of the block

    Returns:
        Cached block if found, None otherwise
    """
    if not self.enable_caching:
        return None

    with self._lock:
        block = self.cached_block_hash_to_block.get_block(block_hash)
        if block:
            self.stats.cache_hits += 1
        else:
            self.stats.cache_misses += 1
        return block

vllm_mlx.paged_cache.PagedCacheManager.cache_full_blocks

cache_full_blocks(blocks: List[CacheBlock], token_ids: List[int], num_cached_blocks: int, num_full_blocks: int) -> None

Cache full blocks for prefix caching (vLLM style).

Computes chain hashes and adds blocks to the cache.

Parameters:

• blocks (List[CacheBlock]) –

  All blocks for the request

• token_ids (List[int]) –

  All token IDs for the request

• num_cached_blocks (int) –

  Number of blocks already cached

• num_full_blocks (int) –

  Number of full blocks to cache

Source code in vllm_mlx/paged_cache.py

def cache_full_blocks(
    self,
    blocks: List[CacheBlock],
    token_ids: List[int],
    num_cached_blocks: int,
    num_full_blocks: int,
) -> None:
    """
    Cache full blocks for prefix caching (vLLM style).

    Computes chain hashes and adds blocks to the cache.

    Args:
        blocks: All blocks for the request
        token_ids: All token IDs for the request
        num_cached_blocks: Number of blocks already cached
        num_full_blocks: Number of full blocks to cache
    """
    if not self.enable_caching:
        return

    if num_cached_blocks >= num_full_blocks:
        return

    with self._lock:
        # Get parent hash from last cached block
        parent_hash = None
        if num_cached_blocks > 0:
            parent_hash = blocks[num_cached_blocks - 1].block_hash

        for i in range(num_cached_blocks, num_full_blocks):
            block = blocks[i]
            if block.block_hash is not None:
                parent_hash = block.block_hash
                continue  # Already cached

            # Get tokens for this block
            start = i * self.block_size
            end = start + self.block_size
            block_tokens = token_ids[start:end]

            # Compute chain hash
            block_hash = compute_block_hash(parent_hash, block_tokens)
            block.block_hash = block_hash
            block.token_count = len(block_tokens)

            # Add to cache
            self.cached_block_hash_to_block.insert(block_hash, block)

            # Also maintain legacy hash for compatibility
            legacy_hash = self.compute_block_hash(block_tokens)
            block.hash_value = legacy_hash
            self.hash_to_block[legacy_hash] = block.block_id

            parent_hash = block_hash

vllm_mlx.paged_cache.PagedCacheManager.get_computed_blocks

get_computed_blocks(token_ids: List[int]) -> Tuple[List[CacheBlock], int]

Find cached blocks for a token prefix (vLLM style).

Parameters:

• token_ids (List[int]) –

  Token IDs to look up

Returns:

• Tuple[List[CacheBlock], int] –

  Tuple of (cached_blocks, num_cached_tokens)

Source code in vllm_mlx/paged_cache.py

def get_computed_blocks(
    self,
    token_ids: List[int],
) -> Tuple[List[CacheBlock], int]:
    """
    Find cached blocks for a token prefix (vLLM style).

    Args:
        token_ids: Token IDs to look up

    Returns:
        Tuple of (cached_blocks, num_cached_tokens)
    """
    if not self.enable_caching:
        return [], 0

    with self._lock:
        cached_blocks = []
        parent_hash = None
        num_cached_tokens = 0

        num_full_blocks = len(token_ids) // self.block_size

        for i in range(num_full_blocks):
            start = i * self.block_size
            end = start + self.block_size
            block_tokens = token_ids[start:end]

            # Compute expected hash
            block_hash = compute_block_hash(parent_hash, block_tokens)

            # Look up in cache
            cached_block = self.cached_block_hash_to_block.get_block(block_hash)
            if cached_block is None:
                self.stats.cache_misses += 1
                break  # Cache miss, stop here

            cached_blocks.append(cached_block)
            parent_hash = block_hash
            num_cached_tokens += self.block_size
            self.stats.cache_hits += 1

        return cached_blocks, num_cached_tokens

vllm_mlx.paged_cache.PagedCacheManager.compute_block_hash staticmethod

compute_block_hash(tokens: List[int]) -> str

Compute legacy string hash for a sequence of tokens.

Source code in vllm_mlx/paged_cache.py

@staticmethod
def compute_block_hash(tokens: List[int]) -> str:
    """Compute legacy string hash for a sequence of tokens."""
    token_bytes = b"".join(t.to_bytes(4, "big") for t in tokens)
    return hashlib.sha256(token_bytes).hexdigest()[:16]

vllm_mlx.paged_cache.PagedCacheManager.find_cached_block

find_cached_block(tokens: List[int]) -> Optional[CacheBlock]

Find a cached block matching the given tokens (legacy method).

Source code in vllm_mlx/paged_cache.py

def find_cached_block(self, tokens: List[int]) -> Optional[CacheBlock]:
    """
    Find a cached block matching the given tokens (legacy method).
    """
    with self._lock:
        hash_value = self.compute_block_hash(tokens)

        if hash_value in self.hash_to_block:
            block_id = self.hash_to_block[hash_value]
            if block_id in self.allocated_blocks:
                block = self.allocated_blocks[block_id]
                block.touch()
                self.stats.cache_hits += 1
                return block

        self.stats.cache_misses += 1
        return None

vllm_mlx.paged_cache.PagedCacheManager.register_block_hash

register_block_hash(block: CacheBlock, tokens: List[int]) -> None

Register a block's hash for deduplication (legacy method).

Source code in vllm_mlx/paged_cache.py

def register_block_hash(self, block: CacheBlock, tokens: List[int]) -> None:
    """Register a block's hash for deduplication (legacy method)."""
    with self._lock:
        hash_value = self.compute_block_hash(tokens)
        block.hash_value = hash_value
        self.hash_to_block[hash_value] = block.block_id

vllm_mlx.paged_cache.PagedCacheManager.create_block_table

create_block_table(request_id: str) -> BlockTable

Create a new block table for a request.

Source code in vllm_mlx/paged_cache.py

def create_block_table(self, request_id: str) -> BlockTable:
    """Create a new block table for a request."""
    with self._lock:
        table = BlockTable(request_id=request_id)
        self.request_tables[request_id] = table
        return table

vllm_mlx.paged_cache.PagedCacheManager.get_block_table

get_block_table(request_id: str) -> Optional[BlockTable]

Get block table for a request.

Source code in vllm_mlx/paged_cache.py

def get_block_table(self, request_id: str) -> Optional[BlockTable]:
    """Get block table for a request."""
    with self._lock:
        return self.request_tables.get(request_id)

vllm_mlx.paged_cache.PagedCacheManager.get_or_create_block_table

get_or_create_block_table(request_id: str) -> BlockTable

Get or create block table for a request.

Source code in vllm_mlx/paged_cache.py

def get_or_create_block_table(self, request_id: str) -> BlockTable:
    """Get or create block table for a request."""
    with self._lock:
        if request_id not in self.request_tables:
            self.request_tables[request_id] = BlockTable(request_id=request_id)
        return self.request_tables[request_id]

vllm_mlx.paged_cache.PagedCacheManager.delete_block_table

delete_block_table(request_id: str) -> None

Delete block table and free associated blocks.

Source code in vllm_mlx/paged_cache.py

def delete_block_table(self, request_id: str) -> None:
    """Delete block table and free associated blocks."""
    with self._lock:
        table = self.request_tables.pop(request_id, None)
        if table:
            for block_id in table.block_ids:
                self.free_block(block_id)

vllm_mlx.paged_cache.PagedCacheManager.add_block_to_table

add_block_to_table(table: BlockTable, block: CacheBlock, tokens_in_block: int) -> None

Add a block to a block table.

Source code in vllm_mlx/paged_cache.py

def add_block_to_table(
    self,
    table: BlockTable,
    block: CacheBlock,
    tokens_in_block: int,
) -> None:
    """Add a block to a block table."""
    with self._lock:
        table.block_ids.append(block.block_id)
        block.token_count = tokens_in_block
        table.num_tokens += tokens_in_block
        self.stats.total_tokens_cached += tokens_in_block

vllm_mlx.paged_cache.PagedCacheManager.find_shared_prefix

find_shared_prefix(tokens: List[int]) -> Tuple[List[int], List[int]]

Find shared prefix blocks for a token sequence.

Source code in vllm_mlx/paged_cache.py

def find_shared_prefix(
    self,
    tokens: List[int],
) -> Tuple[List[int], List[int]]:
    """
    Find shared prefix blocks for a token sequence.
    """
    with self._lock:
        shared_blocks = []
        remaining_tokens = tokens.copy()

        while len(remaining_tokens) >= self.block_size:
            chunk = remaining_tokens[: self.block_size]
            cached_block = self.find_cached_block(chunk)

            if cached_block:
                shared_blocks.append(cached_block.block_id)
                remaining_tokens = remaining_tokens[self.block_size :]
            else:
                break

        return shared_blocks, remaining_tokens

vllm_mlx.paged_cache.PagedCacheManager.fork_block_table

fork_block_table(source_table: BlockTable, new_request_id: str) -> BlockTable

Fork a block table for a new request (COW).

Source code in vllm_mlx/paged_cache.py

def fork_block_table(
    self,
    source_table: BlockTable,
    new_request_id: str,
) -> BlockTable:
    """
    Fork a block table for a new request (COW).
    """
    with self._lock:
        new_table = source_table.copy(new_request_id)

        for block_id in new_table.block_ids:
            self.increment_ref(block_id)

        self.request_tables[new_request_id] = new_table

        logger.debug(
            f"Forked block table: {source_table.request_id} -> {new_request_id}, "
            f"blocks={len(new_table.block_ids)}"
        )

        return new_table

vllm_mlx.paged_cache.PagedCacheManager.get_blocks_for_generation

get_blocks_for_generation(table: BlockTable) -> Tuple[List[CacheBlock], bool]

Get blocks for generation, applying COW if needed.

Source code in vllm_mlx/paged_cache.py

def get_blocks_for_generation(
    self,
    table: BlockTable,
) -> Tuple[List[CacheBlock], bool]:
    """
    Get blocks for generation, applying COW if needed.
    """
    with self._lock:
        blocks = []
        was_copied = False

        for i, block_id in enumerate(table.block_ids):
            block = self.allocated_blocks.get(block_id)
            if not block:
                continue

            if block.is_shared():
                new_block = self._cow_copy_block(block)
                if new_block:
                    table.block_ids[i] = new_block.block_id
                    blocks.append(new_block)
                    was_copied = True
                    self.stats.cow_copies += 1
                else:
                    blocks.append(block)
            else:
                blocks.append(block)

            block.touch()

        return blocks, was_copied

vllm_mlx.paged_cache.PagedCacheManager._cow_copy_block

_cow_copy_block(source_block: CacheBlock) -> Optional[CacheBlock]

Create a copy of a block for COW.

Source code in vllm_mlx/paged_cache.py

def _cow_copy_block(self, source_block: CacheBlock) -> Optional[CacheBlock]:
    """Create a copy of a block for COW."""
    new_block = self.allocate_block()
    if not new_block:
        return None

    new_block.token_count = source_block.token_count
    new_block.cache_data = source_block.cache_data

    source_block.ref_count -= 1
    if source_block.ref_count == 1:
        self.stats.shared_blocks -= 1

    logger.debug(f"COW copy: block {source_block.block_id} -> {new_block.block_id}")

    return new_block

vllm_mlx.paged_cache.PagedCacheManager.allocate_blocks_for_tokens

allocate_blocks_for_tokens(num_tokens: int) -> List[CacheBlock]

Allocate enough blocks to hold num_tokens.

Source code in vllm_mlx/paged_cache.py

def allocate_blocks_for_tokens(self, num_tokens: int) -> List[CacheBlock]:
    """Allocate enough blocks to hold num_tokens."""
    num_blocks_needed = (num_tokens + self.block_size - 1) // self.block_size
    return self.get_new_blocks(num_blocks_needed)

vllm_mlx.paged_cache.PagedCacheManager.evict_lru_blocks

evict_lru_blocks(num_blocks: int) -> int

Evict least recently used blocks.

With the doubly linked list, LRU blocks are already at the front of the free queue. We just need to pop from front.

Source code in vllm_mlx/paged_cache.py

def evict_lru_blocks(self, num_blocks: int) -> int:
    """
    Evict least recently used blocks.

    With the doubly linked list, LRU blocks are already at the front
    of the free queue. We just need to pop from front.
    """
    with self._lock:
        evicted = 0

        # Get evictable blocks from free queue (they're already LRU ordered)
        for _ in range(min(num_blocks, self.free_block_queue.num_free_blocks)):
            try:
                block = self.free_block_queue.popleft()
                self._maybe_evict_cached_block(block)
                # Put back at end (now available for allocation)
                self.free_block_queue.append(block)
                evicted += 1
            except ValueError:
                break

        if evicted > 0:
            logger.info(f"Evicted {evicted} LRU blocks from cache")

        return evicted

vllm_mlx.paged_cache.PagedCacheManager.handle_memory_pressure

handle_memory_pressure(requested_blocks: int) -> bool

Handle memory pressure by evicting blocks.

Source code in vllm_mlx/paged_cache.py

def handle_memory_pressure(self, requested_blocks: int) -> bool:
    """Handle memory pressure by evicting blocks."""
    with self._lock:
        if self.free_block_queue.num_free_blocks >= requested_blocks:
            return True

        needed = requested_blocks - self.free_block_queue.num_free_blocks
        self.evict_lru_blocks(needed)

        return self.free_block_queue.num_free_blocks >= requested_blocks

vllm_mlx.paged_cache.PagedCacheManager.get_stats

get_stats() -> CacheStats

Get current cache statistics.

Source code in vllm_mlx/paged_cache.py

def get_stats(self) -> CacheStats:
    """Get current cache statistics."""
    with self._lock:
        self.stats.shared_blocks = sum(
            1 for b in self.allocated_blocks.values() if b.ref_count > 1
        )
        self.stats.free_blocks = self.free_block_queue.num_free_blocks
        return self.stats

vllm_mlx.paged_cache.PagedCacheManager.get_memory_usage

get_memory_usage() -> Dict[str, Any]

Get memory usage information.

Source code in vllm_mlx/paged_cache.py

def get_memory_usage(self) -> Dict[str, Any]:
    """Get memory usage information."""
    with self._lock:
        stats = self.get_stats()
        return {
            "block_size": self.block_size,
            "max_blocks": self.max_blocks,
            "allocated_blocks": stats.allocated_blocks,
            "free_blocks": stats.free_blocks,
            "shared_blocks": stats.shared_blocks,
            "total_tokens_cached": stats.total_tokens_cached,
            "utilization": stats.allocated_blocks / self.max_blocks,
            "cache_hit_rate": (
                stats.cache_hits / (stats.cache_hits + stats.cache_misses)
                if (stats.cache_hits + stats.cache_misses) > 0
                else 0
            ),
        }

vllm_mlx.paged_cache.PagedCacheManager.reset_stats

reset_stats() -> None

Reset statistics counters.

Source code in vllm_mlx/paged_cache.py

def reset_stats(self) -> None:
    """Reset statistics counters."""
    with self._lock:
        self.stats.cache_hits = 0
        self.stats.cache_misses = 0
        self.stats.cow_copies = 0
        self.stats.evictions = 0

vllm_mlx.paged_cache.PagedCacheManager.reset_prefix_cache

reset_prefix_cache() -> bool

Reset the prefix cache.

Source code in vllm_mlx/paged_cache.py

def reset_prefix_cache(self) -> bool:
    """Reset the prefix cache."""
    with self._lock:
        num_used = self.max_blocks - self.free_block_queue.num_free_blocks
        if num_used > 1:  # null_block is always "used"
            logger.warning(f"Cannot reset cache: {num_used - 1} blocks in use")
            return False

        self.cached_block_hash_to_block.clear()
        self.hash_to_block.clear()

        for block in self.blocks:
            block.reset_hash()
            block.cache_data = None

        self.stats.evictions = 0
        self.stats.cache_hits = 0
        self.stats.cache_misses = 0

        logger.info("Prefix cache reset successfully")
        return True

vllm_mlx.paged_cache.PagedCacheManager.clear

clear() -> None

Clear all cached data.

Source code in vllm_mlx/paged_cache.py

def clear(self) -> None:
    """Clear all cached data."""
    with self._lock:
        # Recreate blocks and queue
        self.blocks = [CacheBlock(block_id=i) for i in range(self.max_blocks)]
        self.free_block_queue = FreeKVCacheBlockQueue(self.blocks)

        self.cached_block_hash_to_block.clear()
        self.hash_to_block.clear()
        self.request_tables.clear()
        self.allocated_blocks.clear()

        # Reserve null block
        self.null_block = self.free_block_queue.popleft()
        self.null_block.is_null = True
        self.null_block.ref_count = 1
        self.allocated_blocks[self.null_block.block_id] = self.null_block

        self.stats = CacheStats(
            total_blocks=self.max_blocks,
            allocated_blocks=1,
            free_blocks=self.max_blocks - 1,
        )

        logger.info("PagedCacheManager cleared")

vllm_mlx.paged_cache.compute_block_hash

compute_block_hash(parent_hash: Optional[BlockHash], token_ids: List[int], extra_keys: Optional[Tuple[Any, ...]] = None) -> BlockHash

Compute hash for a block based on its content and parent block.

This enables prefix caching by creating a chain of hashes where each block's hash depends on all previous blocks (similar to vLLM).

Parameters:

• parent_hash (Optional[BlockHash]) –

  Hash of the previous block, or None for first block

• token_ids (List[int]) –

  Token IDs in this block

• extra_keys (Optional[Tuple[Any, ...]], default: None ) –

  Additional keys (e.g., LoRA, multimodal)

Returns:

• BlockHash –

  Content-based hash for this block

Source code in vllm_mlx/paged_cache.py

def compute_block_hash(
    parent_hash: Optional[BlockHash],
    token_ids: List[int],
    extra_keys: Optional[Tuple[Any, ...]] = None,
) -> BlockHash:
    """
    Compute hash for a block based on its content and parent block.

    This enables prefix caching by creating a chain of hashes where
    each block's hash depends on all previous blocks (similar to vLLM).

    Args:
        parent_hash: Hash of the previous block, or None for first block
        token_ids: Token IDs in this block
        extra_keys: Additional keys (e.g., LoRA, multimodal)

    Returns:
        Content-based hash for this block
    """
    hasher = hashlib.sha256()

    # Include parent hash for chain
    if parent_hash:
        hasher.update(parent_hash)
    else:
        # Use fixed seed for reproducibility
        hasher.update(b"vllm-mlx-root")

    # Include token content
    hasher.update(bytes(str(tuple(token_ids)), "utf-8"))

    # Include extra keys if present
    if extra_keys:
        hasher.update(bytes(str(extra_keys), "utf-8"))

    return BlockHash(hasher.digest())

Complete contract reference

Expand any definition for its exact inputs, annotations, defaults, return contract, directly raised exceptions, source-grounded behavior, and immutable line link. This section includes private and nested definitions that ordinary API generators omit.

vllm_mlx.paged_cache.compute_block_hash · function

vllm_mlx.paged_cache.compute_block_hash(parent_hash: Optional[BlockHash], token_ids: List[int], extra_keys: Optional[Tuple[Any, ...]] = None) -> BlockHash

Compute hash for a block based on its content and parent block.

Parameters

Name	Type	Required	Default	Description
parent_hash	Optional[BlockHash]	yes	none	Hash of the previous block, or None for first block
token_ids	List[int]	yes	none	Token IDs in this block
extra_keys	Optional[Tuple[Any, ...]]	no	None	Additional keys (e.g., LoRA, multimodal)

Returns

• Type: BlockHash
• Direct return expressions: BlockHash(hasher.digest())

Exceptions and behavior

Function compute_block_hash calls hashlib.sha256, hasher.update, bytes, str; returns BlockHash(hasher.digest()). No direct raise statement appears in this definition.

View source #L40-L75.

vllm_mlx.paged_cache.CacheBlock · class

vllm_mlx.paged_cache.CacheBlock(block_id: int, ref_count: int = 0, block_hash: Optional[BlockHash] = None, prev_free_block: Optional['CacheBlock'] = None, next_free_block: Optional['CacheBlock'] = None, is_null: bool = False, cache_data: Optional[List[Tuple[Any, Any]]] = None, token_count: int = 0, hash_value: Optional[str] = None, last_access: float = field(default_factory=time.time))

KV cache block metadata following vLLM's design.

Parameters

Name	Type	Required	Default	Description
block_id	int	yes	none	Required constructor field.
ref_count	int	no	0	Optional constructor field; defaults to 0.
block_hash	Optional[BlockHash]	no	None	Optional constructor field; defaults to None.
prev_free_block	Optional['CacheBlock']	no	None	Optional constructor field; defaults to None.
next_free_block	Optional['CacheBlock']	no	None	Optional constructor field; defaults to None.
is_null	bool	no	False	Optional constructor field; defaults to False.
cache_data	Optional[List[Tuple[Any, Any]]]	no	None	Optional constructor field; defaults to None.
token_count	int	no	0	Optional constructor field; defaults to 0.
hash_value	Optional[str]	no	None	Optional constructor field; defaults to None.
last_access	float	no	field(default_factory=time.time)	Optional constructor field; defaults to field(default_factory=time.time).

Returns

• Constructs: vllm_mlx.paged_cache.CacheBlock

Exceptions and behavior

Class CacheBlock declares 5 direct member(s). No direct raise statement appears in this definition.

View source #L84-L146.

vllm_mlx.paged_cache.CacheBlock.is_full · method

vllm_mlx.paged_cache.CacheBlock.is_full(block_size: int) -> bool

Check if block is at capacity.

Parameters

Name	Type	Required	Default	Description
block_size	int	yes	none	Required positional or keyword input.

Returns

• Type: bool
• Direct return expressions: self.token_count >= block_size

Exceptions and behavior

Method CacheBlock.is_full returns self.token_count >= block_size. No direct raise statement appears in this definition.

View source #L123-L125.

vllm_mlx.paged_cache.CacheBlock.is_shared · method

vllm_mlx.paged_cache.CacheBlock.is_shared() -> bool

Check if block is shared (ref_count > 1).

Parameters

This callable has no explicit inputs.

Returns

• Type: bool
• Direct return expressions: self.ref_count > 1

Exceptions and behavior

Method CacheBlock.is_shared returns self.ref_count > 1. No direct raise statement appears in this definition.

View source #L127-L129.

vllm_mlx.paged_cache.CacheBlock.reset_hash · method

vllm_mlx.paged_cache.CacheBlock.reset_hash() -> None

Reset block hash when evicted from cache.

Parameters

This callable has no explicit inputs.

Returns

• Type: None

Exceptions and behavior

Method CacheBlock.reset_hash updates self.block_hash, self.hash_value. No direct raise statement appears in this definition.

View source #L131-L134.

vllm_mlx.paged_cache.CacheBlock.touch · method

vllm_mlx.paged_cache.CacheBlock.touch() -> None

Update last access time.

Parameters

This callable has no explicit inputs.

Returns

• Type: None

Exceptions and behavior

Method CacheBlock.touch updates self.last_access; calls time.time. No direct raise statement appears in this definition.

View source #L136-L138.

vllm_mlx.paged_cache.CacheBlock.__repr__ · method

vllm_mlx.paged_cache.CacheBlock.__repr__() -> str

Method CacheBlock.__repr__ returns f'CacheBlock(id={self.block_id}, ref={self.ref_count}, tokens={self.token_count}, prev={prev_id}, next={next_id})'.

Parameters

This callable has no explicit inputs.

Returns

• Type: str
• Direct return expressions: f'CacheBlock(id={self.block_id}, ref={self.ref_count}, tokens={self.token_count}, prev={prev_id}, next={next_id})'

Exceptions and behavior

Method CacheBlock.__repr__ returns f'CacheBlock(id={self.block_id}, ref={self.ref_count}, tokens={self.token_count}, prev={prev_id}, next={next_id})'. No direct raise statement appears in this definition.

View source #L140-L146.

vllm_mlx.paged_cache.FreeKVCacheBlockQueue · class

vllm_mlx.paged_cache.FreeKVCacheBlockQueue(blocks: List[CacheBlock])

Doubly linked list of free blocks following vLLM's design.

Parameters

Name	Type	Required	Default	Description
blocks	List[CacheBlock]	yes	none	List of all CacheBlock objects

Returns

• Constructs: vllm_mlx.paged_cache.FreeKVCacheBlockQueue

Exceptions and behavior

Class FreeKVCacheBlockQueue declares 7 direct member(s). No direct raise statement appears in this definition.

View source #L158-L337.

vllm_mlx.paged_cache.FreeKVCacheBlockQueue.__init__ · method

vllm_mlx.paged_cache.FreeKVCacheBlockQueue.__init__(blocks: List[CacheBlock]) -> None

Initialize queue with all blocks as free.

Parameters

Name	Type	Required	Default	Description
blocks	List[CacheBlock]	yes	none	List of all CacheBlock objects

Returns

• Type: None

Exceptions and behavior

Method FreeKVCacheBlockQueue.__init__ updates self.num_free_blocks, self.fake_head, self.fake_tail, self.fake_head.next_free_block; calls len, range, CacheBlock. No direct raise statement appears in this definition.

View source #L174-L201.

vllm_mlx.paged_cache.FreeKVCacheBlockQueue.popleft · method

vllm_mlx.paged_cache.FreeKVCacheBlockQueue.popleft() -> CacheBlock

Pop and return the first (LRU) free block.

Parameters

This callable has no explicit inputs.

Returns

• Type: CacheBlock
• Direct return expressions: block

Exceptions and behavior

Method FreeKVCacheBlockQueue.popleft updates self.fake_head.next_free_block, self.num_free_blocks; calls ValueError; can raise ValueError; returns block. Directly raised exceptions: ValueError.

View source #L203-L225.

vllm_mlx.paged_cache.FreeKVCacheBlockQueue.popleft_n · method

vllm_mlx.paged_cache.FreeKVCacheBlockQueue.popleft_n(n: int) -> List[CacheBlock]

Pop n blocks from the front.

Parameters

Name	Type	Required	Default	Description
n	int	yes	none	Number of blocks to allocate

Returns

• Type: List[CacheBlock]
• Direct return expressions: []; result

Exceptions and behavior

Method FreeKVCacheBlockQueue.popleft_n updates self.fake_head.next_free_block, self.num_free_blocks; calls range, result.append; has 2 explicit return paths. No direct raise statement appears in this definition.

View source #L227-L265.

vllm_mlx.paged_cache.FreeKVCacheBlockQueue.remove · method

vllm_mlx.paged_cache.FreeKVCacheBlockQueue.remove(block: CacheBlock) -> None

Remove a block from the middle of the queue.

Parameters

Name	Type	Required	Default	Description
block	CacheBlock	yes	none	Block to remove

Returns

• Type: None

Exceptions and behavior

Method FreeKVCacheBlockQueue.remove updates self.num_free_blocks; calls RuntimeError; can raise RuntimeError. Directly raised exceptions: RuntimeError.

View source #L267-L288.

vllm_mlx.paged_cache.FreeKVCacheBlockQueue.append · method

vllm_mlx.paged_cache.FreeKVCacheBlockQueue.append(block: CacheBlock) -> None

Append a block to the end (MRU position).

Parameters

Name	Type	Required	Default	Description
block	CacheBlock	yes	none	Block to append

Returns

• Type: None

Exceptions and behavior

Method FreeKVCacheBlockQueue.append updates self.fake_tail.prev_free_block, self.num_free_blocks. No direct raise statement appears in this definition.

View source #L290-L305.

vllm_mlx.paged_cache.FreeKVCacheBlockQueue.append_n · method

vllm_mlx.paged_cache.FreeKVCacheBlockQueue.append_n(blocks: List[CacheBlock]) -> None

Append multiple blocks to the end.

Parameters

Name	Type	Required	Default	Description
blocks	List[CacheBlock]	yes	none	Blocks to append (in order)

Returns

• Type: None
• Direct return expressions: None

Exceptions and behavior

Method FreeKVCacheBlockQueue.append_n updates self.fake_tail.prev_free_block, self.num_free_blocks; calls len; returns None. No direct raise statement appears in this definition.

View source #L307-L328.

vllm_mlx.paged_cache.FreeKVCacheBlockQueue.get_all_free_blocks · method

vllm_mlx.paged_cache.FreeKVCacheBlockQueue.get_all_free_blocks() -> List[CacheBlock]

Get all free blocks (for testing).

Parameters

This callable has no explicit inputs.

Returns

• Type: List[CacheBlock]
• Direct return expressions: result

Exceptions and behavior

Method FreeKVCacheBlockQueue.get_all_free_blocks calls result.append; returns result. No direct raise statement appears in this definition.

View source #L330-L337.

vllm_mlx.paged_cache.BlockHashToBlockMap · class

vllm_mlx.paged_cache.BlockHashToBlockMap()

Cache mapping block hashes to blocks for prefix caching.

Parameters

This callable has no explicit inputs.

Returns

• Constructs: vllm_mlx.paged_cache.BlockHashToBlockMap

Exceptions and behavior

Class BlockHashToBlockMap declares 6 direct member(s). No direct raise statement appears in this definition.

View source #L345-L407.

vllm_mlx.paged_cache.BlockHashToBlockMap.__init__ · method

vllm_mlx.paged_cache.BlockHashToBlockMap.__init__() -> None

Method BlockHashToBlockMap.__init__ updates self._cache.

Parameters

This callable has no explicit inputs.

Returns

• Type: None

Exceptions and behavior

Method BlockHashToBlockMap.__init__ updates self._cache. No direct raise statement appears in this definition.

View source #L353-L354.

vllm_mlx.paged_cache.BlockHashToBlockMap.get_block · method

vllm_mlx.paged_cache.BlockHashToBlockMap.get_block(block_hash: BlockHash) -> Optional[CacheBlock]

Get any block with the given hash.

Parameters

Name	Type	Required	Default	Description
block_hash	BlockHash	yes	none	Required positional or keyword input.

Returns

• Type: Optional[CacheBlock]
• Direct return expressions: None; blocks; next(iter(blocks.values()))

Exceptions and behavior

Method BlockHashToBlockMap.get_block calls self._cache.get, isinstance, next, iter; has 3 explicit return paths. No direct raise statement appears in this definition.

View source #L356-L365.

vllm_mlx.paged_cache.BlockHashToBlockMap.insert · method

vllm_mlx.paged_cache.BlockHashToBlockMap.insert(block_hash: BlockHash, block: CacheBlock) -> None

Insert a block into the cache.

Parameters

Name	Type	Required	Default	Description
block_hash	BlockHash	yes	none	Required positional or keyword input.
block	CacheBlock	yes	none	Required positional or keyword input.

Returns

• Type: None

Exceptions and behavior

Method BlockHashToBlockMap.insert calls self._cache.get, isinstance. No direct raise statement appears in this definition.

View source #L367-L378.

vllm_mlx.paged_cache.BlockHashToBlockMap.pop · method

vllm_mlx.paged_cache.BlockHashToBlockMap.pop(block_hash: BlockHash, block_id: int) -> Optional[CacheBlock]

Remove and return a specific block from the cache.

Parameters

Name	Type	Required	Default	Description
block_hash	BlockHash	yes	none	Required positional or keyword input.
block_id	int	yes	none	Required positional or keyword input.

Returns

• Type: Optional[CacheBlock]
• Direct return expressions: None; blocks; block

Exceptions and behavior

Method BlockHashToBlockMap.pop calls self._cache.pop, isinstance, blocks.pop; has 3 explicit return paths. No direct raise statement appears in this definition.

View source #L380-L399.

vllm_mlx.paged_cache.BlockHashToBlockMap.__len__ · method

vllm_mlx.paged_cache.BlockHashToBlockMap.__len__() -> int

Method BlockHashToBlockMap.__len__ calls len; returns len(self._cache).

Parameters

This callable has no explicit inputs.

Returns

• Type: int
• Direct return expressions: len(self._cache)

Exceptions and behavior

Method BlockHashToBlockMap.__len__ calls len; returns len(self._cache). No direct raise statement appears in this definition.

View source #L401-L402.

vllm_mlx.paged_cache.BlockHashToBlockMap.clear · method

vllm_mlx.paged_cache.BlockHashToBlockMap.clear() -> None

Remove every block-hash mapping without mutating the blocks.

Parameters

This callable has no explicit inputs.

Returns

• Type: None

Exceptions and behavior

Method BlockHashToBlockMap.clear calls self._cache.clear. No direct raise statement appears in this definition.

View source #L404-L407.

vllm_mlx.paged_cache.BlockTable · class

vllm_mlx.paged_cache.BlockTable(request_id: str, block_ids: List[int] = field(default_factory=list), num_tokens: int = 0)

Per-request block table mapping logical to physical blocks.

Parameters

Name	Type	Required	Default	Description
request_id	str	yes	none	Required constructor field.
block_ids	List[int]	no	field(default_factory=list)	Optional constructor field; defaults to field(default_factory=list).
num_tokens	int	no	0	Optional constructor field; defaults to 0.

Returns

• Constructs: vllm_mlx.paged_cache.BlockTable

Exceptions and behavior

Class BlockTable declares 3 direct member(s). No direct raise statement appears in this definition.

View source #L416-L447.

vllm_mlx.paged_cache.BlockTable.add_block · method

vllm_mlx.paged_cache.BlockTable.add_block(block_id: int, num_tokens: int) -> None

Add a block to the table.

Parameters

Name	Type	Required	Default	Description
block_id	int	yes	none	Required positional or keyword input.
num_tokens	int	yes	none	Required positional or keyword input.

Returns

• Type: None

Exceptions and behavior

Method BlockTable.add_block updates self.num_tokens; calls self.block_ids.append. No direct raise statement appears in this definition.

View source #L433-L436.

vllm_mlx.paged_cache.BlockTable.__len__ · method

vllm_mlx.paged_cache.BlockTable.__len__() -> int

Method BlockTable.__len__ calls len; returns len(self.block_ids).

Parameters

This callable has no explicit inputs.

Returns

• Type: int
• Direct return expressions: len(self.block_ids)

Exceptions and behavior

Method BlockTable.__len__ calls len; returns len(self.block_ids). No direct raise statement appears in this definition.

View source #L438-L439.

vllm_mlx.paged_cache.BlockTable.copy · method

vllm_mlx.paged_cache.BlockTable.copy(new_request_id: str) -> 'BlockTable'

Create a copy with new request ID.

Parameters

Name	Type	Required	Default	Description
new_request_id	str	yes	none	Required positional or keyword input.

Returns

• Type: 'BlockTable'
• Direct return expressions: BlockTable(request_id=new_request_id, block_ids=self.block_ids.copy(), num_tokens=self.num_tokens)

Exceptions and behavior

Method BlockTable.copy calls BlockTable, self.block_ids.copy; returns BlockTable(request_id=new_request_id, block_ids=self.block_ids.copy(), num_tokens=self.num_tokens). No direct raise statement appears in this definition.

View source #L441-L447.

vllm_mlx.paged_cache.CacheStats · class

vllm_mlx.paged_cache.CacheStats(total_blocks: int = 0, allocated_blocks: int = 0, free_blocks: int = 0, shared_blocks: int = 0, total_tokens_cached: int = 0, cache_hits: int = 0, cache_misses: int = 0, cow_copies: int = 0, evictions: int = 0)

Statistics for cache monitoring.

Parameters

Name	Type	Required	Default	Description
total_blocks	int	no	0	Optional constructor field; defaults to 0.
allocated_blocks	int	no	0	Optional constructor field; defaults to 0.
free_blocks	int	no	0	Optional constructor field; defaults to 0.
shared_blocks	int	no	0	Optional constructor field; defaults to 0.
total_tokens_cached	int	no	0	Optional constructor field; defaults to 0.
cache_hits	int	no	0	Optional constructor field; defaults to 0.
cache_misses	int	no	0	Optional constructor field; defaults to 0.
cow_copies	int	no	0	Optional constructor field; defaults to 0.
evictions	int	no	0	Optional constructor field; defaults to 0.

Returns

• Constructs: vllm_mlx.paged_cache.CacheStats

Exceptions and behavior

Class CacheStats declares 0 direct member(s). No direct raise statement appears in this definition.

View source #L456-L467.

vllm_mlx.paged_cache.PagedCacheManager · class

vllm_mlx.paged_cache.PagedCacheManager(block_size: int = 64, max_blocks: int = 1000, enable_caching: bool = True)

Paged KV cache manager following vLLM's BlockPool architecture.

Parameters

Name	Type	Required	Default	Description
block_size	int	no	64	Number of tokens per block (default: 64)
max_blocks	int	no	1000	Maximum number of blocks to allocate (default: 1000)
enable_caching	bool	no	True	Whether to enable prefix caching (default: True)

Returns

• Constructs: vllm_mlx.paged_cache.PagedCacheManager

Exceptions and behavior

Class PagedCacheManager declares 34 direct member(s). No direct raise statement appears in this definition.

View source #L475-L1197.

vllm_mlx.paged_cache.PagedCacheManager.__init__ · method

vllm_mlx.paged_cache.PagedCacheManager.__init__(block_size: int = 64, max_blocks: int = 1000, enable_caching: bool = True) -> not annotated

Method PagedCacheManager.__init__ updates self.block_size, self.max_blocks, self.enable_caching, self.blocks; calls CacheBlock, range, FreeKVCacheBlockQueue, BlockHashToBlockMap.

Parameters

Name	Type	Required	Default	Description
block_size	int	no	64	Optional positional or keyword input; defaults to 64.
max_blocks	int	no	1000	Optional positional or keyword input; defaults to 1000.
enable_caching	bool	no	True	Optional positional or keyword input; defaults to True.

Returns

• Type: not annotated

Exceptions and behavior

Method PagedCacheManager.__init__ updates self.block_size, self.max_blocks, self.enable_caching, self.blocks; calls CacheBlock, range, FreeKVCacheBlockQueue, BlockHashToBlockMap. No direct raise statement appears in this definition.

View source #L491-L540.

vllm_mlx.paged_cache.PagedCacheManager.allocate_block · method

vllm_mlx.paged_cache.PagedCacheManager.allocate_block() -> Optional[CacheBlock]

Allocate a new cache block.

Parameters

This callable has no explicit inputs.

Returns

• Type: Optional[CacheBlock]
• Direct return expressions: None; block

Exceptions and behavior

Method PagedCacheManager.allocate_block updates self.stats.allocated_blocks, self.stats.free_blocks; calls logger.warning, self.free_block_queue.popleft, self._maybe_evict_cached_block, block.touch; has 2 explicit return paths. No direct raise statement appears in this definition.

View source #L546-L571.

vllm_mlx.paged_cache.PagedCacheManager.get_new_blocks · method

vllm_mlx.paged_cache.PagedCacheManager.get_new_blocks(num_blocks: int) -> List[CacheBlock]

Allocate multiple blocks at once (vLLM style).

Parameters

Name	Type	Required	Default	Description
num_blocks	int	yes	none	Number of blocks to allocate

Returns

• Type: List[CacheBlock]
• Direct return expressions: blocks

Exceptions and behavior

Method PagedCacheManager.get_new_blocks updates self.stats.allocated_blocks, self.stats.free_blocks; calls ValueError, self.free_block_queue.popleft_n, self._maybe_evict_cached_block, block.touch; can raise ValueError; returns blocks. Directly raised exceptions: ValueError.

View source #L573-L606.

vllm_mlx.paged_cache.PagedCacheManager._maybe_evict_cached_block · method

vllm_mlx.paged_cache.PagedCacheManager._maybe_evict_cached_block(block: CacheBlock) -> bool

Evict a block from the hash cache if present.

Parameters

Name	Type	Required	Default	Description
block	CacheBlock	yes	none	Block to evict

Returns

• Type: bool
• Direct return expressions: False; True

Exceptions and behavior

Method PagedCacheManager._maybe_evict_cached_block updates self.stats.evictions; calls self.cached_block_hash_to_block.pop, block.reset_hash; has 2 explicit return paths. No direct raise statement appears in this definition.

View source #L608-L634.

vllm_mlx.paged_cache.PagedCacheManager.free_block · method

vllm_mlx.paged_cache.PagedCacheManager.free_block(block_id: int) -> bool

Free a cache block (decrements ref_count, frees if 0).

Parameters

Name	Type	Required	Default	Description
block_id	int	yes	none	Required positional or keyword input.

Returns

• Type: bool
• Direct return expressions: False; True

Exceptions and behavior

Method PagedCacheManager.free_block updates self.stats.allocated_blocks, self.stats.free_blocks, self.stats.total_tokens_cached; calls logger.warning, self.free_block_queue.append; has 2 explicit return paths. No direct raise statement appears in this definition.

View source #L636-L667.

vllm_mlx.paged_cache.PagedCacheManager.free_blocks · method

vllm_mlx.paged_cache.PagedCacheManager.free_blocks(blocks: Iterable[CacheBlock]) -> None

Free multiple blocks (vLLM style).

Parameters

Name	Type	Required	Default	Description
blocks	Iterable[CacheBlock]	yes	none	Blocks to free (in eviction order)

Returns

• Type: None

Exceptions and behavior

Method PagedCacheManager.free_blocks updates self.stats.allocated_blocks, self.stats.free_blocks, self.stats.total_tokens_cached; calls list, to_free.append, self.free_block_queue.append_n. No direct raise statement appears in this definition.

View source #L669-L696.

vllm_mlx.paged_cache.PagedCacheManager.touch · method

vllm_mlx.paged_cache.PagedCacheManager.touch(blocks: Iterable[CacheBlock]) -> None

Touch blocks to prevent eviction (cache hit, vLLM style).

Parameters

Name	Type	Required	Default	Description
blocks	Iterable[CacheBlock]	yes	none	Blocks to touch

Returns

• Type: None

Exceptions and behavior

Method PagedCacheManager.touch updates self.stats.free_blocks, self.stats.allocated_blocks; calls self.free_block_queue.remove, block.touch. No direct raise statement appears in this definition.

View source #L698-L720.

vllm_mlx.paged_cache.PagedCacheManager.increment_ref · method

vllm_mlx.paged_cache.PagedCacheManager.increment_ref(block_id: int) -> bool

Increment reference count for a block.

Parameters

Name	Type	Required	Default	Description
block_id	int	yes	none	Required positional or keyword input.

Returns

• Type: bool
• Direct return expressions: False; True

Exceptions and behavior

Method PagedCacheManager.increment_ref updates self.stats.shared_blocks; calls block.touch; has 2 explicit return paths. No direct raise statement appears in this definition.

View source #L726-L739.

vllm_mlx.paged_cache.PagedCacheManager.decrement_ref · method

vllm_mlx.paged_cache.PagedCacheManager.decrement_ref(block_id: int) -> bool

Decrement reference count (alias for free_block).

Parameters

Name	Type	Required	Default	Description
block_id	int	yes	none	Required positional or keyword input.

Returns

• Type: bool
• Direct return expressions: self.free_block(block_id)

Exceptions and behavior

Method PagedCacheManager.decrement_ref calls self.free_block; returns self.free_block(block_id). No direct raise statement appears in this definition.

View source #L741-L743.

vllm_mlx.paged_cache.PagedCacheManager.get_cached_block · method

vllm_mlx.paged_cache.PagedCacheManager.get_cached_block(block_hash: BlockHash) -> Optional[CacheBlock]

Get a cached block by its hash (vLLM style).

Parameters

Name	Type	Required	Default	Description
block_hash	BlockHash	yes	none	Content hash of the block

Returns

• Type: Optional[CacheBlock]
• Direct return expressions: None; block

Exceptions and behavior

Method PagedCacheManager.get_cached_block updates self.stats.cache_hits, self.stats.cache_misses; calls self.cached_block_hash_to_block.get_block; has 2 explicit return paths. No direct raise statement appears in this definition.

View source #L749-L768.

vllm_mlx.paged_cache.PagedCacheManager.cache_full_blocks · method

vllm_mlx.paged_cache.PagedCacheManager.cache_full_blocks(blocks: List[CacheBlock], token_ids: List[int], num_cached_blocks: int, num_full_blocks: int) -> None

Cache full blocks for prefix caching (vLLM style).

Parameters

Name	Type	Required	Default	Description
blocks	List[CacheBlock]	yes	none	All blocks for the request
token_ids	List[int]	yes	none	All token IDs for the request
num_cached_blocks	int	yes	none	Number of blocks already cached
num_full_blocks	int	yes	none	Number of full blocks to cache

Returns

• Type: None
• Direct return expressions: None

Exceptions and behavior

Method PagedCacheManager.cache_full_blocks calls range, compute_block_hash, len, self.cached_block_hash_to_block.insert; returns None. No direct raise statement appears in this definition.

View source #L770-L824.

vllm_mlx.paged_cache.PagedCacheManager.get_computed_blocks · method

vllm_mlx.paged_cache.PagedCacheManager.get_computed_blocks(token_ids: List[int]) -> Tuple[List[CacheBlock], int]

Find cached blocks for a token prefix (vLLM style).

Parameters

Name	Type	Required	Default	Description
token_ids	List[int]	yes	none	Token IDs to look up

Returns

• Type: Tuple[List[CacheBlock], int]
• Direct return expressions: ([], 0); (cached_blocks, num_cached_tokens)

Exceptions and behavior

Method PagedCacheManager.get_computed_blocks updates self.stats.cache_misses, self.stats.cache_hits; calls len, range, compute_block_hash, self.cached_block_hash_to_block.get_block; has 2 explicit return paths. No direct raise statement appears in this definition.

View source #L826-L868.

vllm_mlx.paged_cache.PagedCacheManager.compute_block_hash · method

vllm_mlx.paged_cache.PagedCacheManager.compute_block_hash(tokens: List[int]) -> str

Compute legacy string hash for a sequence of tokens.

Parameters

Name	Type	Required	Default	Description
tokens	List[int]	yes	none	Required positional or keyword input.

Returns

• Type: str
• Direct return expressions: hashlib.sha256(token_bytes).hexdigest()[:16]

Exceptions and behavior

Method PagedCacheManager.compute_block_hash calls b''.join, t.to_bytes, hashlib.sha256(token_bytes).hexdigest, hashlib.sha256; returns hashlib.sha256(token_bytes).hexdigest()[:16]. No direct raise statement appears in this definition.

View source #L875-L878.

vllm_mlx.paged_cache.PagedCacheManager.find_cached_block · method

vllm_mlx.paged_cache.PagedCacheManager.find_cached_block(tokens: List[int]) -> Optional[CacheBlock]

Find a cached block matching the given tokens (legacy method).

Parameters

Name	Type	Required	Default	Description
tokens	List[int]	yes	none	Required positional or keyword input.

Returns

• Type: Optional[CacheBlock]
• Direct return expressions: block; None

Exceptions and behavior

Method PagedCacheManager.find_cached_block updates self.stats.cache_hits, self.stats.cache_misses; calls self.compute_block_hash, block.touch; has 2 explicit return paths. No direct raise statement appears in this definition.

View source #L880-L896.

vllm_mlx.paged_cache.PagedCacheManager.register_block_hash · method

vllm_mlx.paged_cache.PagedCacheManager.register_block_hash(block: CacheBlock, tokens: List[int]) -> None

Register a block's hash for deduplication (legacy method).

Parameters

Name	Type	Required	Default	Description
block	CacheBlock	yes	none	Required positional or keyword input.
tokens	List[int]	yes	none	Required positional or keyword input.

Returns

• Type: None

Exceptions and behavior

Method PagedCacheManager.register_block_hash calls self.compute_block_hash. No direct raise statement appears in this definition.

View source #L898-L903.

vllm_mlx.paged_cache.PagedCacheManager.create_block_table · method

vllm_mlx.paged_cache.PagedCacheManager.create_block_table(request_id: str) -> BlockTable

Create a new block table for a request.

Parameters

Name	Type	Required	Default	Description
request_id	str	yes	none	Required positional or keyword input.

Returns

• Type: BlockTable
• Direct return expressions: table

Exceptions and behavior

Method PagedCacheManager.create_block_table calls BlockTable; returns table. No direct raise statement appears in this definition.

View source #L909-L914.

vllm_mlx.paged_cache.PagedCacheManager.get_block_table · method

vllm_mlx.paged_cache.PagedCacheManager.get_block_table(request_id: str) -> Optional[BlockTable]

Get block table for a request.

Parameters

Name	Type	Required	Default	Description
request_id	str	yes	none	Required positional or keyword input.

Returns

• Type: Optional[BlockTable]
• Direct return expressions: self.request_tables.get(request_id)

Exceptions and behavior

Method PagedCacheManager.get_block_table calls self.request_tables.get; returns self.request_tables.get(request_id). No direct raise statement appears in this definition.

View source #L916-L919.

vllm_mlx.paged_cache.PagedCacheManager.get_or_create_block_table · method

vllm_mlx.paged_cache.PagedCacheManager.get_or_create_block_table(request_id: str) -> BlockTable

Get or create block table for a request.

Parameters

Name	Type	Required	Default	Description
request_id	str	yes	none	Required positional or keyword input.

Returns

• Type: BlockTable
• Direct return expressions: self.request_tables[request_id]

Exceptions and behavior

Method PagedCacheManager.get_or_create_block_table calls BlockTable; returns self.request_tables[request_id]. No direct raise statement appears in this definition.

View source #L921-L926.

vllm_mlx.paged_cache.PagedCacheManager.delete_block_table · method

vllm_mlx.paged_cache.PagedCacheManager.delete_block_table(request_id: str) -> None

Delete block table and free associated blocks.

Parameters

Name	Type	Required	Default	Description
request_id	str	yes	none	Required positional or keyword input.

Returns

• Type: None

Exceptions and behavior

Method PagedCacheManager.delete_block_table calls self.request_tables.pop, self.free_block. No direct raise statement appears in this definition.

View source #L928-L934.

vllm_mlx.paged_cache.PagedCacheManager.add_block_to_table · method

vllm_mlx.paged_cache.PagedCacheManager.add_block_to_table(table: BlockTable, block: CacheBlock, tokens_in_block: int) -> None

Add a block to a block table.

Parameters

Name	Type	Required	Default	Description
table	BlockTable	yes	none	Required positional or keyword input.
block	CacheBlock	yes	none	Required positional or keyword input.
tokens_in_block	int	yes	none	Required positional or keyword input.

Returns

• Type: None

Exceptions and behavior

Method PagedCacheManager.add_block_to_table updates self.stats.total_tokens_cached; calls table.block_ids.append. No direct raise statement appears in this definition.

View source #L936-L947.

vllm_mlx.paged_cache.PagedCacheManager.find_shared_prefix · method

vllm_mlx.paged_cache.PagedCacheManager.find_shared_prefix(tokens: List[int]) -> Tuple[List[int], List[int]]

Find shared prefix blocks for a token sequence.

Parameters

Name	Type	Required	Default	Description
tokens	List[int]	yes	none	Required positional or keyword input.

Returns

• Type: Tuple[List[int], List[int]]
• Direct return expressions: (shared_blocks, remaining_tokens)

Exceptions and behavior

Method PagedCacheManager.find_shared_prefix calls tokens.copy, len, self.find_cached_block, shared_blocks.append; returns (shared_blocks, remaining_tokens). No direct raise statement appears in this definition.

View source #L953-L974.

vllm_mlx.paged_cache.PagedCacheManager.fork_block_table · method

vllm_mlx.paged_cache.PagedCacheManager.fork_block_table(source_table: BlockTable, new_request_id: str) -> BlockTable

Fork a block table for a new request (COW).

Parameters

Name	Type	Required	Default	Description
source_table	BlockTable	yes	none	Required positional or keyword input.
new_request_id	str	yes	none	Required positional or keyword input.

Returns

• Type: BlockTable
• Direct return expressions: new_table

Exceptions and behavior

Method PagedCacheManager.fork_block_table calls source_table.copy, self.increment_ref, logger.debug, len; returns new_table. No direct raise statement appears in this definition.

View source #L976-L997.

vllm_mlx.paged_cache.PagedCacheManager.get_blocks_for_generation · method

vllm_mlx.paged_cache.PagedCacheManager.get_blocks_for_generation(table: BlockTable) -> Tuple[List[CacheBlock], bool]

Get blocks for generation, applying COW if needed.

Parameters

Name	Type	Required	Default	Description
table	BlockTable	yes	none	Required positional or keyword input.

Returns

• Type: Tuple[List[CacheBlock], bool]
• Direct return expressions: (blocks, was_copied)

Exceptions and behavior

Method PagedCacheManager.get_blocks_for_generation updates self.stats.cow_copies; calls enumerate, self.allocated_blocks.get, block.is_shared, self._cow_copy_block; returns (blocks, was_copied). No direct raise statement appears in this definition.

View source #L999-L1029.

vllm_mlx.paged_cache.PagedCacheManager._cow_copy_block · method

vllm_mlx.paged_cache.PagedCacheManager._cow_copy_block(source_block: CacheBlock) -> Optional[CacheBlock]

Create a copy of a block for COW.

Parameters

Name	Type	Required	Default	Description
source_block	CacheBlock	yes	none	Required positional or keyword input.

Returns

• Type: Optional[CacheBlock]
• Direct return expressions: None; new_block

Exceptions and behavior

Method PagedCacheManager._cow_copy_block updates self.stats.shared_blocks; calls self.allocate_block, logger.debug; has 2 explicit return paths. No direct raise statement appears in this definition.

View source #L1031-L1046.

vllm_mlx.paged_cache.PagedCacheManager.allocate_blocks_for_tokens · method

vllm_mlx.paged_cache.PagedCacheManager.allocate_blocks_for_tokens(num_tokens: int) -> List[CacheBlock]

Allocate enough blocks to hold num_tokens.

Parameters

Name	Type	Required	Default	Description
num_tokens	int	yes	none	Required positional or keyword input.

Returns

• Type: List[CacheBlock]
• Direct return expressions: self.get_new_blocks(num_blocks_needed)

Exceptions and behavior

Method PagedCacheManager.allocate_blocks_for_tokens calls self.get_new_blocks; returns self.get_new_blocks(num_blocks_needed). No direct raise statement appears in this definition.

View source #L1052-L1055.

vllm_mlx.paged_cache.PagedCacheManager.evict_lru_blocks · method

vllm_mlx.paged_cache.PagedCacheManager.evict_lru_blocks(num_blocks: int) -> int

Evict least recently used blocks.

Parameters

Name	Type	Required	Default	Description
num_blocks	int	yes	none	Required positional or keyword input.

Returns

• Type: int
• Direct return expressions: evicted

Exceptions and behavior

Method PagedCacheManager.evict_lru_blocks calls range, min, self.free_block_queue.popleft, self._maybe_evict_cached_block; returns evicted. No direct raise statement appears in this definition.

View source #L1061-L1085.

vllm_mlx.paged_cache.PagedCacheManager.handle_memory_pressure · method

vllm_mlx.paged_cache.PagedCacheManager.handle_memory_pressure(requested_blocks: int) -> bool

Handle memory pressure by evicting blocks.

Parameters

Name	Type	Required	Default	Description
requested_blocks	int	yes	none	Required positional or keyword input.

Returns

• Type: bool
• Direct return expressions: True; self.free_block_queue.num_free_blocks >= requested_blocks

Exceptions and behavior

Method PagedCacheManager.handle_memory_pressure calls self.evict_lru_blocks; has 2 explicit return paths. No direct raise statement appears in this definition.

View source #L1087-L1096.

vllm_mlx.paged_cache.PagedCacheManager.free_blocks · method

vllm_mlx.paged_cache.PagedCacheManager.free_blocks() -> int

Number of free blocks available.

Parameters

This callable has no explicit inputs.

Returns

• Type: int
• Direct return expressions: self.free_block_queue.num_free_blocks

Exceptions and behavior

Method PagedCacheManager.free_blocks returns self.free_block_queue.num_free_blocks. No direct raise statement appears in this definition.

View source #L1103-L1105.

vllm_mlx.paged_cache.PagedCacheManager.usage · method

vllm_mlx.paged_cache.PagedCacheManager.usage() -> float

Cache usage ratio (0.0 to 1.0).

Parameters

This callable has no explicit inputs.

Returns

• Type: float
• Direct return expressions: 0.0; 1.0 - self.free_blocks / total

Exceptions and behavior

Method PagedCacheManager.usage has 2 explicit return paths. No direct raise statement appears in this definition.

View source #L1108-L1113.

vllm_mlx.paged_cache.PagedCacheManager.get_stats · method

vllm_mlx.paged_cache.PagedCacheManager.get_stats() -> CacheStats

Get current cache statistics.

Parameters

This callable has no explicit inputs.

Returns

• Type: CacheStats
• Direct return expressions: self.stats

Exceptions and behavior

Method PagedCacheManager.get_stats updates self.stats.shared_blocks, self.stats.free_blocks; calls sum, self.allocated_blocks.values; returns self.stats. No direct raise statement appears in this definition.

View source #L1115-L1122.

vllm_mlx.paged_cache.PagedCacheManager.get_memory_usage · method

vllm_mlx.paged_cache.PagedCacheManager.get_memory_usage() -> Dict[str, Any]

Get memory usage information.

Parameters

This callable has no explicit inputs.

Returns

• Type: Dict[str, Any]
• Direct return expressions: {'block_size': self.block_size, 'max_blocks': self.max_blocks, 'allocated_blocks': stats.allocated_blocks, 'free_blocks…

Exceptions and behavior

Method PagedCacheManager.get_memory_usage calls self.get_stats; returns {'block_size': self.block_size, 'max_blocks': self.max_blocks, 'allocated_blocks': stats.allocated_blocks, 'free_blocks…. No direct raise statement appears in this definition.

View source #L1124-L1141.

vllm_mlx.paged_cache.PagedCacheManager.reset_stats · method

vllm_mlx.paged_cache.PagedCacheManager.reset_stats() -> None

Reset statistics counters.

Parameters

This callable has no explicit inputs.

Returns

• Type: None

Exceptions and behavior

Method PagedCacheManager.reset_stats updates self.stats.cache_hits, self.stats.cache_misses, self.stats.cow_copies, self.stats.evictions. No direct raise statement appears in this definition.

View source #L1143-L1149.

vllm_mlx.paged_cache.PagedCacheManager.reset_prefix_cache · method

vllm_mlx.paged_cache.PagedCacheManager.reset_prefix_cache() -> bool

Reset the prefix cache.

Parameters

This callable has no explicit inputs.

Returns

• Type: bool
• Direct return expressions: False; True

Exceptions and behavior

Method PagedCacheManager.reset_prefix_cache updates self.stats.evictions, self.stats.cache_hits, self.stats.cache_misses; calls logger.warning, self.cached_block_hash_to_block.clear, self.hash_to_block.clear, block.reset_hash; has 2 explicit return paths. No direct raise statement appears in this definition.

View source #L1151-L1171.

vllm_mlx.paged_cache.PagedCacheManager.clear · method

vllm_mlx.paged_cache.PagedCacheManager.clear() -> None

Clear all cached data.

Parameters

This callable has no explicit inputs.

Returns

• Type: None

Exceptions and behavior

Method PagedCacheManager.clear updates self.blocks, self.free_block_queue, self.null_block, self.null_block.is_null; calls CacheBlock, range, FreeKVCacheBlockQueue, self.cached_block_hash_to_block.clear. No direct raise statement appears in this definition.

View source #L1173-L1197.

Complete symbol map

This map also includes private definitions and nested helpers. The signature column exposes every explicit input even when an internal helper has no dedicated parameter prose.

Symbol	Kind	Signature and inputs	What it does	Source
compute_block_hash	function	compute_block_hash(parent_hash: Optional[BlockHash], token_ids: List[int], extra_keys: Optional[Tuple[Any, ...]] = None) -> BlockHash	Compute hash for a block based on its content and parent block.	#L40-L75
CacheBlock	class	CacheBlock(block_id: int, ref_count: int = 0, block_hash: Optional[BlockHash] = None, prev_free_block: Optional['CacheBlock'] = None, next_free_block: Optional['CacheBlock'] = None, is_null: bool = False, cache_data: Optional[List[Tuple[Any, Any]]] = None, token_count: int = 0, hash_value: Optional[str] = None, last_access: float = field(default_factory=time.time))	KV cache block metadata following vLLM's design.	#L84-L146
CacheBlock.is_full	method	CacheBlock.is_full(block_size: int) -> bool	Check if block is at capacity.	#L123-L125
CacheBlock.is_shared	method	CacheBlock.is_shared() -> bool	Check if block is shared (ref_count > 1).	#L127-L129
CacheBlock.reset_hash	method	CacheBlock.reset_hash() -> None	Reset block hash when evicted from cache.	#L131-L134
CacheBlock.touch	method	CacheBlock.touch() -> None	Update last access time.	#L136-L138
CacheBlock.__repr__	method	CacheBlock.__repr__() -> str	Method CacheBlock.__repr__ returns f'CacheBlock(id={self.block_id}, ref={self.ref_count}, tokens={self.token_count}, prev={prev_id}, next={next_id})'.	#L140-L146
FreeKVCacheBlockQueue	class	FreeKVCacheBlockQueue(blocks: List[CacheBlock])	Doubly linked list of free blocks following vLLM's design.	#L158-L337
FreeKVCacheBlockQueue.__init__	method	FreeKVCacheBlockQueue.__init__(blocks: List[CacheBlock]) -> None	Initialize queue with all blocks as free.	#L174-L201
FreeKVCacheBlockQueue.popleft	method	FreeKVCacheBlockQueue.popleft() -> CacheBlock	Pop and return the first (LRU) free block.	#L203-L225
FreeKVCacheBlockQueue.popleft_n	method	FreeKVCacheBlockQueue.popleft_n(n: int) -> List[CacheBlock]	Pop n blocks from the front.	#L227-L265
FreeKVCacheBlockQueue.remove	method	FreeKVCacheBlockQueue.remove(block: CacheBlock) -> None	Remove a block from the middle of the queue.	#L267-L288
FreeKVCacheBlockQueue.append	method	FreeKVCacheBlockQueue.append(block: CacheBlock) -> None	Append a block to the end (MRU position).	#L290-L305
FreeKVCacheBlockQueue.append_n	method	FreeKVCacheBlockQueue.append_n(blocks: List[CacheBlock]) -> None	Append multiple blocks to the end.	#L307-L328
FreeKVCacheBlockQueue.get_all_free_blocks	method	FreeKVCacheBlockQueue.get_all_free_blocks() -> List[CacheBlock]	Get all free blocks (for testing).	#L330-L337
BlockHashToBlockMap	class	BlockHashToBlockMap()	Cache mapping block hashes to blocks for prefix caching.	#L345-L407
BlockHashToBlockMap.__init__	method	BlockHashToBlockMap.__init__() -> None	Method BlockHashToBlockMap.__init__ updates self._cache.	#L353-L354
BlockHashToBlockMap.get_block	method	BlockHashToBlockMap.get_block(block_hash: BlockHash) -> Optional[CacheBlock]	Get any block with the given hash.	#L356-L365
BlockHashToBlockMap.insert	method	BlockHashToBlockMap.insert(block_hash: BlockHash, block: CacheBlock) -> None	Insert a block into the cache.	#L367-L378
BlockHashToBlockMap.pop	method	BlockHashToBlockMap.pop(block_hash: BlockHash, block_id: int) -> Optional[CacheBlock]	Remove and return a specific block from the cache.	#L380-L399
BlockHashToBlockMap.__len__	method	BlockHashToBlockMap.__len__() -> int	Method BlockHashToBlockMap.__len__ calls len; returns len(self._cache).	#L401-L402
BlockHashToBlockMap.clear	method	BlockHashToBlockMap.clear() -> None	Remove every block-hash mapping without mutating the blocks.	#L404-L407
BlockTable	class	BlockTable(request_id: str, block_ids: List[int] = field(default_factory=list), num_tokens: int = 0)	Per-request block table mapping logical to physical blocks.	#L416-L447
BlockTable.add_block	method	BlockTable.add_block(block_id: int, num_tokens: int) -> None	Add a block to the table.	#L433-L436
BlockTable.__len__	method	BlockTable.__len__() -> int	Method BlockTable.__len__ calls len; returns len(self.block_ids).	#L438-L439
BlockTable.copy	method	BlockTable.copy(new_request_id: str) -> 'BlockTable'	Create a copy with new request ID.	#L441-L447
CacheStats	class	CacheStats(total_blocks: int = 0, allocated_blocks: int = 0, free_blocks: int = 0, shared_blocks: int = 0, total_tokens_cached: int = 0, cache_hits: int = 0, cache_misses: int = 0, cow_copies: int = 0, evictions: int = 0)	Statistics for cache monitoring.	#L456-L467
PagedCacheManager	class	PagedCacheManager(block_size: int = 64, max_blocks: int = 1000, enable_caching: bool = True)	Paged KV cache manager following vLLM's BlockPool architecture.	#L475-L1197
PagedCacheManager.__init__	method	PagedCacheManager.__init__(block_size: int = 64, max_blocks: int = 1000, enable_caching: bool = True) -> not annotated	Method PagedCacheManager.__init__ updates self.block_size, self.max_blocks, self.enable_caching, self.blocks; calls CacheBlock, range, FreeKVCacheBlockQueue, BlockHashToBlockMap.	#L491-L540
PagedCacheManager.allocate_block	method	PagedCacheManager.allocate_block() -> Optional[CacheBlock]	Allocate a new cache block.	#L546-L571
PagedCacheManager.get_new_blocks	method	PagedCacheManager.get_new_blocks(num_blocks: int) -> List[CacheBlock]	Allocate multiple blocks at once (vLLM style).	#L573-L606
PagedCacheManager._maybe_evict_cached_block	method	PagedCacheManager._maybe_evict_cached_block(block: CacheBlock) -> bool	Evict a block from the hash cache if present.	#L608-L634
PagedCacheManager.free_block	method	PagedCacheManager.free_block(block_id: int) -> bool	Free a cache block (decrements ref_count, frees if 0).	#L636-L667
PagedCacheManager.free_blocks	method	PagedCacheManager.free_blocks(blocks: Iterable[CacheBlock]) -> None	Free multiple blocks (vLLM style).	#L669-L696
PagedCacheManager.touch	method	PagedCacheManager.touch(blocks: Iterable[CacheBlock]) -> None	Touch blocks to prevent eviction (cache hit, vLLM style).	#L698-L720
PagedCacheManager.increment_ref	method	PagedCacheManager.increment_ref(block_id: int) -> bool	Increment reference count for a block.	#L726-L739
PagedCacheManager.decrement_ref	method	PagedCacheManager.decrement_ref(block_id: int) -> bool	Decrement reference count (alias for free_block).	#L741-L743
PagedCacheManager.get_cached_block	method	PagedCacheManager.get_cached_block(block_hash: BlockHash) -> Optional[CacheBlock]	Get a cached block by its hash (vLLM style).	#L749-L768
PagedCacheManager.cache_full_blocks	method	PagedCacheManager.cache_full_blocks(blocks: List[CacheBlock], token_ids: List[int], num_cached_blocks: int, num_full_blocks: int) -> None	Cache full blocks for prefix caching (vLLM style).	#L770-L824
PagedCacheManager.get_computed_blocks	method	PagedCacheManager.get_computed_blocks(token_ids: List[int]) -> Tuple[List[CacheBlock], int]	Find cached blocks for a token prefix (vLLM style).	#L826-L868
PagedCacheManager.compute_block_hash	method	PagedCacheManager.compute_block_hash(tokens: List[int]) -> str	Compute legacy string hash for a sequence of tokens.	#L875-L878
PagedCacheManager.find_cached_block	method	PagedCacheManager.find_cached_block(tokens: List[int]) -> Optional[CacheBlock]	Find a cached block matching the given tokens (legacy method).	#L880-L896
PagedCacheManager.register_block_hash	method	PagedCacheManager.register_block_hash(block: CacheBlock, tokens: List[int]) -> None	Register a block's hash for deduplication (legacy method).	#L898-L903
PagedCacheManager.create_block_table	method	PagedCacheManager.create_block_table(request_id: str) -> BlockTable	Create a new block table for a request.	#L909-L914
PagedCacheManager.get_block_table	method	PagedCacheManager.get_block_table(request_id: str) -> Optional[BlockTable]	Get block table for a request.	#L916-L919
PagedCacheManager.get_or_create_block_table	method	PagedCacheManager.get_or_create_block_table(request_id: str) -> BlockTable	Get or create block table for a request.	#L921-L926
PagedCacheManager.delete_block_table	method	PagedCacheManager.delete_block_table(request_id: str) -> None	Delete block table and free associated blocks.	#L928-L934
PagedCacheManager.add_block_to_table	method	PagedCacheManager.add_block_to_table(table: BlockTable, block: CacheBlock, tokens_in_block: int) -> None	Add a block to a block table.	#L936-L947
PagedCacheManager.find_shared_prefix	method	PagedCacheManager.find_shared_prefix(tokens: List[int]) -> Tuple[List[int], List[int]]	Find shared prefix blocks for a token sequence.	#L953-L974
PagedCacheManager.fork_block_table	method	PagedCacheManager.fork_block_table(source_table: BlockTable, new_request_id: str) -> BlockTable	Fork a block table for a new request (COW).	#L976-L997
PagedCacheManager.get_blocks_for_generation	method	PagedCacheManager.get_blocks_for_generation(table: BlockTable) -> Tuple[List[CacheBlock], bool]	Get blocks for generation, applying COW if needed.	#L999-L1029
PagedCacheManager._cow_copy_block	method	PagedCacheManager._cow_copy_block(source_block: CacheBlock) -> Optional[CacheBlock]	Create a copy of a block for COW.	#L1031-L1046
PagedCacheManager.allocate_blocks_for_tokens	method	PagedCacheManager.allocate_blocks_for_tokens(num_tokens: int) -> List[CacheBlock]	Allocate enough blocks to hold num_tokens.	#L1052-L1055
PagedCacheManager.evict_lru_blocks	method	PagedCacheManager.evict_lru_blocks(num_blocks: int) -> int	Evict least recently used blocks.	#L1061-L1085
PagedCacheManager.handle_memory_pressure	method	PagedCacheManager.handle_memory_pressure(requested_blocks: int) -> bool	Handle memory pressure by evicting blocks.	#L1087-L1096
PagedCacheManager.free_blocks	method	PagedCacheManager.free_blocks() -> int	Number of free blocks available.	#L1103-L1105
PagedCacheManager.usage	method	PagedCacheManager.usage() -> float	Cache usage ratio (0.0 to 1.0).	#L1108-L1113
PagedCacheManager.get_stats	method	PagedCacheManager.get_stats() -> CacheStats	Get current cache statistics.	#L1115-L1122
PagedCacheManager.get_memory_usage	method	PagedCacheManager.get_memory_usage() -> Dict[str, Any]	Get memory usage information.	#L1124-L1141
PagedCacheManager.reset_stats	method	PagedCacheManager.reset_stats() -> None	Reset statistics counters.	#L1143-L1149
PagedCacheManager.reset_prefix_cache	method	PagedCacheManager.reset_prefix_cache() -> bool	Reset the prefix cache.	#L1151-L1171
PagedCacheManager.clear	method	PagedCacheManager.clear() -> None	Clear all cached data.	#L1173-L1197