vllm_mlx.mllm_cache

MLLM (Multimodal Language Model) Prefix Cache Manager.

View the complete module source at #L1-L459.

API details

Each callable below includes its exact signature, type annotations, inputs, defaults, return contract, documented exceptions, implementation source, and parsed docstring sections when the source provides them.

vllm_mlx.mllm_cache

MLLM (Multimodal Language Model) Prefix Cache Manager.

This module provides advanced caching for MLLM inference, implementing the LMCache-style approach for multimodal prefix caching:

Features: - Image content hashing for cache keys (LMCache style) - Vision embedding caching (skip encoder on hit) - KV cache state caching with prefix matching - Token ID tracking for partial prefix reuse - LRU eviction policy with memory limits - Stats tracking (hits, misses, tokens saved, encoder skips)

Based on research from: - LMCache: https://blog.lmcache.ai/2025-07-03-multimodal-models/ - vLLM Prefix Caching: https://docs.vllm.ai/en/stable/design/prefix_caching/ - mlx-lm cache_prompt: https://github.com/ml-explore/mlx-lm

vllm_mlx.mllm_cache.logger module-attribute

logger = logging.getLogger(__name__)

vllm_mlx.mllm_cache.MLLMCacheManager module-attribute

MLLMCacheManager = MLLMPrefixCacheManager

vllm_mlx.mllm_cache.VLMCacheStats module-attribute

VLMCacheStats = MLLMCacheStats

vllm_mlx.mllm_cache.VLMPrefixCacheEntry module-attribute

VLMPrefixCacheEntry = MLLMPrefixCacheEntry

vllm_mlx.mllm_cache.VLMCacheEntry module-attribute

VLMCacheEntry = MLLMPrefixCacheEntry

vllm_mlx.mllm_cache.VLMPrefixCacheManager module-attribute

VLMPrefixCacheManager = MLLMPrefixCacheManager

vllm_mlx.mllm_cache.VLMCacheManager module-attribute

VLMCacheManager = MLLMPrefixCacheManager

vllm_mlx.mllm_cache.MLLMCacheStats dataclass

MLLMCacheStats(hits: int = 0, misses: int = 0, partial_hits: int = 0, tokens_saved: int = 0, image_cache_hits: int = 0, vision_encoder_skips: int = 0, total_queries: int = 0, evictions: int = 0)

Statistics for MLLM cache performance.

vllm_mlx.mllm_cache.MLLMCacheStats.hits class-attribute instance-attribute

hits: int = 0

vllm_mlx.mllm_cache.MLLMCacheStats.misses class-attribute instance-attribute

misses: int = 0

vllm_mlx.mllm_cache.MLLMCacheStats.partial_hits class-attribute instance-attribute

partial_hits: int = 0

vllm_mlx.mllm_cache.MLLMCacheStats.tokens_saved class-attribute instance-attribute

tokens_saved: int = 0

vllm_mlx.mllm_cache.MLLMCacheStats.image_cache_hits class-attribute instance-attribute

image_cache_hits: int = 0

vllm_mlx.mllm_cache.MLLMCacheStats.vision_encoder_skips class-attribute instance-attribute

vision_encoder_skips: int = 0

vllm_mlx.mllm_cache.MLLMCacheStats.total_queries class-attribute instance-attribute

total_queries: int = 0

vllm_mlx.mllm_cache.MLLMCacheStats.evictions class-attribute instance-attribute

evictions: int = 0

vllm_mlx.mllm_cache.MLLMCacheStats.hit_rate property

hit_rate: float

Calculate cache hit rate.

vllm_mlx.mllm_cache.MLLMCacheStats.to_dict

to_dict() -> dict

Convert stats to dictionary.

Source code in vllm_mlx/mllm_cache.py

def to_dict(self) -> dict:
    """Convert stats to dictionary."""
    return {
        "hits": self.hits,
        "misses": self.misses,
        "partial_hits": self.partial_hits,
        "hit_rate": self.hit_rate,
        "tokens_saved": self.tokens_saved,
        "image_cache_hits": self.image_cache_hits,
        "vision_encoder_skips": self.vision_encoder_skips,
        "total_queries": self.total_queries,
        "evictions": self.evictions,
    }

vllm_mlx.mllm_cache.MLLMPrefixCacheEntry dataclass

MLLMPrefixCacheEntry(image_hash: str, prompt_hash: str, vision_embeddings: Any = None, kv_cache: list[Any] = list(), token_ids: list[int] = list(), num_image_tokens: int = 0, num_text_tokens: int = 0, prompt_tokens: int = 0, created_at: float = time(), hit_count: int = 0, model_name: str = '')

Enhanced cache entry storing vision embeddings, KV cache, and token IDs.

This enables: 1. Skipping vision encoder on image cache hit (saves ~1-2s per image) 2. Skipping prefix computation on token match (saves ~0.5s per 1k tokens) 3. Partial prefix reuse for multi-turn conversations

vllm_mlx.mllm_cache.MLLMPrefixCacheEntry.image_hash instance-attribute

image_hash: str

vllm_mlx.mllm_cache.MLLMPrefixCacheEntry.prompt_hash instance-attribute

prompt_hash: str

vllm_mlx.mllm_cache.MLLMPrefixCacheEntry.vision_embeddings class-attribute instance-attribute

vision_embeddings: Any = None

vllm_mlx.mllm_cache.MLLMPrefixCacheEntry.kv_cache class-attribute instance-attribute

kv_cache: list[Any] = field(default_factory=list)

vllm_mlx.mllm_cache.MLLMPrefixCacheEntry.token_ids class-attribute instance-attribute

token_ids: list[int] = field(default_factory=list)

vllm_mlx.mllm_cache.MLLMPrefixCacheEntry.num_image_tokens class-attribute instance-attribute

num_image_tokens: int = 0

vllm_mlx.mllm_cache.MLLMPrefixCacheEntry.num_text_tokens class-attribute instance-attribute

num_text_tokens: int = 0

vllm_mlx.mllm_cache.MLLMPrefixCacheEntry.prompt_tokens class-attribute instance-attribute

prompt_tokens: int = 0

vllm_mlx.mllm_cache.MLLMPrefixCacheEntry.created_at class-attribute instance-attribute

created_at: float = field(default_factory=time.time)

vllm_mlx.mllm_cache.MLLMPrefixCacheEntry.hit_count class-attribute instance-attribute

hit_count: int = 0

vllm_mlx.mllm_cache.MLLMPrefixCacheEntry.model_name class-attribute instance-attribute

model_name: str = ''

vllm_mlx.mllm_cache.MLLMPrefixCacheEntry.total_tokens property

total_tokens: int

Return the number of token IDs represented by this cache entry.

vllm_mlx.mllm_cache.MLLMPrefixCacheEntry.memory_size property

memory_size: int

Estimate memory usage in bytes.

vllm_mlx.mllm_cache.MLLMPrefixCacheEntry.get_prefix_match_length

get_prefix_match_length(new_token_ids: list[int]) -> int

Find how many tokens match between cached prefix and new input.

This is the key to prefix caching - if the first N tokens match, we can skip computing KV states for those N tokens.

Source code in vllm_mlx/mllm_cache.py

def get_prefix_match_length(self, new_token_ids: list[int]) -> int:
    """
    Find how many tokens match between cached prefix and new input.

    This is the key to prefix caching - if the first N tokens match,
    we can skip computing KV states for those N tokens.
    """
    match_length = 0
    for i, (cached, new) in enumerate(zip(self.token_ids, new_token_ids)):
        if cached != new:
            break
        match_length = i + 1
    return match_length

vllm_mlx.mllm_cache.MLLMPrefixCacheManager

MLLMPrefixCacheManager(max_entries: int = 50, max_memory_mb: int = 2048)

LRU Cache manager for MLLM prefix states with vision embedding caching.

Implements the LMCache approach for multimodal caching: 1. Hash-based identification of image+prompt combinations 2. Vision embedding caching (skip encoder on hit - saves 1-2s!) 3. KV cache reuse for matching prefixes 4. Token ID tracking for partial prefix matching 5. Memory-based eviction (configurable limit)

Example

cache = MLLMPrefixCacheManager(max_memory_mb=2048)

First request - cache miss, full computation

entry, match_len = cache.fetch(["image.jpg"], prompt, token_ids)

... run full forward pass ...

cache.store(["image.jpg"], prompt, vision_emb, kv_cache, token_ids)

Second request with same image - cache hit!

entry, match_len = cache.fetch(["image.jpg"], prompt, token_ids)

entry.vision_embeddings available - skip encoder!

match_len > 0 - skip prefix computation!

Performance (Gemma 3 27B, 256 image tokens): - Vision encoder: ~1.5s -> 0s (skip on hit) - Prefix computation: ~0.5s/1k tokens -> 0s (skip on match) - Multi-turn speedup: 8-12x for subsequent turns

Initialize MLLM prefix cache manager.

Parameters:

• max_entries (int, default: 50 ) –

  Maximum number of cache entries (default: 50)

• max_memory_mb (int, default: 2048 ) –

  Maximum memory in MB (default: 2048)

Source code in vllm_mlx/mllm_cache.py

def __init__(
    self,
    max_entries: int = 50,
    max_memory_mb: int = 2048,
):
    """
    Initialize MLLM prefix cache manager.

    Args:
        max_entries: Maximum number of cache entries (default: 50)
        max_memory_mb: Maximum memory in MB (default: 2048)
    """
    self.max_size = max_entries
    self.max_memory = max_memory_mb * 1024 * 1024
    self._cache: OrderedDict[str, MLLMPrefixCacheEntry] = OrderedDict()
    self._current_memory = 0
    self.stats = MLLMCacheStats()

vllm_mlx.mllm_cache.MLLMPrefixCacheManager.max_size instance-attribute

max_size = max_entries

vllm_mlx.mllm_cache.MLLMPrefixCacheManager.max_memory instance-attribute

max_memory = max_memory_mb * 1024 * 1024

vllm_mlx.mllm_cache.MLLMPrefixCacheManager._cache instance-attribute

_cache: OrderedDict[str, MLLMPrefixCacheEntry] = OrderedDict()

vllm_mlx.mllm_cache.MLLMPrefixCacheManager._current_memory instance-attribute

_current_memory = 0

vllm_mlx.mllm_cache.MLLMPrefixCacheManager.stats instance-attribute

stats = MLLMCacheStats()

vllm_mlx.mllm_cache.MLLMPrefixCacheManager._make_cache_key

_make_cache_key(images: list[str], prompt: str) -> str

Create cache key from images and prompt.

Source code in vllm_mlx/mllm_cache.py

def _make_cache_key(self, images: list[str], prompt: str) -> str:
    """Create cache key from images and prompt."""
    image_hash = compute_images_hash(images)
    prompt_hash = hashlib.sha256(prompt.encode()).hexdigest()[:16]
    return f"{image_hash}_{prompt_hash}"

vllm_mlx.mllm_cache.MLLMPrefixCacheManager._make_image_only_key

_make_image_only_key(images: list[str]) -> str

Create cache key for image-only lookup (vision embedding reuse).

Source code in vllm_mlx/mllm_cache.py

def _make_image_only_key(self, images: list[str]) -> str:
    """Create cache key for image-only lookup (vision embedding reuse)."""
    return compute_images_hash(images)

vllm_mlx.mllm_cache.MLLMPrefixCacheManager._evict_by_memory

_evict_by_memory(required_size: int) -> None

Evict entries until we have enough memory.

Source code in vllm_mlx/mllm_cache.py

def _evict_by_memory(self, required_size: int) -> None:
    """Evict entries until we have enough memory."""
    while self._current_memory + required_size > self.max_memory and self._cache:
        oldest_key = next(iter(self._cache))
        oldest_entry = self._cache.pop(oldest_key)
        self._current_memory -= oldest_entry.memory_size
        self.stats.evictions += 1
        logger.debug(f"MLLM cache evicted (memory): {oldest_key[:20]}...")

vllm_mlx.mllm_cache.MLLMPrefixCacheManager._evict_by_count

_evict_by_count() -> None

Evict entries until we're under max_size.

Source code in vllm_mlx/mllm_cache.py

def _evict_by_count(self) -> None:
    """Evict entries until we're under max_size."""
    while len(self._cache) >= self.max_size and self._cache:
        oldest_key = next(iter(self._cache))
        oldest_entry = self._cache.pop(oldest_key)
        self._current_memory -= oldest_entry.memory_size
        self.stats.evictions += 1
        logger.debug(f"MLLM cache evicted (count): {oldest_key[:20]}...")

vllm_mlx.mllm_cache.MLLMPrefixCacheManager.fetch

fetch(images: list[str], prompt: str, token_ids: list[int] | None = None) -> tuple[MLLMPrefixCacheEntry | None, int]

Fetch cached prefix state with prefix matching.

This is the main entry point for cache lookups. Returns both the cache entry (if found) and the prefix match length.

Parameters:

• images (list[str]) –

  List of image paths

• prompt (str) –

  Text prompt

• token_ids (list[int] | None, default: None ) –

  Optional token IDs for prefix matching

Returns:

• MLLMPrefixCacheEntry | None –

  Tuple of (entry, prefix_match_length) where:

• int –
  • entry: The cache entry if found, None otherwise
• tuple[MLLMPrefixCacheEntry | None, int] –
  • prefix_match_length: Number of tokens that match (0 if miss)

Source code in vllm_mlx/mllm_cache.py

def fetch(
    self,
    images: list[str],
    prompt: str,
    token_ids: list[int] | None = None,
) -> tuple[MLLMPrefixCacheEntry | None, int]:
    """
    Fetch cached prefix state with prefix matching.

    This is the main entry point for cache lookups. Returns both
    the cache entry (if found) and the prefix match length.

    Args:
        images: List of image paths
        prompt: Text prompt
        token_ids: Optional token IDs for prefix matching

    Returns:
        Tuple of (entry, prefix_match_length) where:
        - entry: The cache entry if found, None otherwise
        - prefix_match_length: Number of tokens that match (0 if miss)
    """
    self.stats.total_queries += 1
    cache_key = self._make_cache_key(images, prompt)

    if cache_key in self._cache:
        # Full cache hit - exact image+prompt match
        entry = self._cache.pop(cache_key)
        self._cache[cache_key] = entry  # Move to end (LRU)
        entry.hit_count += 1

        self.stats.hits += 1
        if images:
            self.stats.image_cache_hits += 1
        if entry.vision_embeddings is not None:
            self.stats.vision_encoder_skips += 1

        # Calculate prefix match length
        match_length = entry.total_tokens
        if token_ids:
            match_length = entry.get_prefix_match_length(token_ids)
            if match_length < entry.total_tokens:
                self.stats.partial_hits += 1

        self.stats.tokens_saved += match_length
        logger.debug(
            f"MLLM cache HIT: {cache_key[:32]}..., prefix_match={match_length}"
        )

        return entry, match_length

    # Check for image-only match (can reuse vision embeddings)
    if images:
        image_key = self._make_image_only_key(images)
        for key, entry in self._cache.items():
            if (
                entry.image_hash == image_key
                and entry.vision_embeddings is not None
            ):
                # Image match - can reuse vision embeddings!
                self.stats.partial_hits += 1
                self.stats.vision_encoder_skips += 1
                logger.debug(
                    f"MLLM cache PARTIAL HIT (vision only): image={image_key[:16]}"
                )

                # Return entry for vision embeddings, but 0 prefix match
                # (prompt is different, so KV cache can't be reused)
                return entry, 0

    self.stats.misses += 1
    logger.debug(f"MLLM cache MISS: {cache_key[:32]}...")
    return None, 0

vllm_mlx.mllm_cache.MLLMPrefixCacheManager.fetch_cache

fetch_cache(images: list[str], prompt: str) -> tuple[list[Any] | None, bool]

Legacy API: Fetch cached KV state for image+prompt combination.

For backwards compatibility with existing code.

Source code in vllm_mlx/mllm_cache.py

def fetch_cache(
    self,
    images: list[str],
    prompt: str,
) -> tuple[list[Any] | None, bool]:
    """
    Legacy API: Fetch cached KV state for image+prompt combination.

    For backwards compatibility with existing code.
    """
    entry, match_len = self.fetch(images, prompt)
    # For legacy API, return hit if entry exists (don't require token match)
    if entry is not None and entry.kv_cache is not None:
        return entry.kv_cache, True
    return None, False

vllm_mlx.mllm_cache.MLLMPrefixCacheManager.store

store(images: list[str], prompt: str, vision_embeddings: Any, kv_cache: list[Any], token_ids: list[int], num_image_tokens: int = 0, model_name: str = '') -> None

Store prefix state in cache.

Parameters:

• images (list[str]) –

  List of image paths

• prompt (str) –

  Text prompt

• vision_embeddings (Any) –

  Output of vision encoder (can be None for text-only)

• kv_cache (list[Any]) –

  Language model KV cache states

• token_ids (list[int]) –

  Full token sequence

• num_image_tokens (int, default: 0 ) –

  Number of image tokens (e.g., 256 for Gemma 3)

• model_name (str, default: '' ) –

  Model name for validation

Source code in vllm_mlx/mllm_cache.py

def store(
    self,
    images: list[str],
    prompt: str,
    vision_embeddings: Any,
    kv_cache: list[Any],
    token_ids: list[int],
    num_image_tokens: int = 0,
    model_name: str = "",
) -> None:
    """
    Store prefix state in cache.

    Args:
        images: List of image paths
        prompt: Text prompt
        vision_embeddings: Output of vision encoder (can be None for text-only)
        kv_cache: Language model KV cache states
        token_ids: Full token sequence
        num_image_tokens: Number of image tokens (e.g., 256 for Gemma 3)
        model_name: Model name for validation
    """
    cache_key = self._make_cache_key(images, prompt)

    entry = MLLMPrefixCacheEntry(
        image_hash=compute_images_hash(images),
        prompt_hash=hashlib.sha256(prompt.encode()).hexdigest()[:16],
        vision_embeddings=vision_embeddings,
        kv_cache=kv_cache,
        token_ids=token_ids,
        num_image_tokens=num_image_tokens,
        num_text_tokens=len(token_ids) - num_image_tokens,
        prompt_tokens=len(token_ids),
        model_name=model_name,
    )

    # Evict by memory first
    self._evict_by_memory(entry.memory_size)

    # Then evict by count
    self._evict_by_count()

    self._cache[cache_key] = entry
    self._current_memory += entry.memory_size

    logger.debug(
        f"MLLM cache STORED: key={cache_key[:32]}..., "
        f"tokens={len(token_ids)}, vision_emb={vision_embeddings is not None}, "
        f"memory={entry.memory_size / 1024 / 1024:.1f}MB"
    )

vllm_mlx.mllm_cache.MLLMPrefixCacheManager.store_cache

store_cache(images: list[str], prompt: str, cache: list[Any] | None, num_tokens: int = 0) -> None

Legacy API: Store KV cache for future reuse.

For backwards compatibility with existing code.

Source code in vllm_mlx/mllm_cache.py

def store_cache(
    self,
    images: list[str],
    prompt: str,
    cache: list[Any] | None,
    num_tokens: int = 0,
) -> None:
    """
    Legacy API: Store KV cache for future reuse.

    For backwards compatibility with existing code.
    """
    # Don't store empty or None caches
    if cache is None or (isinstance(cache, list) and len(cache) == 0):
        return

    self.store(
        images=images,
        prompt=prompt,
        vision_embeddings=None,
        kv_cache=cache,
        token_ids=[0] * num_tokens,  # Dummy token IDs
        num_image_tokens=0,
    )

vllm_mlx.mllm_cache.MLLMPrefixCacheManager.get_stats

get_stats() -> dict[str, Any]

Get cache statistics.

Source code in vllm_mlx/mllm_cache.py

def get_stats(self) -> dict[str, Any]:
    """Get cache statistics."""
    stats = self.stats.to_dict()
    stats["entries"] = len(self._cache)
    stats["max_entries"] = self.max_size
    stats["memory_used_mb"] = self._current_memory / 1024 / 1024
    stats["max_memory_mb"] = self.max_memory / 1024 / 1024
    return stats

vllm_mlx.mllm_cache.MLLMPrefixCacheManager.reset_stats

reset_stats() -> None

Reset statistics counters.

Source code in vllm_mlx/mllm_cache.py

def reset_stats(self) -> None:
    """Reset statistics counters."""
    self.stats = MLLMCacheStats()

vllm_mlx.mllm_cache.MLLMPrefixCacheManager.clear

clear() -> None

Clear all cached entries and reset stats.

Source code in vllm_mlx/mllm_cache.py

def clear(self) -> None:
    """Clear all cached entries and reset stats."""
    self._cache.clear()
    self._current_memory = 0
    self.reset_stats()

vllm_mlx.mllm_cache.MLLMPrefixCacheManager.__len__

__len__() -> int

Return number of cached entries.

Source code in vllm_mlx/mllm_cache.py

def __len__(self) -> int:
    """Return number of cached entries."""
    return len(self._cache)

vllm_mlx.mllm_cache.MLLMPrefixCacheManager.__repr__

__repr__() -> str

Source code in vllm_mlx/mllm_cache.py

def __repr__(self) -> str:
    mem_mb = self._current_memory / 1024 / 1024
    return f"<MLLMPrefixCacheManager entries={len(self)} memory={mem_mb:.1f}MB>"

vllm_mlx.mllm_cache.compute_image_hash

compute_image_hash(image_path: str) -> str

Compute hash of image content for cache key.

Following LMCache approach: hash the actual image bytes, not the path. This ensures cache hits even when the same image is loaded from different paths or as base64.

Parameters:

• image_path (str) –

  Path to image file

Returns:

• str –

  SHA256 hash of image content (first 16 chars)

Source code in vllm_mlx/mllm_cache.py

def compute_image_hash(image_path: str) -> str:
    """
    Compute hash of image content for cache key.

    Following LMCache approach: hash the actual image bytes, not the path.
    This ensures cache hits even when the same image is loaded from
    different paths or as base64.

    Args:
        image_path: Path to image file

    Returns:
        SHA256 hash of image content (first 16 chars)
    """
    try:
        path = Path(image_path)
        if path.exists():
            # Hash file content - this is the LMCache approach
            content = path.read_bytes()
            return hashlib.sha256(content).hexdigest()[:16]
        else:
            # Hash the string itself (for URLs or base64)
            return hashlib.sha256(image_path.encode()).hexdigest()[:16]
    except Exception as e:
        logger.warning(f"Failed to hash image: {e}")
        return hashlib.sha256(str(image_path).encode()).hexdigest()[:16]

vllm_mlx.mllm_cache.compute_images_hash

compute_images_hash(images: list[str]) -> str

Compute combined hash for multiple images.

Parameters:

• images (list[str]) –

  List of image paths/URLs

Returns:

• str –

  Combined hash string

Source code in vllm_mlx/mllm_cache.py

def compute_images_hash(images: list[str]) -> str:
    """
    Compute combined hash for multiple images.

    Args:
        images: List of image paths/URLs

    Returns:
        Combined hash string
    """
    if not images:
        return "no_images"

    hashes = [compute_image_hash(img) for img in images]
    combined = "_".join(sorted(hashes))
    return hashlib.sha256(combined.encode()).hexdigest()[:16]

Complete contract reference

Expand any definition for its exact inputs, annotations, defaults, return contract, directly raised exceptions, source-grounded behavior, and immutable line link. This section includes private and nested definitions that ordinary API generators omit.

vllm_mlx.mllm_cache.MLLMCacheStats · class

vllm_mlx.mllm_cache.MLLMCacheStats(hits: int = 0, misses: int = 0, partial_hits: int = 0, tokens_saved: int = 0, image_cache_hits: int = 0, vision_encoder_skips: int = 0, total_queries: int = 0, evictions: int = 0)

Statistics for MLLM cache performance.

Parameters

Name	Type	Required	Default	Description
hits	int	no	0	Optional constructor field; defaults to 0.
misses	int	no	0	Optional constructor field; defaults to 0.
partial_hits	int	no	0	Optional constructor field; defaults to 0.
tokens_saved	int	no	0	Optional constructor field; defaults to 0.
image_cache_hits	int	no	0	Optional constructor field; defaults to 0.
vision_encoder_skips	int	no	0	Optional constructor field; defaults to 0.
total_queries	int	no	0	Optional constructor field; defaults to 0.
evictions	int	no	0	Optional constructor field; defaults to 0.

Returns

• Constructs: vllm_mlx.mllm_cache.MLLMCacheStats

Exceptions and behavior

Class MLLMCacheStats declares 2 direct member(s). No direct raise statement appears in this definition.

View source #L34-L65.

vllm_mlx.mllm_cache.MLLMCacheStats.hit_rate · method

vllm_mlx.mllm_cache.MLLMCacheStats.hit_rate() -> float

Calculate cache hit rate.

Parameters

This callable has no explicit inputs.

Returns

• Type: float
• Direct return expressions: 0.0; self.hits / self.total_queries

Exceptions and behavior

Method MLLMCacheStats.hit_rate has 2 explicit return paths. No direct raise statement appears in this definition.

View source #L47-L51.

vllm_mlx.mllm_cache.MLLMCacheStats.to_dict · method

vllm_mlx.mllm_cache.MLLMCacheStats.to_dict() -> dict

Convert stats to dictionary.

Parameters

This callable has no explicit inputs.

Returns

• Type: dict
• Direct return expressions: {'hits': self.hits, 'misses': self.misses, 'partial_hits': self.partial_hits, 'hit_rate': self.hit_rate, 'tokens_saved'…

Exceptions and behavior

Method MLLMCacheStats.to_dict returns {'hits': self.hits, 'misses': self.misses, 'partial_hits': self.partial_hits, 'hit_rate': self.hit_rate, 'tokens_saved'…. No direct raise statement appears in this definition.

View source #L53-L65.

vllm_mlx.mllm_cache.MLLMPrefixCacheEntry · class

vllm_mlx.mllm_cache.MLLMPrefixCacheEntry(image_hash: str, prompt_hash: str, vision_embeddings: Any = None, kv_cache: list[Any] = field(default_factory=list), token_ids: list[int] = field(default_factory=list), num_image_tokens: int = 0, num_text_tokens: int = 0, prompt_tokens: int = 0, created_at: float = field(default_factory=time.time), hit_count: int = 0, model_name: str = '')

Enhanced cache entry storing vision embeddings, KV cache, and token IDs.

Parameters

Name	Type	Required	Default	Description
image_hash	str	yes	none	Required constructor field.
prompt_hash	str	yes	none	Required constructor field.
vision_embeddings	Any	no	None	Optional constructor field; defaults to None.
kv_cache	list[Any]	no	field(default_factory=list)	Optional constructor field; defaults to field(default_factory=list).
token_ids	list[int]	no	field(default_factory=list)	Optional constructor field; defaults to field(default_factory=list).
num_image_tokens	int	no	0	Optional constructor field; defaults to 0.
num_text_tokens	int	no	0	Optional constructor field; defaults to 0.
prompt_tokens	int	no	0	Optional constructor field; defaults to 0.
created_at	float	no	field(default_factory=time.time)	Optional constructor field; defaults to field(default_factory=time.time).
hit_count	int	no	0	Optional constructor field; defaults to 0.
model_name	str	no	''	Optional constructor field; defaults to ''.

Returns

• Constructs: vllm_mlx.mllm_cache.MLLMPrefixCacheEntry

Exceptions and behavior

Class MLLMPrefixCacheEntry declares 3 direct member(s). No direct raise statement appears in this definition.

View source #L69-L133.

vllm_mlx.mllm_cache.MLLMPrefixCacheEntry.total_tokens · method

vllm_mlx.mllm_cache.MLLMPrefixCacheEntry.total_tokens() -> int

Return the number of token IDs represented by this cache entry.

Parameters

This callable has no explicit inputs.

Returns

• Type: int
• Direct return expressions: len(self.token_ids)

Exceptions and behavior

Method MLLMPrefixCacheEntry.total_tokens calls len; returns len(self.token_ids). No direct raise statement appears in this definition.

View source #L99-L102.

vllm_mlx.mllm_cache.MLLMPrefixCacheEntry.memory_size · method

vllm_mlx.mllm_cache.MLLMPrefixCacheEntry.memory_size() -> int

Estimate memory usage in bytes.

Parameters

This callable has no explicit inputs.

Returns

• Type: int
• Direct return expressions: size

Exceptions and behavior

Method MLLMPrefixCacheEntry.memory_size calls hasattr; returns size. No direct raise statement appears in this definition.

View source #L105-L119.

vllm_mlx.mllm_cache.MLLMPrefixCacheEntry.get_prefix_match_length · method

vllm_mlx.mllm_cache.MLLMPrefixCacheEntry.get_prefix_match_length(new_token_ids: list[int]) -> int

Find how many tokens match between cached prefix and new input.

Parameters

Name	Type	Required	Default	Description
new_token_ids	list[int]	yes	none	Required positional or keyword input.

Returns

• Type: int
• Direct return expressions: match_length

Exceptions and behavior

Method MLLMPrefixCacheEntry.get_prefix_match_length calls enumerate, zip; returns match_length. No direct raise statement appears in this definition.

View source #L121-L133.

vllm_mlx.mllm_cache.compute_image_hash · function

vllm_mlx.mllm_cache.compute_image_hash(image_path: str) -> str

Compute hash of image content for cache key.

Parameters

Name	Type	Required	Default	Description
image_path	str	yes	none	Path to image file

Returns

• Type: str
• Direct return expressions: hashlib.sha256(content).hexdigest()[:16]; hashlib.sha256(image_path.encode()).hexdigest()[:16]; hashlib.sha256(str(image_path).encode()).hexdigest()[:16]

Exceptions and behavior

Function compute_image_hash calls Path, path.exists, path.read_bytes, hashlib.sha256(content).hexdigest; has 3 explicit return paths. No direct raise statement appears in this definition.

View source #L136-L161.

vllm_mlx.mllm_cache.compute_images_hash · function

vllm_mlx.mllm_cache.compute_images_hash(images: list[str]) -> str

Compute combined hash for multiple images.

Parameters

Name	Type	Required	Default	Description
images	list[str]	yes	none	List of image paths/URLs

Returns

• Type: str
• Direct return expressions: 'no_images'; hashlib.sha256(combined.encode()).hexdigest()[:16]

Exceptions and behavior

Function compute_images_hash calls compute_image_hash, '_'.join, sorted, hashlib.sha256(combined.encode()).hexdigest; has 2 explicit return paths. No direct raise statement appears in this definition.

View source #L164-L179.

vllm_mlx.mllm_cache.MLLMPrefixCacheManager · class

vllm_mlx.mllm_cache.MLLMPrefixCacheManager(max_entries: int = 50, max_memory_mb: int = 2048)

LRU Cache manager for MLLM prefix states with vision embedding caching.

Parameters

Name	Type	Required	Default	Description
max_entries	int	no	50	Maximum number of cache entries (default: 50)
max_memory_mb	int	no	2048	Maximum memory in MB (default: 2048)

Returns

• Constructs: vllm_mlx.mllm_cache.MLLMPrefixCacheManager

Exceptions and behavior

Class MLLMPrefixCacheManager declares 14 direct member(s). No direct raise statement appears in this definition.

View source #L182-L448.

vllm_mlx.mllm_cache.MLLMPrefixCacheManager.__init__ · method

vllm_mlx.mllm_cache.MLLMPrefixCacheManager.__init__(max_entries: int = 50, max_memory_mb: int = 2048) -> not annotated

Initialize MLLM prefix cache manager.

Parameters

Name	Type	Required	Default	Description
max_entries	int	no	50	Maximum number of cache entries (default: 50)
max_memory_mb	int	no	2048	Maximum memory in MB (default: 2048)

Returns

• Type: not annotated

Exceptions and behavior

Method MLLMPrefixCacheManager.__init__ updates self.max_size, self.max_memory, self._cache, self._current_memory; calls OrderedDict, MLLMCacheStats. No direct raise statement appears in this definition.

View source #L211-L227.

vllm_mlx.mllm_cache.MLLMPrefixCacheManager._make_cache_key · method

vllm_mlx.mllm_cache.MLLMPrefixCacheManager._make_cache_key(images: list[str], prompt: str) -> str

Create cache key from images and prompt.

Parameters

Name	Type	Required	Default	Description
images	list[str]	yes	none	Required positional or keyword input.
prompt	str	yes	none	Required positional or keyword input.

Returns

• Type: str
• Direct return expressions: f'{image_hash}_{prompt_hash}'

Exceptions and behavior

Method MLLMPrefixCacheManager._make_cache_key calls compute_images_hash, hashlib.sha256(prompt.encode()).hexdigest, hashlib.sha256, prompt.encode; returns f'{image_hash}_{prompt_hash}'. No direct raise statement appears in this definition.

View source #L229-L233.

vllm_mlx.mllm_cache.MLLMPrefixCacheManager._make_image_only_key · method

vllm_mlx.mllm_cache.MLLMPrefixCacheManager._make_image_only_key(images: list[str]) -> str

Create cache key for image-only lookup (vision embedding reuse).

Parameters

Name	Type	Required	Default	Description
images	list[str]	yes	none	Required positional or keyword input.

Returns

• Type: str
• Direct return expressions: compute_images_hash(images)

Exceptions and behavior

Method MLLMPrefixCacheManager._make_image_only_key calls compute_images_hash; returns compute_images_hash(images). No direct raise statement appears in this definition.

View source #L235-L237.

vllm_mlx.mllm_cache.MLLMPrefixCacheManager._evict_by_memory · method

vllm_mlx.mllm_cache.MLLMPrefixCacheManager._evict_by_memory(required_size: int) -> None

Evict entries until we have enough memory.

Parameters

Name	Type	Required	Default	Description
required_size	int	yes	none	Required positional or keyword input.

Returns

• Type: None

Exceptions and behavior

Method MLLMPrefixCacheManager._evict_by_memory updates self._current_memory, self.stats.evictions; calls next, iter, self._cache.pop, logger.debug. No direct raise statement appears in this definition.

View source #L239-L246.

vllm_mlx.mllm_cache.MLLMPrefixCacheManager._evict_by_count · method

vllm_mlx.mllm_cache.MLLMPrefixCacheManager._evict_by_count() -> None

Evict entries until we're under max_size.

Parameters

This callable has no explicit inputs.

Returns

• Type: None

Exceptions and behavior

Method MLLMPrefixCacheManager._evict_by_count updates self._current_memory, self.stats.evictions; calls len, next, iter, self._cache.pop. No direct raise statement appears in this definition.

View source #L248-L255.

vllm_mlx.mllm_cache.MLLMPrefixCacheManager.fetch · method

vllm_mlx.mllm_cache.MLLMPrefixCacheManager.fetch(images: list[str], prompt: str, token_ids: list[int] | None = None) -> tuple[MLLMPrefixCacheEntry | None, int]

Fetch cached prefix state with prefix matching.

Parameters

Name	Type	Required	Default	Description
images	list[str]	yes	none	List of image paths
prompt	str	yes	none	Text prompt
token_ids	list[int] \| None	no	None	Optional token IDs for prefix matching

Returns

• Type: tuple[MLLMPrefixCacheEntry | None, int]
• Direct return expressions: (entry, match_length); (entry, 0); (None, 0)

Exceptions and behavior

Method MLLMPrefixCacheManager.fetch updates self.stats.total_queries, self.stats.hits, self.stats.image_cache_hits, self.stats.vision_encoder_skips; calls self._make_cache_key, self._cache.pop, entry.get_prefix_match_length, logger.debug; has 3 explicit return paths. No direct raise statement appears in this definition.

View source #L257-L329.

vllm_mlx.mllm_cache.MLLMPrefixCacheManager.fetch_cache · method

vllm_mlx.mllm_cache.MLLMPrefixCacheManager.fetch_cache(images: list[str], prompt: str) -> tuple[list[Any] | None, bool]

Legacy API: Fetch cached KV state for image+prompt combination.

Parameters

Name	Type	Required	Default	Description
images	list[str]	yes	none	Required positional or keyword input.
prompt	str	yes	none	Required positional or keyword input.

Returns

• Type: tuple[list[Any] | None, bool]
• Direct return expressions: (entry.kv_cache, True); (None, False)

Exceptions and behavior

Method MLLMPrefixCacheManager.fetch_cache calls self.fetch; has 2 explicit return paths. No direct raise statement appears in this definition.

View source #L331-L345.

vllm_mlx.mllm_cache.MLLMPrefixCacheManager.store · method

vllm_mlx.mllm_cache.MLLMPrefixCacheManager.store(images: list[str], prompt: str, vision_embeddings: Any, kv_cache: list[Any], token_ids: list[int], num_image_tokens: int = 0, model_name: str = '') -> None

Store prefix state in cache.

Parameters

Name	Type	Required	Default	Description
images	list[str]	yes	none	List of image paths
prompt	str	yes	none	Text prompt
vision_embeddings	Any	yes	none	Output of vision encoder (can be None for text-only)
kv_cache	list[Any]	yes	none	Language model KV cache states
token_ids	list[int]	yes	none	Full token sequence
num_image_tokens	int	no	0	Number of image tokens (e.g., 256 for Gemma 3)
model_name	str	no	''	Model name for validation

Returns

• Type: None

Exceptions and behavior

Method MLLMPrefixCacheManager.store updates self._current_memory; calls self._make_cache_key, MLLMPrefixCacheEntry, compute_images_hash, hashlib.sha256(prompt.encode()).hexdigest. No direct raise statement appears in this definition.

View source #L347-L396.

vllm_mlx.mllm_cache.MLLMPrefixCacheManager.store_cache · method

vllm_mlx.mllm_cache.MLLMPrefixCacheManager.store_cache(images: list[str], prompt: str, cache: list[Any] | None, num_tokens: int = 0) -> None

Legacy API: Store KV cache for future reuse.

Parameters

Name	Type	Required	Default	Description
images	list[str]	yes	none	Required positional or keyword input.
prompt	str	yes	none	Required positional or keyword input.
cache	list[Any] \| None	yes	none	Required positional or keyword input.
num_tokens	int	no	0	Optional positional or keyword input; defaults to 0.

Returns

• Type: None
• Direct return expressions: None

Exceptions and behavior

Method MLLMPrefixCacheManager.store_cache calls isinstance, len, self.store; returns None. No direct raise statement appears in this definition.

View source #L398-L421.

vllm_mlx.mllm_cache.MLLMPrefixCacheManager.get_stats · method

vllm_mlx.mllm_cache.MLLMPrefixCacheManager.get_stats() -> dict[str, Any]

Get cache statistics.

Parameters

This callable has no explicit inputs.

Returns

• Type: dict[str, Any]
• Direct return expressions: stats

Exceptions and behavior

Method MLLMPrefixCacheManager.get_stats calls self.stats.to_dict, len; returns stats. No direct raise statement appears in this definition.

View source #L423-L430.

vllm_mlx.mllm_cache.MLLMPrefixCacheManager.reset_stats · method

vllm_mlx.mllm_cache.MLLMPrefixCacheManager.reset_stats() -> None

Reset statistics counters.

Parameters

This callable has no explicit inputs.

Returns

• Type: None

Exceptions and behavior

Method MLLMPrefixCacheManager.reset_stats updates self.stats; calls MLLMCacheStats. No direct raise statement appears in this definition.

View source #L432-L434.

vllm_mlx.mllm_cache.MLLMPrefixCacheManager.clear · method

vllm_mlx.mllm_cache.MLLMPrefixCacheManager.clear() -> None

Clear all cached entries and reset stats.

Parameters

This callable has no explicit inputs.

Returns

• Type: None

Exceptions and behavior

Method MLLMPrefixCacheManager.clear updates self._current_memory; calls self._cache.clear, self.reset_stats. No direct raise statement appears in this definition.

View source #L436-L440.

vllm_mlx.mllm_cache.MLLMPrefixCacheManager.__len__ · method

vllm_mlx.mllm_cache.MLLMPrefixCacheManager.__len__() -> int

Return number of cached entries.

Parameters

This callable has no explicit inputs.

Returns

• Type: int
• Direct return expressions: len(self._cache)

Exceptions and behavior

Method MLLMPrefixCacheManager.__len__ calls len; returns len(self._cache). No direct raise statement appears in this definition.

View source #L442-L444.

vllm_mlx.mllm_cache.MLLMPrefixCacheManager.__repr__ · method

vllm_mlx.mllm_cache.MLLMPrefixCacheManager.__repr__() -> str

Method MLLMPrefixCacheManager.__repr__ calls len; returns f'<MLLMPrefixCacheManager entries={len(self)} memory={mem_mb:.1f}MB>'.

Parameters

This callable has no explicit inputs.

Returns

• Type: str
• Direct return expressions: f'<MLLMPrefixCacheManager entries={len(self)} memory={mem_mb:.1f}MB>'

Exceptions and behavior

Method MLLMPrefixCacheManager.__repr__ calls len; returns f'<MLLMPrefixCacheManager entries={len(self)} memory={mem_mb:.1f}MB>'. No direct raise statement appears in this definition.

View source #L446-L448.

Complete symbol map

This map also includes private definitions and nested helpers. The signature column exposes every explicit input even when an internal helper has no dedicated parameter prose.

Symbol	Kind	Signature and inputs	What it does	Source
MLLMCacheStats	class	MLLMCacheStats(hits: int = 0, misses: int = 0, partial_hits: int = 0, tokens_saved: int = 0, image_cache_hits: int = 0, vision_encoder_skips: int = 0, total_queries: int = 0, evictions: int = 0)	Statistics for MLLM cache performance.	#L34-L65
MLLMCacheStats.hit_rate	method	MLLMCacheStats.hit_rate() -> float	Calculate cache hit rate.	#L47-L51
MLLMCacheStats.to_dict	method	MLLMCacheStats.to_dict() -> dict	Convert stats to dictionary.	#L53-L65
MLLMPrefixCacheEntry	class	MLLMPrefixCacheEntry(image_hash: str, prompt_hash: str, vision_embeddings: Any = None, kv_cache: list[Any] = field(default_factory=list), token_ids: list[int] = field(default_factory=list), num_image_tokens: int = 0, num_text_tokens: int = 0, prompt_tokens: int = 0, created_at: float = field(default_factory=time.time), hit_count: int = 0, model_name: str = '')	Enhanced cache entry storing vision embeddings, KV cache, and token IDs.	#L69-L133
MLLMPrefixCacheEntry.total_tokens	method	MLLMPrefixCacheEntry.total_tokens() -> int	Return the number of token IDs represented by this cache entry.	#L99-L102
MLLMPrefixCacheEntry.memory_size	method	MLLMPrefixCacheEntry.memory_size() -> int	Estimate memory usage in bytes.	#L105-L119
MLLMPrefixCacheEntry.get_prefix_match_length	method	MLLMPrefixCacheEntry.get_prefix_match_length(new_token_ids: list[int]) -> int	Find how many tokens match between cached prefix and new input.	#L121-L133
compute_image_hash	function	compute_image_hash(image_path: str) -> str	Compute hash of image content for cache key.	#L136-L161
compute_images_hash	function	compute_images_hash(images: list[str]) -> str	Compute combined hash for multiple images.	#L164-L179
MLLMPrefixCacheManager	class	MLLMPrefixCacheManager(max_entries: int = 50, max_memory_mb: int = 2048)	LRU Cache manager for MLLM prefix states with vision embedding caching.	#L182-L448
MLLMPrefixCacheManager.__init__	method	MLLMPrefixCacheManager.__init__(max_entries: int = 50, max_memory_mb: int = 2048) -> not annotated	Initialize MLLM prefix cache manager.	#L211-L227
MLLMPrefixCacheManager._make_cache_key	method	MLLMPrefixCacheManager._make_cache_key(images: list[str], prompt: str) -> str	Create cache key from images and prompt.	#L229-L233
MLLMPrefixCacheManager._make_image_only_key	method	MLLMPrefixCacheManager._make_image_only_key(images: list[str]) -> str	Create cache key for image-only lookup (vision embedding reuse).	#L235-L237
MLLMPrefixCacheManager._evict_by_memory	method	MLLMPrefixCacheManager._evict_by_memory(required_size: int) -> None	Evict entries until we have enough memory.	#L239-L246
MLLMPrefixCacheManager._evict_by_count	method	MLLMPrefixCacheManager._evict_by_count() -> None	Evict entries until we're under max_size.	#L248-L255
MLLMPrefixCacheManager.fetch	method	MLLMPrefixCacheManager.fetch(images: list[str], prompt: str, token_ids: list[int] \| None = None) -> tuple[MLLMPrefixCacheEntry \| None, int]	Fetch cached prefix state with prefix matching.	#L257-L329
MLLMPrefixCacheManager.fetch_cache	method	MLLMPrefixCacheManager.fetch_cache(images: list[str], prompt: str) -> tuple[list[Any] \| None, bool]	Legacy API: Fetch cached KV state for image+prompt combination.	#L331-L345
MLLMPrefixCacheManager.store	method	MLLMPrefixCacheManager.store(images: list[str], prompt: str, vision_embeddings: Any, kv_cache: list[Any], token_ids: list[int], num_image_tokens: int = 0, model_name: str = '') -> None	Store prefix state in cache.	#L347-L396
MLLMPrefixCacheManager.store_cache	method	MLLMPrefixCacheManager.store_cache(images: list[str], prompt: str, cache: list[Any] \| None, num_tokens: int = 0) -> None	Legacy API: Store KV cache for future reuse.	#L398-L421
MLLMPrefixCacheManager.get_stats	method	MLLMPrefixCacheManager.get_stats() -> dict[str, Any]	Get cache statistics.	#L423-L430
MLLMPrefixCacheManager.reset_stats	method	MLLMPrefixCacheManager.reset_stats() -> None	Reset statistics counters.	#L432-L434
MLLMPrefixCacheManager.clear	method	MLLMPrefixCacheManager.clear() -> None	Clear all cached entries and reset stats.	#L436-L440
MLLMPrefixCacheManager.__len__	method	MLLMPrefixCacheManager.__len__() -> int	Return number of cached entries.	#L442-L444
MLLMPrefixCacheManager.__repr__	method	MLLMPrefixCacheManager.__repr__() -> str	Method MLLMPrefixCacheManager.__repr__ calls len; returns f'<MLLMPrefixCacheManager entries={len(self)} memory={mem_mb:.1f}MB>'.	#L446-L448