vllm_mlx.vision_embedding_cache

Vision Embedding Cache for MLLM continuous batching.

View the complete module source at #L1-L413.

API details

Each callable below includes its exact signature, type annotations, inputs, defaults, return contract, documented exceptions, implementation source, and parsed docstring sections when the source provides them.

vllm_mlx.vision_embedding_cache

Vision Embedding Cache for MLLM continuous batching.

This module provides caching for vision embeddings to avoid redundant computation when the same images are processed multiple times.

Cache Levels: 1. Pixel Values Cache - Caches processed image tensors (prepare_inputs output) 2. Vision Encoding Cache - Caches VLM forward pass output (logits + cache state)

Performance Impact: - Without cache: ~2s per image for vision encoding - With cache hit: ~0.01s (100x speedup for repeated images)

vllm_mlx.vision_embedding_cache.logger module-attribute

logger = logging.getLogger(__name__)

vllm_mlx.vision_embedding_cache.VisionCacheStats dataclass

VisionCacheStats(pixel_cache_hits: int = 0, pixel_cache_misses: int = 0, encoding_cache_hits: int = 0, encoding_cache_misses: int = 0, total_time_saved: float = 0.0, total_images_processed: int = 0)

Statistics for vision cache performance.

vllm_mlx.vision_embedding_cache.VisionCacheStats.pixel_cache_hits class-attribute instance-attribute

pixel_cache_hits: int = 0

vllm_mlx.vision_embedding_cache.VisionCacheStats.pixel_cache_misses class-attribute instance-attribute

pixel_cache_misses: int = 0

vllm_mlx.vision_embedding_cache.VisionCacheStats.encoding_cache_hits class-attribute instance-attribute

encoding_cache_hits: int = 0

vllm_mlx.vision_embedding_cache.VisionCacheStats.encoding_cache_misses class-attribute instance-attribute

encoding_cache_misses: int = 0

vllm_mlx.vision_embedding_cache.VisionCacheStats.total_time_saved class-attribute instance-attribute

total_time_saved: float = 0.0

vllm_mlx.vision_embedding_cache.VisionCacheStats.total_images_processed class-attribute instance-attribute

total_images_processed: int = 0

vllm_mlx.vision_embedding_cache.VisionCacheStats.pixel_hit_rate property

pixel_hit_rate: float

Return successful pixel-cache lookups divided by all pixel lookups.

vllm_mlx.vision_embedding_cache.VisionCacheStats.encoding_hit_rate property

encoding_hit_rate: float

Return successful encoding lookups divided by all encoding lookups.

vllm_mlx.vision_embedding_cache.VisionCacheStats.to_dict

to_dict() -> dict

Return pixel, encoding, timing, and image counters.

Source code in vllm_mlx/vision_embedding_cache.py

def to_dict(self) -> dict:
    """Return pixel, encoding, timing, and image counters."""

    return {
        "pixel_cache_hits": self.pixel_cache_hits,
        "pixel_cache_misses": self.pixel_cache_misses,
        "pixel_hit_rate": self.pixel_hit_rate,
        "encoding_cache_hits": self.encoding_cache_hits,
        "encoding_cache_misses": self.encoding_cache_misses,
        "encoding_hit_rate": self.encoding_hit_rate,
        "total_time_saved": self.total_time_saved,
        "total_images_processed": self.total_images_processed,
    }

vllm_mlx.vision_embedding_cache.PixelCacheEntry dataclass

PixelCacheEntry(pixel_values: array, input_ids: array, attention_mask: Optional[array], image_grid_thw: Optional[array], extra_kwargs: Dict[str, Any], processing_time: float = 0.0)

Cached pixel values from prepare_inputs.

vllm_mlx.vision_embedding_cache.PixelCacheEntry.pixel_values instance-attribute

pixel_values: array

vllm_mlx.vision_embedding_cache.PixelCacheEntry.input_ids instance-attribute

input_ids: array

vllm_mlx.vision_embedding_cache.PixelCacheEntry.attention_mask instance-attribute

attention_mask: Optional[array]

vllm_mlx.vision_embedding_cache.PixelCacheEntry.image_grid_thw instance-attribute

image_grid_thw: Optional[array]

vllm_mlx.vision_embedding_cache.PixelCacheEntry.extra_kwargs instance-attribute

extra_kwargs: Dict[str, Any]

vllm_mlx.vision_embedding_cache.PixelCacheEntry.processing_time class-attribute instance-attribute

processing_time: float = 0.0

vllm_mlx.vision_embedding_cache.PixelOnlyCacheEntry dataclass

PixelOnlyCacheEntry(pixel_values: array, image_grid_thw: Optional[array], processing_time: float = 0.0)

Cached pixel values only (prompt-independent).

This cache stores only the image-dependent data that doesn't change with different prompts. Useful when the same images are used with different prompts.

vllm_mlx.vision_embedding_cache.PixelOnlyCacheEntry.pixel_values instance-attribute

pixel_values: array

vllm_mlx.vision_embedding_cache.PixelOnlyCacheEntry.image_grid_thw instance-attribute

image_grid_thw: Optional[array]

vllm_mlx.vision_embedding_cache.PixelOnlyCacheEntry.processing_time class-attribute instance-attribute

processing_time: float = 0.0

vllm_mlx.vision_embedding_cache.EncodingCacheEntry dataclass

EncodingCacheEntry(logits: array, first_token: int, logprobs: array, encoding_time: float = 0.0)

Cached vision encoding output.

vllm_mlx.vision_embedding_cache.EncodingCacheEntry.logits instance-attribute

logits: array

vllm_mlx.vision_embedding_cache.EncodingCacheEntry.first_token instance-attribute

first_token: int

vllm_mlx.vision_embedding_cache.EncodingCacheEntry.logprobs instance-attribute

logprobs: array

vllm_mlx.vision_embedding_cache.EncodingCacheEntry.encoding_time class-attribute instance-attribute

encoding_time: float = 0.0

vllm_mlx.vision_embedding_cache.VisionEmbeddingCache

VisionEmbeddingCache(max_pixel_entries: int = 100, max_encoding_entries: int = 50, enabled: bool = True)

Two-level cache for vision processing in MLLM.

Level 1 (Pixel Cache): - Caches output of prepare_inputs() (pixel_values, input_ids, etc.) - Key: hash(images) + hash(prompt) - Saves: Image loading, resizing, normalization time (~0.5-1s)

Level 2 (Encoding Cache): - Caches output of VLM forward pass (logits, first token) - Key: hash(images) + hash(prompt) - Saves: Vision encoder computation time (~1-2s)

Example

cache = VisionEmbeddingCache(max_pixel_entries=50, max_encoding_entries=20)

First request - cache miss

pixel_entry = cache.get_pixel_cache(images, prompt) if pixel_entry is None: ... # Process images... ... cache.set_pixel_cache(images, prompt, pixel_values, ...)

Second request with same image - cache hit!

pixel_entry = cache.get_pixel_cache(images, prompt) # Returns cached data

Initialize the vision embedding cache.

Parameters:

• max_pixel_entries (int, default: 100 ) –

  Max entries in pixel cache (LRU eviction)

• max_encoding_entries (int, default: 50 ) –

  Max entries in encoding cache

• enabled (bool, default: True ) –

  Whether caching is enabled

Source code in vllm_mlx/vision_embedding_cache.py

def __init__(
    self,
    max_pixel_entries: int = 100,
    max_encoding_entries: int = 50,
    enabled: bool = True,
):
    """
    Initialize the vision embedding cache.

    Args:
        max_pixel_entries: Max entries in pixel cache (LRU eviction)
        max_encoding_entries: Max entries in encoding cache
        enabled: Whether caching is enabled
    """
    self.max_pixel_entries = max_pixel_entries
    self.max_encoding_entries = max_encoding_entries
    self.enabled = enabled

    # LRU caches using OrderedDict
    self._pixel_cache: OrderedDict[str, PixelCacheEntry] = OrderedDict()
    self._pixel_only_cache: OrderedDict[str, PixelOnlyCacheEntry] = OrderedDict()
    self._encoding_cache: OrderedDict[str, EncodingCacheEntry] = OrderedDict()

    self.stats = VisionCacheStats()

vllm_mlx.vision_embedding_cache.VisionEmbeddingCache.max_pixel_entries instance-attribute

max_pixel_entries = max_pixel_entries

vllm_mlx.vision_embedding_cache.VisionEmbeddingCache.max_encoding_entries instance-attribute

max_encoding_entries = max_encoding_entries

vllm_mlx.vision_embedding_cache.VisionEmbeddingCache.enabled instance-attribute

enabled = enabled

vllm_mlx.vision_embedding_cache.VisionEmbeddingCache._pixel_cache instance-attribute

_pixel_cache: OrderedDict[str, PixelCacheEntry] = OrderedDict()

vllm_mlx.vision_embedding_cache.VisionEmbeddingCache._pixel_only_cache instance-attribute

_pixel_only_cache: OrderedDict[str, PixelOnlyCacheEntry] = OrderedDict()

vllm_mlx.vision_embedding_cache.VisionEmbeddingCache._encoding_cache instance-attribute

_encoding_cache: OrderedDict[str, EncodingCacheEntry] = OrderedDict()

vllm_mlx.vision_embedding_cache.VisionEmbeddingCache.stats instance-attribute

stats = VisionCacheStats()

vllm_mlx.vision_embedding_cache.VisionEmbeddingCache._make_key

_make_key(images: List[str], prompt: str) -> str

Create cache key from images and prompt.

Source code in vllm_mlx/vision_embedding_cache.py

def _make_key(self, images: List[str], prompt: str) -> str:
    """Create cache key from images and prompt."""
    img_hash = compute_images_hash(images)
    # Use shorter prompt hash for cache key
    prompt_hash = hashlib.sha256(prompt.encode()).hexdigest()[:12]
    return f"{img_hash}_{prompt_hash}"

vllm_mlx.vision_embedding_cache.VisionEmbeddingCache._make_image_only_key

_make_image_only_key(images: List[str]) -> str

Create cache key from images only (prompt-independent).

Source code in vllm_mlx/vision_embedding_cache.py

def _make_image_only_key(self, images: List[str]) -> str:
    """Create cache key from images only (prompt-independent)."""
    return compute_images_hash(images)

vllm_mlx.vision_embedding_cache.VisionEmbeddingCache.get_pixel_cache

get_pixel_cache(images: List[str], prompt: str) -> Optional[PixelCacheEntry]

Get cached pixel values for images+prompt.

Returns:

• Optional[PixelCacheEntry] –

  PixelCacheEntry if found, None otherwise

Source code in vllm_mlx/vision_embedding_cache.py

def get_pixel_cache(
    self,
    images: List[str],
    prompt: str,
) -> Optional[PixelCacheEntry]:
    """
    Get cached pixel values for images+prompt.

    Returns:
        PixelCacheEntry if found, None otherwise
    """
    if not self.enabled or not images:
        return None

    key = self._make_key(images, prompt)

    if key in self._pixel_cache:
        # Move to end (most recently used)
        entry = self._pixel_cache.pop(key)
        self._pixel_cache[key] = entry

        self.stats.pixel_cache_hits += 1
        self.stats.total_time_saved += entry.processing_time
        logger.debug(
            f"Pixel cache hit: {key[:20]}... (saved {entry.processing_time:.2f}s)"
        )
        return entry

    self.stats.pixel_cache_misses += 1
    return None

vllm_mlx.vision_embedding_cache.VisionEmbeddingCache.set_pixel_cache

set_pixel_cache(images: List[str], prompt: str, pixel_values: array, input_ids: array, attention_mask: Optional[array] = None, image_grid_thw: Optional[array] = None, extra_kwargs: Optional[Dict[str, Any]] = None, processing_time: float = 0.0) -> None

Store pixel values in cache.

Source code in vllm_mlx/vision_embedding_cache.py

def set_pixel_cache(
    self,
    images: List[str],
    prompt: str,
    pixel_values: mx.array,
    input_ids: mx.array,
    attention_mask: Optional[mx.array] = None,
    image_grid_thw: Optional[mx.array] = None,
    extra_kwargs: Optional[Dict[str, Any]] = None,
    processing_time: float = 0.0,
) -> None:
    """Store pixel values in cache."""
    if not self.enabled or not images:
        return

    key = self._make_key(images, prompt)

    # Evict oldest if at capacity
    while len(self._pixel_cache) >= self.max_pixel_entries:
        oldest_key = next(iter(self._pixel_cache))
        del self._pixel_cache[oldest_key]
        logger.debug(f"Pixel cache evicted: {oldest_key[:20]}...")

    entry = PixelCacheEntry(
        pixel_values=pixel_values,
        input_ids=input_ids,
        attention_mask=attention_mask,
        image_grid_thw=image_grid_thw,
        extra_kwargs=extra_kwargs or {},
        processing_time=processing_time,
    )
    self._pixel_cache[key] = entry
    self.stats.total_images_processed += len(images)
    logger.debug(f"Pixel cache stored: {key[:20]}...")

vllm_mlx.vision_embedding_cache.VisionEmbeddingCache.get_pixel_values

get_pixel_values(images: List[str]) -> Optional[PixelOnlyCacheEntry]

Get cached pixel values for images (prompt-independent).

This is useful when the same images are used with different prompts. Only the pixel_values and image_grid_thw are cached (no input_ids).

Returns:

• Optional[PixelOnlyCacheEntry] –

  PixelOnlyCacheEntry if found, None otherwise

Source code in vllm_mlx/vision_embedding_cache.py

def get_pixel_values(
    self,
    images: List[str],
) -> Optional[PixelOnlyCacheEntry]:
    """
    Get cached pixel values for images (prompt-independent).

    This is useful when the same images are used with different prompts.
    Only the pixel_values and image_grid_thw are cached (no input_ids).

    Returns:
        PixelOnlyCacheEntry if found, None otherwise
    """
    if not self.enabled or not images:
        return None

    key = self._make_image_only_key(images)

    if key in self._pixel_only_cache:
        # Move to end (most recently used)
        entry = self._pixel_only_cache.pop(key)
        self._pixel_only_cache[key] = entry

        self.stats.pixel_cache_hits += 1
        self.stats.total_time_saved += entry.processing_time
        logger.debug(
            f"Pixel-only cache hit: {key[:16]}... (saved {entry.processing_time:.2f}s)"
        )
        return entry

    self.stats.pixel_cache_misses += 1
    return None

vllm_mlx.vision_embedding_cache.VisionEmbeddingCache.set_pixel_values

set_pixel_values(images: List[str], pixel_values: array, image_grid_thw: Optional[array] = None, processing_time: float = 0.0) -> None

Store pixel values in cache (prompt-independent).

Source code in vllm_mlx/vision_embedding_cache.py

def set_pixel_values(
    self,
    images: List[str],
    pixel_values: mx.array,
    image_grid_thw: Optional[mx.array] = None,
    processing_time: float = 0.0,
) -> None:
    """Store pixel values in cache (prompt-independent)."""
    if not self.enabled or not images:
        return

    key = self._make_image_only_key(images)

    # Evict oldest if at capacity
    while len(self._pixel_only_cache) >= self.max_pixel_entries:
        oldest_key = next(iter(self._pixel_only_cache))
        del self._pixel_only_cache[oldest_key]
        logger.debug(f"Pixel-only cache evicted: {oldest_key[:16]}...")

    entry = PixelOnlyCacheEntry(
        pixel_values=pixel_values,
        image_grid_thw=image_grid_thw,
        processing_time=processing_time,
    )
    self._pixel_only_cache[key] = entry
    logger.debug(f"Pixel-only cache stored: {key[:16]}...")

vllm_mlx.vision_embedding_cache.VisionEmbeddingCache.get_encoding_cache

get_encoding_cache(images: List[str], prompt: str) -> Optional[EncodingCacheEntry]

Get cached vision encoding output.

Returns:

• Optional[EncodingCacheEntry] –

  EncodingCacheEntry if found, None otherwise

Source code in vllm_mlx/vision_embedding_cache.py

def get_encoding_cache(
    self,
    images: List[str],
    prompt: str,
) -> Optional[EncodingCacheEntry]:
    """
    Get cached vision encoding output.

    Returns:
        EncodingCacheEntry if found, None otherwise
    """
    if not self.enabled or not images:
        return None

    key = self._make_key(images, prompt)

    if key in self._encoding_cache:
        entry = self._encoding_cache.pop(key)
        self._encoding_cache[key] = entry

        self.stats.encoding_cache_hits += 1
        self.stats.total_time_saved += entry.encoding_time
        logger.debug(
            f"Encoding cache hit: {key[:20]}... (saved {entry.encoding_time:.2f}s)"
        )
        return entry

    self.stats.encoding_cache_misses += 1
    return None

vllm_mlx.vision_embedding_cache.VisionEmbeddingCache.set_encoding_cache

set_encoding_cache(images: List[str], prompt: str, logits: array, first_token: int, logprobs: array, encoding_time: float = 0.0) -> None

Store vision encoding output in cache.

Source code in vllm_mlx/vision_embedding_cache.py

def set_encoding_cache(
    self,
    images: List[str],
    prompt: str,
    logits: mx.array,
    first_token: int,
    logprobs: mx.array,
    encoding_time: float = 0.0,
) -> None:
    """Store vision encoding output in cache."""
    if not self.enabled or not images:
        return

    key = self._make_key(images, prompt)

    # Evict oldest if at capacity
    while len(self._encoding_cache) >= self.max_encoding_entries:
        oldest_key = next(iter(self._encoding_cache))
        del self._encoding_cache[oldest_key]
        logger.debug(f"Encoding cache evicted: {oldest_key[:20]}...")

    entry = EncodingCacheEntry(
        logits=logits,
        first_token=first_token,
        logprobs=logprobs,
        encoding_time=encoding_time,
    )
    self._encoding_cache[key] = entry
    logger.debug(f"Encoding cache stored: {key[:20]}...")

vllm_mlx.vision_embedding_cache.VisionEmbeddingCache.get_stats

get_stats() -> dict

Get cache statistics.

Source code in vllm_mlx/vision_embedding_cache.py

def get_stats(self) -> dict:
    """Get cache statistics."""
    stats = self.stats.to_dict()
    stats["pixel_cache_size"] = len(self._pixel_cache)
    stats["pixel_only_cache_size"] = len(self._pixel_only_cache)
    stats["encoding_cache_size"] = len(self._encoding_cache)
    return stats

vllm_mlx.vision_embedding_cache.VisionEmbeddingCache.clear

clear() -> None

Clear all caches and reset stats.

Source code in vllm_mlx/vision_embedding_cache.py

def clear(self) -> None:
    """Clear all caches and reset stats."""
    self._pixel_cache.clear()
    self._pixel_only_cache.clear()
    self._encoding_cache.clear()
    self.stats = VisionCacheStats()

vllm_mlx.vision_embedding_cache.VisionEmbeddingCache.__repr__

__repr__() -> str

Source code in vllm_mlx/vision_embedding_cache.py

def __repr__(self) -> str:
    return (
        f"<VisionEmbeddingCache "
        f"pixel={len(self._pixel_cache)}/{self.max_pixel_entries} "
        f"pixel_only={len(self._pixel_only_cache)}/{self.max_pixel_entries} "
        f"encoding={len(self._encoding_cache)}/{self.max_encoding_entries}>"
    )

vllm_mlx.vision_embedding_cache.compute_image_hash

compute_image_hash(image_path: str) -> str

Compute hash of image content.

For files: hash the actual content For URLs/base64: hash the string

Source code in vllm_mlx/vision_embedding_cache.py

def compute_image_hash(image_path: str) -> str:
    """
    Compute hash of image content.

    For files: hash the actual content
    For URLs/base64: hash the string
    """
    try:
        path = Path(image_path)
        if path.exists() and path.is_file():
            # Hash full file content (not truncated — truncation can
            # cause collisions for images with identical headers)
            with open(path, "rb") as f:
                content = f.read()
            return hashlib.sha256(content).hexdigest()[:16]
        else:
            # Hash the string (URL or base64)
            return hashlib.sha256(image_path.encode()).hexdigest()[:16]
    except Exception:
        return hashlib.sha256(str(image_path).encode()).hexdigest()[:16]

vllm_mlx.vision_embedding_cache.compute_images_hash

compute_images_hash(images: List[str]) -> str

Compute combined hash for multiple images.

Source code in vllm_mlx/vision_embedding_cache.py

def compute_images_hash(images: List[str]) -> str:
    """Compute combined hash for multiple images."""
    if not images:
        return "no_images"
    hashes = sorted(compute_image_hash(img) for img in images)
    return hashlib.sha256("_".join(hashes).encode()).hexdigest()[:16]

Complete contract reference

Expand any definition for its exact inputs, annotations, defaults, return contract, directly raised exceptions, source-grounded behavior, and immutable line link. This section includes private and nested definitions that ordinary API generators omit.

vllm_mlx.vision_embedding_cache.VisionCacheStats · class

vllm_mlx.vision_embedding_cache.VisionCacheStats(pixel_cache_hits: int = 0, pixel_cache_misses: int = 0, encoding_cache_hits: int = 0, encoding_cache_misses: int = 0, total_time_saved: float = 0.0, total_images_processed: int = 0)

Statistics for vision cache performance.

Parameters

Name	Type	Required	Default	Description
pixel_cache_hits	int	no	0	Optional constructor field; defaults to 0.
pixel_cache_misses	int	no	0	Optional constructor field; defaults to 0.
encoding_cache_hits	int	no	0	Optional constructor field; defaults to 0.
encoding_cache_misses	int	no	0	Optional constructor field; defaults to 0.
total_time_saved	float	no	0.0	Optional constructor field; defaults to 0.0.
total_images_processed	int	no	0	Optional constructor field; defaults to 0.

Returns

• Constructs: vllm_mlx.vision_embedding_cache.VisionCacheStats

Exceptions and behavior

Class VisionCacheStats declares 3 direct member(s). No direct raise statement appears in this definition.

View source #L30-L66.

vllm_mlx.vision_embedding_cache.VisionCacheStats.pixel_hit_rate · method

vllm_mlx.vision_embedding_cache.VisionCacheStats.pixel_hit_rate() -> float

Return successful pixel-cache lookups divided by all pixel lookups.

Parameters

This callable has no explicit inputs.

Returns

• Type: float
• Direct return expressions: self.pixel_cache_hits / total if total > 0 else 0.0

Exceptions and behavior

Method VisionCacheStats.pixel_hit_rate returns self.pixel_cache_hits / total if total > 0 else 0.0. No direct raise statement appears in this definition.

View source #L41-L45.

vllm_mlx.vision_embedding_cache.VisionCacheStats.encoding_hit_rate · method

vllm_mlx.vision_embedding_cache.VisionCacheStats.encoding_hit_rate() -> float

Return successful encoding lookups divided by all encoding lookups.

Parameters

This callable has no explicit inputs.

Returns

• Type: float
• Direct return expressions: self.encoding_cache_hits / total if total > 0 else 0.0

Exceptions and behavior

Method VisionCacheStats.encoding_hit_rate returns self.encoding_cache_hits / total if total > 0 else 0.0. No direct raise statement appears in this definition.

View source #L48-L52.

vllm_mlx.vision_embedding_cache.VisionCacheStats.to_dict · method

vllm_mlx.vision_embedding_cache.VisionCacheStats.to_dict() -> dict

Return pixel, encoding, timing, and image counters.

Parameters

This callable has no explicit inputs.

Returns

• Type: dict
• Direct return expressions: {'pixel_cache_hits': self.pixel_cache_hits, 'pixel_cache_misses': self.pixel_cache_misses, 'pixel_hit_rate': self.pixel…

Exceptions and behavior

Method VisionCacheStats.to_dict returns {'pixel_cache_hits': self.pixel_cache_hits, 'pixel_cache_misses': self.pixel_cache_misses, 'pixel_hit_rate': self.pixel…. No direct raise statement appears in this definition.

View source #L54-L66.

vllm_mlx.vision_embedding_cache.PixelCacheEntry · class

vllm_mlx.vision_embedding_cache.PixelCacheEntry(pixel_values: mx.array, input_ids: mx.array, attention_mask: Optional[mx.array], image_grid_thw: Optional[mx.array], extra_kwargs: Dict[str, Any], processing_time: float = 0.0)

Cached pixel values from prepare_inputs.

Parameters

Name	Type	Required	Default	Description
pixel_values	mx.array	yes	none	Required constructor field.
input_ids	mx.array	yes	none	Required constructor field.
attention_mask	Optional[mx.array]	yes	none	Required constructor field.
image_grid_thw	Optional[mx.array]	yes	none	Required constructor field.
extra_kwargs	Dict[str, Any]	yes	none	Required constructor field.
processing_time	float	no	0.0	Optional constructor field; defaults to 0.0.

Returns

• Constructs: vllm_mlx.vision_embedding_cache.PixelCacheEntry

Exceptions and behavior

Class PixelCacheEntry declares 0 direct member(s). No direct raise statement appears in this definition.

View source #L70-L78.

vllm_mlx.vision_embedding_cache.PixelOnlyCacheEntry · class

vllm_mlx.vision_embedding_cache.PixelOnlyCacheEntry(pixel_values: mx.array, image_grid_thw: Optional[mx.array], processing_time: float = 0.0)

Cached pixel values only (prompt-independent).

Parameters

Name	Type	Required	Default	Description
pixel_values	mx.array	yes	none	Required constructor field.
image_grid_thw	Optional[mx.array]	yes	none	Required constructor field.
processing_time	float	no	0.0	Optional constructor field; defaults to 0.0.

Returns

• Constructs: vllm_mlx.vision_embedding_cache.PixelOnlyCacheEntry

Exceptions and behavior

Class PixelOnlyCacheEntry declares 0 direct member(s). No direct raise statement appears in this definition.

View source #L82-L92.

vllm_mlx.vision_embedding_cache.EncodingCacheEntry · class

vllm_mlx.vision_embedding_cache.EncodingCacheEntry(logits: mx.array, first_token: int, logprobs: mx.array, encoding_time: float = 0.0)

Cached vision encoding output.

Parameters

Name	Type	Required	Default	Description
logits	mx.array	yes	none	Required constructor field.
first_token	int	yes	none	Required constructor field.
logprobs	mx.array	yes	none	Required constructor field.
encoding_time	float	no	0.0	Optional constructor field; defaults to 0.0.

Returns

• Constructs: vllm_mlx.vision_embedding_cache.EncodingCacheEntry

Exceptions and behavior

Class EncodingCacheEntry declares 0 direct member(s). No direct raise statement appears in this definition.

View source #L96-L102.

vllm_mlx.vision_embedding_cache.compute_image_hash · function

vllm_mlx.vision_embedding_cache.compute_image_hash(image_path: str) -> str

Compute hash of image content.

Parameters

Name	Type	Required	Default	Description
image_path	str	yes	none	Required positional or keyword input.

Returns

• Type: str
• Direct return expressions: hashlib.sha256(content).hexdigest()[:16]; hashlib.sha256(image_path.encode()).hexdigest()[:16]; hashlib.sha256(str(image_path).encode()).hexdigest()[:16]

Exceptions and behavior

Function compute_image_hash calls Path, path.exists, path.is_file, open; has 3 explicit return paths. No direct raise statement appears in this definition.

View source #L105-L124.

vllm_mlx.vision_embedding_cache.compute_images_hash · function

vllm_mlx.vision_embedding_cache.compute_images_hash(images: List[str]) -> str

Compute combined hash for multiple images.

Parameters

Name	Type	Required	Default	Description
images	List[str]	yes	none	Required positional or keyword input.

Returns

• Type: str
• Direct return expressions: 'no_images'; hashlib.sha256('_'.join(hashes).encode()).hexdigest()[:16]

Exceptions and behavior

Function compute_images_hash calls sorted, compute_image_hash, hashlib.sha256('_'.join(hashes).encode()).hexdigest, hashlib.sha256; has 2 explicit return paths. No direct raise statement appears in this definition.

View source #L127-L132.

vllm_mlx.vision_embedding_cache.VisionEmbeddingCache · class

vllm_mlx.vision_embedding_cache.VisionEmbeddingCache(max_pixel_entries: int = 100, max_encoding_entries: int = 50, enabled: bool = True)

Two-level cache for vision processing in MLLM.

Parameters

Name	Type	Required	Default	Description
max_pixel_entries	int	no	100	Max entries in pixel cache (LRU eviction)
max_encoding_entries	int	no	50	Max entries in encoding cache
enabled	bool	no	True	Whether caching is enabled

Returns

• Constructs: vllm_mlx.vision_embedding_cache.VisionEmbeddingCache

Exceptions and behavior

Class VisionEmbeddingCache declares 12 direct member(s). No direct raise statement appears in this definition.

View source #L135-L413.

vllm_mlx.vision_embedding_cache.VisionEmbeddingCache.__init__ · method

vllm_mlx.vision_embedding_cache.VisionEmbeddingCache.__init__(max_pixel_entries: int = 100, max_encoding_entries: int = 50, enabled: bool = True) -> not annotated

Initialize the vision embedding cache.

Parameters

Name	Type	Required	Default	Description
max_pixel_entries	int	no	100	Max entries in pixel cache (LRU eviction)
max_encoding_entries	int	no	50	Max entries in encoding cache
enabled	bool	no	True	Whether caching is enabled

Returns

• Type: not annotated

Exceptions and behavior

Method VisionEmbeddingCache.__init__ updates self.max_pixel_entries, self.max_encoding_entries, self.enabled, self._pixel_cache; calls OrderedDict, VisionCacheStats. No direct raise statement appears in this definition.

View source #L162-L185.

vllm_mlx.vision_embedding_cache.VisionEmbeddingCache._make_key · method

vllm_mlx.vision_embedding_cache.VisionEmbeddingCache._make_key(images: List[str], prompt: str) -> str

Create cache key from images and prompt.

Parameters

Name	Type	Required	Default	Description
images	List[str]	yes	none	Required positional or keyword input.
prompt	str	yes	none	Required positional or keyword input.

Returns

• Type: str
• Direct return expressions: f'{img_hash}_{prompt_hash}'

Exceptions and behavior

Method VisionEmbeddingCache._make_key calls compute_images_hash, hashlib.sha256(prompt.encode()).hexdigest, hashlib.sha256, prompt.encode; returns f'{img_hash}_{prompt_hash}'. No direct raise statement appears in this definition.

View source #L187-L192.

vllm_mlx.vision_embedding_cache.VisionEmbeddingCache._make_image_only_key · method

vllm_mlx.vision_embedding_cache.VisionEmbeddingCache._make_image_only_key(images: List[str]) -> str

Create cache key from images only (prompt-independent).

Parameters

Name	Type	Required	Default	Description
images	List[str]	yes	none	Required positional or keyword input.

Returns

• Type: str
• Direct return expressions: compute_images_hash(images)

Exceptions and behavior

Method VisionEmbeddingCache._make_image_only_key calls compute_images_hash; returns compute_images_hash(images). No direct raise statement appears in this definition.

View source #L194-L196.

vllm_mlx.vision_embedding_cache.VisionEmbeddingCache.get_pixel_cache · method

vllm_mlx.vision_embedding_cache.VisionEmbeddingCache.get_pixel_cache(images: List[str], prompt: str) -> Optional[PixelCacheEntry]

Get cached pixel values for images+prompt.

Parameters

Name	Type	Required	Default	Description
images	List[str]	yes	none	Required positional or keyword input.
prompt	str	yes	none	Required positional or keyword input.

Returns

• Type: Optional[PixelCacheEntry]
• Direct return expressions: None; entry

Exceptions and behavior

Method VisionEmbeddingCache.get_pixel_cache updates self.stats.pixel_cache_hits, self.stats.total_time_saved, self.stats.pixel_cache_misses; calls self._make_key, self._pixel_cache.pop, logger.debug; has 2 explicit return paths. No direct raise statement appears in this definition.

View source #L200-L229.

vllm_mlx.vision_embedding_cache.VisionEmbeddingCache.set_pixel_cache · method

vllm_mlx.vision_embedding_cache.VisionEmbeddingCache.set_pixel_cache(images: List[str], prompt: str, pixel_values: mx.array, input_ids: mx.array, attention_mask: Optional[mx.array] = None, image_grid_thw: Optional[mx.array] = None, extra_kwargs: Optional[Dict[str, Any]] = None, processing_time: float = 0.0) -> None

Store pixel values in cache.

Parameters

Name	Type	Required	Default	Description
images	List[str]	yes	none	Required positional or keyword input.
prompt	str	yes	none	Required positional or keyword input.
pixel_values	mx.array	yes	none	Required positional or keyword input.
input_ids	mx.array	yes	none	Required positional or keyword input.
attention_mask	Optional[mx.array]	no	None	Optional positional or keyword input; defaults to None.
image_grid_thw	Optional[mx.array]	no	None	Optional positional or keyword input; defaults to None.
extra_kwargs	Optional[Dict[str, Any]]	no	None	Optional positional or keyword input; defaults to None.
processing_time	float	no	0.0	Optional positional or keyword input; defaults to 0.0.

Returns

• Type: None
• Direct return expressions: None

Exceptions and behavior

Method VisionEmbeddingCache.set_pixel_cache updates self.stats.total_images_processed; calls self._make_key, len, next, iter; returns None. No direct raise statement appears in this definition.

View source #L231-L264.

vllm_mlx.vision_embedding_cache.VisionEmbeddingCache.get_pixel_values · method

vllm_mlx.vision_embedding_cache.VisionEmbeddingCache.get_pixel_values(images: List[str]) -> Optional[PixelOnlyCacheEntry]

Get cached pixel values for images (prompt-independent).

Parameters

Name	Type	Required	Default	Description
images	List[str]	yes	none	Required positional or keyword input.

Returns

• Type: Optional[PixelOnlyCacheEntry]
• Direct return expressions: None; entry

Exceptions and behavior

Method VisionEmbeddingCache.get_pixel_values updates self.stats.pixel_cache_hits, self.stats.total_time_saved, self.stats.pixel_cache_misses; calls self._make_image_only_key, self._pixel_only_cache.pop, logger.debug; has 2 explicit return paths. No direct raise statement appears in this definition.

View source #L268-L299.

vllm_mlx.vision_embedding_cache.VisionEmbeddingCache.set_pixel_values · method

vllm_mlx.vision_embedding_cache.VisionEmbeddingCache.set_pixel_values(images: List[str], pixel_values: mx.array, image_grid_thw: Optional[mx.array] = None, processing_time: float = 0.0) -> None

Store pixel values in cache (prompt-independent).

Parameters

Name	Type	Required	Default	Description
images	List[str]	yes	none	Required positional or keyword input.
pixel_values	mx.array	yes	none	Required positional or keyword input.
image_grid_thw	Optional[mx.array]	no	None	Optional positional or keyword input; defaults to None.
processing_time	float	no	0.0	Optional positional or keyword input; defaults to 0.0.

Returns

• Type: None
• Direct return expressions: None

Exceptions and behavior

Method VisionEmbeddingCache.set_pixel_values calls self._make_image_only_key, len, next, iter; returns None. No direct raise statement appears in this definition.

View source #L301-L326.

vllm_mlx.vision_embedding_cache.VisionEmbeddingCache.get_encoding_cache · method

vllm_mlx.vision_embedding_cache.VisionEmbeddingCache.get_encoding_cache(images: List[str], prompt: str) -> Optional[EncodingCacheEntry]

Get cached vision encoding output.

Parameters

Name	Type	Required	Default	Description
images	List[str]	yes	none	Required positional or keyword input.
prompt	str	yes	none	Required positional or keyword input.

Returns

• Type: Optional[EncodingCacheEntry]
• Direct return expressions: None; entry

Exceptions and behavior

Method VisionEmbeddingCache.get_encoding_cache updates self.stats.encoding_cache_hits, self.stats.total_time_saved, self.stats.encoding_cache_misses; calls self._make_key, self._encoding_cache.pop, logger.debug; has 2 explicit return paths. No direct raise statement appears in this definition.

View source #L330-L358.

vllm_mlx.vision_embedding_cache.VisionEmbeddingCache.set_encoding_cache · method

vllm_mlx.vision_embedding_cache.VisionEmbeddingCache.set_encoding_cache(images: List[str], prompt: str, logits: mx.array, first_token: int, logprobs: mx.array, encoding_time: float = 0.0) -> None

Store vision encoding output in cache.

Parameters

Name	Type	Required	Default	Description
images	List[str]	yes	none	Required positional or keyword input.
prompt	str	yes	none	Required positional or keyword input.
logits	mx.array	yes	none	Required positional or keyword input.
first_token	int	yes	none	Required positional or keyword input.
logprobs	mx.array	yes	none	Required positional or keyword input.
encoding_time	float	no	0.0	Optional positional or keyword input; defaults to 0.0.

Returns

• Type: None
• Direct return expressions: None

Exceptions and behavior

Method VisionEmbeddingCache.set_encoding_cache calls self._make_key, len, next, iter; returns None. No direct raise statement appears in this definition.

View source #L360-L388.

vllm_mlx.vision_embedding_cache.VisionEmbeddingCache.get_stats · method

vllm_mlx.vision_embedding_cache.VisionEmbeddingCache.get_stats() -> dict

Get cache statistics.

Parameters

This callable has no explicit inputs.

Returns

• Type: dict
• Direct return expressions: stats

Exceptions and behavior

Method VisionEmbeddingCache.get_stats calls self.stats.to_dict, len; returns stats. No direct raise statement appears in this definition.

View source #L392-L398.

vllm_mlx.vision_embedding_cache.VisionEmbeddingCache.clear · method

vllm_mlx.vision_embedding_cache.VisionEmbeddingCache.clear() -> None

Clear all caches and reset stats.

Parameters

This callable has no explicit inputs.

Returns

• Type: None

Exceptions and behavior

Method VisionEmbeddingCache.clear updates self.stats; calls self._pixel_cache.clear, self._pixel_only_cache.clear, self._encoding_cache.clear, VisionCacheStats. No direct raise statement appears in this definition.

View source #L400-L405.

vllm_mlx.vision_embedding_cache.VisionEmbeddingCache.__repr__ · method

vllm_mlx.vision_embedding_cache.VisionEmbeddingCache.__repr__() -> str

Method VisionEmbeddingCache.__repr__ calls len; returns f'<VisionEmbeddingCache pixel={len(self._pixel_cache)}/{self.max_pixel_entries} pixel_only={len(self._pixel_only_cache)….

Parameters

This callable has no explicit inputs.

Returns

• Type: str
• Direct return expressions: f'<VisionEmbeddingCache pixel={len(self._pixel_cache)}/{self.max_pixel_entries} pixel_only={len(self._pixel_only_cache)…

Exceptions and behavior

Method VisionEmbeddingCache.__repr__ calls len; returns f'<VisionEmbeddingCache pixel={len(self._pixel_cache)}/{self.max_pixel_entries} pixel_only={len(self._pixel_only_cache)…. No direct raise statement appears in this definition.

View source #L407-L413.

Complete symbol map

This map also includes private definitions and nested helpers. The signature column exposes every explicit input even when an internal helper has no dedicated parameter prose.

Symbol	Kind	Signature and inputs	What it does	Source
VisionCacheStats	class	VisionCacheStats(pixel_cache_hits: int = 0, pixel_cache_misses: int = 0, encoding_cache_hits: int = 0, encoding_cache_misses: int = 0, total_time_saved: float = 0.0, total_images_processed: int = 0)	Statistics for vision cache performance.	#L30-L66
VisionCacheStats.pixel_hit_rate	method	VisionCacheStats.pixel_hit_rate() -> float	Return successful pixel-cache lookups divided by all pixel lookups.	#L41-L45
VisionCacheStats.encoding_hit_rate	method	VisionCacheStats.encoding_hit_rate() -> float	Return successful encoding lookups divided by all encoding lookups.	#L48-L52
VisionCacheStats.to_dict	method	VisionCacheStats.to_dict() -> dict	Return pixel, encoding, timing, and image counters.	#L54-L66
PixelCacheEntry	class	PixelCacheEntry(pixel_values: mx.array, input_ids: mx.array, attention_mask: Optional[mx.array], image_grid_thw: Optional[mx.array], extra_kwargs: Dict[str, Any], processing_time: float = 0.0)	Cached pixel values from prepare_inputs.	#L70-L78
PixelOnlyCacheEntry	class	PixelOnlyCacheEntry(pixel_values: mx.array, image_grid_thw: Optional[mx.array], processing_time: float = 0.0)	Cached pixel values only (prompt-independent).	#L82-L92
EncodingCacheEntry	class	EncodingCacheEntry(logits: mx.array, first_token: int, logprobs: mx.array, encoding_time: float = 0.0)	Cached vision encoding output.	#L96-L102
compute_image_hash	function	compute_image_hash(image_path: str) -> str	Compute hash of image content.	#L105-L124
compute_images_hash	function	compute_images_hash(images: List[str]) -> str	Compute combined hash for multiple images.	#L127-L132
VisionEmbeddingCache	class	VisionEmbeddingCache(max_pixel_entries: int = 100, max_encoding_entries: int = 50, enabled: bool = True)	Two-level cache for vision processing in MLLM.	#L135-L413
VisionEmbeddingCache.__init__	method	VisionEmbeddingCache.__init__(max_pixel_entries: int = 100, max_encoding_entries: int = 50, enabled: bool = True) -> not annotated	Initialize the vision embedding cache.	#L162-L185
VisionEmbeddingCache._make_key	method	VisionEmbeddingCache._make_key(images: List[str], prompt: str) -> str	Create cache key from images and prompt.	#L187-L192
VisionEmbeddingCache._make_image_only_key	method	VisionEmbeddingCache._make_image_only_key(images: List[str]) -> str	Create cache key from images only (prompt-independent).	#L194-L196
VisionEmbeddingCache.get_pixel_cache	method	VisionEmbeddingCache.get_pixel_cache(images: List[str], prompt: str) -> Optional[PixelCacheEntry]	Get cached pixel values for images+prompt.	#L200-L229
VisionEmbeddingCache.set_pixel_cache	method	VisionEmbeddingCache.set_pixel_cache(images: List[str], prompt: str, pixel_values: mx.array, input_ids: mx.array, attention_mask: Optional[mx.array] = None, image_grid_thw: Optional[mx.array] = None, extra_kwargs: Optional[Dict[str, Any]] = None, processing_time: float = 0.0) -> None	Store pixel values in cache.	#L231-L264
VisionEmbeddingCache.get_pixel_values	method	VisionEmbeddingCache.get_pixel_values(images: List[str]) -> Optional[PixelOnlyCacheEntry]	Get cached pixel values for images (prompt-independent).	#L268-L299
VisionEmbeddingCache.set_pixel_values	method	VisionEmbeddingCache.set_pixel_values(images: List[str], pixel_values: mx.array, image_grid_thw: Optional[mx.array] = None, processing_time: float = 0.0) -> None	Store pixel values in cache (prompt-independent).	#L301-L326
VisionEmbeddingCache.get_encoding_cache	method	VisionEmbeddingCache.get_encoding_cache(images: List[str], prompt: str) -> Optional[EncodingCacheEntry]	Get cached vision encoding output.	#L330-L358
VisionEmbeddingCache.set_encoding_cache	method	VisionEmbeddingCache.set_encoding_cache(images: List[str], prompt: str, logits: mx.array, first_token: int, logprobs: mx.array, encoding_time: float = 0.0) -> None	Store vision encoding output in cache.	#L360-L388
VisionEmbeddingCache.get_stats	method	VisionEmbeddingCache.get_stats() -> dict	Get cache statistics.	#L392-L398
VisionEmbeddingCache.clear	method	VisionEmbeddingCache.clear() -> None	Clear all caches and reset stats.	#L400-L405
VisionEmbeddingCache.__repr__	method	VisionEmbeddingCache.__repr__() -> str	Method VisionEmbeddingCache.__repr__ calls len; returns f'<VisionEmbeddingCache pixel={len(self._pixel_cache)}/{self.max_pixel_entries} pixel_only={len(self._pixel_only_cache)….	#L407-L413