Skip to content

vllm_mlx.model_registry

Registry-backed multi-model serving with memory-budget eviction.

View the complete module source at #L1-L1201.

API details

Each callable below includes its exact signature, type annotations, inputs, defaults, return contract, documented exceptions, implementation source, and parsed docstring sections when the source provides them.

vllm_mlx.model_registry

Registry-backed multi-model serving with memory-budget eviction.

The registry maps OpenAI-compatible model names to concrete local paths or declared HuggingFace IDs. Models are loaded lazily, optionally preloaded, and evicted according to a memory-budget policy with configurable wait/fail/preempt behaviour.

vllm_mlx.model_registry.logger module-attribute 

logger = logging.getLogger(__name__)

vllm_mlx.model_registry._ownership_registry module-attribute 

_ownership_registry = _ModelOwnershipRegistry()

vllm_mlx.model_registry.ContentionStrategy module-attribute 

ContentionStrategy = Literal['fail', 'wait', 'preempt', 'wait_then_fail', 'wait_then_preempt']

vllm_mlx.model_registry.EngineFactory module-attribute 

EngineFactory = Callable[['ResolvedModelConfig'], BaseEngine]

vllm_mlx.model_registry.ModelOwnershipError

Bases: RuntimeError

Raised when an EngineCore attempts to use a model already in use.

vllm_mlx.model_registry._ModelOwnershipRegistry 

_ModelOwnershipRegistry()

Process-local model ownership guard used by EngineCore.

Source code in vllm_mlx/model_registry.py 
45
46
def __init__(self) -> None:
    self._owners: dict[int, str] = {}

vllm_mlx.model_registry._ModelOwnershipRegistry._owners instance-attribute 

_owners: dict[int, str] = {}

vllm_mlx.model_registry._ModelOwnershipRegistry.acquire 

acquire(*, model: Any, engine: Any, engine_id: str, force: bool = True) -> None
Source code in vllm_mlx/model_registry.py 
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
def acquire(
    self,
    *,
    model: Any,
    engine: Any,
    engine_id: str,
    force: bool = True,
) -> None:
    key = id(model)
    owner = self._owners.get(key)
    if owner is not None and owner != engine_id and not force:
        raise ModelOwnershipError(
            f"Model is already owned by engine {owner}; "
            f"engine {engine_id} cannot acquire it"
        )
    self._owners[key] = engine_id

vllm_mlx.model_registry._ModelOwnershipRegistry.release 

release(model: Any, engine_id: str) -> None
Source code in vllm_mlx/model_registry.py 
65
66
67
68
69
def release(self, model: Any, engine_id: str) -> None:
    key = id(model)
    owner = self._owners.get(key)
    if owner == engine_id:
        self._owners.pop(key, None)

vllm_mlx.model_registry._ModelOwnershipRegistry.is_owned 

is_owned(model: Any) -> tuple[bool, str | None]
Source code in vllm_mlx/model_registry.py 
71
72
73
74
75
76
def is_owned(self, model: Any) -> tuple[bool, str | None]:
    key = id(model)
    owner = self._owners.get(key)
    if owner is not None:
        return (True, owner)
    return (False, None)

vllm_mlx.model_registry._ModelOwnershipRegistry.get_stats 

get_stats() -> dict[str, Any]
Source code in vllm_mlx/model_registry.py 
78
79
80
81
82
def get_stats(self) -> dict[str, Any]:
    return {
        "total_entries": len(self._owners),
        "active_owners": len(self._owners),
    }

vllm_mlx.model_registry.RegistryServeDefaults dataclass 

RegistryServeDefaults(continuous_batching: bool, force_mllm: bool, enable_mtp: bool, prefill_step_size: int, specprefill_enabled: bool, specprefill_threshold: int, specprefill_keep_pct: float, specprefill_backbone_pct: float, specprefill_draft_model: str | None, stream_interval: int, gpu_memory_utilization: float, scheduler_config: SchedulerConfig | None, max_tokens: int, download_config: DownloadConfig)

Global serve defaults inherited by registry entries.

vllm_mlx.model_registry.RegistryServeDefaults.continuous_batching instance-attribute 

continuous_batching: bool

vllm_mlx.model_registry.RegistryServeDefaults.force_mllm instance-attribute 

force_mllm: bool

vllm_mlx.model_registry.RegistryServeDefaults.enable_mtp instance-attribute 

enable_mtp: bool

vllm_mlx.model_registry.RegistryServeDefaults.prefill_step_size instance-attribute 

prefill_step_size: int

vllm_mlx.model_registry.RegistryServeDefaults.specprefill_enabled instance-attribute 

specprefill_enabled: bool

vllm_mlx.model_registry.RegistryServeDefaults.specprefill_threshold instance-attribute 

specprefill_threshold: int

vllm_mlx.model_registry.RegistryServeDefaults.specprefill_keep_pct instance-attribute 

specprefill_keep_pct: float

vllm_mlx.model_registry.RegistryServeDefaults.specprefill_backbone_pct instance-attribute 

specprefill_backbone_pct: float

vllm_mlx.model_registry.RegistryServeDefaults.specprefill_draft_model instance-attribute 

specprefill_draft_model: str | None

vllm_mlx.model_registry.RegistryServeDefaults.stream_interval instance-attribute 

stream_interval: int

vllm_mlx.model_registry.RegistryServeDefaults.gpu_memory_utilization instance-attribute 

gpu_memory_utilization: float

vllm_mlx.model_registry.RegistryServeDefaults.scheduler_config instance-attribute 

scheduler_config: SchedulerConfig | None

vllm_mlx.model_registry.RegistryServeDefaults.max_tokens instance-attribute 

max_tokens: int

vllm_mlx.model_registry.RegistryServeDefaults.download_config instance-attribute 

download_config: DownloadConfig

vllm_mlx.model_registry.ContentionPolicy dataclass 

ContentionPolicy(strategy: ContentionStrategy = 'wait_then_fail', wait_timeout_s: float | None = 30.0, preempt_after_s: float | None = None)

Policy used when a new model cannot fit inside the memory budget.

vllm_mlx.model_registry.ContentionPolicy.strategy class-attribute instance-attribute 

strategy: ContentionStrategy = 'wait_then_fail'

vllm_mlx.model_registry.ContentionPolicy.wait_timeout_s class-attribute instance-attribute 

wait_timeout_s: float | None = 30.0

vllm_mlx.model_registry.ContentionPolicy.preempt_after_s class-attribute instance-attribute 

preempt_after_s: float | None = None

vllm_mlx.model_registry.RegistryManagerConfig dataclass 

RegistryManagerConfig(memory_budget_bytes: int, policy: ContentionPolicy)

Global registry manager configuration.

vllm_mlx.model_registry.RegistryManagerConfig.memory_budget_bytes instance-attribute 

memory_budget_bytes: int

vllm_mlx.model_registry.RegistryManagerConfig.policy instance-attribute 

policy: ContentionPolicy

vllm_mlx.model_registry.RegisteredModel dataclass 

RegisteredModel(name: str, source: str, preload: bool = False, continuous_batching: bool | None = None, force_mllm: bool | None = None, enable_mtp: bool | None = None, prefill_step_size: int | None = None, specprefill_enabled: bool | None = None, specprefill_threshold: int | None = None, specprefill_keep_pct: float | None = None, specprefill_backbone_pct: float | None = None, specprefill_draft_model: str | None = None, stream_interval: int | None = None, gpu_memory_utilization: float | None = None, estimated_memory_bytes: int | None = None)

One configured model entry.

vllm_mlx.model_registry.RegisteredModel.name instance-attribute 

name: str

vllm_mlx.model_registry.RegisteredModel.source instance-attribute 

source: str

vllm_mlx.model_registry.RegisteredModel.preload class-attribute instance-attribute 

preload: bool = False

vllm_mlx.model_registry.RegisteredModel.continuous_batching class-attribute instance-attribute 

continuous_batching: bool | None = None

vllm_mlx.model_registry.RegisteredModel.force_mllm class-attribute instance-attribute 

force_mllm: bool | None = None

vllm_mlx.model_registry.RegisteredModel.enable_mtp class-attribute instance-attribute 

enable_mtp: bool | None = None

vllm_mlx.model_registry.RegisteredModel.prefill_step_size class-attribute instance-attribute 

prefill_step_size: int | None = None

vllm_mlx.model_registry.RegisteredModel.specprefill_enabled class-attribute instance-attribute 

specprefill_enabled: bool | None = None

vllm_mlx.model_registry.RegisteredModel.specprefill_threshold class-attribute instance-attribute 

specprefill_threshold: int | None = None

vllm_mlx.model_registry.RegisteredModel.specprefill_keep_pct class-attribute instance-attribute 

specprefill_keep_pct: float | None = None

vllm_mlx.model_registry.RegisteredModel.specprefill_backbone_pct class-attribute instance-attribute 

specprefill_backbone_pct: float | None = None

vllm_mlx.model_registry.RegisteredModel.specprefill_draft_model class-attribute instance-attribute 

specprefill_draft_model: str | None = None

vllm_mlx.model_registry.RegisteredModel.stream_interval class-attribute instance-attribute 

stream_interval: int | None = None

vllm_mlx.model_registry.RegisteredModel.gpu_memory_utilization class-attribute instance-attribute 

gpu_memory_utilization: float | None = None

vllm_mlx.model_registry.RegisteredModel.estimated_memory_bytes class-attribute instance-attribute 

estimated_memory_bytes: int | None = None

vllm_mlx.model_registry.ResolvedModelConfig dataclass 

ResolvedModelConfig(entry: RegisteredModel, resolved_source: str, continuous_batching: bool, force_mllm: bool, enable_mtp: bool, prefill_step_size: int, specprefill_enabled: bool, specprefill_threshold: int, specprefill_keep_pct: float, specprefill_backbone_pct: float, specprefill_draft_model: str | None, stream_interval: int, gpu_memory_utilization: float, scheduler_config: SchedulerConfig | None, estimated_memory_bytes: int)

Effective configuration for a loaded model.

vllm_mlx.model_registry.ResolvedModelConfig.entry instance-attribute 

entry: RegisteredModel

vllm_mlx.model_registry.ResolvedModelConfig.resolved_source instance-attribute 

resolved_source: str

vllm_mlx.model_registry.ResolvedModelConfig.continuous_batching instance-attribute 

continuous_batching: bool

vllm_mlx.model_registry.ResolvedModelConfig.force_mllm instance-attribute 

force_mllm: bool

vllm_mlx.model_registry.ResolvedModelConfig.enable_mtp instance-attribute 

enable_mtp: bool

vllm_mlx.model_registry.ResolvedModelConfig.prefill_step_size instance-attribute 

prefill_step_size: int

vllm_mlx.model_registry.ResolvedModelConfig.specprefill_enabled instance-attribute 

specprefill_enabled: bool

vllm_mlx.model_registry.ResolvedModelConfig.specprefill_threshold instance-attribute 

specprefill_threshold: int

vllm_mlx.model_registry.ResolvedModelConfig.specprefill_keep_pct instance-attribute 

specprefill_keep_pct: float

vllm_mlx.model_registry.ResolvedModelConfig.specprefill_backbone_pct instance-attribute 

specprefill_backbone_pct: float

vllm_mlx.model_registry.ResolvedModelConfig.specprefill_draft_model instance-attribute 

specprefill_draft_model: str | None

vllm_mlx.model_registry.ResolvedModelConfig.stream_interval instance-attribute 

stream_interval: int

vllm_mlx.model_registry.ResolvedModelConfig.gpu_memory_utilization instance-attribute 

gpu_memory_utilization: float

vllm_mlx.model_registry.ResolvedModelConfig.scheduler_config instance-attribute 

scheduler_config: SchedulerConfig | None

vllm_mlx.model_registry.ResolvedModelConfig.estimated_memory_bytes instance-attribute 

estimated_memory_bytes: int

vllm_mlx.model_registry.LoadedModel dataclass 

LoadedModel(config: ResolvedModelConfig, engine: BaseEngine, loaded_at: float = time(), last_used_at: float = time(), active_requests: int = 0, active_tasks: set[Task[Any]] = set(), preempting: bool = False)

Runtime state for a loaded engine.

vllm_mlx.model_registry.LoadedModel.config instance-attribute 

config: ResolvedModelConfig

vllm_mlx.model_registry.LoadedModel.engine instance-attribute 

engine: BaseEngine

vllm_mlx.model_registry.LoadedModel.loaded_at class-attribute instance-attribute 

loaded_at: float = field(default_factory=time.time)

vllm_mlx.model_registry.LoadedModel.last_used_at class-attribute instance-attribute 

last_used_at: float = field(default_factory=time.time)

vllm_mlx.model_registry.LoadedModel.active_requests class-attribute instance-attribute 

active_requests: int = 0

vllm_mlx.model_registry.LoadedModel.active_tasks class-attribute instance-attribute 

active_tasks: set[Task[Any]] = field(default_factory=set)

vllm_mlx.model_registry.LoadedModel.preempting class-attribute instance-attribute 

preempting: bool = False

vllm_mlx.model_registry.PendingLoad dataclass 

PendingLoad(model_name: str, required_bytes: int, future: Future[LoadedModel])

A reserved model load in progress.

vllm_mlx.model_registry.PendingLoad.model_name instance-attribute 

model_name: str

vllm_mlx.model_registry.PendingLoad.required_bytes instance-attribute 

required_bytes: int

vllm_mlx.model_registry.PendingLoad.future instance-attribute 

future: Future[LoadedModel]

vllm_mlx.model_registry.ModelLease dataclass 

ModelLease(manager: 'ModelManager | None', model_name: str, engine: BaseEngine, release_cb: Callable[[], Awaitable[None]])

Active lease for a loaded model.

vllm_mlx.model_registry.ModelLease.manager instance-attribute 

manager: 'ModelManager | None'

vllm_mlx.model_registry.ModelLease.model_name instance-attribute 

model_name: str

vllm_mlx.model_registry.ModelLease.engine instance-attribute 

engine: BaseEngine

vllm_mlx.model_registry.ModelLease.release_cb instance-attribute 

release_cb: Callable[[], Awaitable[None]]

vllm_mlx.model_registry.ModelLease.release async 

release() -> None

Release this lease once and allow the model to become evictable.

Source code in vllm_mlx/model_registry.py 
218
219
220
221
222
223
224
225
async def release(self) -> None:
    """Release this lease once and allow the model to become evictable."""

    if self.manager is None:
        return
    manager = self.manager
    self.manager = None
    await self.release_cb()

vllm_mlx.model_registry.ModelLease.__aenter__ async 

__aenter__() -> 'ModelLease'
Source code in vllm_mlx/model_registry.py 
227
228
async def __aenter__(self) -> "ModelLease":
    return self

vllm_mlx.model_registry.ModelLease.__aexit__ async 

__aexit__(exc_type, exc, tb) -> None
Source code in vllm_mlx/model_registry.py 
230
231
async def __aexit__(self, exc_type, exc, tb) -> None:
    await self.release()

vllm_mlx.model_registry.MemoryBudgetReport dataclass 

MemoryBudgetReport(budget_bytes: int, device_working_set_bytes: int | None, gpu_memory_utilization: float | None, gpu_memory_utilization_source: str | None, per_engine_cache_limit_bytes: int | None, per_engine_cache_percent: float | None, continuous_batching_entries: int, total_entries: int)

Reconciliation of the manager weight budget with the Metal ceiling.

The manager budget counts model weights only, and the Metal allocation ceiling (gpu_memory_utilization x device working set) is process-wide. Those two are directly comparable, so a budget above the ceiling is a deterministic conflict: the manager will keep models resident that MLX cannot allocate, and the load fails instead of evicting.

The prefix-cache limit is deliberately not folded into that comparison. cache_memory_mb is a per-engine maximum — it is cloned into each resident continuous-batching engine and allocated lazily, and simple-mode entries never receive it at all — so it is neither a single process-wide reservation nor a bound that can be subtracted once. It is reported alongside the ceiling instead, with its own conflict check.

vllm_mlx.model_registry.MemoryBudgetReport.budget_bytes instance-attribute 

budget_bytes: int

vllm_mlx.model_registry.MemoryBudgetReport.device_working_set_bytes instance-attribute 

device_working_set_bytes: int | None

vllm_mlx.model_registry.MemoryBudgetReport.gpu_memory_utilization instance-attribute 

gpu_memory_utilization: float | None

vllm_mlx.model_registry.MemoryBudgetReport.gpu_memory_utilization_source instance-attribute 

gpu_memory_utilization_source: str | None

vllm_mlx.model_registry.MemoryBudgetReport.per_engine_cache_limit_bytes instance-attribute 

per_engine_cache_limit_bytes: int | None

vllm_mlx.model_registry.MemoryBudgetReport.per_engine_cache_percent instance-attribute 

per_engine_cache_percent: float | None

vllm_mlx.model_registry.MemoryBudgetReport.continuous_batching_entries instance-attribute 

continuous_batching_entries: int

vllm_mlx.model_registry.MemoryBudgetReport.total_entries instance-attribute 

total_entries: int

vllm_mlx.model_registry.MemoryBudgetReport.allocation_ceiling_bytes property 

allocation_ceiling_bytes: int | None

Metal soft allocation limit that will be installed at engine start.

None when no ceiling can be attributed: either MLX cannot report a device working set, or no entry will install one (only BatchedEngine calls mx.set_memory_limit).

vllm_mlx.model_registry.MemoryBudgetReport.exceeds_ceiling property 

exceeds_ceiling: bool

True when the weights budget alone cannot fit under the ceiling.

Both sides are process-wide totals, so this is the deterministic check.

vllm_mlx.model_registry.MemoryBudgetReport.cache_limit_exceeds_ceiling property 

cache_limit_exceeds_ceiling: bool

True when one engine's prefix cache could alone fill the ceiling.

vllm_mlx.model_registry.ModelManager 

ModelManager(manager_config: RegistryManagerConfig, registry: dict[str, RegisteredModel], defaults: RegistryServeDefaults, *, engine_factory: EngineFactory | None = None)

Registry-backed model manager with lazy load and memory-budget eviction.

Source code in vllm_mlx/model_registry.py 
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
def __init__(
    self,
    manager_config: RegistryManagerConfig,
    registry: dict[str, RegisteredModel],
    defaults: RegistryServeDefaults,
    *,
    engine_factory: EngineFactory | None = None,
) -> None:
    self._config = manager_config
    self._registry = registry
    self._defaults = defaults
    self._engine_factory = engine_factory
    self._loaded: dict[str, LoadedModel] = {}
    self._loading: dict[str, PendingLoad] = {}
    self._unloading: dict[str, LoadedModel] = {}
    self._condition = asyncio.Condition()
    self._shutting_down = False

vllm_mlx.model_registry.ModelManager._config instance-attribute 

_config = manager_config

vllm_mlx.model_registry.ModelManager._registry instance-attribute 

_registry = registry

vllm_mlx.model_registry.ModelManager._defaults instance-attribute 

_defaults = defaults

vllm_mlx.model_registry.ModelManager._engine_factory instance-attribute 

_engine_factory = engine_factory

vllm_mlx.model_registry.ModelManager._loaded instance-attribute 

_loaded: dict[str, LoadedModel] = {}

vllm_mlx.model_registry.ModelManager._loading instance-attribute 

_loading: dict[str, PendingLoad] = {}

vllm_mlx.model_registry.ModelManager._unloading instance-attribute 

_unloading: dict[str, LoadedModel] = {}

vllm_mlx.model_registry.ModelManager._condition instance-attribute 

_condition = asyncio.Condition()

vllm_mlx.model_registry.ModelManager._shutting_down instance-attribute 

_shutting_down = False

vllm_mlx.model_registry.ModelManager.memory_budget_bytes property 

memory_budget_bytes: int

Return the registry's configured resident-model memory budget.

vllm_mlx.model_registry.ModelManager.registered_model_names property 

registered_model_names: list[str]

Return sorted list of all registered model names.

vllm_mlx.model_registry.ModelManager.has_model 

has_model(model_name: str) -> bool

Return whether a model name is present in the serving registry.

Source code in vllm_mlx/model_registry.py 
656
657
658
659
def has_model(self, model_name: str) -> bool:
    """Return whether a model name is present in the serving registry."""

    return model_name in self._registry

vllm_mlx.model_registry.ModelManager.list_models 

list_models() -> list[dict[str, Any]]

Return registry state for /v1/models.

Source code in vllm_mlx/model_registry.py 
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
def list_models(self) -> list[dict[str, Any]]:
    """Return registry state for /v1/models."""
    data = []
    for name, entry in self._registry.items():
        loaded = self._loaded.get(name)
        unloading = self._unloading.get(name)
        loading = self._loading.get(name)
        state = "unloaded"
        if loaded is not None:
            state = "preempting" if loaded.preempting else "loaded"
        elif loading is not None:
            state = "loading"
        elif unloading is not None:
            state = "unloading"

        estimated = (
            loaded.config.estimated_memory_bytes
            if loaded is not None
            else (
                unloading.config.estimated_memory_bytes
                if unloading is not None
                else (
                    loading.required_bytes
                    if loading is not None
                    else self._resolve_estimated_bytes(entry, entry.source)
                )
            )
        )
        data.append(
            {
                "id": name,
                "status": state,
                "loaded": loaded is not None,
                "owned_by": "vllm-mlx",
                "source": entry.source,
                "memory_gb": round(estimated / (1024**3), 2) if estimated else None,
            }
        )
    return data

vllm_mlx.model_registry.ModelManager.preload async 

preload() -> None

Preload any entries marked preload=true.

Source code in vllm_mlx/model_registry.py 
701
702
703
704
705
706
async def preload(self) -> None:
    """Preload any entries marked preload=true."""
    for entry in self._registry.values():
        if entry.preload:
            lease = await self.acquire(entry.name)
            await lease.release()

vllm_mlx.model_registry.ModelManager.shutdown async 

shutdown() -> None

Stop and unload every loaded engine.

Source code in vllm_mlx/model_registry.py 
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
async def shutdown(self) -> None:
    """Stop and unload every loaded engine."""
    pending: list[asyncio.Future[LoadedModel]] = []
    unloads: list[LoadedModel] = []
    cancel_tasks: set[asyncio.Task[Any]] = set()

    async with self._condition:
        self._shutting_down = True
        pending = [item.future for item in self._loading.values()]
        for loaded in self._loaded.values():
            loaded.preempting = True
            cancel_tasks.update(loaded.active_tasks)
        for name in list(self._loaded.keys()):
            if self._loaded[name].active_requests == 0:
                unloads.append(self._begin_unload_locked(name))
        self._condition.notify_all()

    for task in cancel_tasks:
        task.cancel()
    await self._run_unloads(unloads)

    if pending:
        await asyncio.gather(*pending, return_exceptions=True)

    remaining: list[LoadedModel] = []
    async with self._condition:
        for name in list(self._loaded.keys()):
            if self._loaded[name].active_requests == 0:
                remaining.append(self._begin_unload_locked(name))
        self._condition.notify_all()

    await self._run_unloads(remaining)

vllm_mlx.model_registry.ModelManager.acquire async 

acquire(model_name: str) -> ModelLease

Acquire a lease for a configured model.

Source code in vllm_mlx/model_registry.py 
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
async def acquire(self, model_name: str) -> ModelLease:
    """Acquire a lease for a configured model."""
    if model_name not in self._registry:
        raise KeyError(model_name)

    start = time.monotonic()
    while True:
        load: PendingLoad | None = None
        unloads: list[LoadedModel] = []
        cancel_tasks: set[asyncio.Task[Any]] = set()
        same_model_future: asyncio.Future[LoadedModel] | None = None
        wait_timeout: float | None = None

        async with self._condition:
            if self._shutting_down:
                raise RuntimeError("Model manager is shutting down")

            claimed = self._claim_loaded_locked(model_name)
            if claimed is not None:
                return claimed

            same_model_future = self._loading.get(model_name, None)
            if same_model_future is not None:
                same_model_future = same_model_future.future
            elif model_name in self._unloading:
                wait_timeout = self._remaining_wait_timeout(start)
            else:
                entry = self._registry[model_name]
                required_bytes = self._resolve_estimated_bytes(entry, entry.source)
                unloads = self._collect_idle_unloads_locked(
                    model_name, required_bytes
                )
                if not unloads and self._can_reserve_locked(required_bytes):
                    load = self._reserve_load_locked(model_name, required_bytes)
                elif not unloads:
                    cancel_tasks = self._maybe_preempt_locked(
                        model_name=model_name,
                        required_bytes=required_bytes,
                        start=start,
                    )
                    if not cancel_tasks and not self._should_wait_locked(start):
                        raise RuntimeError(
                            f"Cannot load '{model_name}' within memory budget "
                            f"({self._config.memory_budget_bytes / (1024**3):.1f} GB)"
                        )
                    wait_timeout = self._remaining_wait_timeout(start)

        if unloads:
            await self._run_unloads(unloads)
            continue

        if cancel_tasks:
            for task in cancel_tasks:
                task.cancel()
            timeout = self._remaining_wait_timeout(start)
            await self._wait_for_change(timeout)
            continue

        if load is not None:
            loaded = await self._execute_load(load)
            async with self._condition:
                claimed = self._claim_loaded_locked(
                    model_name, loaded_override=loaded
                )
                if claimed is not None:
                    return claimed
            continue

        if same_model_future is not None:
            loaded = await same_model_future
            async with self._condition:
                claimed = self._claim_loaded_locked(
                    model_name, loaded_override=loaded
                )
                if claimed is not None:
                    return claimed
            continue

        await self._wait_for_change(wait_timeout)

vllm_mlx.model_registry.ModelManager.release async 

release(model_name: str) -> None

Release a previously acquired model lease.

Source code in vllm_mlx/model_registry.py 
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
async def release(self, model_name: str) -> None:
    """Release a previously acquired model lease."""
    unload: LoadedModel | None = None

    async with self._condition:
        loaded = self._loaded.get(model_name)
        if loaded is None:
            return

        loaded.active_requests = max(0, loaded.active_requests - 1)
        loaded.last_used_at = time.time()
        task = asyncio.current_task()
        if task is not None:
            loaded.active_tasks.discard(task)

        if loaded.preempting and loaded.active_requests == 0:
            unload = self._begin_unload_locked(model_name)

        self._condition.notify_all()

    if unload is not None:
        await self._run_unloads([unload])

vllm_mlx.model_registry.ModelManager._claim_loaded_locked 

_claim_loaded_locked(model_name: str, *, loaded_override: LoadedModel | None = None) -> ModelLease | None
Source code in vllm_mlx/model_registry.py 
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
def _claim_loaded_locked(
    self,
    model_name: str,
    *,
    loaded_override: LoadedModel | None = None,
) -> ModelLease | None:
    loaded = loaded_override or self._loaded.get(model_name)
    if loaded is None:
        return None

    if loaded_override is not None and model_name not in self._loaded:
        self._loaded[model_name] = loaded

    if loaded.preempting:
        return None

    loaded.active_requests += 1
    loaded.last_used_at = time.time()
    task = asyncio.current_task()
    if task is not None:
        loaded.active_tasks.add(task)

    async def _release() -> None:
        await self.release(model_name)

    return ModelLease(
        manager=self,
        model_name=model_name,
        engine=loaded.engine,
        release_cb=_release,
    )

vllm_mlx.model_registry.ModelManager._execute_load async 

_execute_load(pending: PendingLoad) -> LoadedModel

Instantiate a reserved model load outside the manager lock.

Source code in vllm_mlx/model_registry.py 
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
async def _execute_load(self, pending: PendingLoad) -> LoadedModel:
    """Instantiate a reserved model load outside the manager lock."""
    entry = self._registry[pending.model_name]
    loaded: LoadedModel | None = None
    unload_after_load: LoadedModel | None = None

    try:
        resolved_source = await self._resolve_source(entry)
        loaded = await self._instantiate_model(entry, resolved_source)
    except Exception as exc:
        async with self._condition:
            current = self._loading.pop(pending.model_name, None)
            if current is pending and not current.future.done():
                current.future.set_exception(exc)
            self._condition.notify_all()
        raise

    async with self._condition:
        current = self._loading.pop(pending.model_name, None)
        if current is not pending:
            unload_after_load = loaded
        elif self._shutting_down:
            unload_after_load = loaded
            if not current.future.done():
                current.future.set_exception(
                    RuntimeError("Model manager is shutting down")
                )
        else:
            self._loaded[pending.model_name] = loaded
            if not current.future.done():
                current.future.set_result(loaded)
        self._condition.notify_all()

    if unload_after_load is not None:
        await unload_after_load.engine.stop()
        raise RuntimeError("Model load was aborted before it became available")

    return loaded

vllm_mlx.model_registry.ModelManager._wait_for_change async 

_wait_for_change(timeout: float | None) -> None
Source code in vllm_mlx/model_registry.py 
915
916
917
918
919
920
921
922
async def _wait_for_change(self, timeout: float | None) -> None:
    async with self._condition:
        if timeout is None:
            await self._condition.wait()
            return
        if timeout <= 0:
            raise RuntimeError("Timed out waiting for model capacity")
        await asyncio.wait_for(self._condition.wait(), timeout=timeout)

vllm_mlx.model_registry.ModelManager._run_unloads async 

_run_unloads(unloads: list[LoadedModel]) -> None
Source code in vllm_mlx/model_registry.py 
924
925
926
927
928
929
930
931
async def _run_unloads(self, unloads: list[LoadedModel]) -> None:
    for loaded in unloads:
        try:
            await loaded.engine.stop()
        finally:
            async with self._condition:
                self._unloading.pop(loaded.config.entry.name, None)
                self._condition.notify_all()

vllm_mlx.model_registry.ModelManager._reserve_load_locked 

_reserve_load_locked(model_name: str, required_bytes: int) -> PendingLoad
Source code in vllm_mlx/model_registry.py 
933
934
935
936
937
938
939
940
941
def _reserve_load_locked(self, model_name: str, required_bytes: int) -> PendingLoad:
    future: asyncio.Future[LoadedModel] = asyncio.get_running_loop().create_future()
    pending = PendingLoad(
        model_name=model_name,
        required_bytes=required_bytes,
        future=future,
    )
    self._loading[model_name] = pending
    return pending

vllm_mlx.model_registry.ModelManager._begin_unload_locked 

_begin_unload_locked(model_name: str) -> LoadedModel
Source code in vllm_mlx/model_registry.py 
943
944
945
946
def _begin_unload_locked(self, model_name: str) -> LoadedModel:
    loaded = self._loaded.pop(model_name)
    self._unloading[model_name] = loaded
    return loaded

vllm_mlx.model_registry.ModelManager._collect_idle_unloads_locked 

_collect_idle_unloads_locked(requested_model: str, required_bytes: int) -> list[LoadedModel]
Source code in vllm_mlx/model_registry.py 
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
def _collect_idle_unloads_locked(
    self, requested_model: str, required_bytes: int
) -> list[LoadedModel]:
    selected: list[LoadedModel] = []
    projected_bytes = self._committed_bytes_locked()
    candidates = sorted(
        (
            loaded
            for name, loaded in self._loaded.items()
            if name != requested_model and loaded.active_requests == 0
        ),
        key=lambda item: item.last_used_at,
    )

    for loaded in candidates:
        if projected_bytes + required_bytes <= self._config.memory_budget_bytes:
            break
        selected.append(self._begin_unload_locked(loaded.config.entry.name))
        projected_bytes -= loaded.config.estimated_memory_bytes

    return selected

vllm_mlx.model_registry.ModelManager._maybe_preempt_locked 

_maybe_preempt_locked(*, model_name: str, required_bytes: int, start: float) -> set[Task[Any]]
Source code in vllm_mlx/model_registry.py 
 970
 971
 972
 973
 974
 975
 976
 977
 978
 979
 980
 981
 982
 983
 984
 985
 986
 987
 988
 989
 990
 991
 992
 993
 994
 995
 996
 997
 998
 999
1000
1001
1002
1003
def _maybe_preempt_locked(
    self,
    *,
    model_name: str,
    required_bytes: int,
    start: float,
) -> set[asyncio.Task[Any]]:
    if not self._should_preempt_locked(start):
        return set()

    cancel_tasks: set[asyncio.Task[Any]] = set()
    projected_bytes = self._committed_bytes_locked()
    candidates = sorted(
        (
            loaded
            for name, loaded in self._loaded.items()
            if name != model_name and loaded.active_requests > 0
        ),
        key=lambda item: item.last_used_at,
    )

    for loaded in candidates:
        if projected_bytes + required_bytes <= self._config.memory_budget_bytes:
            break
        if loaded.preempting:
            projected_bytes -= loaded.config.estimated_memory_bytes
            continue
        loaded.preempting = True
        cancel_tasks.update(loaded.active_tasks)
        projected_bytes -= loaded.config.estimated_memory_bytes

    if cancel_tasks:
        self._condition.notify_all()
    return cancel_tasks

vllm_mlx.model_registry.ModelManager._should_wait_locked 

_should_wait_locked(start: float) -> bool
Source code in vllm_mlx/model_registry.py 
1005
1006
1007
1008
1009
1010
def _should_wait_locked(self, start: float) -> bool:
    strategy = self._config.policy.strategy
    if strategy == "fail":
        return False
    timeout = self._remaining_wait_timeout(start)
    return timeout is None or timeout > 0

vllm_mlx.model_registry.ModelManager._should_preempt_locked 

_should_preempt_locked(start: float) -> bool
Source code in vllm_mlx/model_registry.py 
1012
1013
1014
1015
1016
1017
1018
1019
1020
def _should_preempt_locked(self, start: float) -> bool:
    policy = self._config.policy
    elapsed = time.monotonic() - start
    if policy.strategy == "preempt":
        return True
    if policy.strategy != "wait_then_preempt":
        return False
    trigger = policy.preempt_after_s if policy.preempt_after_s is not None else 0.0
    return elapsed >= trigger

vllm_mlx.model_registry.ModelManager._remaining_wait_timeout 

_remaining_wait_timeout(start: float) -> float | None
Source code in vllm_mlx/model_registry.py 
1022
1023
1024
1025
1026
def _remaining_wait_timeout(self, start: float) -> float | None:
    timeout = self._config.policy.wait_timeout_s
    if timeout is None or timeout <= 0:
        return None
    return max(timeout - (time.monotonic() - start), 0.0)

vllm_mlx.model_registry.ModelManager._can_reserve_locked 

_can_reserve_locked(required_bytes: int) -> bool
Source code in vllm_mlx/model_registry.py 
1028
1029
1030
1031
1032
def _can_reserve_locked(self, required_bytes: int) -> bool:
    return (
        self._committed_bytes_locked() + required_bytes
        <= self._config.memory_budget_bytes
    )

vllm_mlx.model_registry.ModelManager._committed_bytes_locked 

_committed_bytes_locked() -> int
Source code in vllm_mlx/model_registry.py 
1034
1035
1036
1037
1038
1039
1040
1041
1042
def _committed_bytes_locked(self) -> int:
    loaded_bytes = sum(
        loaded.config.estimated_memory_bytes for loaded in self._loaded.values()
    )
    loading_bytes = sum(item.required_bytes for item in self._loading.values())
    unloading_bytes = sum(
        loaded.config.estimated_memory_bytes for loaded in self._unloading.values()
    )
    return loaded_bytes + loading_bytes + unloading_bytes

vllm_mlx.model_registry.ModelManager._instantiate_model async 

_instantiate_model(entry: RegisteredModel, resolved_source: str) -> LoadedModel
Source code in vllm_mlx/model_registry.py 
1044
1045
1046
1047
1048
1049
1050
1051
1052
1053
1054
1055
1056
1057
1058
1059
1060
1061
1062
1063
1064
1065
1066
1067
1068
1069
1070
1071
1072
1073
async def _instantiate_model(
    self, entry: RegisteredModel, resolved_source: str
) -> LoadedModel:
    config = self._resolve_model_config(entry, resolved_source)

    if self._engine_factory is not None:
        engine = self._engine_factory(config)
    elif config.continuous_batching:
        engine = BatchedEngine(
            model_name=config.resolved_source,
            scheduler_config=config.scheduler_config,
            stream_interval=config.stream_interval,
            force_mllm=config.force_mllm,
            gpu_memory_utilization=config.gpu_memory_utilization,
        )
    else:
        engine = SimpleEngine(
            model_name=config.resolved_source,
            force_mllm=config.force_mllm,
            mtp=config.enable_mtp,
            prefill_step_size=config.prefill_step_size,
            specprefill_enabled=config.specprefill_enabled,
            specprefill_threshold=config.specprefill_threshold,
            specprefill_keep_pct=config.specprefill_keep_pct,
            specprefill_backbone_pct=config.specprefill_backbone_pct,
            specprefill_draft_model=config.specprefill_draft_model,
        )

    await engine.start()
    return LoadedModel(config=config, engine=engine)

vllm_mlx.model_registry.ModelManager._resolve_source async 

_resolve_source(entry: RegisteredModel) -> str
Source code in vllm_mlx/model_registry.py 
1075
1076
async def _resolve_source(self, entry: RegisteredModel) -> str:
    return await asyncio.to_thread(self._resolve_source_sync, entry)

vllm_mlx.model_registry.ModelManager._resolve_source_sync 

_resolve_source_sync(entry: RegisteredModel) -> str
Source code in vllm_mlx/model_registry.py 
1078
1079
1080
1081
1082
1083
1084
1085
1086
1087
def _resolve_source_sync(self, entry: RegisteredModel) -> str:
    source = entry.source
    if Path(source).exists():
        return source
    downloaded = ensure_model_downloaded(
        source,
        config=self._defaults.download_config,
        is_mllm=is_mllm_model(source) or bool(entry.force_mllm),
    )
    return str(downloaded)

vllm_mlx.model_registry.ModelManager._resolve_estimated_bytes 

_resolve_estimated_bytes(entry: RegisteredModel, resolved_source: str) -> int
Source code in vllm_mlx/model_registry.py 
1089
1090
1091
1092
1093
1094
1095
1096
1097
1098
1099
1100
1101
1102
1103
1104
1105
1106
1107
1108
1109
1110
1111
1112
1113
1114
1115
1116
1117
1118
1119
1120
1121
def _resolve_estimated_bytes(
    self, entry: RegisteredModel, resolved_source: str
) -> int:
    if entry.estimated_memory_bytes is not None:
        return entry.estimated_memory_bytes
    estimated = _estimate_model_bytes_from_source(resolved_source)
    if estimated > 0:
        return estimated
    source_path = Path(resolved_source)
    if not source_path.exists():
        raise ValueError(
            "models-config entry "
            f"'{entry.name}' uses non-local source '{entry.source}' without "
            "estimated_memory_gb. Registry-backed loading requires an explicit "
            "memory estimate for non-local models so eviction remains deterministic."
        )

    available = _safe_available_memory_bytes()
    if available > 0:
        logger.warning(
            "Falling back to a coarse memory estimate for registry entry '%s' "
            "because no weight files were found under %s; set estimated_memory_gb "
            "explicitly for deterministic eviction.",
            entry.name,
            resolved_source,
        )
        return max(available // 8, 1)

    raise ValueError(
        "Cannot estimate memory for registry entry "
        f"'{entry.name}' from '{resolved_source}'. Set estimated_memory_gb "
        "explicitly in the models config."
    )

vllm_mlx.model_registry.ModelManager._resolve_model_config 

_resolve_model_config(entry: RegisteredModel, resolved_source: str) -> ResolvedModelConfig
Source code in vllm_mlx/model_registry.py 
1123
1124
1125
1126
1127
1128
1129
1130
1131
1132
1133
1134
1135
1136
1137
1138
1139
1140
1141
1142
1143
1144
1145
1146
1147
1148
1149
1150
1151
1152
1153
1154
1155
1156
1157
1158
1159
1160
1161
1162
1163
1164
1165
1166
1167
1168
1169
1170
1171
1172
1173
1174
1175
1176
1177
1178
1179
1180
1181
1182
1183
1184
1185
1186
1187
1188
1189
1190
1191
1192
1193
1194
1195
1196
1197
1198
1199
1200
1201
def _resolve_model_config(
    self, entry: RegisteredModel, resolved_source: str
) -> ResolvedModelConfig:
    scheduler_config = _clone_scheduler_config(self._defaults.scheduler_config)

    continuous_batching = (
        entry.continuous_batching
        if entry.continuous_batching is not None
        else self._defaults.continuous_batching
    )
    force_mllm = (
        entry.force_mllm
        if entry.force_mllm is not None
        else self._defaults.force_mllm
    )
    enable_mtp = (
        entry.enable_mtp
        if entry.enable_mtp is not None
        else self._defaults.enable_mtp
    )
    prefill_step_size = (
        entry.prefill_step_size
        if entry.prefill_step_size is not None
        else self._defaults.prefill_step_size
    )
    specprefill_enabled = (
        entry.specprefill_enabled
        if entry.specprefill_enabled is not None
        else self._defaults.specprefill_enabled
    )
    specprefill_threshold = (
        entry.specprefill_threshold
        if entry.specprefill_threshold is not None
        else self._defaults.specprefill_threshold
    )
    specprefill_keep_pct = (
        entry.specprefill_keep_pct
        if entry.specprefill_keep_pct is not None
        else self._defaults.specprefill_keep_pct
    )
    specprefill_backbone_pct = (
        entry.specprefill_backbone_pct
        if entry.specprefill_backbone_pct is not None
        else self._defaults.specprefill_backbone_pct
    )
    specprefill_draft_model = (
        entry.specprefill_draft_model
        if entry.specprefill_draft_model is not None
        else self._defaults.specprefill_draft_model
    )
    stream_interval = (
        entry.stream_interval
        if entry.stream_interval is not None
        else self._defaults.stream_interval
    )
    gpu_memory_utilization = (
        entry.gpu_memory_utilization
        if entry.gpu_memory_utilization is not None
        else self._defaults.gpu_memory_utilization
    )
    estimated_memory_bytes = self._resolve_estimated_bytes(entry, resolved_source)

    return ResolvedModelConfig(
        entry=entry,
        resolved_source=resolved_source,
        continuous_batching=continuous_batching,
        force_mllm=force_mllm,
        enable_mtp=enable_mtp,
        prefill_step_size=prefill_step_size,
        specprefill_enabled=specprefill_enabled,
        specprefill_threshold=specprefill_threshold,
        specprefill_keep_pct=specprefill_keep_pct,
        specprefill_backbone_pct=specprefill_backbone_pct,
        specprefill_draft_model=specprefill_draft_model,
        stream_interval=stream_interval,
        gpu_memory_utilization=gpu_memory_utilization,
        scheduler_config=scheduler_config,
        estimated_memory_bytes=estimated_memory_bytes,
    )

vllm_mlx.model_registry.get_registry 

get_registry() -> _ModelOwnershipRegistry

Return the global model ownership registry used by EngineCore.

Source code in vllm_mlx/model_registry.py 
88
89
90
def get_registry() -> _ModelOwnershipRegistry:
    """Return the global model ownership registry used by EngineCore."""
    return _ownership_registry

vllm_mlx.model_registry._clone_scheduler_config 

_clone_scheduler_config(config: SchedulerConfig | None) -> SchedulerConfig | None

Clone a SchedulerConfig so per-model overrides do not mutate globals.

Source code in vllm_mlx/model_registry.py 
234
235
236
237
238
def _clone_scheduler_config(config: SchedulerConfig | None) -> SchedulerConfig | None:
    """Clone a SchedulerConfig so per-model overrides do not mutate globals."""
    if config is None:
        return None
    return SchedulerConfig(**vars(config))

vllm_mlx.model_registry._parse_memory_budget_bytes 

_parse_memory_budget_bytes(value: Any) -> int

Parse a memory budget from bytes, MB, or GB.

Source code in vllm_mlx/model_registry.py 
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
def _parse_memory_budget_bytes(value: Any) -> int:
    """Parse a memory budget from bytes, MB, or GB."""
    if value is None:
        raise ValueError("models-config manager.memory_budget_gb is required")
    if isinstance(value, (int, float)):
        return int(float(value) * (1024**3))
    if isinstance(value, str):
        raw = value.strip().lower()
        if raw.endswith("gb"):
            return int(float(raw[:-2]) * (1024**3))
        if raw.endswith("mb"):
            return int(float(raw[:-2]) * (1024**2))
        if raw.endswith("b"):
            return int(float(raw[:-1]))
        return int(float(raw) * (1024**3))
    raise TypeError(f"Unsupported memory budget value: {value!r}")

vllm_mlx.model_registry._safe_available_memory_bytes 

_safe_available_memory_bytes() -> int

Best-effort available system memory.

Source code in vllm_mlx/model_registry.py 
259
260
261
262
263
def _safe_available_memory_bytes() -> int:
    """Best-effort available system memory."""
    if psutil is None:  # pragma: no cover - fallback only
        return 0
    return int(psutil.virtual_memory().available)

vllm_mlx.model_registry._device_working_set_bytes 

_device_working_set_bytes() -> int | None

Best-effort Metal recommended working-set size, or None when unavailable.

Source code in vllm_mlx/model_registry.py 
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
def _device_working_set_bytes() -> int | None:
    """Best-effort Metal recommended working-set size, or None when unavailable."""
    try:
        import mlx.core as mx

        if not mx.metal.is_available():
            return None
        info = mx.device_info()
        raw = info.get(
            "max_recommended_working_set_size",
            info.get("memory_size", 0),
        )
        working_set = int(raw or 0)
    except Exception as exc:  # pragma: no cover - platform dependent
        logger.debug("Could not query MLX device memory: %s", exc)
        return None
    return working_set or None

vllm_mlx.model_registry.build_memory_budget_report 

build_memory_budget_report(manager_config: RegistryManagerConfig, registry: dict[str, RegisteredModel], defaults: RegistryServeDefaults, *, device_working_set_bytes: int | None = None) -> MemoryBudgetReport

Reconcile the manager weight budget against the Metal allocation ceiling.

The Metal limit is process-wide but is re-installed by every BatchedEngine start, so the ceiling the manager has to live under is the lowest utilization among the entries that actually install one. Only continuous-batching entries qualify: SimpleEngine never calls mx.set_memory_limit and is not even given a gpu_memory_utilization. A registry with no continuous-batching entries therefore gets no attributed ceiling rather than one derived from a value nothing installs.

Source code in vllm_mlx/model_registry.py 
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
def build_memory_budget_report(
    manager_config: RegistryManagerConfig,
    registry: dict[str, RegisteredModel],
    defaults: RegistryServeDefaults,
    *,
    device_working_set_bytes: int | None = None,
) -> MemoryBudgetReport:
    """Reconcile the manager weight budget against the Metal allocation ceiling.

    The Metal limit is process-wide but is re-installed by every
    ``BatchedEngine`` start, so the ceiling the manager has to live under is the
    *lowest* utilization among the entries that actually install one. Only
    continuous-batching entries qualify: ``SimpleEngine`` never calls
    ``mx.set_memory_limit`` and is not even given a ``gpu_memory_utilization``.
    A registry with no continuous-batching entries therefore gets no attributed
    ceiling rather than one derived from a value nothing installs.
    """
    if device_working_set_bytes is None:
        device_working_set_bytes = _device_working_set_bytes()

    # Only BatchedEngine installs a Metal allocation limit — SimpleEngine is not
    # even constructed with a gpu_memory_utilization — so a simple-mode entry's
    # override is inert and must not be treated as a ceiling candidate.
    candidates: list[tuple[float, int, str]] = []
    for name in sorted(registry):
        entry = registry[name]
        entry_continuous_batching = (
            entry.continuous_batching
            if entry.continuous_batching is not None
            else defaults.continuous_batching
        )
        if not entry_continuous_batching:
            continue
        if entry.gpu_memory_utilization is not None:
            # Rank 1: an override is only attributable to the entry declaring it.
            candidates.append(
                (entry.gpu_memory_utilization, 1, f"models-config entry '{name}'")
            )
        else:
            # Rank 0: prefer the serve default as the named source on ties.
            candidates.append((defaults.gpu_memory_utilization, 0, "serve default"))

    continuous_batching_entries = len(candidates)

    utilization: float | None = None
    utilization_source: str | None = None
    if candidates:
        utilization, _, utilization_source = min(candidates)

    # cache_memory_mb only binds for continuous-batching engines, and only when
    # the memory-aware prefix cache is the one actually in use.
    scheduler_config = defaults.scheduler_config
    per_engine_cache_limit_bytes: int | None = None
    per_engine_cache_percent: float | None = None
    cache_applies = (
        scheduler_config is not None
        and continuous_batching_entries > 0
        and getattr(scheduler_config, "enable_prefix_cache", False)
        and not getattr(scheduler_config, "use_paged_cache", False)
        and getattr(scheduler_config, "use_memory_aware_cache", False)
    )
    if cache_applies:
        cache_memory_mb = getattr(scheduler_config, "cache_memory_mb", None)
        if cache_memory_mb:
            per_engine_cache_limit_bytes = int(cache_memory_mb) * (1024**2)
        else:
            percent = getattr(scheduler_config, "cache_memory_percent", None)
            if percent:
                per_engine_cache_percent = float(percent)

    return MemoryBudgetReport(
        budget_bytes=manager_config.memory_budget_bytes,
        device_working_set_bytes=device_working_set_bytes,
        gpu_memory_utilization=utilization,
        gpu_memory_utilization_source=utilization_source,
        per_engine_cache_limit_bytes=per_engine_cache_limit_bytes,
        per_engine_cache_percent=per_engine_cache_percent,
        continuous_batching_entries=continuous_batching_entries,
        total_entries=len(registry),
    )

vllm_mlx.model_registry.log_memory_budget_report 

log_memory_budget_report(report: MemoryBudgetReport) -> None

Log the budget/ceiling reconciliation, warning when they conflict.

Source code in vllm_mlx/model_registry.py 
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
def log_memory_budget_report(report: MemoryBudgetReport) -> None:
    """Log the budget/ceiling reconciliation, warning when they conflict."""
    gb = 1024**3
    ceiling = report.allocation_ceiling_bytes

    if ceiling is None:
        if report.gpu_memory_utilization is None:
            reason = (
                "no continuous-batching entries, and --gpu-memory-utilization "
                "installs a Metal limit only for those"
            )
        else:
            reason = "MLX cannot report a device working set on this host"
        logger.info(
            "Registry memory budget: %.1f GB of model weights; no Metal "
            "allocation ceiling to reconcile it with (%s)",
            report.budget_bytes / gb,
            reason,
        )
        return

    engines = f"{report.continuous_batching_entries} of {report.total_entries} entries"
    if report.per_engine_cache_limit_bytes is not None:
        cache_desc = (
            f"{report.per_engine_cache_limit_bytes / gb:.1f} GB per "
            f"continuous-batching engine (--cache-memory-mb, {engines})"
        )
    elif report.per_engine_cache_percent is not None:
        cache_desc = (
            f"~{report.per_engine_cache_percent * 100:.0f}% of available RAM per "
            f"continuous-batching engine (--cache-memory-percent, {engines}); "
            "scales at runtime"
        )
    else:
        cache_desc = "none configured"

    logger.info(
        "Registry memory budget: %.1f GB of model weights; "
        "Metal allocation ceiling %.1f GB (%.0f%% of %.1f GB, from %s); "
        "prefix-cache maximum %s",
        report.budget_bytes / gb,
        ceiling / gb,
        report.gpu_memory_utilization * 100,
        (report.device_working_set_bytes or 0) / gb,
        report.gpu_memory_utilization_source,
        cache_desc,
    )

    if report.exceeds_ceiling:
        logger.warning(
            "models-config manager.memory_budget_gb (%.1f GB) exceeds the Metal "
            "allocation ceiling (%.1f GB). The budget counts model weights only, "
            "so the manager will keep models resident that MLX cannot allocate, "
            "and a load can fail with an out-of-memory error instead of evicting. "
            "Lower the budget below %.1f GB — further still, since the KV cache "
            "and activations also come out of the ceiling — or raise "
            "--gpu-memory-utilization.",
            report.budget_bytes / gb,
            ceiling / gb,
            ceiling / gb,
        )

    if report.cache_limit_exceeds_ceiling:
        logger.warning(
            "--cache-memory-mb (%.1f GB per continuous-batching engine) is at or "
            "above the Metal allocation ceiling (%.1f GB) on its own, leaving no "
            "room for model weights. Note this is a per-engine maximum: it is "
            "cloned into every resident continuous-batching engine, so the "
            "aggregate grows with the number of resident models.",
            (report.per_engine_cache_limit_bytes or 0) / gb,
            ceiling / gb,
        )

    if not report.exceeds_ceiling and not report.cache_limit_exceeds_ceiling:
        logger.info(
            "The registry budget covers model weights only; the KV cache, the "
            "prefix cache and activations are additional and are not reserved "
            "by it."
        )

vllm_mlx.model_registry._estimate_model_bytes_from_source 

_estimate_model_bytes_from_source(source: str) -> int

Estimate model footprint from local artifact size when possible.

Source code in vllm_mlx/model_registry.py 
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
def _estimate_model_bytes_from_source(source: str) -> int:
    """Estimate model footprint from local artifact size when possible."""
    path = Path(source)
    if not path.exists():
        return 0

    if path.is_file():
        return path.stat().st_size if path.suffix in {".safetensors", ".gguf"} else 0

    total = 0
    for pattern in ("*.safetensors", "*.gguf"):
        for fp in path.rglob(pattern):
            try:
                total += fp.stat().st_size
            except OSError:
                continue
    return total

vllm_mlx.model_registry.load_registry_config 

load_registry_config(config_path: str | PathLike[str], defaults: RegistryServeDefaults) -> tuple[RegistryManagerConfig, dict[str, RegisteredModel]]

Load and validate the models registry YAML file.

Source code in vllm_mlx/model_registry.py 
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
def load_registry_config(
    config_path: str | os.PathLike[str],
    defaults: RegistryServeDefaults,
) -> tuple[RegistryManagerConfig, dict[str, RegisteredModel]]:
    """Load and validate the models registry YAML file."""
    import yaml  # lazy: only needed when a registry config is provided

    raw = yaml.safe_load(Path(config_path).read_text()) or {}
    models = raw.get("models")
    if not isinstance(models, list) or not models:
        raise ValueError("models-config must define a non-empty 'models' list")

    manager_raw = raw.get("manager") or {}
    policy_raw = manager_raw.get("contention_policy") or {}
    policy = ContentionPolicy(
        strategy=policy_raw.get("strategy", "wait_then_fail"),
        wait_timeout_s=(
            float(policy_raw["wait_timeout_s"])
            if policy_raw.get("wait_timeout_s") is not None
            else 30.0
        ),
        preempt_after_s=(
            float(policy_raw["preempt_after_s"])
            if policy_raw.get("preempt_after_s") is not None
            else None
        ),
    )
    if policy.strategy not in {
        "fail",
        "wait",
        "preempt",
        "wait_then_fail",
        "wait_then_preempt",
    }:
        raise ValueError(f"Unsupported contention strategy: {policy.strategy}")

    manager = RegistryManagerConfig(
        memory_budget_bytes=_parse_memory_budget_bytes(
            manager_raw.get("memory_budget_gb", manager_raw.get("memory_budget"))
        ),
        policy=policy,
    )

    registry: dict[str, RegisteredModel] = {}
    for item in models:
        if not isinstance(item, dict):
            raise ValueError(f"Invalid model entry: {item!r}")
        name = item.get("name")
        source = item.get("path") or item.get("source") or item.get("model")
        if not name or not source:
            raise ValueError(
                f"Each model entry must define 'name' and one of 'path'/'source'/'model': {item!r}"
            )
        if name in registry:
            raise ValueError(f"Duplicate model name in registry: {name}")

        estimated = item.get("estimated_memory_gb")
        estimated_bytes = (
            int(float(estimated) * (1024**3)) if estimated is not None else None
        )

        raw_gpu_memory_utilization = item.get("gpu_memory_utilization")
        gpu_memory_utilization = None
        if raw_gpu_memory_utilization is not None:
            try:
                gpu_memory_utilization = float(raw_gpu_memory_utilization)
            except (TypeError, ValueError):
                raise ValueError(
                    f"models-config entry '{name}' gpu_memory_utilization must "
                    "be finite and within (0, 1]"
                ) from None
            if not math.isfinite(gpu_memory_utilization) or not (
                0.0 < gpu_memory_utilization <= 1.0
            ):
                raise ValueError(
                    f"models-config entry '{name}' gpu_memory_utilization must "
                    "be finite and within (0, 1]"
                )

        registry[name] = RegisteredModel(
            name=name,
            source=str(source),
            preload=bool(item.get("preload", False)),
            continuous_batching=item.get("continuous_batching"),
            force_mllm=item.get("mllm"),
            enable_mtp=item.get("enable_mtp"),
            prefill_step_size=item.get("prefill_step_size"),
            specprefill_enabled=item.get("specprefill"),
            specprefill_threshold=item.get("specprefill_threshold"),
            specprefill_keep_pct=item.get("specprefill_keep_pct"),
            specprefill_backbone_pct=item.get("specprefill_backbone_pct"),
            specprefill_draft_model=item.get("specprefill_draft_model"),
            stream_interval=item.get("stream_interval"),
            gpu_memory_utilization=gpu_memory_utilization,
            estimated_memory_bytes=estimated_bytes,
        )

    return manager, registry

Complete contract reference

Expand any definition for its exact inputs, annotations, defaults, return contract, directly raised exceptions, source-grounded behavior, and immutable line link. This section includes private and nested definitions that ordinary API generators omit.

vllm_mlx.model_registry.ModelOwnershipError · class 
vllm_mlx.model_registry.ModelOwnershipError()

Raised when an EngineCore attempts to use a model already in use.

Parameters

This callable has no explicit inputs.

Returns

  • Constructs: vllm_mlx.model_registry.ModelOwnershipError

Exceptions and behavior

Class ModelOwnershipError derives from RuntimeError and declares 0 direct member(s). No direct raise statement appears in this definition.

View source #L38-L39.

vllm_mlx.model_registry._ModelOwnershipRegistry · class 
vllm_mlx.model_registry._ModelOwnershipRegistry()

Process-local model ownership guard used by EngineCore.

Parameters

This callable has no explicit inputs.

Returns

  • Constructs: vllm_mlx.model_registry._ModelOwnershipRegistry

Exceptions and behavior

Class _ModelOwnershipRegistry declares 5 direct member(s). No direct raise statement appears in this definition.

View source #L42-L82.

vllm_mlx.model_registry._ModelOwnershipRegistry.__init__ · method 
vllm_mlx.model_registry._ModelOwnershipRegistry.__init__() -> None

Method _ModelOwnershipRegistry.__init__ updates self._owners.

Parameters

This callable has no explicit inputs.

Returns

  • Type: None

Exceptions and behavior

Method _ModelOwnershipRegistry.__init__ updates self._owners. No direct raise statement appears in this definition.

View source #L45-L46.

vllm_mlx.model_registry._ModelOwnershipRegistry.acquire · method 
vllm_mlx.model_registry._ModelOwnershipRegistry.acquire(*, model: Any, engine: Any, engine_id: str, force: bool = True) -> None

Method _ModelOwnershipRegistry.acquire calls id, self._owners.get, ModelOwnershipError; can raise ModelOwnershipError.

Parameters

Name Type Required Default Description
model Any yes none Required keyword-only input.
engine Any yes none Required keyword-only input.
engine_id str yes none Required keyword-only input.
force bool no True Optional keyword-only input; defaults to True.

Returns

  • Type: None

Exceptions and behavior

Method _ModelOwnershipRegistry.acquire calls id, self._owners.get, ModelOwnershipError; can raise ModelOwnershipError. Directly raised exceptions: ModelOwnershipError.

View source #L48-L63.

vllm_mlx.model_registry._ModelOwnershipRegistry.release · method 
vllm_mlx.model_registry._ModelOwnershipRegistry.release(model: Any, engine_id: str) -> None

Method _ModelOwnershipRegistry.release calls id, self._owners.get, self._owners.pop.

Parameters

Name Type Required Default Description
model Any yes none Required positional or keyword input.
engine_id str yes none Required positional or keyword input.

Returns

  • Type: None

Exceptions and behavior

Method _ModelOwnershipRegistry.release calls id, self._owners.get, self._owners.pop. No direct raise statement appears in this definition.

View source #L65-L69.

vllm_mlx.model_registry._ModelOwnershipRegistry.is_owned · method 
vllm_mlx.model_registry._ModelOwnershipRegistry.is_owned(model: Any) -> tuple[bool, str | None]

Method _ModelOwnershipRegistry.is_owned calls id, self._owners.get; has 2 explicit return paths.

Parameters

Name Type Required Default Description
model Any yes none Required positional or keyword input.

Returns

  • Type: tuple[bool, str | None]
  • Direct return expressions: (True, owner); (False, None)

Exceptions and behavior

Method _ModelOwnershipRegistry.is_owned calls id, self._owners.get; has 2 explicit return paths. No direct raise statement appears in this definition.

View source #L71-L76.

vllm_mlx.model_registry._ModelOwnershipRegistry.get_stats · method 
vllm_mlx.model_registry._ModelOwnershipRegistry.get_stats() -> dict[str, Any]

Method _ModelOwnershipRegistry.get_stats calls len; returns {'total_entries': len(self._owners), 'active_owners': len(self._owners)}.

Parameters

This callable has no explicit inputs.

Returns

  • Type: dict[str, Any]
  • Direct return expressions: {'total_entries': len(self._owners), 'active_owners': len(self._owners)}

Exceptions and behavior

Method _ModelOwnershipRegistry.get_stats calls len; returns {'total_entries': len(self._owners), 'active_owners': len(self._owners)}. No direct raise statement appears in this definition.

View source #L78-L82.

vllm_mlx.model_registry.get_registry · function 
vllm_mlx.model_registry.get_registry() -> _ModelOwnershipRegistry

Return the global model ownership registry used by EngineCore.

Parameters

This callable has no explicit inputs.

Returns

  • Type: _ModelOwnershipRegistry
  • Direct return expressions: _ownership_registry

Exceptions and behavior

Function get_registry returns _ownership_registry. No direct raise statement appears in this definition.

View source #L88-L90.

vllm_mlx.model_registry.RegistryServeDefaults · class 
vllm_mlx.model_registry.RegistryServeDefaults(continuous_batching: bool, force_mllm: bool, enable_mtp: bool, prefill_step_size: int, specprefill_enabled: bool, specprefill_threshold: int, specprefill_keep_pct: float, specprefill_backbone_pct: float, specprefill_draft_model: str | None, stream_interval: int, gpu_memory_utilization: float, scheduler_config: SchedulerConfig | None, max_tokens: int, download_config: DownloadConfig)

Global serve defaults inherited by registry entries.

Parameters

Name Type Required Default Description
continuous_batching bool yes none Required constructor field.
force_mllm bool yes none Required constructor field.
enable_mtp bool yes none Required constructor field.
prefill_step_size int yes none Required constructor field.
specprefill_enabled bool yes none Required constructor field.
specprefill_threshold int yes none Required constructor field.
specprefill_keep_pct float yes none Required constructor field.
specprefill_backbone_pct float yes none Required constructor field.
specprefill_draft_model str \| None yes none Required constructor field.
stream_interval int yes none Required constructor field.
gpu_memory_utilization float yes none Required constructor field.
scheduler_config SchedulerConfig \| None yes none Required constructor field.
max_tokens int yes none Required constructor field.
download_config DownloadConfig yes none Required constructor field.

Returns

  • Constructs: vllm_mlx.model_registry.RegistryServeDefaults

Exceptions and behavior

Class RegistryServeDefaults declares 0 direct member(s). No direct raise statement appears in this definition.

View source #L109-L125.

vllm_mlx.model_registry.ContentionPolicy · class 
vllm_mlx.model_registry.ContentionPolicy(strategy: ContentionStrategy = 'wait_then_fail', wait_timeout_s: float | None = 30.0, preempt_after_s: float | None = None)

Policy used when a new model cannot fit inside the memory budget.

Parameters

Name Type Required Default Description
strategy ContentionStrategy no 'wait_then_fail' Optional constructor field; defaults to 'wait_then_fail'.
wait_timeout_s float \| None no 30.0 Optional constructor field; defaults to 30.0.
preempt_after_s float \| None no None Optional constructor field; defaults to None.

Returns

  • Constructs: vllm_mlx.model_registry.ContentionPolicy

Exceptions and behavior

Class ContentionPolicy declares 0 direct member(s). No direct raise statement appears in this definition.

View source #L129-L134.

vllm_mlx.model_registry.RegistryManagerConfig · class 
vllm_mlx.model_registry.RegistryManagerConfig(memory_budget_bytes: int, policy: ContentionPolicy)

Global registry manager configuration.

Parameters

Name Type Required Default Description
memory_budget_bytes int yes none Required constructor field.
policy ContentionPolicy yes none Required constructor field.

Returns

  • Constructs: vllm_mlx.model_registry.RegistryManagerConfig

Exceptions and behavior

Class RegistryManagerConfig declares 0 direct member(s). No direct raise statement appears in this definition.

View source #L138-L142.

vllm_mlx.model_registry.RegisteredModel · class 
vllm_mlx.model_registry.RegisteredModel(name: str, source: str, preload: bool = False, continuous_batching: bool | None = None, force_mllm: bool | None = None, enable_mtp: bool | None = None, prefill_step_size: int | None = None, specprefill_enabled: bool | None = None, specprefill_threshold: int | None = None, specprefill_keep_pct: float | None = None, specprefill_backbone_pct: float | None = None, specprefill_draft_model: str | None = None, stream_interval: int | None = None, gpu_memory_utilization: float | None = None, estimated_memory_bytes: int | None = None)

One configured model entry.

Parameters

Name Type Required Default Description
name str yes none Required constructor field.
source str yes none Required constructor field.
preload bool no False Optional constructor field; defaults to False.
continuous_batching bool \| None no None Optional constructor field; defaults to None.
force_mllm bool \| None no None Optional constructor field; defaults to None.
enable_mtp bool \| None no None Optional constructor field; defaults to None.
prefill_step_size int \| None no None Optional constructor field; defaults to None.
specprefill_enabled bool \| None no None Optional constructor field; defaults to None.
specprefill_threshold int \| None no None Optional constructor field; defaults to None.
specprefill_keep_pct float \| None no None Optional constructor field; defaults to None.
specprefill_backbone_pct float \| None no None Optional constructor field; defaults to None.
specprefill_draft_model str \| None no None Optional constructor field; defaults to None.
stream_interval int \| None no None Optional constructor field; defaults to None.
gpu_memory_utilization float \| None no None Optional constructor field; defaults to None.
estimated_memory_bytes int \| None no None Optional constructor field; defaults to None.

Returns

  • Constructs: vllm_mlx.model_registry.RegisteredModel

Exceptions and behavior

Class RegisteredModel declares 0 direct member(s). No direct raise statement appears in this definition.

View source #L146-L163.

vllm_mlx.model_registry.ResolvedModelConfig · class 
vllm_mlx.model_registry.ResolvedModelConfig(entry: RegisteredModel, resolved_source: str, continuous_batching: bool, force_mllm: bool, enable_mtp: bool, prefill_step_size: int, specprefill_enabled: bool, specprefill_threshold: int, specprefill_keep_pct: float, specprefill_backbone_pct: float, specprefill_draft_model: str | None, stream_interval: int, gpu_memory_utilization: float, scheduler_config: SchedulerConfig | None, estimated_memory_bytes: int)

Effective configuration for a loaded model.

Parameters

Name Type Required Default Description
entry RegisteredModel yes none Required constructor field.
resolved_source str yes none Required constructor field.
continuous_batching bool yes none Required constructor field.
force_mllm bool yes none Required constructor field.
enable_mtp bool yes none Required constructor field.
prefill_step_size int yes none Required constructor field.
specprefill_enabled bool yes none Required constructor field.
specprefill_threshold int yes none Required constructor field.
specprefill_keep_pct float yes none Required constructor field.
specprefill_backbone_pct float yes none Required constructor field.
specprefill_draft_model str \| None yes none Required constructor field.
stream_interval int yes none Required constructor field.
gpu_memory_utilization float yes none Required constructor field.
scheduler_config SchedulerConfig \| None yes none Required constructor field.
estimated_memory_bytes int yes none Required constructor field.

Returns

  • Constructs: vllm_mlx.model_registry.ResolvedModelConfig

Exceptions and behavior

Class ResolvedModelConfig declares 0 direct member(s). No direct raise statement appears in this definition.

View source #L167-L184.

vllm_mlx.model_registry.LoadedModel · class 
vllm_mlx.model_registry.LoadedModel(config: ResolvedModelConfig, engine: BaseEngine, loaded_at: float = field(default_factory=time.time), last_used_at: float = field(default_factory=time.time), active_requests: int = 0, active_tasks: set[asyncio.Task[Any]] = field(default_factory=set), preempting: bool = False)

Runtime state for a loaded engine.

Parameters

Name Type Required Default Description
config ResolvedModelConfig yes none Required constructor field.
engine BaseEngine yes none Required constructor field.
loaded_at float no field(default_factory=time.time) Optional constructor field; defaults to field(default_factory=time.time).
last_used_at float no field(default_factory=time.time) Optional constructor field; defaults to field(default_factory=time.time).
active_requests int no 0 Optional constructor field; defaults to 0.
active_tasks set[asyncio.Task[Any]] no field(default_factory=set) Optional constructor field; defaults to field(default_factory=set).
preempting bool no False Optional constructor field; defaults to False.

Returns

  • Constructs: vllm_mlx.model_registry.LoadedModel

Exceptions and behavior

Class LoadedModel declares 0 direct member(s). No direct raise statement appears in this definition.

View source #L188-L197.

vllm_mlx.model_registry.PendingLoad · class 
vllm_mlx.model_registry.PendingLoad(model_name: str, required_bytes: int, future: asyncio.Future[LoadedModel])

A reserved model load in progress.

Parameters

Name Type Required Default Description
model_name str yes none Required constructor field.
required_bytes int yes none Required constructor field.
future asyncio.Future[LoadedModel] yes none Required constructor field.

Returns

  • Constructs: vllm_mlx.model_registry.PendingLoad

Exceptions and behavior

Class PendingLoad declares 0 direct member(s). No direct raise statement appears in this definition.

View source #L201-L206.

vllm_mlx.model_registry.ModelLease · class 
vllm_mlx.model_registry.ModelLease(manager: 'ModelManager | None', model_name: str, engine: BaseEngine, release_cb: Callable[[], Awaitable[None]])

Active lease for a loaded model.

Parameters

Name Type Required Default Description
manager 'ModelManager \| None' yes none Required constructor field.
model_name str yes none Required constructor field.
engine BaseEngine yes none Required constructor field.
release_cb Callable[[], Awaitable[None]] yes none Required constructor field.

Returns

  • Constructs: vllm_mlx.model_registry.ModelLease

Exceptions and behavior

Class ModelLease declares 3 direct member(s). No direct raise statement appears in this definition.

View source #L210-L231.

vllm_mlx.model_registry.ModelLease.release · method 
async vllm_mlx.model_registry.ModelLease.release() -> None

Release this lease once and allow the model to become evictable.

Parameters

This callable has no explicit inputs.

Returns

  • Type: None
  • Direct return expressions: None

Exceptions and behavior

Method ModelLease.release updates self.manager; calls self.release_cb; awaits asynchronous work; returns None. No direct raise statement appears in this definition.

View source #L218-L225.

vllm_mlx.model_registry.ModelLease.__aenter__ · method 
async vllm_mlx.model_registry.ModelLease.__aenter__() -> 'ModelLease'

Method ModelLease.__aenter__ returns self.

Parameters

This callable has no explicit inputs.

Returns

  • Type: 'ModelLease'
  • Direct return expressions: self

Exceptions and behavior

Method ModelLease.__aenter__ returns self. No direct raise statement appears in this definition.

View source #L227-L228.

vllm_mlx.model_registry.ModelLease.__aexit__ · method 
async vllm_mlx.model_registry.ModelLease.__aexit__(exc_type, exc, tb) -> None

Method ModelLease.__aexit__ calls self.release; awaits asynchronous work.

Parameters

Name Type Required Default Description
exc_type not annotated yes none Required positional or keyword input.
exc not annotated yes none Required positional or keyword input.
tb not annotated yes none Required positional or keyword input.

Returns

  • Type: None

Exceptions and behavior

Method ModelLease.__aexit__ calls self.release; awaits asynchronous work. No direct raise statement appears in this definition.

View source #L230-L231.

vllm_mlx.model_registry._clone_scheduler_config · function 
vllm_mlx.model_registry._clone_scheduler_config(config: SchedulerConfig | None) -> SchedulerConfig | None

Clone a SchedulerConfig so per-model overrides do not mutate globals.

Parameters

Name Type Required Default Description
config SchedulerConfig \| None yes none Required positional or keyword input.

Returns

  • Type: SchedulerConfig | None
  • Direct return expressions: None; SchedulerConfig(**vars(config))

Exceptions and behavior

Function _clone_scheduler_config calls SchedulerConfig, vars; has 2 explicit return paths. No direct raise statement appears in this definition.

View source #L234-L238.

vllm_mlx.model_registry._parse_memory_budget_bytes · function 
vllm_mlx.model_registry._parse_memory_budget_bytes(value: Any) -> int

Parse a memory budget from bytes, MB, or GB.

Parameters

Name Type Required Default Description
value Any yes none Required positional or keyword input.

Returns

  • Type: int
  • Direct return expressions: int(float(value) * 1024 ** 3); int(float(raw[:-2]) * 1024 ** 3); int(float(raw[:-2]) * 1024 ** 2); int(float(raw[:-1])); int(float(raw) * 1024 ** 3)

Exceptions and behavior

Function _parse_memory_budget_bytes calls ValueError, isinstance, int, float; can raise ValueError, TypeError; has 5 explicit return paths. Directly raised exceptions: ValueError, TypeError.

View source #L241-L256.

vllm_mlx.model_registry._safe_available_memory_bytes · function 
vllm_mlx.model_registry._safe_available_memory_bytes() -> int

Best-effort available system memory.

Parameters

This callable has no explicit inputs.

Returns

  • Type: int
  • Direct return expressions: 0; int(psutil.virtual_memory().available)

Exceptions and behavior

Function _safe_available_memory_bytes calls int, psutil.virtual_memory; has 2 explicit return paths. No direct raise statement appears in this definition.

View source #L259-L263.

vllm_mlx.model_registry._device_working_set_bytes · function 
vllm_mlx.model_registry._device_working_set_bytes() -> int | None

Best-effort Metal recommended working-set size, or None when unavailable.

Parameters

This callable has no explicit inputs.

Returns

  • Type: int | None
  • Direct return expressions: None; working_set or None

Exceptions and behavior

Function _device_working_set_bytes calls mx.metal.is_available, mx.device_info, info.get, int; has 2 explicit return paths. No direct raise statement appears in this definition.

View source #L266-L282.

vllm_mlx.model_registry.MemoryBudgetReport · class 
vllm_mlx.model_registry.MemoryBudgetReport(budget_bytes: int, device_working_set_bytes: int | None, gpu_memory_utilization: float | None, gpu_memory_utilization_source: str | None, per_engine_cache_limit_bytes: int | None, per_engine_cache_percent: float | None, continuous_batching_entries: int, total_entries: int)

Reconciliation of the manager weight budget with the Metal ceiling.

Parameters

Name Type Required Default Description
budget_bytes int yes none Required constructor field.
device_working_set_bytes int \| None yes none Required constructor field.
gpu_memory_utilization float \| None yes none Required constructor field.
gpu_memory_utilization_source str \| None yes none Required constructor field.
per_engine_cache_limit_bytes int \| None yes none Required constructor field.
per_engine_cache_percent float \| None yes none Required constructor field.
continuous_batching_entries int yes none Required constructor field.
total_entries int yes none Required constructor field.

Returns

  • Constructs: vllm_mlx.model_registry.MemoryBudgetReport

Exceptions and behavior

Class MemoryBudgetReport declares 3 direct member(s). No direct raise statement appears in this definition.

View source #L286-L339.

vllm_mlx.model_registry.MemoryBudgetReport.allocation_ceiling_bytes · method 
vllm_mlx.model_registry.MemoryBudgetReport.allocation_ceiling_bytes() -> int | None

Metal soft allocation limit that will be installed at engine start.

Parameters

This callable has no explicit inputs.

Returns

  • Type: int | None
  • Direct return expressions: None; int(self.device_working_set_bytes * self.gpu_memory_utilization)

Exceptions and behavior

Method MemoryBudgetReport.allocation_ceiling_bytes calls int; has 2 explicit return paths. No direct raise statement appears in this definition.

View source #L313-L322.

vllm_mlx.model_registry.MemoryBudgetReport.exceeds_ceiling · method 
vllm_mlx.model_registry.MemoryBudgetReport.exceeds_ceiling() -> bool

True when the weights budget alone cannot fit under the ceiling.

Parameters

This callable has no explicit inputs.

Returns

  • Type: bool
  • Direct return expressions: ceiling is not None and self.budget_bytes > ceiling

Exceptions and behavior

Method MemoryBudgetReport.exceeds_ceiling returns ceiling is not None and self.budget_bytes > ceiling. No direct raise statement appears in this definition.

View source #L325-L331.

vllm_mlx.model_registry.MemoryBudgetReport.cache_limit_exceeds_ceiling · method 
vllm_mlx.model_registry.MemoryBudgetReport.cache_limit_exceeds_ceiling() -> bool

True when one engine's prefix cache could alone fill the ceiling.

Parameters

This callable has no explicit inputs.

Returns

  • Type: bool
  • Direct return expressions: False; self.per_engine_cache_limit_bytes >= ceiling

Exceptions and behavior

Method MemoryBudgetReport.cache_limit_exceeds_ceiling has 2 explicit return paths. No direct raise statement appears in this definition.

View source #L334-L339.

vllm_mlx.model_registry.build_memory_budget_report · function 
vllm_mlx.model_registry.build_memory_budget_report(manager_config: RegistryManagerConfig, registry: dict[str, RegisteredModel], defaults: RegistryServeDefaults, *, device_working_set_bytes: int | None = None) -> MemoryBudgetReport

Reconcile the manager weight budget against the Metal allocation ceiling.

Parameters

Name Type Required Default Description
manager_config RegistryManagerConfig yes none Required positional or keyword input.
registry dict[str, RegisteredModel] yes none Required positional or keyword input.
defaults RegistryServeDefaults yes none Required positional or keyword input.
device_working_set_bytes int \| None no None Optional keyword-only input; defaults to None.

Returns

  • Type: MemoryBudgetReport
  • Direct return expressions: MemoryBudgetReport(budget_bytes=manager_config.memory_budget_bytes, device_working_set_bytes=device_working_set_bytes, …

Exceptions and behavior

Function build_memory_budget_report calls _device_working_set_bytes, sorted, candidates.append, len; returns MemoryBudgetReport(budget_bytes=manager_config.memory_budget_bytes, device_working_set_bytes=device_working_set_bytes, …. No direct raise statement appears in this definition.

View source #L342-L421.

vllm_mlx.model_registry.log_memory_budget_report · function 
vllm_mlx.model_registry.log_memory_budget_report(report: MemoryBudgetReport) -> None

Log the budget/ceiling reconciliation, warning when they conflict.

Parameters

Name Type Required Default Description
report MemoryBudgetReport yes none Required positional or keyword input.

Returns

  • Type: None
  • Direct return expressions: None

Exceptions and behavior

Function log_memory_budget_report calls logger.info, logger.warning; returns None. No direct raise statement appears in this definition.

View source #L424-L502.

vllm_mlx.model_registry._estimate_model_bytes_from_source · function 
vllm_mlx.model_registry._estimate_model_bytes_from_source(source: str) -> int

Estimate model footprint from local artifact size when possible.

Parameters

Name Type Required Default Description
source str yes none Required positional or keyword input.

Returns

  • Type: int
  • Direct return expressions: 0; path.stat().st_size if path.suffix in {'.safetensors', '.gguf'} else 0; total

Exceptions and behavior

Function _estimate_model_bytes_from_source calls Path, path.exists, path.is_file, path.stat; has 3 explicit return paths. No direct raise statement appears in this definition.

View source #L505-L521.

vllm_mlx.model_registry.load_registry_config · function 
vllm_mlx.model_registry.load_registry_config(config_path: str | os.PathLike[str], defaults: RegistryServeDefaults) -> tuple[RegistryManagerConfig, dict[str, RegisteredModel]]

Load and validate the models registry YAML file.

Parameters

Name Type Required Default Description
config_path str \| os.PathLike[str] yes none Required positional or keyword input.
defaults RegistryServeDefaults yes none Required positional or keyword input.

Returns

  • Type: tuple[RegistryManagerConfig, dict[str, RegisteredModel]]
  • Direct return expressions: (manager, registry)

Exceptions and behavior

Function load_registry_config calls yaml.safe_load, Path(config_path).read_text, Path, raw.get; can raise ValueError; returns (manager, registry). Directly raised exceptions: ValueError.

View source #L524-L621.

vllm_mlx.model_registry.ModelManager · class 
vllm_mlx.model_registry.ModelManager(manager_config: RegistryManagerConfig, registry: dict[str, RegisteredModel], defaults: RegistryServeDefaults, *, engine_factory: EngineFactory | None = None)

Registry-backed model manager with lazy load and memory-budget eviction.

Parameters

Name Type Required Default Description
manager_config RegistryManagerConfig yes none Required positional or keyword input.
registry dict[str, RegisteredModel] yes none Required positional or keyword input.
defaults RegistryServeDefaults yes none Required positional or keyword input.
engine_factory EngineFactory \| None no None Optional keyword-only input; defaults to None.

Returns

  • Constructs: vllm_mlx.model_registry.ModelManager

Exceptions and behavior

Class ModelManager declares 27 direct member(s). No direct raise statement appears in this definition.

View source #L624-L1201.

vllm_mlx.model_registry.ModelManager.__init__ · method 
vllm_mlx.model_registry.ModelManager.__init__(manager_config: RegistryManagerConfig, registry: dict[str, RegisteredModel], defaults: RegistryServeDefaults, *, engine_factory: EngineFactory | None = None) -> None

Method ModelManager.__init__ updates self._config, self._registry, self._defaults, self._engine_factory; calls asyncio.Condition.

Parameters

Name Type Required Default Description
manager_config RegistryManagerConfig yes none Required positional or keyword input.
registry dict[str, RegisteredModel] yes none Required positional or keyword input.
defaults RegistryServeDefaults yes none Required positional or keyword input.
engine_factory EngineFactory \| None no None Optional keyword-only input; defaults to None.

Returns

  • Type: None

Exceptions and behavior

Method ModelManager.__init__ updates self._config, self._registry, self._defaults, self._engine_factory; calls asyncio.Condition. No direct raise statement appears in this definition.

View source #L627-L643.

vllm_mlx.model_registry.ModelManager.memory_budget_bytes · method 
vllm_mlx.model_registry.ModelManager.memory_budget_bytes() -> int

Return the registry's configured resident-model memory budget.

Parameters

This callable has no explicit inputs.

Returns

  • Type: int
  • Direct return expressions: self._config.memory_budget_bytes

Exceptions and behavior

Method ModelManager.memory_budget_bytes returns self._config.memory_budget_bytes. No direct raise statement appears in this definition.

View source #L646-L649.

vllm_mlx.model_registry.ModelManager.registered_model_names · method 
vllm_mlx.model_registry.ModelManager.registered_model_names() -> list[str]

Return sorted list of all registered model names.

Parameters

This callable has no explicit inputs.

Returns

  • Type: list[str]
  • Direct return expressions: sorted(self._registry.keys())

Exceptions and behavior

Method ModelManager.registered_model_names calls sorted, self._registry.keys; returns sorted(self._registry.keys()). No direct raise statement appears in this definition.

View source #L652-L654.

vllm_mlx.model_registry.ModelManager.has_model · method 
vllm_mlx.model_registry.ModelManager.has_model(model_name: str) -> bool

Return whether a model name is present in the serving registry.

Parameters

Name Type Required Default Description
model_name str yes none Required positional or keyword input.

Returns

  • Type: bool
  • Direct return expressions: model_name in self._registry

Exceptions and behavior

Method ModelManager.has_model returns model_name in self._registry. No direct raise statement appears in this definition.

View source #L656-L659.

vllm_mlx.model_registry.ModelManager.list_models · method 
vllm_mlx.model_registry.ModelManager.list_models() -> list[dict[str, Any]]

Return registry state for /v1/models.

Parameters

This callable has no explicit inputs.

Returns

  • Type: list[dict[str, Any]]
  • Direct return expressions: data

Exceptions and behavior

Method ModelManager.list_models calls self._registry.items, self._loaded.get, self._unloading.get, self._loading.get; returns data. No direct raise statement appears in this definition.

View source #L661-L699.

vllm_mlx.model_registry.ModelManager.preload · method 
async vllm_mlx.model_registry.ModelManager.preload() -> None

Preload any entries marked preload=true.

Parameters

This callable has no explicit inputs.

Returns

  • Type: None

Exceptions and behavior

Method ModelManager.preload calls self._registry.values, self.acquire, lease.release; awaits asynchronous work. No direct raise statement appears in this definition.

View source #L701-L706.

vllm_mlx.model_registry.ModelManager.shutdown · method 
async vllm_mlx.model_registry.ModelManager.shutdown() -> None

Stop and unload every loaded engine.

Parameters

This callable has no explicit inputs.

Returns

  • Type: None

Exceptions and behavior

Method ModelManager.shutdown updates self._shutting_down; calls set, self._loading.values, self._loaded.values, cancel_tasks.update; awaits asynchronous work. No direct raise statement appears in this definition.

View source #L708-L739.

vllm_mlx.model_registry.ModelManager.acquire · method 
async vllm_mlx.model_registry.ModelManager.acquire(model_name: str) -> ModelLease

Acquire a lease for a configured model.

Parameters

Name Type Required Default Description
model_name str yes none Required positional or keyword input.

Returns

  • Type: ModelLease
  • Direct return expressions: claimed

Exceptions and behavior

Method ModelManager.acquire calls KeyError, time.monotonic, set, RuntimeError; awaits asynchronous work; can raise KeyError, RuntimeError; returns claimed. Directly raised exceptions: KeyError, RuntimeError.

View source #L741-L819.

vllm_mlx.model_registry.ModelManager.release · method 
async vllm_mlx.model_registry.ModelManager.release(model_name: str) -> None

Release a previously acquired model lease.

Parameters

Name Type Required Default Description
model_name str yes none Required positional or keyword input.

Returns

  • Type: None
  • Direct return expressions: None

Exceptions and behavior

Method ModelManager.release calls self._loaded.get, max, time.time, asyncio.current_task; awaits asynchronous work; returns None. No direct raise statement appears in this definition.

View source #L821-L842.

vllm_mlx.model_registry.ModelManager._claim_loaded_locked · method 
vllm_mlx.model_registry.ModelManager._claim_loaded_locked(model_name: str, *, loaded_override: LoadedModel | None = None) -> ModelLease | None

Method ModelManager._claim_loaded_locked calls self._loaded.get, time.time, asyncio.current_task, loaded.active_tasks.add; has 2 explicit return paths.

Parameters

Name Type Required Default Description
model_name str yes none Required positional or keyword input.
loaded_override LoadedModel \| None no None Optional keyword-only input; defaults to None.

Returns

  • Type: ModelLease | None
  • Direct return expressions: None; ModelLease(manager=self, model_name=model_name, engine=loaded.engine, release_cb=_release)

Exceptions and behavior

Method ModelManager._claim_loaded_locked calls self._loaded.get, time.time, asyncio.current_task, loaded.active_tasks.add; has 2 explicit return paths. No direct raise statement appears in this definition.

View source #L844-L874.

vllm_mlx.model_registry.ModelManager._claim_loaded_locked._release · nested function 
async vllm_mlx.model_registry.ModelManager._claim_loaded_locked._release() -> None

Nested Function ModelManager._claim_loaded_locked._release calls self.release; awaits asynchronous work.

Parameters

This callable has no explicit inputs.

Returns

  • Type: None

Exceptions and behavior

Nested Function ModelManager._claim_loaded_locked._release calls self.release; awaits asynchronous work. No direct raise statement appears in this definition.

View source #L866-L867.

vllm_mlx.model_registry.ModelManager._execute_load · method 
async vllm_mlx.model_registry.ModelManager._execute_load(pending: PendingLoad) -> LoadedModel

Instantiate a reserved model load outside the manager lock.

Parameters

Name Type Required Default Description
pending PendingLoad yes none Required positional or keyword input.

Returns

  • Type: LoadedModel
  • Direct return expressions: loaded

Exceptions and behavior

Method ModelManager._execute_load calls self._resolve_source, self._instantiate_model, self._loading.pop, current.future.done; awaits asynchronous work; can raise RuntimeError; returns loaded. Directly raised exceptions: RuntimeError.

View source #L876-L913.

vllm_mlx.model_registry.ModelManager._wait_for_change · method 
async vllm_mlx.model_registry.ModelManager._wait_for_change(timeout: float | None) -> None

Method ModelManager._wait_for_change calls self._condition.wait, RuntimeError, asyncio.wait_for; awaits asynchronous work; can raise RuntimeError; returns None.

Parameters

Name Type Required Default Description
timeout float \| None yes none Required positional or keyword input.

Returns

  • Type: None
  • Direct return expressions: None

Exceptions and behavior

Method ModelManager._wait_for_change calls self._condition.wait, RuntimeError, asyncio.wait_for; awaits asynchronous work; can raise RuntimeError; returns None. Directly raised exceptions: RuntimeError.

View source #L915-L922.

vllm_mlx.model_registry.ModelManager._run_unloads · method 
async vllm_mlx.model_registry.ModelManager._run_unloads(unloads: list[LoadedModel]) -> None

Method ModelManager._run_unloads calls loaded.engine.stop, self._unloading.pop, self._condition.notify_all; awaits asynchronous work.

Parameters

Name Type Required Default Description
unloads list[LoadedModel] yes none Required positional or keyword input.

Returns

  • Type: None

Exceptions and behavior

Method ModelManager._run_unloads calls loaded.engine.stop, self._unloading.pop, self._condition.notify_all; awaits asynchronous work. No direct raise statement appears in this definition.

View source #L924-L931.

vllm_mlx.model_registry.ModelManager._reserve_load_locked · method 
vllm_mlx.model_registry.ModelManager._reserve_load_locked(model_name: str, required_bytes: int) -> PendingLoad

Method ModelManager._reserve_load_locked calls asyncio.get_running_loop().create_future, asyncio.get_running_loop, PendingLoad; returns pending.

Parameters

Name Type Required Default Description
model_name str yes none Required positional or keyword input.
required_bytes int yes none Required positional or keyword input.

Returns

  • Type: PendingLoad
  • Direct return expressions: pending

Exceptions and behavior

Method ModelManager._reserve_load_locked calls asyncio.get_running_loop().create_future, asyncio.get_running_loop, PendingLoad; returns pending. No direct raise statement appears in this definition.

View source #L933-L941.

vllm_mlx.model_registry.ModelManager._begin_unload_locked · method 
vllm_mlx.model_registry.ModelManager._begin_unload_locked(model_name: str) -> LoadedModel

Method ModelManager._begin_unload_locked calls self._loaded.pop; returns loaded.

Parameters

Name Type Required Default Description
model_name str yes none Required positional or keyword input.

Returns

  • Type: LoadedModel
  • Direct return expressions: loaded

Exceptions and behavior

Method ModelManager._begin_unload_locked calls self._loaded.pop; returns loaded. No direct raise statement appears in this definition.

View source #L943-L946.

vllm_mlx.model_registry.ModelManager._collect_idle_unloads_locked · method 
vllm_mlx.model_registry.ModelManager._collect_idle_unloads_locked(requested_model: str, required_bytes: int) -> list[LoadedModel]

Method ModelManager._collect_idle_unloads_locked calls self._committed_bytes_locked, sorted, self._loaded.items, selected.append; returns selected.

Parameters

Name Type Required Default Description
requested_model str yes none Required positional or keyword input.
required_bytes int yes none Required positional or keyword input.

Returns

  • Type: list[LoadedModel]
  • Direct return expressions: selected

Exceptions and behavior

Method ModelManager._collect_idle_unloads_locked calls self._committed_bytes_locked, sorted, self._loaded.items, selected.append; returns selected. No direct raise statement appears in this definition.

View source #L948-L968.

vllm_mlx.model_registry.ModelManager._maybe_preempt_locked · method 
vllm_mlx.model_registry.ModelManager._maybe_preempt_locked(*, model_name: str, required_bytes: int, start: float) -> set[asyncio.Task[Any]]

Method ModelManager._maybe_preempt_locked calls self._should_preempt_locked, set, self._committed_bytes_locked, sorted; has 2 explicit return paths.

Parameters

Name Type Required Default Description
model_name str yes none Required keyword-only input.
required_bytes int yes none Required keyword-only input.
start float yes none Required keyword-only input.

Returns

  • Type: set[asyncio.Task[Any]]
  • Direct return expressions: set(); cancel_tasks

Exceptions and behavior

Method ModelManager._maybe_preempt_locked calls self._should_preempt_locked, set, self._committed_bytes_locked, sorted; has 2 explicit return paths. No direct raise statement appears in this definition.

View source #L970-L1003.

vllm_mlx.model_registry.ModelManager._should_wait_locked · method 
vllm_mlx.model_registry.ModelManager._should_wait_locked(start: float) -> bool

Method ModelManager._should_wait_locked calls self._remaining_wait_timeout; has 2 explicit return paths.

Parameters

Name Type Required Default Description
start float yes none Required positional or keyword input.

Returns

  • Type: bool
  • Direct return expressions: False; timeout is None or timeout > 0

Exceptions and behavior

Method ModelManager._should_wait_locked calls self._remaining_wait_timeout; has 2 explicit return paths. No direct raise statement appears in this definition.

View source #L1005-L1010.

vllm_mlx.model_registry.ModelManager._should_preempt_locked · method 
vllm_mlx.model_registry.ModelManager._should_preempt_locked(start: float) -> bool

Method ModelManager._should_preempt_locked calls time.monotonic; has 3 explicit return paths.

Parameters

Name Type Required Default Description
start float yes none Required positional or keyword input.

Returns

  • Type: bool
  • Direct return expressions: True; False; elapsed >= trigger

Exceptions and behavior

Method ModelManager._should_preempt_locked calls time.monotonic; has 3 explicit return paths. No direct raise statement appears in this definition.

View source #L1012-L1020.

vllm_mlx.model_registry.ModelManager._remaining_wait_timeout · method 
vllm_mlx.model_registry.ModelManager._remaining_wait_timeout(start: float) -> float | None

Method ModelManager._remaining_wait_timeout calls max, time.monotonic; has 2 explicit return paths.

Parameters

Name Type Required Default Description
start float yes none Required positional or keyword input.

Returns

  • Type: float | None
  • Direct return expressions: None; max(timeout - (time.monotonic() - start), 0.0)

Exceptions and behavior

Method ModelManager._remaining_wait_timeout calls max, time.monotonic; has 2 explicit return paths. No direct raise statement appears in this definition.

View source #L1022-L1026.

vllm_mlx.model_registry.ModelManager._can_reserve_locked · method 
vllm_mlx.model_registry.ModelManager._can_reserve_locked(required_bytes: int) -> bool

Method ModelManager._can_reserve_locked calls self._committed_bytes_locked; returns self._committed_bytes_locked() + required_bytes <= self._config.memory_budget_bytes.

Parameters

Name Type Required Default Description
required_bytes int yes none Required positional or keyword input.

Returns

  • Type: bool
  • Direct return expressions: self._committed_bytes_locked() + required_bytes <= self._config.memory_budget_bytes

Exceptions and behavior

Method ModelManager._can_reserve_locked calls self._committed_bytes_locked; returns self._committed_bytes_locked() + required_bytes <= self._config.memory_budget_bytes. No direct raise statement appears in this definition.

View source #L1028-L1032.

vllm_mlx.model_registry.ModelManager._committed_bytes_locked · method 
vllm_mlx.model_registry.ModelManager._committed_bytes_locked() -> int

Method ModelManager._committed_bytes_locked calls sum, self._loaded.values, self._loading.values, self._unloading.values; returns loaded_bytes + loading_bytes + unloading_bytes.

Parameters

This callable has no explicit inputs.

Returns

  • Type: int
  • Direct return expressions: loaded_bytes + loading_bytes + unloading_bytes

Exceptions and behavior

Method ModelManager._committed_bytes_locked calls sum, self._loaded.values, self._loading.values, self._unloading.values; returns loaded_bytes + loading_bytes + unloading_bytes. No direct raise statement appears in this definition.

View source #L1034-L1042.

vllm_mlx.model_registry.ModelManager._instantiate_model · method 
async vllm_mlx.model_registry.ModelManager._instantiate_model(entry: RegisteredModel, resolved_source: str) -> LoadedModel

Method ModelManager._instantiate_model calls self._resolve_model_config, self._engine_factory, BatchedEngine, SimpleEngine; awaits asynchronous work; returns LoadedModel(config=config, engine=engine).

Parameters

Name Type Required Default Description
entry RegisteredModel yes none Required positional or keyword input.
resolved_source str yes none Required positional or keyword input.

Returns

  • Type: LoadedModel
  • Direct return expressions: LoadedModel(config=config, engine=engine)

Exceptions and behavior

Method ModelManager._instantiate_model calls self._resolve_model_config, self._engine_factory, BatchedEngine, SimpleEngine; awaits asynchronous work; returns LoadedModel(config=config, engine=engine). No direct raise statement appears in this definition.

View source #L1044-L1073.

vllm_mlx.model_registry.ModelManager._resolve_source · method 
async vllm_mlx.model_registry.ModelManager._resolve_source(entry: RegisteredModel) -> str

Method ModelManager._resolve_source calls asyncio.to_thread; awaits asynchronous work; returns await asyncio.to_thread(self._resolve_source_sync, entry).

Parameters

Name Type Required Default Description
entry RegisteredModel yes none Required positional or keyword input.

Returns

  • Type: str
  • Direct return expressions: await asyncio.to_thread(self._resolve_source_sync, entry)

Exceptions and behavior

Method ModelManager._resolve_source calls asyncio.to_thread; awaits asynchronous work; returns await asyncio.to_thread(self._resolve_source_sync, entry). No direct raise statement appears in this definition.

View source #L1075-L1076.

vllm_mlx.model_registry.ModelManager._resolve_source_sync · method 
vllm_mlx.model_registry.ModelManager._resolve_source_sync(entry: RegisteredModel) -> str

Method ModelManager._resolve_source_sync calls Path(source).exists, Path, ensure_model_downloaded, is_mllm_model; has 2 explicit return paths.

Parameters

Name Type Required Default Description
entry RegisteredModel yes none Required positional or keyword input.

Returns

  • Type: str
  • Direct return expressions: source; str(downloaded)

Exceptions and behavior

Method ModelManager._resolve_source_sync calls Path(source).exists, Path, ensure_model_downloaded, is_mllm_model; has 2 explicit return paths. No direct raise statement appears in this definition.

View source #L1078-L1087.

vllm_mlx.model_registry.ModelManager._resolve_estimated_bytes · method 
vllm_mlx.model_registry.ModelManager._resolve_estimated_bytes(entry: RegisteredModel, resolved_source: str) -> int

Method ModelManager._resolve_estimated_bytes calls _estimate_model_bytes_from_source, Path, source_path.exists, ValueError; can raise ValueError; has 3 explicit return paths.

Parameters

Name Type Required Default Description
entry RegisteredModel yes none Required positional or keyword input.
resolved_source str yes none Required positional or keyword input.

Returns

  • Type: int
  • Direct return expressions: entry.estimated_memory_bytes; estimated; max(available // 8, 1)

Exceptions and behavior

Method ModelManager._resolve_estimated_bytes calls _estimate_model_bytes_from_source, Path, source_path.exists, ValueError; can raise ValueError; has 3 explicit return paths. Directly raised exceptions: ValueError.

View source #L1089-L1121.

vllm_mlx.model_registry.ModelManager._resolve_model_config · method 
vllm_mlx.model_registry.ModelManager._resolve_model_config(entry: RegisteredModel, resolved_source: str) -> ResolvedModelConfig

Method ModelManager._resolve_model_config calls _clone_scheduler_config, self._resolve_estimated_bytes, ResolvedModelConfig; returns ResolvedModelConfig(entry=entry, resolved_source=resolved_source, continuous_batching=continuous_batching, force_mllm=f….

Parameters

Name Type Required Default Description
entry RegisteredModel yes none Required positional or keyword input.
resolved_source str yes none Required positional or keyword input.

Returns

  • Type: ResolvedModelConfig
  • Direct return expressions: ResolvedModelConfig(entry=entry, resolved_source=resolved_source, continuous_batching=continuous_batching, force_mllm=f…

Exceptions and behavior

Method ModelManager._resolve_model_config calls _clone_scheduler_config, self._resolve_estimated_bytes, ResolvedModelConfig; returns ResolvedModelConfig(entry=entry, resolved_source=resolved_source, continuous_batching=continuous_batching, force_mllm=f…. No direct raise statement appears in this definition.

View source #L1123-L1201.

Complete symbol map

This map also includes private definitions and nested helpers. The signature column exposes every explicit input even when an internal helper has no dedicated parameter prose.

Symbol Kind Signature and inputs What it does Source
ModelOwnershipError class ModelOwnershipError() Raised when an EngineCore attempts to use a model already in use. #L38-L39
_ModelOwnershipRegistry class _ModelOwnershipRegistry() Process-local model ownership guard used by EngineCore. #L42-L82
_ModelOwnershipRegistry.__init__ method _ModelOwnershipRegistry.__init__() -> None Method _ModelOwnershipRegistry.__init__ updates self._owners. #L45-L46
_ModelOwnershipRegistry.acquire method _ModelOwnershipRegistry.acquire(*, model: Any, engine: Any, engine_id: str, force: bool = True) -> None Method _ModelOwnershipRegistry.acquire calls id, self._owners.get, ModelOwnershipError; can raise ModelOwnershipError. #L48-L63
_ModelOwnershipRegistry.release method _ModelOwnershipRegistry.release(model: Any, engine_id: str) -> None Method _ModelOwnershipRegistry.release calls id, self._owners.get, self._owners.pop. #L65-L69
_ModelOwnershipRegistry.is_owned method _ModelOwnershipRegistry.is_owned(model: Any) -> tuple[bool, str \| None] Method _ModelOwnershipRegistry.is_owned calls id, self._owners.get; has 2 explicit return paths. #L71-L76
_ModelOwnershipRegistry.get_stats method _ModelOwnershipRegistry.get_stats() -> dict[str, Any] Method _ModelOwnershipRegistry.get_stats calls len; returns {'total_entries': len(self._owners), 'active_owners': len(self._owners)}. #L78-L82
get_registry function get_registry() -> _ModelOwnershipRegistry Return the global model ownership registry used by EngineCore. #L88-L90
RegistryServeDefaults class RegistryServeDefaults(continuous_batching: bool, force_mllm: bool, enable_mtp: bool, prefill_step_size: int, specprefill_enabled: bool, specprefill_threshold: int, specprefill_keep_pct: float, specprefill_backbone_pct: float, specprefill_draft_model: str \| None, stream_interval: int, gpu_memory_utilization: float, scheduler_config: SchedulerConfig \| None, max_tokens: int, download_config: DownloadConfig) Global serve defaults inherited by registry entries. #L109-L125
ContentionPolicy class ContentionPolicy(strategy: ContentionStrategy = 'wait_then_fail', wait_timeout_s: float \| None = 30.0, preempt_after_s: float \| None = None) Policy used when a new model cannot fit inside the memory budget. #L129-L134
RegistryManagerConfig class RegistryManagerConfig(memory_budget_bytes: int, policy: ContentionPolicy) Global registry manager configuration. #L138-L142
RegisteredModel class RegisteredModel(name: str, source: str, preload: bool = False, continuous_batching: bool \| None = None, force_mllm: bool \| None = None, enable_mtp: bool \| None = None, prefill_step_size: int \| None = None, specprefill_enabled: bool \| None = None, specprefill_threshold: int \| None = None, specprefill_keep_pct: float \| None = None, specprefill_backbone_pct: float \| None = None, specprefill_draft_model: str \| None = None, stream_interval: int \| None = None, gpu_memory_utilization: float \| None = None, estimated_memory_bytes: int \| None = None) One configured model entry. #L146-L163
ResolvedModelConfig class ResolvedModelConfig(entry: RegisteredModel, resolved_source: str, continuous_batching: bool, force_mllm: bool, enable_mtp: bool, prefill_step_size: int, specprefill_enabled: bool, specprefill_threshold: int, specprefill_keep_pct: float, specprefill_backbone_pct: float, specprefill_draft_model: str \| None, stream_interval: int, gpu_memory_utilization: float, scheduler_config: SchedulerConfig \| None, estimated_memory_bytes: int) Effective configuration for a loaded model. #L167-L184
LoadedModel class LoadedModel(config: ResolvedModelConfig, engine: BaseEngine, loaded_at: float = field(default_factory=time.time), last_used_at: float = field(default_factory=time.time), active_requests: int = 0, active_tasks: set[asyncio.Task[Any]] = field(default_factory=set), preempting: bool = False) Runtime state for a loaded engine. #L188-L197
PendingLoad class PendingLoad(model_name: str, required_bytes: int, future: asyncio.Future[LoadedModel]) A reserved model load in progress. #L201-L206
ModelLease class ModelLease(manager: 'ModelManager \| None', model_name: str, engine: BaseEngine, release_cb: Callable[[], Awaitable[None]]) Active lease for a loaded model. #L210-L231
ModelLease.release method async ModelLease.release() -> None Release this lease once and allow the model to become evictable. #L218-L225
ModelLease.__aenter__ method async ModelLease.__aenter__() -> 'ModelLease' Method ModelLease.__aenter__ returns self. #L227-L228
ModelLease.__aexit__ method async ModelLease.__aexit__(exc_type, exc, tb) -> None Method ModelLease.__aexit__ calls self.release; awaits asynchronous work. #L230-L231
_clone_scheduler_config function _clone_scheduler_config(config: SchedulerConfig \| None) -> SchedulerConfig \| None Clone a SchedulerConfig so per-model overrides do not mutate globals. #L234-L238
_parse_memory_budget_bytes function _parse_memory_budget_bytes(value: Any) -> int Parse a memory budget from bytes, MB, or GB. #L241-L256
_safe_available_memory_bytes function _safe_available_memory_bytes() -> int Best-effort available system memory. #L259-L263
_device_working_set_bytes function _device_working_set_bytes() -> int \| None Best-effort Metal recommended working-set size, or None when unavailable. #L266-L282
MemoryBudgetReport class MemoryBudgetReport(budget_bytes: int, device_working_set_bytes: int \| None, gpu_memory_utilization: float \| None, gpu_memory_utilization_source: str \| None, per_engine_cache_limit_bytes: int \| None, per_engine_cache_percent: float \| None, continuous_batching_entries: int, total_entries: int) Reconciliation of the manager weight budget with the Metal ceiling. #L286-L339
MemoryBudgetReport.allocation_ceiling_bytes method MemoryBudgetReport.allocation_ceiling_bytes() -> int \| None Metal soft allocation limit that will be installed at engine start. #L313-L322
MemoryBudgetReport.exceeds_ceiling method MemoryBudgetReport.exceeds_ceiling() -> bool True when the weights budget alone cannot fit under the ceiling. #L325-L331
MemoryBudgetReport.cache_limit_exceeds_ceiling method MemoryBudgetReport.cache_limit_exceeds_ceiling() -> bool True when one engine's prefix cache could alone fill the ceiling. #L334-L339
build_memory_budget_report function build_memory_budget_report(manager_config: RegistryManagerConfig, registry: dict[str, RegisteredModel], defaults: RegistryServeDefaults, *, device_working_set_bytes: int \| None = None) -> MemoryBudgetReport Reconcile the manager weight budget against the Metal allocation ceiling. #L342-L421
log_memory_budget_report function log_memory_budget_report(report: MemoryBudgetReport) -> None Log the budget/ceiling reconciliation, warning when they conflict. #L424-L502
_estimate_model_bytes_from_source function _estimate_model_bytes_from_source(source: str) -> int Estimate model footprint from local artifact size when possible. #L505-L521
load_registry_config function load_registry_config(config_path: str \| os.PathLike[str], defaults: RegistryServeDefaults) -> tuple[RegistryManagerConfig, dict[str, RegisteredModel]] Load and validate the models registry YAML file. #L524-L621
ModelManager class ModelManager(manager_config: RegistryManagerConfig, registry: dict[str, RegisteredModel], defaults: RegistryServeDefaults, *, engine_factory: EngineFactory \| None = None) Registry-backed model manager with lazy load and memory-budget eviction. #L624-L1201
ModelManager.__init__ method ModelManager.__init__(manager_config: RegistryManagerConfig, registry: dict[str, RegisteredModel], defaults: RegistryServeDefaults, *, engine_factory: EngineFactory \| None = None) -> None Method ModelManager.__init__ updates self._config, self._registry, self._defaults, self._engine_factory; calls asyncio.Condition. #L627-L643
ModelManager.memory_budget_bytes method ModelManager.memory_budget_bytes() -> int Return the registry's configured resident-model memory budget. #L646-L649
ModelManager.registered_model_names method ModelManager.registered_model_names() -> list[str] Return sorted list of all registered model names. #L652-L654
ModelManager.has_model method ModelManager.has_model(model_name: str) -> bool Return whether a model name is present in the serving registry. #L656-L659
ModelManager.list_models method ModelManager.list_models() -> list[dict[str, Any]] Return registry state for /v1/models. #L661-L699
ModelManager.preload method async ModelManager.preload() -> None Preload any entries marked preload=true. #L701-L706
ModelManager.shutdown method async ModelManager.shutdown() -> None Stop and unload every loaded engine. #L708-L739
ModelManager.acquire method async ModelManager.acquire(model_name: str) -> ModelLease Acquire a lease for a configured model. #L741-L819
ModelManager.release method async ModelManager.release(model_name: str) -> None Release a previously acquired model lease. #L821-L842
ModelManager._claim_loaded_locked method ModelManager._claim_loaded_locked(model_name: str, *, loaded_override: LoadedModel \| None = None) -> ModelLease \| None Method ModelManager._claim_loaded_locked calls self._loaded.get, time.time, asyncio.current_task, loaded.active_tasks.add; has 2 explicit return paths. #L844-L874
ModelManager._claim_loaded_locked._release nested function async ModelManager._claim_loaded_locked._release() -> None Nested Function ModelManager._claim_loaded_locked._release calls self.release; awaits asynchronous work. #L866-L867
ModelManager._execute_load method async ModelManager._execute_load(pending: PendingLoad) -> LoadedModel Instantiate a reserved model load outside the manager lock. #L876-L913
ModelManager._wait_for_change method async ModelManager._wait_for_change(timeout: float \| None) -> None Method ModelManager._wait_for_change calls self._condition.wait, RuntimeError, asyncio.wait_for; awaits asynchronous work; can raise RuntimeError; returns None. #L915-L922
ModelManager._run_unloads method async ModelManager._run_unloads(unloads: list[LoadedModel]) -> None Method ModelManager._run_unloads calls loaded.engine.stop, self._unloading.pop, self._condition.notify_all; awaits asynchronous work. #L924-L931
ModelManager._reserve_load_locked method ModelManager._reserve_load_locked(model_name: str, required_bytes: int) -> PendingLoad Method ModelManager._reserve_load_locked calls asyncio.get_running_loop().create_future, asyncio.get_running_loop, PendingLoad; returns pending. #L933-L941
ModelManager._begin_unload_locked method ModelManager._begin_unload_locked(model_name: str) -> LoadedModel Method ModelManager._begin_unload_locked calls self._loaded.pop; returns loaded. #L943-L946
ModelManager._collect_idle_unloads_locked method ModelManager._collect_idle_unloads_locked(requested_model: str, required_bytes: int) -> list[LoadedModel] Method ModelManager._collect_idle_unloads_locked calls self._committed_bytes_locked, sorted, self._loaded.items, selected.append; returns selected. #L948-L968
ModelManager._maybe_preempt_locked method ModelManager._maybe_preempt_locked(*, model_name: str, required_bytes: int, start: float) -> set[asyncio.Task[Any]] Method ModelManager._maybe_preempt_locked calls self._should_preempt_locked, set, self._committed_bytes_locked, sorted; has 2 explicit return paths. #L970-L1003
ModelManager._should_wait_locked method ModelManager._should_wait_locked(start: float) -> bool Method ModelManager._should_wait_locked calls self._remaining_wait_timeout; has 2 explicit return paths. #L1005-L1010
ModelManager._should_preempt_locked method ModelManager._should_preempt_locked(start: float) -> bool Method ModelManager._should_preempt_locked calls time.monotonic; has 3 explicit return paths. #L1012-L1020
ModelManager._remaining_wait_timeout method ModelManager._remaining_wait_timeout(start: float) -> float \| None Method ModelManager._remaining_wait_timeout calls max, time.monotonic; has 2 explicit return paths. #L1022-L1026
ModelManager._can_reserve_locked method ModelManager._can_reserve_locked(required_bytes: int) -> bool Method ModelManager._can_reserve_locked calls self._committed_bytes_locked; returns self._committed_bytes_locked() + required_bytes <= self._config.memory_budget_bytes. #L1028-L1032
ModelManager._committed_bytes_locked method ModelManager._committed_bytes_locked() -> int Method ModelManager._committed_bytes_locked calls sum, self._loaded.values, self._loading.values, self._unloading.values; returns loaded_bytes + loading_bytes + unloading_bytes. #L1034-L1042
ModelManager._instantiate_model method async ModelManager._instantiate_model(entry: RegisteredModel, resolved_source: str) -> LoadedModel Method ModelManager._instantiate_model calls self._resolve_model_config, self._engine_factory, BatchedEngine, SimpleEngine; awaits asynchronous work; returns LoadedModel(config=config, engine=engine). #L1044-L1073
ModelManager._resolve_source method async ModelManager._resolve_source(entry: RegisteredModel) -> str Method ModelManager._resolve_source calls asyncio.to_thread; awaits asynchronous work; returns await asyncio.to_thread(self._resolve_source_sync, entry). #L1075-L1076
ModelManager._resolve_source_sync method ModelManager._resolve_source_sync(entry: RegisteredModel) -> str Method ModelManager._resolve_source_sync calls Path(source).exists, Path, ensure_model_downloaded, is_mllm_model; has 2 explicit return paths. #L1078-L1087
ModelManager._resolve_estimated_bytes method ModelManager._resolve_estimated_bytes(entry: RegisteredModel, resolved_source: str) -> int Method ModelManager._resolve_estimated_bytes calls _estimate_model_bytes_from_source, Path, source_path.exists, ValueError; can raise ValueError; has 3 explicit return paths. #L1089-L1121
ModelManager._resolve_model_config method ModelManager._resolve_model_config(entry: RegisteredModel, resolved_source: str) -> ResolvedModelConfig Method ModelManager._resolve_model_config calls _clone_scheduler_config, self._resolve_estimated_bytes, ResolvedModelConfig; returns ResolvedModelConfig(entry=entry, resolved_source=resolved_source, continuous_batching=continuous_batching, force_mllm=f…. #L1123-L1201