vllm_mlx.lifecycle

Model lifecycle / residency management for vllm-mlx.

View the complete module source at #L1-L493.

API details

Each callable below includes its exact signature, type annotations, inputs, defaults, return contract, documented exceptions, implementation source, and parsed docstring sections when the source provides them.

vllm_mlx.lifecycle

Model lifecycle / residency management for vllm-mlx.

vllm_mlx.lifecycle.ResidentState

Bases: str, Enum

Runtime residency state for a configured model.

vllm_mlx.lifecycle.ResidentState.UNLOADED class-attribute instance-attribute

UNLOADED = 'unloaded'

vllm_mlx.lifecycle.ResidentState.LOADING class-attribute instance-attribute

LOADING = 'loading'

vllm_mlx.lifecycle.ResidentState.LOADED class-attribute instance-attribute

LOADED = 'loaded'

vllm_mlx.lifecycle.ResidentState.UNLOADING class-attribute instance-attribute

UNLOADING = 'unloading'

vllm_mlx.lifecycle.ResidentState.FAILED class-attribute instance-attribute

FAILED = 'failed'

vllm_mlx.lifecycle.ModelSpec dataclass

ModelSpec(model_key: str, model_name: str, use_batching: bool = False, scheduler_config: Any | None = None, stream_interval: int = 1, max_tokens: int = 32768, force_mllm: bool = False, mtp: bool = False, prefill_step_size: int = 2048, specprefill_enabled: bool = False, specprefill_threshold: int = 8192, specprefill_keep_pct: float = 0.3, specprefill_backbone_pct: float = 0.0, specprefill_draft_model: str | None = None)

Immutable engine construction inputs for a resident model.

vllm_mlx.lifecycle.ModelSpec.model_key instance-attribute

model_key: str

vllm_mlx.lifecycle.ModelSpec.model_name instance-attribute

model_name: str

vllm_mlx.lifecycle.ModelSpec.use_batching class-attribute instance-attribute

use_batching: bool = False

vllm_mlx.lifecycle.ModelSpec.scheduler_config class-attribute instance-attribute

scheduler_config: Any | None = None

vllm_mlx.lifecycle.ModelSpec.stream_interval class-attribute instance-attribute

stream_interval: int = 1

vllm_mlx.lifecycle.ModelSpec.max_tokens class-attribute instance-attribute

max_tokens: int = 32768

vllm_mlx.lifecycle.ModelSpec.force_mllm class-attribute instance-attribute

force_mllm: bool = False

vllm_mlx.lifecycle.ModelSpec.mtp class-attribute instance-attribute

mtp: bool = False

vllm_mlx.lifecycle.ModelSpec.prefill_step_size class-attribute instance-attribute

prefill_step_size: int = 2048

vllm_mlx.lifecycle.ModelSpec.specprefill_enabled class-attribute instance-attribute

specprefill_enabled: bool = False

vllm_mlx.lifecycle.ModelSpec.specprefill_threshold class-attribute instance-attribute

specprefill_threshold: int = 8192

vllm_mlx.lifecycle.ModelSpec.specprefill_keep_pct class-attribute instance-attribute

specprefill_keep_pct: float = 0.3

vllm_mlx.lifecycle.ModelSpec.specprefill_backbone_pct class-attribute instance-attribute

specprefill_backbone_pct: float = 0.0

vllm_mlx.lifecycle.ModelSpec.specprefill_draft_model class-attribute instance-attribute

specprefill_draft_model: str | None = None

vllm_mlx.lifecycle.ResidentModel dataclass

ResidentModel(spec: ModelSpec, state: ResidentState = UNLOADED, engine: BaseEngine | None = None, active_requests: int = 0, last_used_at: float | None = None, loaded_at: float | None = None, last_error: str | None = None, estimated_memory_bytes: int | None = None, _load_waiters: int = 0, _load_waiter_task: Task[BaseEngine] | None = None, _prepare_task: Task[None] | None = None, _abandoned_loading_task: Task[BaseEngine] | None = None, _loading_task: Task[BaseEngine] | None = None, _unloading_task: Task[bool] | None = None)

Runtime state for a single resident model.

vllm_mlx.lifecycle.ResidentModel.spec instance-attribute

spec: ModelSpec

vllm_mlx.lifecycle.ResidentModel.state class-attribute instance-attribute

state: ResidentState = ResidentState.UNLOADED

vllm_mlx.lifecycle.ResidentModel.engine class-attribute instance-attribute

engine: BaseEngine | None = None

vllm_mlx.lifecycle.ResidentModel.active_requests class-attribute instance-attribute

active_requests: int = 0

vllm_mlx.lifecycle.ResidentModel.last_used_at class-attribute instance-attribute

last_used_at: float | None = None

vllm_mlx.lifecycle.ResidentModel.loaded_at class-attribute instance-attribute

loaded_at: float | None = None

vllm_mlx.lifecycle.ResidentModel.last_error class-attribute instance-attribute

last_error: str | None = None

vllm_mlx.lifecycle.ResidentModel.estimated_memory_bytes class-attribute instance-attribute

estimated_memory_bytes: int | None = None

vllm_mlx.lifecycle.ResidentModel._load_waiters class-attribute instance-attribute

_load_waiters: int = field(default=0, repr=False)

vllm_mlx.lifecycle.ResidentModel._load_waiter_task class-attribute instance-attribute

_load_waiter_task: Task[BaseEngine] | None = field(default=None, repr=False)

vllm_mlx.lifecycle.ResidentModel._prepare_task class-attribute instance-attribute

_prepare_task: Task[None] | None = field(default=None, repr=False)

vllm_mlx.lifecycle.ResidentModel._abandoned_loading_task class-attribute instance-attribute

_abandoned_loading_task: Task[BaseEngine] | None = field(default=None, repr=False)

vllm_mlx.lifecycle.ResidentModel._loading_task class-attribute instance-attribute

_loading_task: Task[BaseEngine] | None = field(default=None, repr=False)

vllm_mlx.lifecycle.ResidentModel._unloading_task class-attribute instance-attribute

_unloading_task: Task[bool] | None = field(default=None, repr=False)

vllm_mlx.lifecycle.ResidencyManager

ResidencyManager(engine_factory: Callable[[ModelSpec], Awaitable[BaseEngine]], *, on_engine_loaded: Callable[[ModelSpec, BaseEngine], Awaitable[None] | None] | None = None, on_engine_unloading: Callable[[ModelSpec, BaseEngine], Awaitable[None] | None] | None = None, time_fn: Callable[[], float] | None = None, auto_unload_idle_seconds: float = 0)

Single-flight lifecycle manager for resident models.

Source code in vllm_mlx/lifecycle.py

def __init__(
    self,
    engine_factory: Callable[[ModelSpec], Awaitable[BaseEngine]],
    *,
    on_engine_loaded: (
        Callable[[ModelSpec, BaseEngine], Awaitable[None] | None] | None
    ) = None,
    on_engine_unloading: (
        Callable[[ModelSpec, BaseEngine], Awaitable[None] | None] | None
    ) = None,
    time_fn: Callable[[], float] | None = None,
    auto_unload_idle_seconds: float = 0,
) -> None:
    self._engine_factory = engine_factory
    self._on_engine_loaded = on_engine_loaded
    self._on_engine_unloading = on_engine_unloading
    self._time_fn = time_fn or __import__("time").time
    self.auto_unload_idle_seconds = auto_unload_idle_seconds
    self._residents: dict[str, ResidentModel] = {}
    self._lock = asyncio.Lock()

vllm_mlx.lifecycle.ResidencyManager._engine_factory instance-attribute

_engine_factory = engine_factory

vllm_mlx.lifecycle.ResidencyManager._on_engine_loaded instance-attribute

_on_engine_loaded = on_engine_loaded

vllm_mlx.lifecycle.ResidencyManager._on_engine_unloading instance-attribute

_on_engine_unloading = on_engine_unloading

vllm_mlx.lifecycle.ResidencyManager._time_fn instance-attribute

_time_fn = time_fn or __import__('time').time

vllm_mlx.lifecycle.ResidencyManager.auto_unload_idle_seconds instance-attribute

auto_unload_idle_seconds = auto_unload_idle_seconds

vllm_mlx.lifecycle.ResidencyManager._residents instance-attribute

_residents: dict[str, ResidentModel] = {}

vllm_mlx.lifecycle.ResidencyManager._lock instance-attribute

_lock = asyncio.Lock()

vllm_mlx.lifecycle.ResidencyManager.register_model

register_model(spec: ModelSpec) -> str

Register a model spec, or replace a dormant resident entry.

Source code in vllm_mlx/lifecycle.py

def register_model(self, spec: ModelSpec) -> str:
    """Register a model spec, or replace a dormant resident entry."""
    existing = self._residents.get(spec.model_key)
    if existing is not None:
        is_dormant = (
            existing.engine is None
            and existing.active_requests == 0
            and existing._load_waiters == 0
            and existing._loading_task is None
            and existing._unloading_task is None
            and existing.state in {ResidentState.UNLOADED, ResidentState.FAILED}
        )
        if not is_dormant:
            raise RuntimeError(
                f"Cannot replace resident model '{spec.model_key}' while it is live"
            )

    self._residents[spec.model_key] = ResidentModel(spec=spec)
    return spec.model_key

vllm_mlx.lifecycle.ResidencyManager.get_engine

get_engine(model_key: str) -> BaseEngine | None

Get the currently loaded engine, if any.

Source code in vllm_mlx/lifecycle.py

def get_engine(self, model_key: str) -> BaseEngine | None:
    """Get the currently loaded engine, if any."""
    return self._resident(model_key).engine

vllm_mlx.lifecycle.ResidencyManager.get_status

get_status(model_key: str) -> dict[str, Any]

Return a serializable snapshot of resident state.

Source code in vllm_mlx/lifecycle.py

def get_status(self, model_key: str) -> dict[str, Any]:
    """Return a serializable snapshot of resident state."""
    resident = self._resident(model_key)
    return {
        "model_key": resident.spec.model_key,
        "model_name": resident.spec.model_name,
        "state": resident.state.value,
        "active_requests": resident.active_requests,
        "last_used_at": resident.last_used_at,
        "loaded_at": resident.loaded_at,
        "last_error": resident.last_error,
        "estimated_memory_bytes": resident.estimated_memory_bytes,
        "auto_unload_idle_seconds": self.auto_unload_idle_seconds,
    }

vllm_mlx.lifecycle.ResidencyManager.ensure_loaded async

ensure_loaded(model_key: str) -> BaseEngine

Load and start a resident engine if needed.

Source code in vllm_mlx/lifecycle.py

async def ensure_loaded(self, model_key: str) -> BaseEngine:
    """Load and start a resident engine if needed."""
    while True:
        task: asyncio.Task[BaseEngine] | None = None
        unloading_task: asyncio.Task[bool] | None = None

        async with self._lock:
            resident = self._resident(model_key)
            if (
                resident.state == ResidentState.LOADED
                and resident.engine is not None
            ):
                return resident.engine

            if resident._unloading_task is not None:
                unloading_task = resident._unloading_task
            else:
                if resident._loading_task is None:
                    resident.state = ResidentState.LOADING
                    resident.last_error = None
                    resident._loading_task = asyncio.create_task(
                        self._load_engine(resident)
                    )
                    resident._load_waiters = 0
                    resident._load_waiter_task = resident._loading_task
                    resident._abandoned_loading_task = None
                task = resident._loading_task
                resident._load_waiters += 1
                resident._load_waiter_task = task

        if unloading_task is not None:
            await asyncio.shield(unloading_task)
            continue

        if task is None:
            raise RuntimeError(f"No load task available for resident {model_key}")
        try:
            return await asyncio.shield(task)
        except asyncio.CancelledError:
            current_task = asyncio.current_task()
            cancelling = getattr(current_task, "cancelling", None)
            if (
                task.done()
                and task.cancelled()
                and (cancelling is None or cancelling() == 0)
            ):
                async with self._lock:
                    resident = self._resident(model_key)
                    if resident._abandoned_loading_task is task:
                        continue
            raise
        finally:
            await self._release_load_waiter(model_key, task)

vllm_mlx.lifecycle.ResidencyManager.acquire async

acquire(model_key: str, *, count_activity: bool = True) -> BaseEngine

Acquire a resident engine for request processing.

Source code in vllm_mlx/lifecycle.py

async def acquire(
    self,
    model_key: str,
    *,
    count_activity: bool = True,
) -> BaseEngine:
    """Acquire a resident engine for request processing."""
    while True:
        engine = await self.ensure_loaded(model_key)
        async with self._lock:
            resident = self._resident(model_key)
            if (
                resident.engine is not engine
                or resident.state != ResidentState.LOADED
                or resident._unloading_task is not None
            ):
                continue
            resident.active_requests += 1
            if count_activity:
                resident.last_used_at = self._time_fn()
            return engine

vllm_mlx.lifecycle.ResidencyManager.release async

release(model_key: str, *, count_activity: bool = True) -> None

Release a previously acquired resident engine.

Source code in vllm_mlx/lifecycle.py

async def release(self, model_key: str, *, count_activity: bool = True) -> None:
    """Release a previously acquired resident engine."""
    async with self._lock:
        resident = self._resident(model_key)
        if resident.active_requests > 0:
            resident.active_requests -= 1
        if count_activity:
            resident.last_used_at = self._time_fn()

vllm_mlx.lifecycle.ResidencyManager.unload_if_idle async

unload_if_idle(model_key: str) -> bool

Unload a resident engine if it has been idle past the threshold.

Source code in vllm_mlx/lifecycle.py

async def unload_if_idle(self, model_key: str) -> bool:
    """Unload a resident engine if it has been idle past the threshold."""
    if self.auto_unload_idle_seconds <= 0:
        return False

    while True:
        unloading_task: asyncio.Task[bool] | None = None
        async with self._lock:
            resident = self._resident(model_key)

            if resident._loading_task is not None:
                return False

            if resident._unloading_task is not None:
                unloading_task = resident._unloading_task
            else:
                if (
                    resident.state != ResidentState.LOADED
                    or resident.engine is None
                    or resident.active_requests > 0
                    or resident.last_used_at is None
                ):
                    return False

                idle_for = self._time_fn() - resident.last_used_at
                if idle_for < self.auto_unload_idle_seconds:
                    return False

                resident.state = ResidentState.UNLOADING
                resident._unloading_task = asyncio.create_task(
                    self._unload_engine(resident)
                )
                unloading_task = resident._unloading_task

        if unloading_task is None:
            return False
        return await asyncio.shield(unloading_task)

vllm_mlx.lifecycle.ResidencyManager.shutdown async

shutdown() -> None

Stop all loaded residents.

Source code in vllm_mlx/lifecycle.py

async def shutdown(self) -> None:
    """Stop all loaded residents."""
    keys = list(self._residents.keys())
    failures: list[str] = []
    for model_key in keys:
        while True:
            loading_task: asyncio.Task[BaseEngine] | None = None
            unloading_task: asyncio.Task[bool] | None = None

            async with self._lock:
                resident = self._resident(model_key)

                if resident._loading_task is not None:
                    resident._loading_task.cancel()
                    loading_task = resident._loading_task
                elif (
                    resident.engine is None
                    or resident.state == ResidentState.UNLOADED
                ):
                    break
                else:
                    if resident._unloading_task is None:
                        resident.state = ResidentState.UNLOADING
                        resident._unloading_task = asyncio.create_task(
                            self._unload_engine(resident)
                        )
                    unloading_task = resident._unloading_task

            if loading_task is not None:
                with suppress(asyncio.CancelledError):
                    await loading_task
                continue

            if unloading_task is not None:
                # Shield the unload so that cancelling shutdown() does not
                # orphan a half-stopped engine in UNLOADING state.
                try:
                    unloaded = await asyncio.shield(unloading_task)
                except asyncio.CancelledError:
                    # Shutdown itself was cancelled — finish the in-flight
                    # unload deterministically before propagating.
                    with suspend_cancellation():
                        unloaded = await unloading_task
                    raise
                if not unloaded:
                    async with self._lock:
                        resident = self._resident(model_key)
                        error = resident.last_error or "resident remained loaded"
                    failures.append(
                        f"Failed to unload resident model '{model_key}' during shutdown: {error}"
                    )
                    break
                break

    if failures:
        if len(failures) == 1:
            raise RuntimeError(failures[0])
        raise RuntimeError("; ".join(failures))

vllm_mlx.lifecycle.ResidencyManager._load_engine async

_load_engine(resident: ResidentModel) -> BaseEngine

Create and start a resident engine.

Source code in vllm_mlx/lifecycle.py

async def _load_engine(self, resident: ResidentModel) -> BaseEngine:
    """Create and start a resident engine."""
    engine: BaseEngine | None = None
    try:
        engine = await self._engine_factory(resident.spec)
        await self._prepare_engine_start(resident, engine)
        await engine.start()
        await self._run_hook(self._on_engine_loaded, resident.spec, engine)
    except asyncio.CancelledError:
        await self._cleanup_cancelled_load(resident, engine)
        raise
    except Exception as exc:
        async with self._lock:
            abandoned = resident._abandoned_loading_task is asyncio.current_task()
        if abandoned:
            await self._cleanup_cancelled_load(resident, engine)
            raise asyncio.CancelledError() from exc
        if engine is not None:
            with suppress(Exception):
                await engine.stop()
        async with self._lock:
            resident.state = ResidentState.FAILED
            resident.last_error = str(exc)
            resident._abandoned_loading_task = None
            resident._loading_task = None
        raise

    try:
        async with self._lock:
            resident.engine = engine
            resident.state = ResidentState.LOADED
            resident.loaded_at = self._time_fn()
            resident.last_used_at = resident.loaded_at
            resident.last_error = None
            resident._abandoned_loading_task = None
            resident._loading_task = None
    except asyncio.CancelledError:
        await self._cleanup_cancelled_load(resident, engine)
        raise

    return engine

vllm_mlx.lifecycle.ResidencyManager._unload_engine async

_unload_engine(resident: ResidentModel) -> bool

Stop and drop a resident engine.

Source code in vllm_mlx/lifecycle.py

async def _unload_engine(self, resident: ResidentModel) -> bool:
    """Stop and drop a resident engine."""
    engine = resident.engine
    if engine is None:
        async with self._lock:
            resident.state = ResidentState.UNLOADED
            resident._unloading_task = None
        return False

    try:
        await self._run_hook(self._on_engine_unloading, resident.spec, engine)
        await engine.stop()
    except asyncio.CancelledError:
        async with self._lock:
            resident.state = ResidentState.LOADED
            resident._unloading_task = None
        raise
    except Exception as exc:
        async with self._lock:
            resident.engine = engine
            resident.state = ResidentState.LOADED
            resident.last_error = str(exc)
            resident._unloading_task = None
        return False

    async with self._lock:
        resident.engine = None
        resident.state = ResidentState.UNLOADED
        resident.loaded_at = None
        resident.last_error = None
        resident._unloading_task = None

    return True

vllm_mlx.lifecycle.ResidencyManager._resident

_resident(model_key: str) -> ResidentModel

Source code in vllm_mlx/lifecycle.py

def _resident(self, model_key: str) -> ResidentModel:
    try:
        return self._residents[model_key]
    except KeyError as exc:
        raise KeyError(f"Resident model '{model_key}' is not registered") from exc

vllm_mlx.lifecycle.ResidencyManager._run_hook async

_run_hook(hook: Callable[[ModelSpec, BaseEngine], Awaitable[None] | None] | None, spec: ModelSpec, engine: BaseEngine) -> None

Source code in vllm_mlx/lifecycle.py

async def _run_hook(
    self,
    hook: Callable[[ModelSpec, BaseEngine], Awaitable[None] | None] | None,
    spec: ModelSpec,
    engine: BaseEngine,
) -> None:
    if hook is None:
        return

    result = hook(spec, engine)
    if inspect.isawaitable(result):
        await result

vllm_mlx.lifecycle.ResidencyManager._prepare_engine_start async

_prepare_engine_start(resident: ResidentModel, engine: BaseEngine) -> None

Run blocking startup work away from the serving event loop.

Source code in vllm_mlx/lifecycle.py

async def _prepare_engine_start(
    self,
    resident: ResidentModel,
    engine: BaseEngine,
) -> None:
    """Run blocking startup work away from the serving event loop."""
    prepare_for_start = getattr(engine, "prepare_for_start", None)
    if prepare_for_start is None:
        return

    uses_default_prepare = getattr(engine, "_uses_default_prepare_for_start", None)
    if callable(uses_default_prepare) and uses_default_prepare():
        # Keep default engine prepare on the event-loop thread so MLX
        # thread-local stream ownership matches subsequent streaming calls.
        prepare_for_start()
        return

    prepare_task = asyncio.create_task(asyncio.to_thread(prepare_for_start))
    async with self._lock:
        resident._prepare_task = prepare_task

    try:
        await asyncio.shield(prepare_task)
    except asyncio.CancelledError:
        with suspend_cancellation():
            while not prepare_task.done():
                try:
                    await asyncio.shield(prepare_task)
                except asyncio.CancelledError:
                    continue
                except Exception:
                    break
        raise
    finally:
        async with self._lock:
            if resident._prepare_task is prepare_task:
                resident._prepare_task = None

vllm_mlx.lifecycle.ResidencyManager._cleanup_cancelled_load async

_cleanup_cancelled_load(resident: ResidentModel, engine: BaseEngine | None) -> None

Stop a partially loaded engine and unwind resident state.

Source code in vllm_mlx/lifecycle.py

async def _cleanup_cancelled_load(
    self,
    resident: ResidentModel,
    engine: BaseEngine | None,
) -> None:
    """Stop a partially loaded engine and unwind resident state."""
    with suspend_cancellation():
        if engine is not None:
            with suppress(Exception):
                await engine.stop()
        async with self._lock:
            resident.engine = None
            resident.state = ResidentState.UNLOADED
            resident.loaded_at = None
            resident.last_error = None
            # Keep the abandoned-load marker until a new load task replaces it
            # so late waiters on the old task can still recognize a retryable
            # cancellation instead of inheriting CancelledError.
            resident._loading_task = None

vllm_mlx.lifecycle.ResidencyManager._release_load_waiter async

_release_load_waiter(model_key: str, task: Task[BaseEngine]) -> None

Drop one waiter from a shared load, canceling abandoned solo loads.

Source code in vllm_mlx/lifecycle.py

async def _release_load_waiter(
    self,
    model_key: str,
    task: asyncio.Task[BaseEngine],
) -> None:
    """Drop one waiter from a shared load, canceling abandoned solo loads."""
    task_to_cancel: asyncio.Task[BaseEngine] | None = None

    async with self._lock:
        resident = self._resident(model_key)
        if resident._load_waiter_task is not task or resident._load_waiters <= 0:
            return

        resident._load_waiters -= 1
        if resident._load_waiters == 0:
            resident._load_waiter_task = None
            if resident._loading_task is task and not task.done():
                resident._abandoned_loading_task = task
                task_to_cancel = task

    if task_to_cancel is None:
        return

    with suspend_cancellation():
        task_to_cancel.cancel()
        with suppress(asyncio.CancelledError):
            await task_to_cancel

Complete contract reference

Expand any definition for its exact inputs, annotations, defaults, return contract, directly raised exceptions, source-grounded behavior, and immutable line link. This section includes private and nested definitions that ordinary API generators omit.

vllm_mlx.lifecycle.ResidentState · class

vllm_mlx.lifecycle.ResidentState()

Runtime residency state for a configured model.

Parameters

This callable has no explicit inputs.

Returns

• Constructs: vllm_mlx.lifecycle.ResidentState

Exceptions and behavior

Class ResidentState derives from str, Enum and declares 0 direct member(s). No direct raise statement appears in this definition.

View source #L17-L24.

vllm_mlx.lifecycle.ModelSpec · class

vllm_mlx.lifecycle.ModelSpec(model_key: str, model_name: str, use_batching: bool = False, scheduler_config: Any | None = None, stream_interval: int = 1, max_tokens: int = 32768, force_mllm: bool = False, mtp: bool = False, prefill_step_size: int = 2048, specprefill_enabled: bool = False, specprefill_threshold: int = 8192, specprefill_keep_pct: float = 0.3, specprefill_backbone_pct: float = 0.0, specprefill_draft_model: str | None = None)

Immutable engine construction inputs for a resident model.

Parameters

Name	Type	Required	Default	Description
model_key	str	yes	none	Required constructor field.
model_name	str	yes	none	Required constructor field.
use_batching	bool	no	False	Optional constructor field; defaults to False.
scheduler_config	Any \| None	no	None	Optional constructor field; defaults to None.
stream_interval	int	no	1	Optional constructor field; defaults to 1.
max_tokens	int	no	32768	Optional constructor field; defaults to 32768.
force_mllm	bool	no	False	Optional constructor field; defaults to False.
mtp	bool	no	False	Optional constructor field; defaults to False.
prefill_step_size	int	no	2048	Optional constructor field; defaults to 2048.
specprefill_enabled	bool	no	False	Optional constructor field; defaults to False.
specprefill_threshold	int	no	8192	Optional constructor field; defaults to 8192.
specprefill_keep_pct	float	no	0.3	Optional constructor field; defaults to 0.3.
specprefill_backbone_pct	float	no	0.0	Optional constructor field; defaults to 0.0.
specprefill_draft_model	str \| None	no	None	Optional constructor field; defaults to None.

Returns

• Constructs: vllm_mlx.lifecycle.ModelSpec

Exceptions and behavior

Class ModelSpec declares 0 direct member(s). No direct raise statement appears in this definition.

View source #L28-L44.

vllm_mlx.lifecycle.ResidentModel · class

vllm_mlx.lifecycle.ResidentModel(spec: ModelSpec, state: ResidentState = ResidentState.UNLOADED, engine: BaseEngine | None = None, active_requests: int = 0, last_used_at: float | None = None, loaded_at: float | None = None, last_error: str | None = None, estimated_memory_bytes: int | None = None, _load_waiters: int = field(default=0, repr=False), _load_waiter_task: asyncio.Task[BaseEngine] | None = field(default=None, repr=False), _prepare_task: asyncio.Task[None] | None = field(default=None, repr=False), _abandoned_loading_task: asyncio.Task[BaseEngine] | None = field(default=None, repr=False), _loading_task: asyncio.Task[BaseEngine] | None = field(default=None, repr=False), _unloading_task: asyncio.Task[bool] | None = field(default=None, repr=False))

Runtime state for a single resident model.

Parameters

Name	Type	Required	Default	Description
spec	ModelSpec	yes	none	Required constructor field.
state	ResidentState	no	ResidentState.UNLOADED	Optional constructor field; defaults to ResidentState.UNLOADED.
engine	BaseEngine \| None	no	None	Optional constructor field; defaults to None.
active_requests	int	no	0	Optional constructor field; defaults to 0.
last_used_at	float \| None	no	None	Optional constructor field; defaults to None.
loaded_at	float \| None	no	None	Optional constructor field; defaults to None.
last_error	str \| None	no	None	Optional constructor field; defaults to None.
estimated_memory_bytes	int \| None	no	None	Optional constructor field; defaults to None.
_load_waiters	int	no	field(default=0, repr=False)	Optional constructor field; defaults to field(default=0, repr=False).
_load_waiter_task	asyncio.Task[BaseEngine] \| None	no	field(default=None, repr=False)	Optional constructor field; defaults to field(default=None, repr=False).
_prepare_task	asyncio.Task[None] \| None	no	field(default=None, repr=False)	Optional constructor field; defaults to field(default=None, repr=False).
_abandoned_loading_task	asyncio.Task[BaseEngine] \| None	no	field(default=None, repr=False)	Optional constructor field; defaults to field(default=None, repr=False).
_loading_task	asyncio.Task[BaseEngine] \| None	no	field(default=None, repr=False)	Optional constructor field; defaults to field(default=None, repr=False).
_unloading_task	asyncio.Task[bool] \| None	no	field(default=None, repr=False)	Optional constructor field; defaults to field(default=None, repr=False).

Returns

• Constructs: vllm_mlx.lifecycle.ResidentModel

Exceptions and behavior

Class ResidentModel declares 0 direct member(s). No direct raise statement appears in this definition.

View source #L48-L66.

vllm_mlx.lifecycle.ResidencyManager · class

vllm_mlx.lifecycle.ResidencyManager(engine_factory: Callable[[ModelSpec], Awaitable[BaseEngine]], *, on_engine_loaded: Callable[[ModelSpec, BaseEngine], Awaitable[None] | None] | None = None, on_engine_unloading: Callable[[ModelSpec, BaseEngine], Awaitable[None] | None] | None = None, time_fn: Callable[[], float] | None = None, auto_unload_idle_seconds: float = 0)

Single-flight lifecycle manager for resident models.

Parameters

Name	Type	Required	Default	Description
engine_factory	Callable[[ModelSpec], Awaitable[BaseEngine]]	yes	none	Required positional or keyword input.
on_engine_loaded	Callable[[ModelSpec, BaseEngine], Awaitable[None] \| None] \| None	no	None	Optional keyword-only input; defaults to None.
on_engine_unloading	Callable[[ModelSpec, BaseEngine], Awaitable[None] \| None] \| None	no	None	Optional keyword-only input; defaults to None.
time_fn	Callable[[], float] \| None	no	None	Optional keyword-only input; defaults to None.
auto_unload_idle_seconds	float	no	0	Optional keyword-only input; defaults to 0.

Returns

• Constructs: vllm_mlx.lifecycle.ResidencyManager

Exceptions and behavior

Class ResidencyManager declares 16 direct member(s). No direct raise statement appears in this definition.

View source #L69-L493.

vllm_mlx.lifecycle.ResidencyManager.__init__ · method

vllm_mlx.lifecycle.ResidencyManager.__init__(engine_factory: Callable[[ModelSpec], Awaitable[BaseEngine]], *, on_engine_loaded: Callable[[ModelSpec, BaseEngine], Awaitable[None] | None] | None = None, on_engine_unloading: Callable[[ModelSpec, BaseEngine], Awaitable[None] | None] | None = None, time_fn: Callable[[], float] | None = None, auto_unload_idle_seconds: float = 0) -> None

Method ResidencyManager.__init__ updates self._engine_factory, self._on_engine_loaded, self._on_engine_unloading, self._time_fn; calls __import__, asyncio.Lock.

Parameters

Name	Type	Required	Default	Description
engine_factory	Callable[[ModelSpec], Awaitable[BaseEngine]]	yes	none	Required positional or keyword input.
on_engine_loaded	Callable[[ModelSpec, BaseEngine], Awaitable[None] \| None] \| None	no	None	Optional keyword-only input; defaults to None.
on_engine_unloading	Callable[[ModelSpec, BaseEngine], Awaitable[None] \| None] \| None	no	None	Optional keyword-only input; defaults to None.
time_fn	Callable[[], float] \| None	no	None	Optional keyword-only input; defaults to None.
auto_unload_idle_seconds	float	no	0	Optional keyword-only input; defaults to 0.

Returns

• Type: None

Exceptions and behavior

Method ResidencyManager.__init__ updates self._engine_factory, self._on_engine_loaded, self._on_engine_unloading, self._time_fn; calls __import__, asyncio.Lock. No direct raise statement appears in this definition.

View source #L72-L91.

vllm_mlx.lifecycle.ResidencyManager.register_model · method

vllm_mlx.lifecycle.ResidencyManager.register_model(spec: ModelSpec) -> str

Register a model spec, or replace a dormant resident entry.

Parameters

Name	Type	Required	Default	Description
spec	ModelSpec	yes	none	Required positional or keyword input.

Returns

• Type: str
• Direct return expressions: spec.model_key

Exceptions and behavior

Method ResidencyManager.register_model calls self._residents.get, RuntimeError, ResidentModel; can raise RuntimeError; returns spec.model_key. Directly raised exceptions: RuntimeError.

View source #L93-L111.

vllm_mlx.lifecycle.ResidencyManager.get_engine · method

vllm_mlx.lifecycle.ResidencyManager.get_engine(model_key: str) -> BaseEngine | None

Get the currently loaded engine, if any.

Parameters

Name	Type	Required	Default	Description
model_key	str	yes	none	Required positional or keyword input.

Returns

• Type: BaseEngine | None
• Direct return expressions: self._resident(model_key).engine

Exceptions and behavior

Method ResidencyManager.get_engine calls self._resident; returns self._resident(model_key).engine. No direct raise statement appears in this definition.

View source #L113-L115.

vllm_mlx.lifecycle.ResidencyManager.get_status · method

vllm_mlx.lifecycle.ResidencyManager.get_status(model_key: str) -> dict[str, Any]

Return a serializable snapshot of resident state.

Parameters

Name	Type	Required	Default	Description
model_key	str	yes	none	Required positional or keyword input.

Returns

• Type: dict[str, Any]
• Direct return expressions: {'model_key': resident.spec.model_key, 'model_name': resident.spec.model_name, 'state': resident.state.value, 'active_r…

Exceptions and behavior

Method ResidencyManager.get_status calls self._resident; returns {'model_key': resident.spec.model_key, 'model_name': resident.spec.model_name, 'state': resident.state.value, 'active_r…. No direct raise statement appears in this definition.

View source #L117-L130.

vllm_mlx.lifecycle.ResidencyManager.ensure_loaded · method

async vllm_mlx.lifecycle.ResidencyManager.ensure_loaded(model_key: str) -> BaseEngine

Load and start a resident engine if needed.

Parameters

Name	Type	Required	Default	Description
model_key	str	yes	none	Required positional or keyword input.

Returns

• Type: BaseEngine
• Direct return expressions: resident.engine; await asyncio.shield(task)

Exceptions and behavior

Method ResidencyManager.ensure_loaded calls self._resident, asyncio.create_task, self._load_engine, asyncio.shield; awaits asynchronous work; can raise RuntimeError; has 2 explicit return paths. Directly raised exceptions: RuntimeError.

View source #L132-L184.

vllm_mlx.lifecycle.ResidencyManager.acquire · method

async vllm_mlx.lifecycle.ResidencyManager.acquire(model_key: str, *, count_activity: bool = True) -> BaseEngine

Acquire a resident engine for request processing.

Parameters

Name	Type	Required	Default	Description
model_key	str	yes	none	Required positional or keyword input.
count_activity	bool	no	True	Optional keyword-only input; defaults to True.

Returns

• Type: BaseEngine
• Direct return expressions: engine

Exceptions and behavior

Method ResidencyManager.acquire calls self.ensure_loaded, self._resident, self._time_fn; awaits asynchronous work; returns engine. No direct raise statement appears in this definition.

View source #L186-L206.

vllm_mlx.lifecycle.ResidencyManager.release · method

async vllm_mlx.lifecycle.ResidencyManager.release(model_key: str, *, count_activity: bool = True) -> None

Release a previously acquired resident engine.

Parameters

Name	Type	Required	Default	Description
model_key	str	yes	none	Required positional or keyword input.
count_activity	bool	no	True	Optional keyword-only input; defaults to True.

Returns

• Type: None

Exceptions and behavior

Method ResidencyManager.release calls self._resident, self._time_fn. No direct raise statement appears in this definition.

View source #L208-L215.

vllm_mlx.lifecycle.ResidencyManager.unload_if_idle · method

async vllm_mlx.lifecycle.ResidencyManager.unload_if_idle(model_key: str) -> bool

Unload a resident engine if it has been idle past the threshold.

Parameters

Name	Type	Required	Default	Description
model_key	str	yes	none	Required positional or keyword input.

Returns

• Type: bool
• Direct return expressions: False; await asyncio.shield(unloading_task)

Exceptions and behavior

Method ResidencyManager.unload_if_idle calls self._resident, self._time_fn, asyncio.create_task, self._unload_engine; awaits asynchronous work; has 2 explicit return paths. No direct raise statement appears in this definition.

View source #L217-L253.

vllm_mlx.lifecycle.ResidencyManager.shutdown · method

async vllm_mlx.lifecycle.ResidencyManager.shutdown() -> None

Stop all loaded residents.

Parameters

This callable has no explicit inputs.

Returns

• Type: None

Exceptions and behavior

Method ResidencyManager.shutdown calls list, self._residents.keys, self._resident, resident._loading_task.cancel; awaits asynchronous work; can raise RuntimeError. Directly raised exceptions: RuntimeError.

View source #L255-L312.

vllm_mlx.lifecycle.ResidencyManager._load_engine · method

async vllm_mlx.lifecycle.ResidencyManager._load_engine(resident: ResidentModel) -> BaseEngine

Create and start a resident engine.

Parameters

Name	Type	Required	Default	Description
resident	ResidentModel	yes	none	Required positional or keyword input.

Returns

• Type: BaseEngine
• Direct return expressions: engine

Exceptions and behavior

Method ResidencyManager._load_engine calls self._engine_factory, self._prepare_engine_start, engine.start, self._run_hook; awaits asynchronous work; can raise asyncio.CancelledError; returns engine. Directly raised exceptions: asyncio.CancelledError.

View source #L314-L354.

vllm_mlx.lifecycle.ResidencyManager._unload_engine · method

async vllm_mlx.lifecycle.ResidencyManager._unload_engine(resident: ResidentModel) -> bool

Stop and drop a resident engine.

Parameters

Name	Type	Required	Default	Description
resident	ResidentModel	yes	none	Required positional or keyword input.

Returns

• Type: bool
• Direct return expressions: False; True

Exceptions and behavior

Method ResidencyManager._unload_engine calls self._run_hook, engine.stop, str; awaits asynchronous work; has 2 explicit return paths. No direct raise statement appears in this definition.

View source #L356-L388.

vllm_mlx.lifecycle.ResidencyManager._resident · method

vllm_mlx.lifecycle.ResidencyManager._resident(model_key: str) -> ResidentModel

Method ResidencyManager._resident calls KeyError; can raise KeyError; returns self._residents[model_key].

Parameters

Name	Type	Required	Default	Description
model_key	str	yes	none	Required positional or keyword input.

Returns

• Type: ResidentModel
• Direct return expressions: self._residents[model_key]

Exceptions and behavior

Method ResidencyManager._resident calls KeyError; can raise KeyError; returns self._residents[model_key]. Directly raised exceptions: KeyError.

View source #L390-L394.

vllm_mlx.lifecycle.ResidencyManager._run_hook · method

async vllm_mlx.lifecycle.ResidencyManager._run_hook(hook: Callable[[ModelSpec, BaseEngine], Awaitable[None] | None] | None, spec: ModelSpec, engine: BaseEngine) -> None

Method ResidencyManager._run_hook calls hook, inspect.isawaitable; awaits asynchronous work; returns None.

Parameters

Name	Type	Required	Default	Description
hook	Callable[[ModelSpec, BaseEngine], Awaitable[None] \| None] \| None	yes	none	Required positional or keyword input.
spec	ModelSpec	yes	none	Required positional or keyword input.
engine	BaseEngine	yes	none	Required positional or keyword input.

Returns

• Type: None
• Direct return expressions: None

Exceptions and behavior

Method ResidencyManager._run_hook calls hook, inspect.isawaitable; awaits asynchronous work; returns None. No direct raise statement appears in this definition.

View source #L396-L407.

vllm_mlx.lifecycle.ResidencyManager._prepare_engine_start · method

async vllm_mlx.lifecycle.ResidencyManager._prepare_engine_start(resident: ResidentModel, engine: BaseEngine) -> None

Run blocking startup work away from the serving event loop.

Parameters

Name	Type	Required	Default	Description
resident	ResidentModel	yes	none	Required positional or keyword input.
engine	BaseEngine	yes	none	Required positional or keyword input.

Returns

• Type: None
• Direct return expressions: None

Exceptions and behavior

Method ResidencyManager._prepare_engine_start calls getattr, callable, uses_default_prepare, prepare_for_start; awaits asynchronous work; returns None. No direct raise statement appears in this definition.

View source #L409-L445.

vllm_mlx.lifecycle.ResidencyManager._cleanup_cancelled_load · method

async vllm_mlx.lifecycle.ResidencyManager._cleanup_cancelled_load(resident: ResidentModel, engine: BaseEngine | None) -> None

Stop a partially loaded engine and unwind resident state.

Parameters

Name	Type	Required	Default	Description
resident	ResidentModel	yes	none	Required positional or keyword input.
engine	BaseEngine \| None	yes	none	Required positional or keyword input.

Returns

• Type: None

Exceptions and behavior

Method ResidencyManager._cleanup_cancelled_load calls suspend_cancellation, suppress, engine.stop; awaits asynchronous work. No direct raise statement appears in this definition.

View source #L447-L465.

vllm_mlx.lifecycle.ResidencyManager._release_load_waiter · method

async vllm_mlx.lifecycle.ResidencyManager._release_load_waiter(model_key: str, task: asyncio.Task[BaseEngine]) -> None

Drop one waiter from a shared load, canceling abandoned solo loads.

Parameters

Name	Type	Required	Default	Description
model_key	str	yes	none	Required positional or keyword input.
task	asyncio.Task[BaseEngine]	yes	none	Required positional or keyword input.

Returns

• Type: None
• Direct return expressions: None

Exceptions and behavior

Method ResidencyManager._release_load_waiter calls self._resident, task.done, suspend_cancellation, task_to_cancel.cancel; awaits asynchronous work; returns None. No direct raise statement appears in this definition.

View source #L467-L493.

Complete symbol map

This map also includes private definitions and nested helpers. The signature column exposes every explicit input even when an internal helper has no dedicated parameter prose.

Symbol	Kind	Signature and inputs	What it does	Source
ResidentState	class	ResidentState()	Runtime residency state for a configured model.	#L17-L24
ModelSpec	class	ModelSpec(model_key: str, model_name: str, use_batching: bool = False, scheduler_config: Any \| None = None, stream_interval: int = 1, max_tokens: int = 32768, force_mllm: bool = False, mtp: bool = False, prefill_step_size: int = 2048, specprefill_enabled: bool = False, specprefill_threshold: int = 8192, specprefill_keep_pct: float = 0.3, specprefill_backbone_pct: float = 0.0, specprefill_draft_model: str \| None = None)	Immutable engine construction inputs for a resident model.	#L28-L44
ResidentModel	class	ResidentModel(spec: ModelSpec, state: ResidentState = ResidentState.UNLOADED, engine: BaseEngine \| None = None, active_requests: int = 0, last_used_at: float \| None = None, loaded_at: float \| None = None, last_error: str \| None = None, estimated_memory_bytes: int \| None = None, _load_waiters: int = field(default=0, repr=False), _load_waiter_task: asyncio.Task[BaseEngine] \| None = field(default=None, repr=False), _prepare_task: asyncio.Task[None] \| None = field(default=None, repr=False), _abandoned_loading_task: asyncio.Task[BaseEngine] \| None = field(default=None, repr=False), _loading_task: asyncio.Task[BaseEngine] \| None = field(default=None, repr=False), _unloading_task: asyncio.Task[bool] \| None = field(default=None, repr=False))	Runtime state for a single resident model.	#L48-L66
ResidencyManager	class	ResidencyManager(engine_factory: Callable[[ModelSpec], Awaitable[BaseEngine]], *, on_engine_loaded: Callable[[ModelSpec, BaseEngine], Awaitable[None] \| None] \| None = None, on_engine_unloading: Callable[[ModelSpec, BaseEngine], Awaitable[None] \| None] \| None = None, time_fn: Callable[[], float] \| None = None, auto_unload_idle_seconds: float = 0)	Single-flight lifecycle manager for resident models.	#L69-L493
ResidencyManager.__init__	method	ResidencyManager.__init__(engine_factory: Callable[[ModelSpec], Awaitable[BaseEngine]], *, on_engine_loaded: Callable[[ModelSpec, BaseEngine], Awaitable[None] \| None] \| None = None, on_engine_unloading: Callable[[ModelSpec, BaseEngine], Awaitable[None] \| None] \| None = None, time_fn: Callable[[], float] \| None = None, auto_unload_idle_seconds: float = 0) -> None	Method ResidencyManager.__init__ updates self._engine_factory, self._on_engine_loaded, self._on_engine_unloading, self._time_fn; calls __import__, asyncio.Lock.	#L72-L91
ResidencyManager.register_model	method	ResidencyManager.register_model(spec: ModelSpec) -> str	Register a model spec, or replace a dormant resident entry.	#L93-L111
ResidencyManager.get_engine	method	ResidencyManager.get_engine(model_key: str) -> BaseEngine \| None	Get the currently loaded engine, if any.	#L113-L115
ResidencyManager.get_status	method	ResidencyManager.get_status(model_key: str) -> dict[str, Any]	Return a serializable snapshot of resident state.	#L117-L130
ResidencyManager.ensure_loaded	method	async ResidencyManager.ensure_loaded(model_key: str) -> BaseEngine	Load and start a resident engine if needed.	#L132-L184
ResidencyManager.acquire	method	async ResidencyManager.acquire(model_key: str, *, count_activity: bool = True) -> BaseEngine	Acquire a resident engine for request processing.	#L186-L206
ResidencyManager.release	method	async ResidencyManager.release(model_key: str, *, count_activity: bool = True) -> None	Release a previously acquired resident engine.	#L208-L215
ResidencyManager.unload_if_idle	method	async ResidencyManager.unload_if_idle(model_key: str) -> bool	Unload a resident engine if it has been idle past the threshold.	#L217-L253
ResidencyManager.shutdown	method	async ResidencyManager.shutdown() -> None	Stop all loaded residents.	#L255-L312
ResidencyManager._load_engine	method	async ResidencyManager._load_engine(resident: ResidentModel) -> BaseEngine	Create and start a resident engine.	#L314-L354
ResidencyManager._unload_engine	method	async ResidencyManager._unload_engine(resident: ResidentModel) -> bool	Stop and drop a resident engine.	#L356-L388
ResidencyManager._resident	method	ResidencyManager._resident(model_key: str) -> ResidentModel	Method ResidencyManager._resident calls KeyError; can raise KeyError; returns self._residents[model_key].	#L390-L394
ResidencyManager._run_hook	method	async ResidencyManager._run_hook(hook: Callable[[ModelSpec, BaseEngine], Awaitable[None] \| None] \| None, spec: ModelSpec, engine: BaseEngine) -> None	Method ResidencyManager._run_hook calls hook, inspect.isawaitable; awaits asynchronous work; returns None.	#L396-L407
ResidencyManager._prepare_engine_start	method	async ResidencyManager._prepare_engine_start(resident: ResidentModel, engine: BaseEngine) -> None	Run blocking startup work away from the serving event loop.	#L409-L445
ResidencyManager._cleanup_cancelled_load	method	async ResidencyManager._cleanup_cancelled_load(resident: ResidentModel, engine: BaseEngine \| None) -> None	Stop a partially loaded engine and unwind resident state.	#L447-L465
ResidencyManager._release_load_waiter	method	async ResidencyManager._release_load_waiter(model_key: str, task: asyncio.Task[BaseEngine]) -> None	Drop one waiter from a shared load, canceling abandoned solo loads.	#L467-L493