Skip to content

vllm_mlx.engine.base

Base engine interface for vllm-mlx inference.

View the complete module source at #L1-L288.

API details

Each callable below includes its exact signature, type annotations, inputs, defaults, return contract, documented exceptions, implementation source, and parsed docstring sections when the source provides them.

vllm_mlx.engine.base

Base engine interface for vllm-mlx inference.

vllm_mlx.engine.base.logger module-attribute 

logger = logging.getLogger(__name__)

vllm_mlx.engine.base.GenerationOutput dataclass 

GenerationOutput(text: str, tokens: list[int] = list(), prompt_tokens: int = 0, completion_tokens: int = 0, finish_reason: str | None = 'stop', mtp_drafts: int = 0, mtp_accepted: int = 0, new_text: str = '', finished: bool = True)

Output from generation.

Compatible with both simple and batched engines.

vllm_mlx.engine.base.GenerationOutput.text instance-attribute 

text: str

vllm_mlx.engine.base.GenerationOutput.tokens class-attribute instance-attribute 

tokens: list[int] = field(default_factory=list)

vllm_mlx.engine.base.GenerationOutput.prompt_tokens class-attribute instance-attribute 

prompt_tokens: int = 0

vllm_mlx.engine.base.GenerationOutput.completion_tokens class-attribute instance-attribute 

completion_tokens: int = 0

vllm_mlx.engine.base.GenerationOutput.finish_reason class-attribute instance-attribute 

finish_reason: str | None = 'stop'

vllm_mlx.engine.base.GenerationOutput.new_text class-attribute instance-attribute 

new_text: str = ''

vllm_mlx.engine.base.GenerationOutput.finished class-attribute instance-attribute 

finished: bool = True

vllm_mlx.engine.base.GenerationOutput.mtp_drafts class-attribute instance-attribute 

mtp_drafts: int = 0

vllm_mlx.engine.base.GenerationOutput.mtp_accepted class-attribute instance-attribute 

mtp_accepted: int = 0

vllm_mlx.engine.base.EngineBusy

Bases: RuntimeError

Raised when a serialized engine route is already serving a request.

vllm_mlx.engine.base.EngineBusy.code class-attribute instance-attribute 

code = 'text_generation_busy'

vllm_mlx.engine.base.BaseEngine

Bases: ABC

Abstract base class for inference engines.

Both SimpleEngine and BatchedEngine implement this interface, allowing the server to use either without code changes.

vllm_mlx.engine.base.BaseEngine.model_name abstractmethod property 

model_name: str

Get the model name.

vllm_mlx.engine.base.BaseEngine.is_mllm abstractmethod property 

is_mllm: bool

Check if this is a multimodal model.

vllm_mlx.engine.base.BaseEngine.tokenizer abstractmethod property 

tokenizer: Any

Get the tokenizer.

vllm_mlx.engine.base.BaseEngine.preserve_native_tool_format property writable 

preserve_native_tool_format: bool

Whether to preserve native tool message format.

When True, role="tool" messages and tool_calls fields are preserved instead of being converted to text. Set by server based on tool parser.

vllm_mlx.engine.base.BaseEngine.prepare_for_start 

prepare_for_start() -> None

Run blocking startup work before async engine start.

Engines can override this to perform heavyweight synchronous model loads off the serving event loop. The default implementation is a no-op so lightweight engines do not need extra plumbing.

Source code in vllm_mlx/engine/base.py 
143
144
145
146
147
148
149
150
def prepare_for_start(self) -> None:
    """Run blocking startup work before async engine start.

    Engines can override this to perform heavyweight synchronous model
    loads off the serving event loop. The default implementation is a
    no-op so lightweight engines do not need extra plumbing.
    """
    return None

vllm_mlx.engine.base.BaseEngine.start abstractmethod async 

start() -> None

Start the engine (load model if not loaded).

Source code in vllm_mlx/engine/base.py 
152
153
154
155
@abstractmethod
async def start(self) -> None:
    """Start the engine (load model if not loaded)."""
    pass

vllm_mlx.engine.base.BaseEngine.stop abstractmethod async 

stop() -> None

Stop the engine and cleanup resources.

Source code in vllm_mlx/engine/base.py 
157
158
159
160
@abstractmethod
async def stop(self) -> None:
    """Stop the engine and cleanup resources."""
    pass

vllm_mlx.engine.base.BaseEngine.generate abstractmethod async 

generate(prompt: str, max_tokens: int = 256, temperature: float = 0.7, top_p: float = 0.9, stop: list[str] | None = None, **kwargs) -> GenerationOutput

Generate a complete response (non-streaming).

Parameters:

  • prompt (str) –

    Input text

  • max_tokens (int, default: 256 ) –

    Maximum tokens to generate

  • temperature (float, default: 0.7 ) –

    Sampling temperature

  • top_p (float, default: 0.9 ) –

    Top-p sampling

  • stop (list[str] | None, default: None ) –

    Stop sequences

  • **kwargs –

    Additional model-specific parameters

Returns:

Source code in vllm_mlx/engine/base.py 
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
@abstractmethod
async def generate(
    self,
    prompt: str,
    max_tokens: int = 256,
    temperature: float = 0.7,
    top_p: float = 0.9,
    stop: list[str] | None = None,
    **kwargs,
) -> GenerationOutput:
    """
    Generate a complete response (non-streaming).

    Args:
        prompt: Input text
        max_tokens: Maximum tokens to generate
        temperature: Sampling temperature
        top_p: Top-p sampling
        stop: Stop sequences
        **kwargs: Additional model-specific parameters

    Returns:
        GenerationOutput with complete text
    """
    pass

vllm_mlx.engine.base.BaseEngine.stream_generate abstractmethod async 

stream_generate(prompt: str, max_tokens: int = 256, temperature: float = 0.7, top_p: float = 0.9, stop: list[str] | None = None, **kwargs) -> AsyncIterator[GenerationOutput]

Stream generation token by token.

Parameters:

  • prompt (str) –

    Input text

  • max_tokens (int, default: 256 ) –

    Maximum tokens to generate

  • temperature (float, default: 0.7 ) –

    Sampling temperature

  • top_p (float, default: 0.9 ) –

    Top-p sampling

  • stop (list[str] | None, default: None ) –

    Stop sequences

  • **kwargs –

    Additional model-specific parameters

Yields:

Source code in vllm_mlx/engine/base.py 
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
@abstractmethod
async def stream_generate(
    self,
    prompt: str,
    max_tokens: int = 256,
    temperature: float = 0.7,
    top_p: float = 0.9,
    stop: list[str] | None = None,
    **kwargs,
) -> AsyncIterator[GenerationOutput]:
    """
    Stream generation token by token.

    Args:
        prompt: Input text
        max_tokens: Maximum tokens to generate
        temperature: Sampling temperature
        top_p: Top-p sampling
        stop: Stop sequences
        **kwargs: Additional model-specific parameters

    Yields:
        GenerationOutput with incremental text
    """
    pass

vllm_mlx.engine.base.BaseEngine.chat abstractmethod async 

chat(messages: list[dict[str, Any]], max_tokens: int = 256, temperature: float = 0.7, top_p: float = 0.9, tools: list[dict] | None = None, images: list[str] | None = None, videos: list[str] | None = None, **kwargs) -> GenerationOutput

Chat completion (non-streaming).

Parameters:

  • messages (list[dict[str, Any]]) –

    List of chat messages

  • max_tokens (int, default: 256 ) –

    Maximum tokens to generate

  • temperature (float, default: 0.7 ) –

    Sampling temperature

  • top_p (float, default: 0.9 ) –

    Top-p sampling

  • tools (list[dict] | None, default: None ) –

    Optional tool definitions

  • images (list[str] | None, default: None ) –

    Optional image URLs/paths

  • videos (list[str] | None, default: None ) –

    Optional video URLs/paths

  • **kwargs –

    Additional model-specific parameters

Returns:

Source code in vllm_mlx/engine/base.py 
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
@abstractmethod
async def chat(
    self,
    messages: list[dict[str, Any]],
    max_tokens: int = 256,
    temperature: float = 0.7,
    top_p: float = 0.9,
    tools: list[dict] | None = None,
    images: list[str] | None = None,
    videos: list[str] | None = None,
    **kwargs,
) -> GenerationOutput:
    """
    Chat completion (non-streaming).

    Args:
        messages: List of chat messages
        max_tokens: Maximum tokens to generate
        temperature: Sampling temperature
        top_p: Top-p sampling
        tools: Optional tool definitions
        images: Optional image URLs/paths
        videos: Optional video URLs/paths
        **kwargs: Additional model-specific parameters

    Returns:
        GenerationOutput with assistant response
    """
    pass

vllm_mlx.engine.base.BaseEngine.stream_chat abstractmethod async 

stream_chat(messages: list[dict[str, Any]], max_tokens: int = 256, temperature: float = 0.7, top_p: float = 0.9, tools: list[dict] | None = None, images: list[str] | None = None, videos: list[str] | None = None, **kwargs) -> AsyncIterator[GenerationOutput]

Stream chat completion token by token.

Parameters:

  • messages (list[dict[str, Any]]) –

    List of chat messages

  • max_tokens (int, default: 256 ) –

    Maximum tokens to generate

  • temperature (float, default: 0.7 ) –

    Sampling temperature

  • top_p (float, default: 0.9 ) –

    Top-p sampling

  • tools (list[dict] | None, default: None ) –

    Optional tool definitions

  • images (list[str] | None, default: None ) –

    Optional image URLs/paths

  • videos (list[str] | None, default: None ) –

    Optional video URLs/paths

  • **kwargs –

    Additional model-specific parameters

Yields:

Source code in vllm_mlx/engine/base.py 
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
@abstractmethod
async def stream_chat(
    self,
    messages: list[dict[str, Any]],
    max_tokens: int = 256,
    temperature: float = 0.7,
    top_p: float = 0.9,
    tools: list[dict] | None = None,
    images: list[str] | None = None,
    videos: list[str] | None = None,
    **kwargs,
) -> AsyncIterator[GenerationOutput]:
    """
    Stream chat completion token by token.

    Args:
        messages: List of chat messages
        max_tokens: Maximum tokens to generate
        temperature: Sampling temperature
        top_p: Top-p sampling
        tools: Optional tool definitions
        images: Optional image URLs/paths
        videos: Optional video URLs/paths
        **kwargs: Additional model-specific parameters

    Yields:
        GenerationOutput with incremental text
    """
    pass

vllm_mlx.engine.base.BaseEngine.get_stats 

get_stats() -> dict[str, Any]

Get engine statistics. Override in subclasses.

Source code in vllm_mlx/engine/base.py 
274
275
276
def get_stats(self) -> dict[str, Any]:
    """Get engine statistics. Override in subclasses."""
    return {}

vllm_mlx.engine.base.BaseEngine.get_cache_stats 

get_cache_stats() -> dict[str, Any] | None

Get cache statistics. Override in subclasses.

Source code in vllm_mlx/engine/base.py 
278
279
280
def get_cache_stats(self) -> dict[str, Any] | None:
    """Get cache statistics. Override in subclasses."""
    return None

vllm_mlx.engine.base.BaseEngine.clear_runtime_caches 

clear_runtime_caches() -> dict[str, Any] | None

Clear engine-managed runtime caches. Override in subclasses.

Source code in vllm_mlx/engine/base.py 
282
283
284
def clear_runtime_caches(self) -> dict[str, Any] | None:
    """Clear engine-managed runtime caches. Override in subclasses."""
    return None

vllm_mlx.engine.base.BaseEngine.abort_request async 

abort_request(request_id: str) -> bool

Abort an active or queued request when the engine supports it.

Source code in vllm_mlx/engine/base.py 
286
287
288
async def abort_request(self, request_id: str) -> bool:
    """Abort an active or queued request when the engine supports it."""
    return False

vllm_mlx.engine.base.suspend_cancellation 

suspend_cancellation()

Temporarily clear task cancellation so cleanup can finish deterministically.

Source code in vllm_mlx/engine/base.py 
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
@contextmanager
def suspend_cancellation():
    """Temporarily clear task cancellation so cleanup can finish deterministically."""
    task = asyncio.current_task()
    if task is None:
        yield
        return

    cancelling = getattr(task, "cancelling", None)
    uncancel = getattr(task, "uncancel", None)
    if cancelling is None or uncancel is None:
        yield
        return

    pending_cancels = cancelling()
    for _ in range(pending_cancels):
        uncancel()
    try:
        yield
    finally:
        for _ in range(pending_cancels):
            task.cancel()

vllm_mlx.engine.base.run_blocking_startup_work async 

run_blocking_startup_work(work: Callable[[], Any]) -> None

Run blocking startup work off-loop without leaking cancellation races.

Source code in vllm_mlx/engine/base.py 
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
async def run_blocking_startup_work(work: Callable[[], Any]) -> None:
    """Run blocking startup work off-loop without leaking cancellation races."""
    task = asyncio.create_task(asyncio.to_thread(work))
    try:
        await asyncio.shield(task)
    except asyncio.CancelledError:
        with suspend_cancellation():
            while not task.done():
                try:
                    await asyncio.shield(task)
                except asyncio.CancelledError:
                    continue
                except Exception:
                    break
        raise

vllm_mlx.engine.base.cleanup_startup_cancellation async 

cleanup_startup_cancellation(cleanup: Callable[[], Awaitable[None]]) -> None

Run startup cleanup without letting cleanup failures replace cancellation.

Source code in vllm_mlx/engine/base.py 
87
88
89
90
91
92
93
94
95
96
97
98
async def cleanup_startup_cancellation(cleanup: Callable[[], Awaitable[None]]) -> None:
    """Run startup cleanup without letting cleanup failures replace cancellation."""
    with suspend_cancellation():
        try:
            await cleanup()
        except BaseException as exc:
            if isinstance(exc, (KeyboardInterrupt, SystemExit)):
                raise
            logger.error(
                "Engine startup cleanup failed while preserving cancellation",
                exc_info=(type(exc), exc, exc.__traceback__),
            )

Complete contract reference

Expand any definition for its exact inputs, annotations, defaults, return contract, directly raised exceptions, source-grounded behavior, and immutable line link. This section includes private and nested definitions that ordinary API generators omit.

vllm_mlx.engine.base.GenerationOutput · class 
vllm_mlx.engine.base.GenerationOutput(text: str, tokens: list[int] = field(default_factory=list), prompt_tokens: int = 0, completion_tokens: int = 0, finish_reason: str | None = 'stop', mtp_drafts: int = 0, mtp_accepted: int = 0, new_text: str = '', finished: bool = True, mtp_drafts: int = 0, mtp_accepted: int = 0)

Output from generation.

Parameters

Name Type Required Default Description
text str yes none Required constructor field.
tokens list[int] no field(default_factory=list) Optional constructor field; defaults to field(default_factory=list).
prompt_tokens int no 0 Optional constructor field; defaults to 0.
completion_tokens int no 0 Optional constructor field; defaults to 0.
finish_reason str \| None no 'stop' Optional constructor field; defaults to 'stop'.
mtp_drafts int no 0 Optional constructor field; defaults to 0.
mtp_accepted int no 0 Optional constructor field; defaults to 0.
new_text str no '' Optional constructor field; defaults to ''.
finished bool no True Optional constructor field; defaults to True.
mtp_drafts int no 0 Optional constructor field; defaults to 0.
mtp_accepted int no 0 Optional constructor field; defaults to 0.

Returns

  • Constructs: vllm_mlx.engine.base.GenerationOutput

Exceptions and behavior

Class GenerationOutput declares 0 direct member(s). No direct raise statement appears in this definition.

View source #L18-L37.

vllm_mlx.engine.base.EngineBusy · class 
vllm_mlx.engine.base.EngineBusy()

Raised when a serialized engine route is already serving a request.

Parameters

This callable has no explicit inputs.

Returns

  • Constructs: vllm_mlx.engine.base.EngineBusy

Exceptions and behavior

Class EngineBusy derives from RuntimeError and declares 0 direct member(s). No direct raise statement appears in this definition.

View source #L40-L43.

vllm_mlx.engine.base.suspend_cancellation · function 
vllm_mlx.engine.base.suspend_cancellation() -> not annotated

Temporarily clear task cancellation so cleanup can finish deterministically.

Parameters

This callable has no explicit inputs.

Returns

  • Type: not annotated
  • Direct return expressions: None
  • Yields values incrementally.

Exceptions and behavior

Function suspend_cancellation calls asyncio.current_task, getattr, cancelling, range; yields values incrementally; returns None. No direct raise statement appears in this definition.

View source #L47-L67.

vllm_mlx.engine.base.run_blocking_startup_work · function 
async vllm_mlx.engine.base.run_blocking_startup_work(work: Callable[[], Any]) -> None

Run blocking startup work off-loop without leaking cancellation races.

Parameters

Name Type Required Default Description
work Callable[[], Any] yes none Required positional or keyword input.

Returns

  • Type: None

Exceptions and behavior

Function run_blocking_startup_work calls asyncio.create_task, asyncio.to_thread, asyncio.shield, suspend_cancellation; awaits asynchronous work. No direct raise statement appears in this definition.

View source #L70-L84.

vllm_mlx.engine.base.cleanup_startup_cancellation · function 
async vllm_mlx.engine.base.cleanup_startup_cancellation(cleanup: Callable[[], Awaitable[None]]) -> None

Run startup cleanup without letting cleanup failures replace cancellation.

Parameters

Name Type Required Default Description
cleanup Callable[[], Awaitable[None]] yes none Required positional or keyword input.

Returns

  • Type: None

Exceptions and behavior

Function cleanup_startup_cancellation calls suspend_cancellation, cleanup, isinstance, logger.error; awaits asynchronous work. No direct raise statement appears in this definition.

View source #L87-L98.

vllm_mlx.engine.base.BaseEngine · class 
vllm_mlx.engine.base.BaseEngine()

Abstract base class for inference engines.

Parameters

This callable has no explicit inputs.

Returns

  • Constructs: vllm_mlx.engine.base.BaseEngine

Exceptions and behavior

Class BaseEngine derives from ABC and declares 16 direct member(s). No direct raise statement appears in this definition.

View source #L101-L288.

vllm_mlx.engine.base.BaseEngine.model_name · method 
vllm_mlx.engine.base.BaseEngine.model_name() -> str

Get the model name.

Parameters

This callable has no explicit inputs.

Returns

  • Type: str

Exceptions and behavior

Method BaseEngine.model_name contains no state mutation, call, raise, return, await, or yield. No direct raise statement appears in this definition.

View source #L111-L113.

vllm_mlx.engine.base.BaseEngine.is_mllm · method 
vllm_mlx.engine.base.BaseEngine.is_mllm() -> bool

Check if this is a multimodal model.

Parameters

This callable has no explicit inputs.

Returns

  • Type: bool

Exceptions and behavior

Method BaseEngine.is_mllm contains no state mutation, call, raise, return, await, or yield. No direct raise statement appears in this definition.

View source #L117-L119.

vllm_mlx.engine.base.BaseEngine.tokenizer · method 
vllm_mlx.engine.base.BaseEngine.tokenizer() -> Any

Get the tokenizer.

Parameters

This callable has no explicit inputs.

Returns

  • Type: Any

Exceptions and behavior

Method BaseEngine.tokenizer contains no state mutation, call, raise, return, await, or yield. No direct raise statement appears in this definition.

View source #L123-L125.

vllm_mlx.engine.base.BaseEngine.preserve_native_tool_format · method 
vllm_mlx.engine.base.BaseEngine.preserve_native_tool_format() -> bool

Whether to preserve native tool message format.

Parameters

This callable has no explicit inputs.

Returns

  • Type: bool
  • Direct return expressions: getattr(self, '_preserve_native_tool_format', False)

Exceptions and behavior

Method BaseEngine.preserve_native_tool_format calls getattr; returns getattr(self, '_preserve_native_tool_format', False). No direct raise statement appears in this definition.

View source #L128-L135.

vllm_mlx.engine.base.BaseEngine.preserve_native_tool_format · method 
vllm_mlx.engine.base.BaseEngine.preserve_native_tool_format(value: bool) -> None

Enable or disable preservation of model-native tool messages.

Parameters

Name Type Required Default Description
value bool yes none Required positional or keyword input.

Returns

  • Type: None

Exceptions and behavior

Method BaseEngine.preserve_native_tool_format updates self._preserve_native_tool_format. No direct raise statement appears in this definition.

View source #L138-L141.

vllm_mlx.engine.base.BaseEngine.prepare_for_start · method 
vllm_mlx.engine.base.BaseEngine.prepare_for_start() -> None

Run blocking startup work before async engine start.

Parameters

This callable has no explicit inputs.

Returns

  • Type: None
  • Direct return expressions: None

Exceptions and behavior

Method BaseEngine.prepare_for_start returns None. No direct raise statement appears in this definition.

View source #L143-L150.

vllm_mlx.engine.base.BaseEngine.start · method 
async vllm_mlx.engine.base.BaseEngine.start() -> None

Start the engine (load model if not loaded).

Parameters

This callable has no explicit inputs.

Returns

  • Type: None

Exceptions and behavior

Method BaseEngine.start contains no state mutation, call, raise, return, await, or yield. No direct raise statement appears in this definition.

View source #L153-L155.

vllm_mlx.engine.base.BaseEngine.stop · method 
async vllm_mlx.engine.base.BaseEngine.stop() -> None

Stop the engine and cleanup resources.

Parameters

This callable has no explicit inputs.

Returns

  • Type: None

Exceptions and behavior

Method BaseEngine.stop contains no state mutation, call, raise, return, await, or yield. No direct raise statement appears in this definition.

View source #L158-L160.

vllm_mlx.engine.base.BaseEngine.generate · method 
async vllm_mlx.engine.base.BaseEngine.generate(prompt: str, max_tokens: int = 256, temperature: float = 0.7, top_p: float = 0.9, stop: list[str] | None = None, **kwargs) -> GenerationOutput

Generate a complete response (non-streaming).

Parameters

Name Type Required Default Description
prompt str yes none Input text
max_tokens int no 256 Maximum tokens to generate
temperature float no 0.7 Sampling temperature
top_p float no 0.9 Top-p sampling
stop list[str] \| None no None Stop sequences
**kwargs not annotated no none Additional model-specific parameters

Returns

  • Type: GenerationOutput

Exceptions and behavior

Method BaseEngine.generate contains no state mutation, call, raise, return, await, or yield. No direct raise statement appears in this definition.

View source #L163-L186.

vllm_mlx.engine.base.BaseEngine.stream_generate · method 
async vllm_mlx.engine.base.BaseEngine.stream_generate(prompt: str, max_tokens: int = 256, temperature: float = 0.7, top_p: float = 0.9, stop: list[str] | None = None, **kwargs) -> AsyncIterator[GenerationOutput]

Stream generation token by token.

Parameters

Name Type Required Default Description
prompt str yes none Input text
max_tokens int no 256 Maximum tokens to generate
temperature float no 0.7 Sampling temperature
top_p float no 0.9 Top-p sampling
stop list[str] \| None no None Stop sequences
**kwargs not annotated no none Additional model-specific parameters

Returns

  • Type: AsyncIterator[GenerationOutput]

Exceptions and behavior

Method BaseEngine.stream_generate contains no state mutation, call, raise, return, await, or yield. No direct raise statement appears in this definition.

View source #L189-L212.

vllm_mlx.engine.base.BaseEngine.chat · method 
async vllm_mlx.engine.base.BaseEngine.chat(messages: list[dict[str, Any]], max_tokens: int = 256, temperature: float = 0.7, top_p: float = 0.9, tools: list[dict] | None = None, images: list[str] | None = None, videos: list[str] | None = None, **kwargs) -> GenerationOutput

Chat completion (non-streaming).

Parameters

Name Type Required Default Description
messages list[dict[str, Any]] yes none List of chat messages
max_tokens int no 256 Maximum tokens to generate
temperature float no 0.7 Sampling temperature
top_p float no 0.9 Top-p sampling
tools list[dict] \| None no None Optional tool definitions
images list[str] \| None no None Optional image URLs/paths
videos list[str] \| None no None Optional video URLs/paths
**kwargs not annotated no none Additional model-specific parameters

Returns

  • Type: GenerationOutput

Exceptions and behavior

Method BaseEngine.chat contains no state mutation, call, raise, return, await, or yield. No direct raise statement appears in this definition.

View source #L215-L242.

vllm_mlx.engine.base.BaseEngine.stream_chat · method 
async vllm_mlx.engine.base.BaseEngine.stream_chat(messages: list[dict[str, Any]], max_tokens: int = 256, temperature: float = 0.7, top_p: float = 0.9, tools: list[dict] | None = None, images: list[str] | None = None, videos: list[str] | None = None, **kwargs) -> AsyncIterator[GenerationOutput]

Stream chat completion token by token.

Parameters

Name Type Required Default Description
messages list[dict[str, Any]] yes none List of chat messages
max_tokens int no 256 Maximum tokens to generate
temperature float no 0.7 Sampling temperature
top_p float no 0.9 Top-p sampling
tools list[dict] \| None no None Optional tool definitions
images list[str] \| None no None Optional image URLs/paths
videos list[str] \| None no None Optional video URLs/paths
**kwargs not annotated no none Additional model-specific parameters

Returns

  • Type: AsyncIterator[GenerationOutput]

Exceptions and behavior

Method BaseEngine.stream_chat contains no state mutation, call, raise, return, await, or yield. No direct raise statement appears in this definition.

View source #L245-L272.

vllm_mlx.engine.base.BaseEngine.get_stats · method 
vllm_mlx.engine.base.BaseEngine.get_stats() -> dict[str, Any]

Get engine statistics.

Parameters

This callable has no explicit inputs.

Returns

  • Type: dict[str, Any]
  • Direct return expressions: {}

Exceptions and behavior

Method BaseEngine.get_stats returns {}. No direct raise statement appears in this definition.

View source #L274-L276.

vllm_mlx.engine.base.BaseEngine.get_cache_stats · method 
vllm_mlx.engine.base.BaseEngine.get_cache_stats() -> dict[str, Any] | None

Get cache statistics.

Parameters

This callable has no explicit inputs.

Returns

  • Type: dict[str, Any] | None
  • Direct return expressions: None

Exceptions and behavior

Method BaseEngine.get_cache_stats returns None. No direct raise statement appears in this definition.

View source #L278-L280.

vllm_mlx.engine.base.BaseEngine.clear_runtime_caches · method 
vllm_mlx.engine.base.BaseEngine.clear_runtime_caches() -> dict[str, Any] | None

Clear engine-managed runtime caches.

Parameters

This callable has no explicit inputs.

Returns

  • Type: dict[str, Any] | None
  • Direct return expressions: None

Exceptions and behavior

Method BaseEngine.clear_runtime_caches returns None. No direct raise statement appears in this definition.

View source #L282-L284.

vllm_mlx.engine.base.BaseEngine.abort_request · method 
async vllm_mlx.engine.base.BaseEngine.abort_request(request_id: str) -> bool

Abort an active or queued request when the engine supports it.

Parameters

Name Type Required Default Description
request_id str yes none Required positional or keyword input.

Returns

  • Type: bool
  • Direct return expressions: False

Exceptions and behavior

Method BaseEngine.abort_request returns False. No direct raise statement appears in this definition.

View source #L286-L288.

Complete symbol map

This map also includes private definitions and nested helpers. The signature column exposes every explicit input even when an internal helper has no dedicated parameter prose.

Symbol Kind Signature and inputs What it does Source
GenerationOutput class GenerationOutput(text: str, tokens: list[int] = field(default_factory=list), prompt_tokens: int = 0, completion_tokens: int = 0, finish_reason: str \| None = 'stop', mtp_drafts: int = 0, mtp_accepted: int = 0, new_text: str = '', finished: bool = True, mtp_drafts: int = 0, mtp_accepted: int = 0) Output from generation. #L18-L37
EngineBusy class EngineBusy() Raised when a serialized engine route is already serving a request. #L40-L43
suspend_cancellation function suspend_cancellation() -> not annotated Temporarily clear task cancellation so cleanup can finish deterministically. #L47-L67
run_blocking_startup_work function async run_blocking_startup_work(work: Callable[[], Any]) -> None Run blocking startup work off-loop without leaking cancellation races. #L70-L84
cleanup_startup_cancellation function async cleanup_startup_cancellation(cleanup: Callable[[], Awaitable[None]]) -> None Run startup cleanup without letting cleanup failures replace cancellation. #L87-L98
BaseEngine class BaseEngine() Abstract base class for inference engines. #L101-L288
BaseEngine.model_name method BaseEngine.model_name() -> str Get the model name. #L111-L113
BaseEngine.is_mllm method BaseEngine.is_mllm() -> bool Check if this is a multimodal model. #L117-L119
BaseEngine.tokenizer method BaseEngine.tokenizer() -> Any Get the tokenizer. #L123-L125
BaseEngine.preserve_native_tool_format method BaseEngine.preserve_native_tool_format() -> bool Whether to preserve native tool message format. #L128-L135
BaseEngine.preserve_native_tool_format method BaseEngine.preserve_native_tool_format(value: bool) -> None Enable or disable preservation of model-native tool messages. #L138-L141
BaseEngine.prepare_for_start method BaseEngine.prepare_for_start() -> None Run blocking startup work before async engine start. #L143-L150
BaseEngine.start method async BaseEngine.start() -> None Start the engine (load model if not loaded). #L153-L155
BaseEngine.stop method async BaseEngine.stop() -> None Stop the engine and cleanup resources. #L158-L160
BaseEngine.generate method async BaseEngine.generate(prompt: str, max_tokens: int = 256, temperature: float = 0.7, top_p: float = 0.9, stop: list[str] \| None = None, **kwargs) -> GenerationOutput Generate a complete response (non-streaming). #L163-L186
BaseEngine.stream_generate method async BaseEngine.stream_generate(prompt: str, max_tokens: int = 256, temperature: float = 0.7, top_p: float = 0.9, stop: list[str] \| None = None, **kwargs) -> AsyncIterator[GenerationOutput] Stream generation token by token. #L189-L212
BaseEngine.chat method async BaseEngine.chat(messages: list[dict[str, Any]], max_tokens: int = 256, temperature: float = 0.7, top_p: float = 0.9, tools: list[dict] \| None = None, images: list[str] \| None = None, videos: list[str] \| None = None, **kwargs) -> GenerationOutput Chat completion (non-streaming). #L215-L242
BaseEngine.stream_chat method async BaseEngine.stream_chat(messages: list[dict[str, Any]], max_tokens: int = 256, temperature: float = 0.7, top_p: float = 0.9, tools: list[dict] \| None = None, images: list[str] \| None = None, videos: list[str] \| None = None, **kwargs) -> AsyncIterator[GenerationOutput] Stream chat completion token by token. #L245-L272
BaseEngine.get_stats method BaseEngine.get_stats() -> dict[str, Any] Get engine statistics. #L274-L276
BaseEngine.get_cache_stats method BaseEngine.get_cache_stats() -> dict[str, Any] \| None Get cache statistics. #L278-L280
BaseEngine.clear_runtime_caches method BaseEngine.clear_runtime_caches() -> dict[str, Any] \| None Clear engine-managed runtime caches. #L282-L284
BaseEngine.abort_request method async BaseEngine.abort_request(request_id: str) -> bool Abort an active or queued request when the engine supports it. #L286-L288