2003 symbols
| Name | Kind | Signature and inputs | What it does | Source |
|---|---|---|---|---|
vllm_mlx.__getattr__ |
function | vllm_mlx.__getattr__(name) -> not annotated |
Lazy load all components to avoid mlx_lm import on non-Apple platforms. | #L21-L90 |
vllm_mlx.api.anthropic_adapter._convert_message |
function | vllm_mlx.api.anthropic_adapter._convert_message(msg: AnthropicMessage) -> list[Message] |
Convert an Anthropic message to one or more OpenAI messages. | #L165-L261 |
vllm_mlx.api.anthropic_adapter._convert_stop_reason |
function | vllm_mlx.api.anthropic_adapter._convert_stop_reason(openai_reason: str | None) -> str |
Convert OpenAI finish_reason to Anthropic stop_reason. | #L305-L321 |
vllm_mlx.api.anthropic_adapter._convert_tool |
function | vllm_mlx.api.anthropic_adapter._convert_tool(tool: AnthropicToolDef) -> ToolDefinition |
Convert an Anthropic tool definition to OpenAI format. | #L264-L278 |
vllm_mlx.api.anthropic_adapter._convert_tool_choice |
function | vllm_mlx.api.anthropic_adapter._convert_tool_choice(tool_choice: dict) -> str | dict | None |
Convert Anthropic tool_choice to OpenAI format. | #L281-L302 |
vllm_mlx.api.anthropic_adapter.anthropic_to_openai |
function | vllm_mlx.api.anthropic_adapter.anthropic_to_openai(request: AnthropicRequest) -> ChatCompletionRequest |
Convert an Anthropic Messages API request to OpenAI Chat Completions format. | #L31-L99 |
vllm_mlx.api.anthropic_adapter.openai_to_anthropic |
function | vllm_mlx.api.anthropic_adapter.openai_to_anthropic(response: ChatCompletionResponse, model: str) -> AnthropicResponse |
Convert an OpenAI Chat Completions response to Anthropic Messages API format. | #L102-L162 |
vllm_mlx.api.anthropic_models.AnthropicContentBlock |
class | vllm_mlx.api.anthropic_models.AnthropicContentBlock(type: str, text: str | None = None, id: str | None = None, name: str | None = None, input: dict | None = None, tool_use_id: str | None = None, content: str | list | None = None, is_error: bool | None = None, source: dict | None = None) |
A content block in an Anthropic message. | #L20-L35 |
vllm_mlx.api.anthropic_models.AnthropicMessage |
class | vllm_mlx.api.anthropic_models.AnthropicMessage(role: str, content: str | list[AnthropicContentBlock]) |
A message in an Anthropic conversation. | #L38-L42 |
vllm_mlx.api.anthropic_models.AnthropicRequest |
class | vllm_mlx.api.anthropic_models.AnthropicRequest(model: str, messages: list[AnthropicMessage], system: str | list[dict] | None = None, max_tokens: int = Field(gt=0), temperature: float | None = None, top_p: float | None = None, stream: bool = False, stop_sequences: list[str] | None = None, tools: list[AnthropicToolDef] | None = None, tool_choice: dict | None = None, metadata: dict | None = None, top_k: int | None = None, response_format: dict | None = None, chat_template_kwargs: dict[str, Any] | None = None) |
Request for Anthropic Messages API. | #L53-L73 |
vllm_mlx.api.anthropic_models.AnthropicResponse |
class | vllm_mlx.api.anthropic_models.AnthropicResponse(id: str = Field(default_factory=lambda: f'msg_{uuid.uuid4().hex[:24]}'), type: str = 'message', role: str = 'assistant', model: str, content: list[AnthropicResponseContentBlock], stop_reason: str | None = None, stop_sequence: str | None = None, usage: AnthropicUsage = Field(default_factory=AnthropicUsage)) |
Response for Anthropic Messages API. | #L103-L113 |
vllm_mlx.api.anthropic_models.AnthropicResponseContentBlock |
class | vllm_mlx.api.anthropic_models.AnthropicResponseContentBlock(type: str, text: str | None = None, thinking: str | None = None, id: str | None = None, name: str | None = None, input: Any | None = None) |
A content block in the Anthropic response. | #L90-L100 |
vllm_mlx.api.anthropic_models.AnthropicToolDef |
class | vllm_mlx.api.anthropic_models.AnthropicToolDef(name: str, description: str | None = None, input_schema: dict | None = None) |
Definition of a tool in Anthropic format. | #L45-L50 |
vllm_mlx.api.anthropic_models.AnthropicUsage |
class | vllm_mlx.api.anthropic_models.AnthropicUsage(input_tokens: int = 0, output_tokens: int = 0, cache_creation_input_tokens: int | None = None, cache_read_input_tokens: int | None = None) |
Token usage for Anthropic response. | #L81-L87 |
vllm_mlx.api.harmony_tools._convert_type |
function | vllm_mlx.api.harmony_tools._convert_type(prop: dict[str, Any]) -> str |
Convert a JSON Schema property to a TypeScript type string. | #L31-L54 |
vllm_mlx.api.harmony_tools.convert_tools_to_typescript |
function | vllm_mlx.api.harmony_tools.convert_tools_to_typescript(tools: list[dict[str, Any]] | None) -> str | None |
Convert OpenAI JSON Schema tool definitions to TypeScript namespace format. | #L57-L109 |
vllm_mlx.api.models.AssistantMessage |
class | vllm_mlx.api.models.AssistantMessage(role: str = 'assistant', content: str | None = None, reasoning_content: str | None = Field(default=None, validation_alias=AliasChoices('reasoning_content', 'reasoning')), tool_calls: list[ToolCall] | None = None) |
Response message from the assistant. | #L219-L248 |
vllm_mlx.api.models.AssistantMessage._serialize |
method | vllm_mlx.api.models.AssistantMessage._serialize() -> dict |
Serialize with OpenAI-compatible schema. | #L237-L248 |
vllm_mlx.api.models.AssistantMessage.reasoning |
method | vllm_mlx.api.models.AssistantMessage.reasoning() -> str | None |
Return reasoning content through the legacy compatibility alias. | #L231-L234 |
vllm_mlx.api.models.AudioSeparationRequest |
class | vllm_mlx.api.models.AudioSeparationRequest(model: str = 'htdemucs', stems: list[str] = Field(default_factory=lambda: ['vocals', 'accompaniment'])) |
Request for audio source separation. | #L443-L447 |
vllm_mlx.api.models.AudioSpeechRequest |
class | vllm_mlx.api.models.AudioSpeechRequest(model: str = 'kokoro', input: str, voice: str = 'af_heart', speed: float = 1.0, response_format: str = 'wav') |
Request for text-to-speech. | #L433-L440 |
vllm_mlx.api.models.AudioTranscriptionRequest |
class | vllm_mlx.api.models.AudioTranscriptionRequest(model: str = 'whisper-large-v3', language: str | None = None, response_format: str = 'json', temperature: float = 0.0, timestamp_granularities: list[str] | None = None) |
Request for audio transcription (STT). | #L414-L421 |
vllm_mlx.api.models.AudioTranscriptionResponse |
class | vllm_mlx.api.models.AudioTranscriptionResponse(text: str, language: str | None = None, duration: float | None = None, segments: list[dict] | None = None) |
Response from audio transcription. | #L424-L430 |
vllm_mlx.api.models.AudioUrl |
class | vllm_mlx.api.models.AudioUrl(url: str) |
Audio URL for audio content. | #L37-L40 |
vllm_mlx.api.models.ChatCompletionChoice |
class | vllm_mlx.api.models.ChatCompletionChoice(index: int = 0, message: AssistantMessage, finish_reason: str | None = 'stop') |
A single choice in chat completion response. | #L251-L256 |
vllm_mlx.api.models.ChatCompletionChunk |
class | vllm_mlx.api.models.ChatCompletionChunk(id: str = Field(default_factory=lambda: f'chatcmpl-{uuid.uuid4().hex[:8]}'), object: str = 'chat.completion.chunk', created: int = Field(default_factory=lambda: int(time.time())), model: str, choices: list[ChatCompletionChunkChoice], usage: Usage | None = None) |
A streaming chunk for chat completion. | #L573-L581 |
vllm_mlx.api.models.ChatCompletionChunkChoice |
class | vllm_mlx.api.models.ChatCompletionChunkChoice(index: int = 0, delta: ChatCompletionChunkDelta, finish_reason: str | None = None) |
A single choice in a streaming chunk. | #L565-L570 |
vllm_mlx.api.models.ChatCompletionChunkDelta |
class | vllm_mlx.api.models.ChatCompletionChunkDelta(role: str | None = None, content: str | None = None, reasoning_content: str | None = Field(default=None, validation_alias=AliasChoices('reasoning_content', 'reasoning')), tool_calls: list[dict] | None = None) |
Delta content in a streaming chunk. | #L529-L562 |
vllm_mlx.api.models.ChatCompletionChunkDelta._serialize |
method | vllm_mlx.api.models.ChatCompletionChunkDelta._serialize() -> dict |
Serialize delta with only non-None fields. | #L547-L562 |
vllm_mlx.api.models.ChatCompletionChunkDelta.reasoning |
method | vllm_mlx.api.models.ChatCompletionChunkDelta.reasoning() -> str | None |
Return incremental reasoning through the compatibility alias. | #L541-L544 |
vllm_mlx.api.models.ChatCompletionRequest |
class | vllm_mlx.api.models.ChatCompletionRequest(model: str, messages: list[Message], temperature: float | None = None, top_p: float | None = None, top_k: int | None = None, min_p: float | None = None, presence_penalty: float | None = None, max_tokens: int | None = Field(default=None, gt=0), stream: bool = False, stream_options: StreamOptions | None = None, stop: list[str] | None = None, tools: list[ToolDefinition] | None = None, tool_choice: str | dict | None = None, response_format: ResponseFormat | dict | None = None, logit_bias: dict[str, float] | None = None, chat_template_kwargs: dict[str, Any] | None = None, video_fps: float | None = None, video_max_frames: int | None = None, repetition_penalty: float | None = None, timeout: float | None = None, specprefill: bool | None = None, specprefill_keep_pct: float | None = None, specprefill_backbone_pct: float | None = None, enable_thinking: bool | None = None, mllm_draft: bool | None = None, thinking_token_budget: int | None = Field(default=None, gt=0)) |
Request for chat completion. | #L170-L216 |
vllm_mlx.api.models.ChatCompletionResponse |
class | vllm_mlx.api.models.ChatCompletionResponse(id: str = Field(default_factory=lambda: f'chatcmpl-{uuid.uuid4().hex[:8]}'), object: str = 'chat.completion', created: int = Field(default_factory=lambda: int(time.time())), model: str, choices: list[ChatCompletionChoice], usage: Usage = Field(default_factory=Usage), generation_metadata: GenerationMetadata | None = None) |
Response for chat completion. | #L274-L283 |
vllm_mlx.api.models.CompletionChoice |
class | vllm_mlx.api.models.CompletionChoice(index: int = 0, text: str, finish_reason: str | None = 'stop') |
A single choice in text completion response. | #L316-L321 |
vllm_mlx.api.models.CompletionRequest |
class | vllm_mlx.api.models.CompletionRequest(model: str, prompt: str | list[str], temperature: float | None = None, top_p: float | None = None, top_k: int | None = None, min_p: float | None = None, presence_penalty: float | None = None, max_tokens: int | None = Field(default=None, gt=0), stream: bool = False, stop: list[str] | None = None, repetition_penalty: float | None = None, timeout: float | None = None, specprefill: bool | None = None, specprefill_keep_pct: float | None = None, specprefill_backbone_pct: float | None = None) |
Request for text completion. | #L291-L313 |
vllm_mlx.api.models.CompletionResponse |
class | vllm_mlx.api.models.CompletionResponse(id: str = Field(default_factory=lambda: f'cmpl-{uuid.uuid4().hex[:8]}'), object: str = 'text_completion', created: int = Field(default_factory=lambda: int(time.time())), model: str, choices: list[CompletionChoice], usage: Usage = Field(default_factory=Usage)) |
Response for text completion. | #L324-L332 |
vllm_mlx.api.models.ContentPart |
class | vllm_mlx.api.models.ContentPart(type: str, text: str | None = None, image_url: ImageUrl | dict | str | None = None, video: str | None = None, video_url: VideoUrl | dict | str | None = None, audio_url: AudioUrl | dict | str | None = None) |
A part of a multimodal message content. | #L43-L60 |
vllm_mlx.api.models.EmbeddingData |
class | vllm_mlx.api.models.EmbeddingData(object: str = 'embedding', index: int, embedding: list[float]) |
A single embedding result. | #L463-L468 |
vllm_mlx.api.models.EmbeddingRequest |
class | vllm_mlx.api.models.EmbeddingRequest(input: str | list[str], model: str, encoding_format: str | None = 'float') |
Request for text embeddings (OpenAI compatible). | #L455-L460 |
vllm_mlx.api.models.EmbeddingResponse |
class | vllm_mlx.api.models.EmbeddingResponse(object: str = 'list', data: list[EmbeddingData], model: str, usage: EmbeddingUsage = Field(default_factory=EmbeddingUsage)) |
Response for embeddings endpoint (OpenAI compatible). | #L478-L484 |
vllm_mlx.api.models.EmbeddingUsage |
class | vllm_mlx.api.models.EmbeddingUsage(prompt_tokens: int = 0, total_tokens: int = 0) |
Token usage for embedding requests. | #L471-L475 |
vllm_mlx.api.models.FunctionCall |
class | vllm_mlx.api.models.FunctionCall(name: str, arguments: str) |
A function call with name and arguments. | #L95-L99 |
vllm_mlx.api.models.GenerationMetadata |
class | vllm_mlx.api.models.GenerationMetadata(no_final_content_watchdog_tokens: int | None = None, no_final_content_watchdog_enforced: bool = False) |
Optional generation diagnostics emitted for feature-bearing requests. | #L267-L271 |
vllm_mlx.api.models.ImageUrl |
class | vllm_mlx.api.models.ImageUrl(url: str, detail: str | None = None) |
Image URL with optional detail level. | #L24-L28 |
vllm_mlx.api.models.MCPExecuteRequest |
class | vllm_mlx.api.models.MCPExecuteRequest(tool_name: str, arguments: dict = Field(default_factory=dict)) |
Request to execute an MCP tool. | #L393-L397 |
vllm_mlx.api.models.MCPExecuteResponse |
class | vllm_mlx.api.models.MCPExecuteResponse(tool_name: str, content: str | list | dict | None = None, is_error: bool = False, error_message: str | None = None) |
Response from executing an MCP tool. | #L400-L406 |
vllm_mlx.api.models.MCPServerInfo |
class | vllm_mlx.api.models.MCPServerInfo(name: str, state: str, transport: str, tools_count: int, error: str | None = None) |
Information about an MCP server. | #L377-L384 |
vllm_mlx.api.models.MCPServersResponse |
class | vllm_mlx.api.models.MCPServersResponse(servers: list[MCPServerInfo]) |
Response for listing MCP servers. | #L387-L390 |
vllm_mlx.api.models.MCPToolInfo |
class | vllm_mlx.api.models.MCPToolInfo(name: str, description: str, server: str, parameters: dict = Field(default_factory=dict)) |
Information about an MCP tool. | #L361-L367 |
vllm_mlx.api.models.MCPToolsResponse |
class | vllm_mlx.api.models.MCPToolsResponse(tools: list[MCPToolInfo], count: int) |
Response for listing MCP tools. | #L370-L374 |
vllm_mlx.api.models.Message |
class | vllm_mlx.api.models.Message(role: str, content: str | list[ContentPart] | list[dict] | None = None, tool_calls: list[dict] | None = None, tool_call_id: str | None = None) |
A message in a chat conversation. | #L68-L84 |
vllm_mlx.api.models.ModelInfo |
class | vllm_mlx.api.models.ModelInfo(id: str, object: str = 'model', created: int = Field(default_factory=lambda: int(time.time())), owned_by: str = 'vllm-mlx') |
Information about an available model. | #L340-L346 |
vllm_mlx.api.models.ModelsResponse |
class | vllm_mlx.api.models.ModelsResponse(object: str = 'list', data: list[ModelInfo]) |
Response for listing models. | #L349-L353 |
vllm_mlx.api.models.RerankRequest |
class | vllm_mlx.api.models.RerankRequest(model: str, query: str, documents: list[str | dict], top_n: int | None = None, return_documents: bool = True) |
Request for reranking documents against a query (Jina/Cohere convention). | #L492-L499 |
vllm_mlx.api.models.RerankResponse |
class | vllm_mlx.api.models.RerankResponse(model: str, results: list[RerankResult], usage: RerankUsage = Field(default_factory=RerankUsage)) |
Response for reranking endpoint (Jina/Cohere convention). | #L516-L521 |
vllm_mlx.api.models.RerankResult |
class | vllm_mlx.api.models.RerankResult(index: int, relevance_score: float, document: dict | None = None) |
A single reranked document result. | #L502-L507 |
vllm_mlx.api.models.RerankUsage |
class | vllm_mlx.api.models.RerankUsage(total_tokens: int = 0) |
Token usage for rerank requests. | #L510-L513 |
vllm_mlx.api.models.ResponseFormat |
class | vllm_mlx.api.models.ResponseFormat(type: str = 'text', json_schema: ResponseFormatJsonSchema | None = None) |
Response format specification for structured output. | #L145-L156 |
vllm_mlx.api.models.ResponseFormatJsonSchema |
class | vllm_mlx.api.models.ResponseFormatJsonSchema(name: str, description: str | None = None, schema_: dict = Field(alias='schema'), strict: bool | None = False) |
JSON Schema definition for structured output. | #L131-L142 |
vllm_mlx.api.models.ResponseFormatJsonSchema.Config |
class | vllm_mlx.api.models.ResponseFormatJsonSchema.Config() |
Allow callers to populate the aliased ``schema`` field by name. | #L139-L142 |
vllm_mlx.api.models.StreamOptions |
class | vllm_mlx.api.models.StreamOptions(include_usage: bool = False) |
Options for streaming responses. | #L164-L167 |
vllm_mlx.api.models.ToolCall |
class | vllm_mlx.api.models.ToolCall(id: str, type: str = 'function', function: FunctionCall) |
A tool call from the model. | #L102-L107 |
vllm_mlx.api.models.ToolDefinition |
class | vllm_mlx.api.models.ToolDefinition(type: str = 'function', function: dict) |
Definition of a tool that can be called by the model. | #L110-L123 |
vllm_mlx.api.models.ToolDefinition._validate_openai_function_name |
method | vllm_mlx.api.models.ToolDefinition._validate_openai_function_name() -> not annotated |
Method `ToolDefinition._validate_openai_function_name` calls `self.function.get`, `isinstance`, `_OPENAI_FUNCTION_NAME_RE.fullmatch`, `ValueError`; can raise `ValueError`; returns `self`. | #L117-L123 |
vllm_mlx.api.models.Usage |
class | vllm_mlx.api.models.Usage(prompt_tokens: int = 0, completion_tokens: int = 0, total_tokens: int = 0) |
Token usage statistics. | #L259-L264 |
vllm_mlx.api.models.VideoUrl |
class | vllm_mlx.api.models.VideoUrl(url: str) |
Video URL. | #L31-L34 |
vllm_mlx.api.prompt_canonicalize.canonicalize_system_messages |
function | vllm_mlx.api.prompt_canonicalize.canonicalize_system_messages(messages: list[dict]) -> list[dict] |
Canonicalize string content on system-role messages without mutation. | #L27-L51 |
vllm_mlx.api.prompt_canonicalize.canonicalize_system_prompt |
function | vllm_mlx.api.prompt_canonicalize.canonicalize_system_prompt(text: str | None) -> str | None |
Remove known non-semantic volatile lines from system prompt text. | #L17-L24 |
vllm_mlx.api.responses_models.ResponseCompletedEvent |
class | vllm_mlx.api.responses_models.ResponseCompletedEvent() |
Carry the terminal completed response object. | #L249-L253 |
vllm_mlx.api.responses_models.ResponseContentPartAddedEvent |
class | vllm_mlx.api.responses_models.ResponseContentPartAddedEvent() |
Announce a content part attached to an output item. | #L272-L279 |
vllm_mlx.api.responses_models.ResponseContentPartDoneEvent |
class | vllm_mlx.api.responses_models.ResponseContentPartDoneEvent() |
Signal that an output item's content part is complete. | #L282-L289 |
vllm_mlx.api.responses_models.ResponseCreatedEvent |
class | vllm_mlx.api.responses_models.ResponseCreatedEvent() |
Signal that a response object has been created. | #L235-L239 |
vllm_mlx.api.responses_models.ResponseError |
class | vllm_mlx.api.responses_models.ResponseError(code: str, message: str) |
Error payload. | #L137-L141 |
vllm_mlx.api.responses_models.ResponseFunctionCallArgumentsDeltaEvent |
class | vllm_mlx.api.responses_models.ResponseFunctionCallArgumentsDeltaEvent() |
Carry an incremental fragment of function-call arguments. | #L334-L342 |
vllm_mlx.api.responses_models.ResponseFunctionCallItem |
class | vllm_mlx.api.responses_models.ResponseFunctionCallItem(id: str | None = None, type: Literal['function_call'] = 'function_call', call_id: str, name: str, arguments: str, status: Literal['in_progress', 'completed', 'incomplete'] = 'completed') |
A function call output item. | #L80-L88 |
vllm_mlx.api.responses_models.ResponseFunctionCallOutputItem |
class | vllm_mlx.api.responses_models.ResponseFunctionCallOutputItem(type: Literal['function_call_output'] = 'function_call_output', call_id: str, output: str) |
A tool result item passed back into a later request. | #L91-L96 |
vllm_mlx.api.responses_models.ResponseFunctionTool |
class | vllm_mlx.api.responses_models.ResponseFunctionTool(type: Literal['function'] = 'function', name: str, description: str | None = '', parameters: dict = Field(default_factory=lambda: {'type': 'object', 'properties': {}}), strict: bool = False) |
A function tool definition. | #L99-L108 |
vllm_mlx.api.responses_models.ResponseIncompleteDetails |
class | vllm_mlx.api.responses_models.ResponseIncompleteDetails(reason: str) |
Incomplete response details. | #L144-L147 |
vllm_mlx.api.responses_models.ResponseInProgressEvent |
class | vllm_mlx.api.responses_models.ResponseInProgressEvent() |
Signal that response generation is in progress. | #L242-L246 |
vllm_mlx.api.responses_models.ResponseMessageItem |
class | vllm_mlx.api.responses_models.ResponseMessageItem(id: str | None = None, type: Literal['message'] = 'message', role: Literal['system', 'user', 'assistant', 'developer'] = 'assistant', content: str | list[ResponseTextContentPart] = Field(default_factory=list), status: Literal['in_progress', 'completed', 'incomplete'] | None = 'completed') |
A Responses API message item. | #L60-L67 |
vllm_mlx.api.responses_models.ResponseObject |
class | vllm_mlx.api.responses_models.ResponseObject(id: str = Field(default_factory=lambda: f'resp_{uuid.uuid4().hex}'), object: Literal['response'] = 'response', created_at: int = Field(default_factory=lambda: int(time.time())), status: Literal['completed', 'failed', 'incomplete', 'in_progress'] = 'completed', background: bool = False, error: ResponseError | None = None, incomplete_details: ResponseIncompleteDetails | None = None, instructions: str | None = None, max_output_tokens: int | None = None, max_tool_calls: int | None = None, metadata: dict = Field(default_factory=dict), model: str, output: list[ResponseMessageItem | ResponseReasoningItem | ResponseFunctionCallItem] = Field(default_factory=list), parallel_tool_calls: bool = True, previous_response_id: str | None = None, text: ResponseTextConfig = Field(default_factory=ResponseTextConfig), tool_choice: str | dict | None = 'auto', tools: list[ResponseFunctionTool | dict] = Field(default_factory=list), top_p: float = 1.0, temperature: float | None = None, truncation: str = 'disabled', usage: ResponsesUsage | None = None, user: str | None = None, store: bool = True) |
Response object for /v1/responses. | #L182-L226 |
vllm_mlx.api.responses_models.ResponseObject.output_text |
method | vllm_mlx.api.responses_models.ResponseObject.output_text() -> str |
Concatenate assistant text content into the convenience field. | #L214-L226 |
vllm_mlx.api.responses_models.ResponseOutputItemAddedEvent |
class | vllm_mlx.api.responses_models.ResponseOutputItemAddedEvent() |
Announce a newly added response output item. | #L256-L261 |
vllm_mlx.api.responses_models.ResponseOutputItemDoneEvent |
class | vllm_mlx.api.responses_models.ResponseOutputItemDoneEvent() |
Signal that a response output item is complete. | #L264-L269 |
vllm_mlx.api.responses_models.ResponseOutputTextDeltaEvent |
class | vllm_mlx.api.responses_models.ResponseOutputTextDeltaEvent() |
Carry an incremental final-answer text fragment. | #L292-L300 |
vllm_mlx.api.responses_models.ResponseOutputTextDoneEvent |
class | vllm_mlx.api.responses_models.ResponseOutputTextDoneEvent() |
Carry the complete final-answer text for one content part. | #L303-L311 |
vllm_mlx.api.responses_models.ResponseReasoningConfig |
class | vllm_mlx.api.responses_models.ResponseReasoningConfig(effort: Literal['none', 'minimal', 'low', 'medium', 'high', 'xhigh'] | None = None) |
Reasoning configuration. | #L31-L34 |
vllm_mlx.api.responses_models.ResponseReasoningItem |
class | vllm_mlx.api.responses_models.ResponseReasoningItem(id: str | None = None, type: Literal['reasoning'] = 'reasoning', summary: list[ResponseReasoningSummaryTextPart] = Field(default_factory=list), content: list[ResponseReasoningTextPart] = Field(default_factory=list), status: Literal['in_progress', 'completed', 'incomplete'] | None = 'completed') |
A reasoning output item. | #L70-L77 |
vllm_mlx.api.responses_models.ResponseReasoningSummaryTextPart |
class | vllm_mlx.api.responses_models.ResponseReasoningSummaryTextPart(type: Literal['summary_text'] = 'summary_text', text: str) |
A reasoning summary item. | #L53-L57 |
vllm_mlx.api.responses_models.ResponseReasoningTextDeltaEvent |
class | vllm_mlx.api.responses_models.ResponseReasoningTextDeltaEvent() |
Carry an incremental reasoning text fragment. | #L314-L321 |
vllm_mlx.api.responses_models.ResponseReasoningTextDoneEvent |
class | vllm_mlx.api.responses_models.ResponseReasoningTextDoneEvent() |
Carry the complete reasoning text for one content part. | #L324-L331 |
vllm_mlx.api.responses_models.ResponseReasoningTextPart |
class | vllm_mlx.api.responses_models.ResponseReasoningTextPart(type: Literal['reasoning_text'] = 'reasoning_text', text: str) |
A reasoning text content part. | #L46-L50 |
vllm_mlx.api.responses_models.ResponsesEventBase |
class | vllm_mlx.api.responses_models.ResponsesEventBase(sequence_number: int) |
Base event fields. | #L229-L232 |
vllm_mlx.api.responses_models.ResponsesInputTokenDetails |
class | vllm_mlx.api.responses_models.ResponsesInputTokenDetails(cached_tokens: int = 0) |
Input token breakdown. | #L111-L114 |
vllm_mlx.api.responses_models.ResponsesOutputTokenDetails |
class | vllm_mlx.api.responses_models.ResponsesOutputTokenDetails(reasoning_tokens: int = 0) |
Output token breakdown. | #L117-L120 |
vllm_mlx.api.responses_models.ResponsesRequest |
class | vllm_mlx.api.responses_models.ResponsesRequest(model: str, input: str | list[ResponseMessageItem | ResponseReasoningItem | ResponseFunctionCallItem | ResponseFunctionCallOutputItem | di…, instructions: str | None = None, max_output_tokens: int | None = None, stream: bool = False, tools: list[ResponseFunctionTool | dict] = Field(default_factory=list), tool_choice: str | dict | None = 'auto', parallel_tool_calls: bool = True, previous_response_id: str | None = None, temperature: float | None = None, top_p: float | None = None, chat_template_kwargs: dict[str, Any] | None = None, metadata: dict = Field(default_factory=dict), text: ResponseTextConfig = Field(default_factory=ResponseTextConfig), reasoning: ResponseReasoningConfig | None = None, store: bool = True, truncation: str = 'disabled', user: str | None = None) |
Request payload for /v1/responses. | #L150-L179 |
vllm_mlx.api.responses_models.ResponsesUsage |
class | vllm_mlx.api.responses_models.ResponsesUsage(input_tokens: int, output_tokens: int, total_tokens: int, input_tokens_details: ResponsesInputTokenDetails = Field(default_factory=ResponsesInputTokenDetails), output_tokens_details: ResponsesOutputTokenDetails = Field(default_factory=ResponsesOutputTokenDetails)) |
Responses API token usage. | #L123-L134 |
vllm_mlx.api.responses_models.ResponseTextConfig |
class | vllm_mlx.api.responses_models.ResponseTextConfig(format: ResponseTextFormat = Field(default_factory=ResponseTextFormat)) |
Text output configuration. | #L25-L28 |
vllm_mlx.api.responses_models.ResponseTextContentPart |
class | vllm_mlx.api.responses_models.ResponseTextContentPart(type: Literal['text', 'input_text', 'output_text'] = 'output_text', text: str, annotations: list[dict] = Field(default_factory=list), logprobs: list[dict] = Field(default_factory=list)) |
A text content part for message items. | #L37-L43 |
vllm_mlx.api.responses_models.ResponseTextFormat |
class | vllm_mlx.api.responses_models.ResponseTextFormat(type: Literal['text', 'json_object'] = 'text') |
Output text format configuration. | #L19-L22 |
vllm_mlx.api.streaming._escape_json_string |
function | vllm_mlx.api.streaming._escape_json_string(s: str) -> str |
Escape a string for JSON without the surrounding quotes. | #L16-L24 |
vllm_mlx.api.streaming.StreamingJSONEncoder |
class | vllm_mlx.api.streaming.StreamingJSONEncoder(response_id: str, model: str, object_type: str, created: int | None = None) |
Optimized JSON encoder for OpenAI-compatible streaming responses. | #L27-L210 |
vllm_mlx.api.streaming.StreamingJSONEncoder.__init__ |
method | vllm_mlx.api.streaming.StreamingJSONEncoder.__init__(response_id: str, model: str, object_type: str, created: int | None = None) -> not annotated |
Initialize the encoder with static response metadata. | #L57-L102 |
vllm_mlx.api.streaming.StreamingJSONEncoder.encode_chat_chunk |
method | vllm_mlx.api.streaming.StreamingJSONEncoder.encode_chat_chunk(role: str | None = None, content: str | None = None, finish_reason: str | None = None, usage: dict[str, int] | None = None) -> str |
Encode a chat completion chunk using pre-computed templates. | #L151-L201 |
vllm_mlx.api.streaming.StreamingJSONEncoder.encode_completion_chunk |
method | vllm_mlx.api.streaming.StreamingJSONEncoder.encode_completion_chunk(text: str, index: int = 0, finish_reason: str | None = None, usage: dict[str, int] | None = None) -> str |
Encode a text completion chunk using pre-computed templates. | #L104-L149 |
vllm_mlx.api.streaming.StreamingJSONEncoder.encode_done |
method | vllm_mlx.api.streaming.StreamingJSONEncoder.encode_done() -> str |
Encode the [DONE] message that signals end of stream. | #L203-L210 |
vllm_mlx.api.tool_calling._looks_like_tool_call |
function | vllm_mlx.api.tool_calling._looks_like_tool_call(obj: Any) -> bool |
Heuristic: decide whether a parsed JSON object really represents a tool call as opposed to user data that happens to carry a ``"name"`` field. | #L32-L57 |
vllm_mlx.api.tool_calling._parse_raw_json_tool_calls |
function | vllm_mlx.api.tool_calling._parse_raw_json_tool_calls(text: str) -> Optional[List[dict]] |
Parse raw JSON tool calls from model output. | #L60-L123 |
vllm_mlx.api.tool_calling._repair_truncated_json |
function | vllm_mlx.api.tool_calling._repair_truncated_json(fragment: str) -> Optional[Dict[str, Any]] |
Attempt to parse a JSON fragment whose closing brackets were cut off (e.g. | #L497-L582 |
vllm_mlx.api.tool_calling._repair_truncated_json._close |
nested function | vllm_mlx.api.tool_calling._repair_truncated_json._close(text: str) -> str |
Nested Function `_repair_truncated_json._close` calls `reversed`; returns `text`. | #L540-L543 |
vllm_mlx.api.tool_calling._scan_balanced_json |
function | vllm_mlx.api.tool_calling._scan_balanced_json(text: str, start: int) -> Optional[str] |
Walk forward from ``start`` (which must point at ``{`` or ``[``) and return the substring that represents the first balanced JSON value, respecting strings and escapes. | #L459-L494 |
vllm_mlx.api.tool_calling.apply_response_format_or_error |
function | vllm_mlx.api.tool_calling.apply_response_format_or_error(text: str, response_format: object, *, ensure_ascii: bool = False) -> str |
Return canonical JSON content or raise for invalid response_format output. | #L858-L873 |
vllm_mlx.api.tool_calling.build_json_logits_processor |
function | vllm_mlx.api.tool_calling.build_json_logits_processor(response_format: ResponseFormat | dict[str, Any] | None, tokenizer: Any) -> not annotated |
Build a logits processor that constrains generation to valid JSON matching ``response_format``. | #L956-L1035 |
vllm_mlx.api.tool_calling.build_json_system_prompt |
function | vllm_mlx.api.tool_calling.build_json_system_prompt(response_format: Optional[Union[ResponseFormat, Dict[str, Any]]] = None, *, thinking_model: bool = False) -> Optional[str] |
Build a system prompt instruction for JSON output. | #L876-L953 |
vllm_mlx.api.tool_calling.convert_tools_for_template |
function | vllm_mlx.api.tool_calling.convert_tools_for_template(tools: Optional[List]) -> Optional[List[dict]] |
Convert OpenAI tools format to format expected by tokenizer.apply_chat_template. | #L354-L409 |
vllm_mlx.api.tool_calling.extract_json_from_text |
function | vllm_mlx.api.tool_calling.extract_json_from_text(text: str) -> Optional[Dict[str, Any]] |
Extract JSON from model output text. | #L585-L666 |
vllm_mlx.api.tool_calling.format_tool_call_for_message |
function | vllm_mlx.api.tool_calling.format_tool_call_for_message(tool_call: ToolCall) -> dict |
Format a ToolCall object for inclusion in a message. | #L412-L429 |
vllm_mlx.api.tool_calling.InvalidResponseFormatOutput |
class | vllm_mlx.api.tool_calling.InvalidResponseFormatOutput(message: str) |
Raised when generated content does not satisfy response_format. | #L24-L29 |
vllm_mlx.api.tool_calling.InvalidResponseFormatOutput.__init__ |
method | vllm_mlx.api.tool_calling.InvalidResponseFormatOutput.__init__(message: str) -> not annotated |
Method `InvalidResponseFormatOutput.__init__` updates `self.message`; calls `super().__init__`, `super`. | #L27-L29 |
vllm_mlx.api.tool_calling.parse_json_output |
function | vllm_mlx.api.tool_calling.parse_json_output(text: str, response_format: Optional[Union[ResponseFormat, Dict[str, Any]]] = None) -> Tuple[str, Optional[Dict[str, Any]], bool, Optional[str]] |
Parse JSON from model output when response_format is set. | #L790-L855 |
vllm_mlx.api.tool_calling.parse_tool_calls |
function | vllm_mlx.api.tool_calling.parse_tool_calls(text: str, request: dict[str, Any] | None = None) -> Tuple[str, Optional[List[ToolCall]]] |
Parse tool calls from model output. | #L126-L351 |
vllm_mlx.api.tool_calling.StreamingJsonFenceStripper |
class | vllm_mlx.api.tool_calling.StreamingJsonFenceStripper() |
Strip markdown code fences from streamed content when response_format is set. | #L669-L787 |
vllm_mlx.api.tool_calling.StreamingJsonFenceStripper.__init__ |
method | vllm_mlx.api.tool_calling.StreamingJsonFenceStripper.__init__() -> None |
Method `StreamingJsonFenceStripper.__init__` updates `self._buf`, `self._past_opening`. | #L695-L697 |
vllm_mlx.api.tool_calling.StreamingJsonFenceStripper.feed |
method | vllm_mlx.api.tool_calling.StreamingJsonFenceStripper.feed(delta: str) -> str |
Append a content delta and return the portion safe to emit now. | #L699-L748 |
vllm_mlx.api.tool_calling.StreamingJsonFenceStripper.finalize |
method | vllm_mlx.api.tool_calling.StreamingJsonFenceStripper.finalize() -> str |
Flush the remaining buffer, dropping any trailing fence. | #L750-L787 |
vllm_mlx.api.tool_calling.validate_json_schema |
function | vllm_mlx.api.tool_calling.validate_json_schema(data: Any, schema: Dict[str, Any]) -> Tuple[bool, Optional[str]] |
Validate JSON data against a JSON Schema. | #L437-L456 |
vllm_mlx.api.utils._check_legacy_string_patterns |
function | vllm_mlx.api.utils._check_legacy_string_patterns(model_name: str) -> bool |
Validation 1: substring match of MLLM_PATTERNS against the input string. | #L456-L466 |
vllm_mlx.api.utils._clean_gpt_oss_output |
function | vllm_mlx.api.utils._clean_gpt_oss_output(text: str) -> str |
Extract final channel content from GPT-OSS channel-based output. | #L39-L73 |
vllm_mlx.api.utils._config_indicates_vlm |
function | vllm_mlx.api.utils._config_indicates_vlm(config: dict) -> bool |
Inspect a parsed config.json dict for multimodal markers. | #L437-L453 |
vllm_mlx.api.utils._content_to_text |
function | vllm_mlx.api.utils._content_to_text(content) -> str |
Extract text from content that can be str, list[ContentPart], or None. | #L544-L560 |
vllm_mlx.api.utils._try_read_config_json |
function | vllm_mlx.api.utils._try_read_config_json(name_or_path: str) -> dict | None |
Read config.json from a local model directory. | #L408-L434 |
vllm_mlx.api.utils.clean_output_text |
function | vllm_mlx.api.utils.clean_output_text(text: str) -> str |
Clean model output by removing special tokens. | #L76-L108 |
vllm_mlx.api.utils.extract_multimodal_content |
function | vllm_mlx.api.utils.extract_multimodal_content(messages: list[Message], preserve_native_format: bool = False) -> tuple[list[dict], list[str], list[str], list[str]] |
Extract text content, images, videos, and audio from OpenAI-format messages. | #L563-L747 |
vllm_mlx.api.utils.has_media_content |
function | vllm_mlx.api.utils.has_media_content(messages: list) -> bool |
Check if any message contains media content (images, video, audio). | #L516-L536 |
vllm_mlx.api.utils.is_mllm_model |
function | vllm_mlx.api.utils.is_mllm_model(model_name: str) -> bool |
Check if a model name or path indicates a multimodal language model. | #L469-L493 |
vllm_mlx.api.utils.StreamingThinkRouter |
class | vllm_mlx.api.utils.StreamingThinkRouter(start_in_thinking: bool = False) |
Route <think>...</think> content to separate Anthropic thinking blocks. | #L237-L327 |
vllm_mlx.api.utils.StreamingThinkRouter.__init__ |
method | vllm_mlx.api.utils.StreamingThinkRouter.__init__(start_in_thinking: bool = False) -> not annotated |
Method `StreamingThinkRouter.__init__` updates `self._buffer`, `self._in_think`. | #L255-L257 |
vllm_mlx.api.utils.StreamingThinkRouter._extract_pieces |
method | vllm_mlx.api.utils.StreamingThinkRouter._extract_pieces(pieces: list[tuple[str, str]]) -> None |
Extract all complete pieces from the buffer. | #L266-L317 |
vllm_mlx.api.utils.StreamingThinkRouter.flush |
method | vllm_mlx.api.utils.StreamingThinkRouter.flush() -> list[tuple[str, str]] |
Flush remaining buffer at end of stream. | #L319-L327 |
vllm_mlx.api.utils.StreamingThinkRouter.process |
method | vllm_mlx.api.utils.StreamingThinkRouter.process(delta: str) -> list[tuple[str, str]] |
Process a delta. | #L259-L264 |
vllm_mlx.api.utils.StreamingToolCallFilter |
class | vllm_mlx.api.utils.StreamingToolCallFilter() |
Buffer streaming text to suppress tool call markup. | #L134-L229 |
vllm_mlx.api.utils.StreamingToolCallFilter.__init__ |
method | vllm_mlx.api.utils.StreamingToolCallFilter.__init__() -> not annotated |
Method `StreamingToolCallFilter.__init__` updates `self._buffer`, `self._in_block`, `self._close_tag`, `self._max_open_len`; calls `max`, `len`. | #L146-L151 |
vllm_mlx.api.utils.StreamingToolCallFilter._consume_block |
method | vllm_mlx.api.utils.StreamingToolCallFilter._consume_block() -> str |
Consume content inside a tool call block. | #L196-L218 |
vllm_mlx.api.utils.StreamingToolCallFilter._scan_for_open |
method | vllm_mlx.api.utils.StreamingToolCallFilter._scan_for_open() -> str |
Scan buffer for tool call open tags. | #L162-L194 |
vllm_mlx.api.utils.StreamingToolCallFilter.flush |
method | vllm_mlx.api.utils.StreamingToolCallFilter.flush() -> str |
Flush remaining buffer at end of stream. | #L220-L229 |
vllm_mlx.api.utils.StreamingToolCallFilter.process |
method | vllm_mlx.api.utils.StreamingToolCallFilter.process(delta: str) -> str |
Process a streaming delta. | #L153-L160 |
vllm_mlx.attention.create_mlx_attention_backend |
function | vllm_mlx.attention.create_mlx_attention_backend() -> type |
Factory function to create MLX attention backend. | #L243-L245 |
vllm_mlx.attention.MLXAttentionBackend |
class | vllm_mlx.attention.MLXAttentionBackend() |
Attention backend using MLX's native attention. | #L42-L135 |
vllm_mlx.attention.MLXAttentionBackend.get_impl_cls |
method | vllm_mlx.attention.MLXAttentionBackend.get_impl_cls() -> type |
Return the implementation class. | #L60-L62 |
vllm_mlx.attention.MLXAttentionBackend.get_kv_cache_shape |
method | vllm_mlx.attention.MLXAttentionBackend.get_kv_cache_shape(num_blocks: int, block_size: int, num_kv_heads: int, head_size: int) -> tuple[int, ...] |
Get the shape of KV cache. | #L70-L89 |
vllm_mlx.attention.MLXAttentionBackend.get_metadata_cls |
method | vllm_mlx.attention.MLXAttentionBackend.get_metadata_cls() -> type |
Return the metadata class. | #L65-L67 |
vllm_mlx.attention.MLXAttentionBackend.get_name |
method | vllm_mlx.attention.MLXAttentionBackend.get_name() -> str |
Return backend name. | #L55-L57 |
vllm_mlx.attention.MLXAttentionBackend.get_supported_head_sizes |
method | vllm_mlx.attention.MLXAttentionBackend.get_supported_head_sizes() -> list[int] |
Return supported attention head sizes. | #L92-L94 |
vllm_mlx.attention.MLXAttentionBackend.supports_attn_type |
method | vllm_mlx.attention.MLXAttentionBackend.supports_attn_type(attn_type: str) -> bool |
Check if attention type is supported. | #L133-L135 |
vllm_mlx.attention.MLXAttentionBackend.supports_block_size |
method | vllm_mlx.attention.MLXAttentionBackend.supports_block_size(block_size: int) -> bool |
Check if block size is supported. | #L128-L130 |
vllm_mlx.attention.MLXAttentionBackend.supports_dtype |
method | vllm_mlx.attention.MLXAttentionBackend.supports_dtype(dtype: 'torch.dtype') -> bool |
Check if dtype is supported. | #L121-L125 |
vllm_mlx.attention.MLXAttentionBackend.validate_configuration |
method | vllm_mlx.attention.MLXAttentionBackend.validate_configuration(num_heads: int, head_size: int, num_kv_heads: int, dtype: 'torch.dtype', block_size: int, **kwargs) -> list[str] |
Validate attention configuration. | #L97-L118 |
vllm_mlx.attention.MLXAttentionImpl |
class | vllm_mlx.attention.MLXAttentionImpl(num_heads: int, head_size: int, scale: float, num_kv_heads: int | None = None, alibi_slopes: list[float] | None = None, sliding_window: int | None = None, kv_cache_dtype: str = 'auto', blocksparse_params: dict | None = None, logits_soft_cap: float | None = None, **kwargs) |
MLX attention implementation. | #L138-L240 |
vllm_mlx.attention.MLXAttentionImpl.__init__ |
method | vllm_mlx.attention.MLXAttentionImpl.__init__(num_heads: int, head_size: int, scale: float, num_kv_heads: int | None = None, alibi_slopes: list[float] | None = None, sliding_window: int | None = None, kv_cache_dtype: str = 'auto', blocksparse_params: dict | None = None, logits_soft_cap: float | None = None, **kwargs) -> not annotated |
Initialize MLX attention. | #L147-L186 |
vllm_mlx.attention.MLXAttentionImpl.forward |
method | vllm_mlx.attention.MLXAttentionImpl.forward(query: Any, key: Any, value: Any, kv_cache: Any | None = None, attn_metadata: MLXAttentionMetadata | None = None, output: Any | None = None, **kwargs) -> Any |
Compute attention. | #L188-L240 |
vllm_mlx.attention.MLXAttentionMetadata |
class | vllm_mlx.attention.MLXAttentionMetadata(seq_lens: list[int], max_seq_len: int, num_prefill_tokens: int = 0, num_decode_tokens: int = 0, block_tables: Any | None = None, slot_mapping: Any | None = None) |
Metadata for MLX attention computation. | #L20-L39 |
vllm_mlx.audio.processor.AudioProcessor |
class | vllm_mlx.audio.processor.AudioProcessor(model_name: str = DEFAULT_SAM_MODEL) |
Audio processor for voice separation and enhancement. | #L33-L192 |
vllm_mlx.audio.processor.AudioProcessor.__init__ |
method | vllm_mlx.audio.processor.AudioProcessor.__init__(model_name: str = DEFAULT_SAM_MODEL) -> not annotated |
Initialize audio processor. | #L48-L66 |
vllm_mlx.audio.processor.AudioProcessor._to_numpy |
method | vllm_mlx.audio.processor.AudioProcessor._to_numpy(audio) -> np.ndarray |
Convert audio to numpy array. | #L153-L157 |
vllm_mlx.audio.processor.AudioProcessor.load |
method | vllm_mlx.audio.processor.AudioProcessor.load() -> None |
Load the SAM-Audio model. | #L68-L88 |
vllm_mlx.audio.processor.AudioProcessor.save |
method | vllm_mlx.audio.processor.AudioProcessor.save(audio: np.ndarray, path: Union[str, Path], sample_rate: Optional[int] = None) -> None |
Save audio to file. | #L159-L185 |
vllm_mlx.audio.processor.AudioProcessor.separate |
method | vllm_mlx.audio.processor.AudioProcessor.separate(audio_path: Union[str, Path], description: str = 'speech', chunk_seconds: Optional[float] = None) -> SeparationResult |
Separate audio based on text description. | #L90-L151 |
vllm_mlx.audio.processor.AudioProcessor.unload |
method | vllm_mlx.audio.processor.AudioProcessor.unload() -> None |
Unload model to free memory. | #L187-L192 |
vllm_mlx.audio.processor.separate_voice |
function | vllm_mlx.audio.processor.separate_voice(audio_path: Union[str, Path], model_name: str = DEFAULT_SAM_MODEL, description: str = 'speech') -> Tuple[np.ndarray, np.ndarray] |
Convenience function to separate voice from audio. | #L195-L214 |
vllm_mlx.audio.processor.SeparationResult |
class | vllm_mlx.audio.processor.SeparationResult(target: np.ndarray, residual: np.ndarray, sample_rate: int, peak_memory: float) |
Result from audio separation. | #L24-L30 |
vllm_mlx.audio.stt.STTEngine |
class | vllm_mlx.audio.stt.STTEngine(model_name: str = DEFAULT_WHISPER_MODEL) |
Speech-to-Text engine supporting Whisper and Parakeet models. | #L32-L139 |
vllm_mlx.audio.stt.STTEngine.__init__ |
method | vllm_mlx.audio.stt.STTEngine.__init__(model_name: str = DEFAULT_WHISPER_MODEL) -> not annotated |
Initialize STT engine. | #L43-L62 |
vllm_mlx.audio.stt.STTEngine.load |
method | vllm_mlx.audio.stt.STTEngine.load() -> None |
Load the STT model. | #L64-L79 |
vllm_mlx.audio.stt.STTEngine.transcribe |
method | vllm_mlx.audio.stt.STTEngine.transcribe(audio_path: Union[str, Path], language: Optional[str] = None, task: str = 'transcribe') -> TranscriptionResult |
Transcribe audio file to text. | #L81-L133 |
vllm_mlx.audio.stt.STTEngine.unload |
method | vllm_mlx.audio.stt.STTEngine.unload() -> None |
Unload model to free memory. | #L135-L139 |
vllm_mlx.audio.stt.transcribe_audio |
function | vllm_mlx.audio.stt.transcribe_audio(audio_path: Union[str, Path], model_name: str = DEFAULT_WHISPER_MODEL, language: Optional[str] = None) -> TranscriptionResult |
Convenience function to transcribe audio without managing engine. | #L142-L160 |
vllm_mlx.audio.stt.TranscriptionResult |
class | vllm_mlx.audio.stt.TranscriptionResult(text: str, language: Optional[str] = None, duration: Optional[float] = None, segments: Optional[list] = None) |
Result from audio transcription. | #L23-L29 |
vllm_mlx.audio.tts.AudioOutput |
class | vllm_mlx.audio.tts.AudioOutput(audio: np.ndarray, sample_rate: int, duration: float) |
Output from TTS generation. | #L44-L49 |
vllm_mlx.audio.tts.generate_speech |
function | vllm_mlx.audio.tts.generate_speech(text: str, model_name: str = DEFAULT_TTS_MODEL, voice: str = 'af_heart', speed: float = 1.0) -> AudioOutput |
Convenience function to generate speech without managing engine. | #L295-L315 |
vllm_mlx.audio.tts.TTSEngine |
class | vllm_mlx.audio.tts.TTSEngine(model_name: str = DEFAULT_TTS_MODEL) |
Text-to-Speech engine supporting multiple model families. | #L52-L292 |
vllm_mlx.audio.tts.TTSEngine.__init__ |
method | vllm_mlx.audio.tts.TTSEngine.__init__(model_name: str = DEFAULT_TTS_MODEL) -> not annotated |
Initialize TTS engine. | #L63-L80 |
vllm_mlx.audio.tts.TTSEngine._detect_family |
method | vllm_mlx.audio.tts.TTSEngine._detect_family(model_name: str) -> str |
Detect model family from name. | #L82-L98 |
vllm_mlx.audio.tts.TTSEngine.generate |
method | vllm_mlx.audio.tts.TTSEngine.generate(text: str, voice: str = 'af_heart', speed: float = 1.0, lang_code: str = 'a') -> AudioOutput |
Generate speech from text. | #L119-L185 |
vllm_mlx.audio.tts.TTSEngine.get_voices |
method | vllm_mlx.audio.tts.TTSEngine.get_voices() -> list |
Get available voices for current model. | #L279-L286 |
vllm_mlx.audio.tts.TTSEngine.load |
method | vllm_mlx.audio.tts.TTSEngine.load() -> None |
Load the TTS model. | #L100-L117 |
vllm_mlx.audio.tts.TTSEngine.save |
method | vllm_mlx.audio.tts.TTSEngine.save(audio: AudioOutput, path: Union[str, Path], format: str = 'wav') -> None |
Save audio to file. | #L229-L255 |
vllm_mlx.audio.tts.TTSEngine.stream_generate |
method | vllm_mlx.audio.tts.TTSEngine.stream_generate(text: str, voice: str = 'af_heart', speed: float = 1.0) -> Iterator[AudioOutput] |
Stream speech generation chunk by chunk. | #L187-L227 |
vllm_mlx.audio.tts.TTSEngine.to_bytes |
method | vllm_mlx.audio.tts.TTSEngine.to_bytes(audio: AudioOutput, format: str = 'wav') -> bytes |
Convert audio to bytes. | #L257-L277 |
vllm_mlx.audio.tts.TTSEngine.unload |
method | vllm_mlx.audio.tts.TTSEngine.unload() -> None |
Unload model to free memory. | #L288-L292 |
vllm_mlx.audio_limits.AsyncReadableUpload |
class | vllm_mlx.audio_limits.AsyncReadableUpload() |
Structural type for an asynchronously readable uploaded file. | #L17-L25 |
vllm_mlx.audio_limits.AsyncReadableUpload.read |
method | async vllm_mlx.audio_limits.AsyncReadableUpload.read(size: int = -1) -> bytes |
Read at most ``size`` bytes, or all remaining bytes when negative. | #L22-L25 |
vllm_mlx.audio_limits.save_upload_with_limit |
function | async vllm_mlx.audio_limits.save_upload_with_limit(file: AsyncReadableUpload, *, max_bytes: int, default_suffix: str = '.wav', chunk_size: int = UPLOAD_CHUNK_SIZE) -> str |
Stream an uploaded file to disk while enforcing a hard byte limit. | #L28-L65 |
vllm_mlx.audio_limits.validate_tts_input_length |
function | vllm_mlx.audio_limits.validate_tts_input_length(text: str, *, max_chars: int) -> None |
Reject oversized TTS requests before synthesis starts. | #L68-L77 |
vllm_mlx.bench_serve._assemble_case_request_kwargs |
function | vllm_mlx.bench_serve._assemble_case_request_kwargs(case: WorkloadCase, workload: Workload, model: str) -> dict |
Build the keyword-arguments dict passed to ``stream_chat_completion`` for one case, applying max_tokens fallback and converting ``policy_timeout_ms`` to seconds. | #L1355-L1372 |
vllm_mlx.bench_serve._build_tool_calls_summary |
function | vllm_mlx.bench_serve._build_tool_calls_summary(tool_calls: Any) -> Optional[dict] |
Compact summary of streamed tool calls for the case record. | #L1421-L1434 |
vllm_mlx.bench_serve._build_workload_case |
function | vllm_mlx.bench_serve._build_workload_case(item: Any, idx: int, *, defaults: dict, workload_path: Path) -> WorkloadCase |
Construct one ``WorkloadCase`` from a raw workload entry. | #L230-L300 |
vllm_mlx.bench_serve._build_workload_record |
function | vllm_mlx.bench_serve._build_workload_record(*, case: WorkloadCase, workload: Workload, model: str, runtime: dict, hardware: dict, run_id: str, timestamp: str, started_wall: str, repetition: int, result: dict, error: str, quality_ok: bool, quality_issues: list[str], content: str, cache_hits_delta: int, cache_misses_delta: int, tokens_saved_delta: int, status_after: dict, cache_reset: Optional[dict], include_content: bool) -> dict |
Assemble the JSON-serializable workload-case record from the raw inputs and the completion result. | #L1437-L1515 |
vllm_mlx.bench_serve._cancel_server_request |
function | async vllm_mlx.bench_serve._cancel_server_request(client: httpx.AsyncClient, base_url: str, request_id: Optional[str]) -> None |
Best-effort server-side cancellation for timed-out workload streams. | #L757-L770 |
vllm_mlx.bench_serve._check_finish_reason |
function | vllm_mlx.bench_serve._check_finish_reason(allowed: Any, finish_reason: Optional[str]) -> list[str] |
Verify ``finish_reason`` is in the allowed set, if one is configured. | #L1052-L1059 |
vllm_mlx.bench_serve._check_json_content |
function | vllm_mlx.bench_serve._check_json_content(should_be_json: Any, content: str) -> list[str] |
Verify ``content`` parses as JSON when ``checks['json']`` is truthy. | #L1099-L1107 |
vllm_mlx.bench_serve._check_length_bounds |
function | vllm_mlx.bench_serve._check_length_bounds(min_chars: Any, max_chars: Any, content: str) -> list[str] |
Apply ``min_chars`` / ``max_chars`` content-length bounds. | #L1062-L1069 |
vllm_mlx.bench_serve._check_regex_patterns |
function | vllm_mlx.bench_serve._check_regex_patterns(patterns: Any, content: str, *, kind: str, expect_match: bool) -> list[str] |
Validate that each pattern either matches or does not, per ``expect_match``. | #L1072-L1096 |
vllm_mlx.bench_serve._check_tool_call_args |
function | vllm_mlx.bench_serve._check_tool_call_args(required_args: Any, tool_calls: list[dict]) -> list[str] |
Validate parsed JSON arguments include the required keys per function. | #L1135-L1174 |
vllm_mlx.bench_serve._check_tool_call_count_and_names |
function | vllm_mlx.bench_serve._check_tool_call_count_and_names(checks: dict, tool_calls: list[dict]) -> list[str] |
Apply ``no_tool_calls`` / ``tool_call_count`` / ``tool_call_names``. | #L1110-L1132 |
vllm_mlx.bench_serve._compute_within_policy_timeout |
function | vllm_mlx.bench_serve._compute_within_policy_timeout(timeout_ms: Optional[int], *, error_present: bool, e2e_latency_ms: float) -> Optional[bool] |
Resolve the ``policy.within_timeout`` field. | #L1405-L1418 |
vllm_mlx.bench_serve._empty_completion_result |
function | vllm_mlx.bench_serve._empty_completion_result() -> dict |
Zero-valued completion result used when ``stream_chat_completion`` raises. | #L1375-L1390 |
vllm_mlx.bench_serve._fetch_post_run_status |
function | async vllm_mlx.bench_serve._fetch_post_run_status(client: httpx.AsyncClient, base_url: str) -> dict |
GET ``/v1/status`` after a case run, swallowing transport errors so a missing or temporarily-unavailable status endpoint does not fail the case record. | #L1393-L1402 |
vllm_mlx.bench_serve._first_not_none |
function | vllm_mlx.bench_serve._first_not_none(*values: Any) -> Any |
Function `_first_not_none` has 2 explicit return paths. | #L174-L178 |
vllm_mlx.bench_serve._group_results_by_case_id |
function | vllm_mlx.bench_serve._group_results_by_case_id(results: list[dict]) -> dict[str, list[dict]] |
Bucket workload case records by their ``case_id`` field, defaulting a missing ``case_id`` to the empty string so the grouping is stable. | #L1596-L1602 |
vllm_mlx.bench_serve._load_case_request |
function | vllm_mlx.bench_serve._load_case_request(path: str, *, workload_path: Path, case_id: str) -> dict |
Function `_load_case_request` calls `Path(path).expanduser`, `Path`, `request_path.is_absolute`, `request_path.open`; can raise `ValueError`; returns `request`. | #L151-L159 |
vllm_mlx.bench_serve._merge_case_checks |
function | vllm_mlx.bench_serve._merge_case_checks(default_checks: Any, case_checks: Any, *, case_id: str) -> Optional[dict] |
Merge a case's ``checks`` over the workload defaults. | #L194-L227 |
vllm_mlx.bench_serve._normalize_cache_policy |
function | vllm_mlx.bench_serve._normalize_cache_policy(value: Optional[str]) -> str |
Normalize cache-policy spelling from CLI or workload JSON. | #L687-L698 |
vllm_mlx.bench_serve._normalize_tags |
function | vllm_mlx.bench_serve._normalize_tags(tags: Any, *, case_id: str) -> tuple[str, ...] |
Coerce a workload case's ``tags`` field to a tuple of strings. | #L181-L191 |
vllm_mlx.bench_serve._request_extra_body |
function | vllm_mlx.bench_serve._request_extra_body(request: dict) -> dict |
Function `_request_extra_body` calls `request.items`; returns `{key: value for key, value in request.items() if key not in reserved}`. | #L162-L171 |
vllm_mlx.bench_serve._require_message_list |
function | vllm_mlx.bench_serve._require_message_list(value: Any, *, label: str) -> list[dict] |
Function `_require_message_list` calls `isinstance`, `ValueError`, `enumerate`; can raise `ValueError`; returns `value`. | #L140-L148 |
vllm_mlx.bench_serve._resolve_max_tokens |
function | vllm_mlx.bench_serve._resolve_max_tokens(case: WorkloadCase, workload: Workload) -> int |
Return the effective ``max_tokens`` for a case, falling back to workload defaults and finally to 256. | #L1349-L1352 |
vllm_mlx.bench_serve._result_to_dict |
function | vllm_mlx.bench_serve._result_to_dict(r: BenchServeResult) -> dict |
Convert a :class:`BenchServeResult` to an ordered dict. | #L1840-L1846 |
vllm_mlx.bench_serve._sql_escape |
function | vllm_mlx.bench_serve._sql_escape(value) -> str |
Escape a Python value for use as a SQL literal. | #L1907-L1927 |
vllm_mlx.bench_serve._summarize_case |
function | vllm_mlx.bench_serve._summarize_case(case_results: list[dict]) -> dict |
Build the per-case summary block. | #L1605-L1648 |
vllm_mlx.bench_serve._summary_or_empty |
function | vllm_mlx.bench_serve._summary_or_empty(values: list[float]) -> dict |
Function `_summary_or_empty` calls `compute_summary_stats`; returns `compute_summary_stats(values) if values else {}`. | #L1345-L1346 |
vllm_mlx.bench_serve._validate_sql_identifier |
function | vllm_mlx.bench_serve._validate_sql_identifier(identifier: str, *, kind: str) -> None |
Reject unsafe SQL identifiers before string interpolation. | #L1993-L1996 |
vllm_mlx.bench_serve._workload_record_to_row |
function | vllm_mlx.bench_serve._workload_record_to_row(record: dict) -> dict |
Function `_workload_record_to_row` calls `record.get`, `metrics.get`, `','.join`, `hardware.get`; returns `{'run_id': record.get('run_id', ''), 'timestamp': record.get('timestamp', ''), 'workload': record.get('workload', ''), …`. | #L2069-L2119 |
vllm_mlx.bench_serve._write_sqlite_rows |
function | vllm_mlx.bench_serve._write_sqlite_rows(output_path: str, *, table: str, schema: str, columns: list[str], rows: list[dict]) -> None |
Append benchmark rows to a SQLite database. | #L1967-L1990 |
vllm_mlx.bench_serve.accumulate_tool_calls |
function | vllm_mlx.bench_serve.accumulate_tool_calls(acc: dict[int, dict], delta_list: list[dict]) -> None |
Merge streamed OpenAI tool-call deltas into *acc* by index. | #L773-L792 |
vllm_mlx.bench_serve.auto_detect_runtime |
function | async vllm_mlx.bench_serve.auto_detect_runtime(client: httpx.AsyncClient, base_url: str) -> dict |
Query the running server and return a runtime descriptor dict. | #L576-L642 |
vllm_mlx.bench_serve.BenchServeResult |
class | vllm_mlx.bench_serve.BenchServeResult(run_id: str = '', timestamp: str = '', tag: str = '', chip: str = '', gpu_cores: int = 0, memory_gb: float = 0.0, bandwidth_gbs: float = 0.0, os_version: str = '', model_id: str = '', model_type: str = '', engine_type: str = '', mtp_enabled: bool = False, specprefill: bool = False, kv_quant: str = '', cache_type: str = '', prompt_set: str = '', concurrency: int = 1, max_tokens: int = 256, enable_thinking: Optional[bool] = None, extra_body: str = '', repetition: int = 0, prompt_tokens: int = 0, ttft_ms: float = 0.0, tpot_ms: float = 0.0, e2e_latency_ms: float = 0.0, gen_tps: float = 0.0, prompt_tps: float = 0.0, throughput_tps: float = 0.0, requests_per_s: float = 0.0, metal_active_gb: float = 0.0, metal_peak_gb: float = 0.0, metal_cache_gb: float = 0.0, cache_hits: int = 0, cache_misses: int = 0, cache_hit_rate: float = 0.0, tokens_saved: int = 0, validated: bool = True) |
Aggregated results from a single bench-serve run configuration. | #L344-L400 |
vllm_mlx.bench_serve.clear_runtime_cache |
function | async vllm_mlx.bench_serve.clear_runtime_cache(client: httpx.AsyncClient, base_url: str) -> dict |
Clear server-side runtime caches and return a JSON-serializable event. | #L664-L684 |
vllm_mlx.bench_serve.compute_request_metrics |
function | vllm_mlx.bench_serve.compute_request_metrics(t_start: float, t_first_token: float, token_times: list, t_end: float, prompt_tokens: int, completion_tokens: int) -> dict |
Compute standard latency and throughput metrics for a single request. | #L800-L852 |
vllm_mlx.bench_serve.compute_summary_stats |
function | vllm_mlx.bench_serve.compute_summary_stats(values: list[float]) -> dict |
Compute summary statistics over a list of floats. | #L1234-L1276 |
vllm_mlx.bench_serve.compute_summary_stats._percentile |
nested function | vllm_mlx.bench_serve.compute_summary_stats._percentile(p: float) -> float |
Nested Function `compute_summary_stats._percentile` calls `int`; has 3 explicit return paths. | #L1256-L1266 |
vllm_mlx.bench_serve.count_prompt_tokens |
function | async vllm_mlx.bench_serve.count_prompt_tokens(client: httpx.AsyncClient, base_url: str, messages: list[dict], model: str) -> int |
Count prompt tokens for a message list by sending a 1-token request. | #L855-L889 |
vllm_mlx.bench_serve.detect_hardware_fingerprint |
function | vllm_mlx.bench_serve.detect_hardware_fingerprint() -> dict |
Return a hardware fingerprint dict for the current machine. | #L524-L573 |
vllm_mlx.bench_serve.expand_sweep |
function | vllm_mlx.bench_serve.expand_sweep(prompt_sets: list[str], concurrencies: list[int], thinking_values: list[Optional[bool]], extra_bodies: list[str], repetitions: int) -> list[SweepConfig] |
Expand sweep parameters into a flat list of configurations. | #L411-L444 |
vllm_mlx.bench_serve.finalize_tool_calls |
function | vllm_mlx.bench_serve.finalize_tool_calls(acc: dict[int, dict]) -> list[dict] |
Return accumulated tool calls in stream index order. | #L795-L797 |
vllm_mlx.bench_serve.format_csv |
function | vllm_mlx.bench_serve.format_csv(results: list[BenchServeResult]) -> str |
Serialize benchmark results as CSV with a header row. | #L1888-L1904 |
vllm_mlx.bench_serve.format_json |
function | vllm_mlx.bench_serve.format_json(results: list[BenchServeResult]) -> str |
Serialize benchmark results as a JSON array. | #L1874-L1885 |
vllm_mlx.bench_serve.format_sql |
function | vllm_mlx.bench_serve.format_sql(results: list[BenchServeResult]) -> str |
Emit a SQL ``CREATE TABLE IF NOT EXISTS`` statement and INSERT rows. | #L1945-L1964 |
vllm_mlx.bench_serve.format_table |
function | vllm_mlx.bench_serve.format_table(results: list[BenchServeResult]) -> str |
Render a human-readable terminal table of benchmark results. | #L1849-L1871 |
vllm_mlx.bench_serve.format_workload_csv |
function | vllm_mlx.bench_serve.format_workload_csv(payload: dict) -> str |
Serialize workload result records with the stable CSV column contract. | #L2143-L2151 |
vllm_mlx.bench_serve.format_workload_json |
function | vllm_mlx.bench_serve.format_workload_json(payload: dict) -> str |
Serialize a workload result payload as indented JSON. | #L2137-L2140 |
vllm_mlx.bench_serve.format_workload_payload |
function | vllm_mlx.bench_serve.format_workload_payload(payload: dict, fmt: str = 'json') -> str |
Serialize a workload payload in the requested text output format. | #L2196-L2211 |
vllm_mlx.bench_serve.format_workload_sql |
function | vllm_mlx.bench_serve.format_workload_sql(payload: dict) -> str |
Render SQL statements that create and populate the workload table. | #L2170-L2180 |
vllm_mlx.bench_serve.format_workload_table |
function | vllm_mlx.bench_serve.format_workload_table(payload: dict) -> str |
Format workload result records as a compact human-readable table. | #L2122-L2134 |
vllm_mlx.bench_serve.load_prompt_set |
function | vllm_mlx.bench_serve.load_prompt_set(name_or_path: str) -> list[list[dict]] |
Load a prompt set by builtin name or file path. | #L75-L137 |
vllm_mlx.bench_serve.load_workload |
function | vllm_mlx.bench_serve.load_workload(path: str | Path) -> Workload |
Load a declarative serving benchmark workload. | #L303-L335 |
vllm_mlx.bench_serve.parse_health_response |
function | vllm_mlx.bench_serve.parse_health_response(data: dict) -> dict |
Extract model identity fields from a GET /health response. | #L452-L467 |
vllm_mlx.bench_serve.parse_metrics_text |
function | vllm_mlx.bench_serve.parse_metrics_text(text: str) -> dict |
Parse Prometheus text exposition format from GET /metrics. | #L499-L521 |
vllm_mlx.bench_serve.parse_metrics_text._extract |
nested function | vllm_mlx.bench_serve.parse_metrics_text._extract(metric_name: str) -> int |
Nested Function `parse_metrics_text._extract` calls `re.escape`, `re.search`, `int`, `m.group`; returns `int(m.group(1)) if m else 0`. | #L512-L515 |
vllm_mlx.bench_serve.parse_sse_line |
function | vllm_mlx.bench_serve.parse_sse_line(line: str) -> Optional[dict] |
Parse one Server-Sent Events line from a streaming chat completion. | #L706-L754 |
vllm_mlx.bench_serve.parse_status_response |
function | vllm_mlx.bench_serve.parse_status_response(data: dict) -> dict |
Extract metal and cache info from a GET /v1/status response. | #L470-L496 |
vllm_mlx.bench_serve.run_bench_serve |
function | async vllm_mlx.bench_serve.run_bench_serve(url: str = 'http://127.0.0.1:8080', model: Optional[str] = None, prompt_sets: list[str] = None, prompt_file: Optional[str] = None, concurrencies: list[int] = None, max_tokens: int = 256, repetitions: int = 3, warmup: int = 1, thinking_values: list[Optional[bool]] = None, extra_bodies: list[str] = None, output_path: Optional[str] = None, fmt: str = 'table', do_validate: bool = True, scrape: bool = True, tag: Optional[str] = None, override_fields: Optional[dict] = None, system_prompt_file: Optional[str] = None, skip_preflight_token_count: bool = False) -> list[BenchServeResult] |
Run the full bench-serve sweep against a running vllm-mlx server. | #L2221-L2638 |
vllm_mlx.bench_serve.run_bench_serve._mean |
nested function | vllm_mlx.bench_serve.run_bench_serve._mean(key: str) -> float |
Nested Function `run_bench_serve._mean` calls `statistics.mean`; returns `statistics.mean(vals) if vals else 0.0`. | #L2522-L2526 |
vllm_mlx.bench_serve.run_bench_serve_workload |
function | async vllm_mlx.bench_serve.run_bench_serve_workload(*, url: str, workload_path: str, model: Optional[str] = None, output_path: Optional[str] = None, output_format: str = 'json', scrape: bool = True, include_content: bool = False, request_timeout_s: Optional[float] = 300.0, repetitions: int = 1, cache_policy: Optional[str] = None) -> dict |
Run a declarative workload against a running server. | #L1692-L1818 |
vllm_mlx.bench_serve.run_concurrent_requests |
function | async vllm_mlx.bench_serve.run_concurrent_requests(client: httpx.AsyncClient, base_url: str, prompts: list[list[dict]], model: str, concurrency: int, max_tokens: int = 256, enable_thinking: Optional[bool] = None, extra_body: Optional[dict] = None, do_validate: bool = True) -> list[dict] |
Fire ``concurrency`` concurrent streaming requests and collect results. | #L1279-L1342 |
vllm_mlx.bench_serve.run_concurrent_requests._single |
nested function | async vllm_mlx.bench_serve.run_concurrent_requests._single(messages: list[dict]) -> dict |
Nested Function `run_concurrent_requests._single` calls `stream_chat_completion`, `validate_response`, `result.get`, `str`; awaits asynchronous work; has 2 explicit return paths. | #L1315-L1339 |
vllm_mlx.bench_serve.run_workload_case |
function | async vllm_mlx.bench_serve.run_workload_case(client: httpx.AsyncClient, base_url: str, *, workload: Workload, case: WorkloadCase, model: str, runtime: dict, hardware: dict, run_id: str, timestamp: str, repetition: int = 0, scrape: bool = True, include_content: bool = False, cache_reset: Optional[dict] = None) -> dict |
Run one workload case and return a JSON-serializable result. | #L1518-L1593 |
vllm_mlx.bench_serve.scrape_metrics |
function | async vllm_mlx.bench_serve.scrape_metrics(client: httpx.AsyncClient, base_url: str) -> dict |
Scrape Prometheus metrics from the server. | #L645-L661 |
vllm_mlx.bench_serve.stream_chat_completion |
function | async vllm_mlx.bench_serve.stream_chat_completion(client: httpx.AsyncClient, base_url: str, messages: list[dict], model: str, max_tokens: int = 256, enable_thinking: Optional[bool] = None, extra_body: Optional[dict] = None, timeout_s: Optional[float] = None) -> dict |
Send a streaming chat completion and collect per-token timing data. | #L892-L1012 |
vllm_mlx.bench_serve.stream_chat_completion._consume_stream |
nested function | async vllm_mlx.bench_serve.stream_chat_completion._consume_stream() -> None |
Nested Function `stream_chat_completion._consume_stream` calls `client.stream`, `response.raise_for_status`, `response.aiter_lines`, `parse_sse_line`. | #L946-L975 |
vllm_mlx.bench_serve.summarize_workload_results |
function | vllm_mlx.bench_serve.summarize_workload_results(results: list[dict]) -> dict |
Aggregate workload case records into stable qualification summary stats. | #L1651-L1689 |
vllm_mlx.bench_serve.validate_quality_checks |
function | vllm_mlx.bench_serve.validate_quality_checks(finish_reason: Optional[str], content: str, checks: Optional[dict], *, status_code: int = 200, tool_calls: Optional[list[dict]] = None) -> tuple[bool, list[str]] |
Validate content against generic workload quality checks. | #L1177-L1231 |
vllm_mlx.bench_serve.validate_response |
function | vllm_mlx.bench_serve.validate_response(finish_reason: Optional[str], content: str, status_code: int, *, tool_calls: Optional[list[dict]] = None) -> tuple[bool, str] |
Validate a single streaming response result. | #L1020-L1049 |
vllm_mlx.bench_serve.Workload |
class | vllm_mlx.bench_serve.Workload(name: str, description: str, defaults: dict, cases: list[WorkloadCase]) |
Normalized bench-serve workload manifest. | #L66-L72 |
vllm_mlx.bench_serve.WorkloadCase |
class | vllm_mlx.bench_serve.WorkloadCase(case_id: str, messages: list[dict], request_path: Optional[str] = None, max_tokens: Optional[int] = None, enable_thinking: Optional[bool] = None, extra_body: Optional[dict] = None, policy_timeout_ms: Optional[int] = None, checks: Optional[dict] = None, tags: tuple[str, ...] = ()) |
One declarative benchmark case for contract-style serving tests. | #L51-L62 |
vllm_mlx.bench_serve.write_sqlite |
function | vllm_mlx.bench_serve.write_sqlite(results: list[BenchServeResult], output_path: str) -> None |
Append prompt-sweep benchmark results to a SQLite database. | #L1999-L2009 |
vllm_mlx.bench_serve.write_workload_sqlite |
function | vllm_mlx.bench_serve.write_workload_sqlite(payload: dict, output_path: str) -> None |
Append workload result records to a SQLite database. | #L2183-L2193 |
vllm_mlx.benchmark.benchmark_mllm_resolution |
function | vllm_mlx.benchmark.benchmark_mllm_resolution(model, processor, config, base_image: Image.Image, width: int, height: int, max_tokens: int = 256, warmup: bool = False) -> MLLMBenchmarkResult |
Run MLLM benchmark for a specific resolution. | #L708-L800 |
vllm_mlx.benchmark.benchmark_single_prompt |
function | vllm_mlx.benchmark.benchmark_single_prompt(model, tokenizer, prompt: str, max_tokens: int = 256, temperature: float = 0.7) -> Optional[BenchmarkResult] |
Benchmark a single prompt with detailed timing. | #L328-L394 |
vllm_mlx.benchmark.benchmark_video_config |
function | vllm_mlx.benchmark.benchmark_video_config(model, video_path: str, fps: float, max_frames: int, config_name: str, video_info: dict, max_tokens: int = 150, warmup: bool = False) -> VideoBenchmarkResult |
Run a single video benchmark configuration. | #L1097-L1162 |
vllm_mlx.benchmark.BenchmarkResult |
class | vllm_mlx.benchmark.BenchmarkResult(prompt: str, prompt_tokens: int, generated_tokens: int, ttft: float, total_time: float, tpot: float = 0.0, generation_tps: float = 0.0, processing_tps: float = 0.0) |
Results from a single benchmark run. | #L235-L268 |
vllm_mlx.benchmark.BenchmarkResult.__post_init__ |
method | vllm_mlx.benchmark.BenchmarkResult.__post_init__() -> not annotated |
Method `BenchmarkResult.__post_init__` updates `self.tpot`, `self.generation_tps`, `self.processing_tps`. | #L251-L268 |
vllm_mlx.benchmark.BenchmarkSummary |
class | vllm_mlx.benchmark.BenchmarkSummary(model_name: str, num_runs: int, total_prompt_tokens: int, total_generated_tokens: int, total_time: float, ttft_mean: float, ttft_min: float, ttft_max: float, ttft_p50: float, ttft_p95: float, tpot_mean: float, tpot_min: float, tpot_max: float, generation_tps_mean: float, generation_tps_max: float, processing_tps_mean: float, latency_mean: float, latency_min: float, latency_max: float, latency_p50: float, latency_p95: float, total_throughput_tps: float, requests_per_second: float, hardware_chip: str = '', hardware_memory_gb: float = 0.0, hardware_bandwidth_gbs: float = 0.0, resources: ResourceMetrics = field(default_factory=ResourceMetrics)) |
Summary statistics across all benchmark runs. | #L272-L315 |
vllm_mlx.benchmark.calculate_percentile |
function | vllm_mlx.benchmark.calculate_percentile(data: list, percentile: float) -> float |
Calculate percentile from a list. | #L318-L325 |
vllm_mlx.benchmark.create_test_video |
function | vllm_mlx.benchmark.create_test_video(duration: float = 10.0, fps: float = 30.0, width: int = 640, height: int = 480) -> str |
Create a synthetic test video with colored frames and text. | #L1002-L1056 |
vllm_mlx.benchmark.download_test_image |
function | vllm_mlx.benchmark.download_test_image(url: str, timeout: int = 30) -> Image.Image |
Download image from URL and return PIL Image. | #L677-L684 |
vllm_mlx.benchmark.download_video |
function | vllm_mlx.benchmark.download_video(url: str, timeout: int = 120) -> str |
Download video from URL and return local path. | #L1059-L1075 |
vllm_mlx.benchmark.get_mlx_memory_info |
function | vllm_mlx.benchmark.get_mlx_memory_info(reset_peak: bool = True) -> dict |
Get MLX memory usage information. | #L98-L135 |
vllm_mlx.benchmark.get_process_memory |
function | vllm_mlx.benchmark.get_process_memory() -> float |
Get current process memory usage in GB. | #L138-L147 |
vllm_mlx.benchmark.get_system_memory |
function | vllm_mlx.benchmark.get_system_memory() -> tuple[float, float] |
Get system memory (used, total) in GB. | #L150-L159 |
vllm_mlx.benchmark.get_video_info |
function | vllm_mlx.benchmark.get_video_info(video_path: str) -> dict |
Get information about a video file. | #L1078-L1094 |
vllm_mlx.benchmark.image_to_base64 |
function | vllm_mlx.benchmark.image_to_base64(img: Image.Image, format: str = 'JPEG') -> str |
Convert PIL Image to base64 data URL. | #L692-L705 |
vllm_mlx.benchmark.is_mllm_model |
function | vllm_mlx.benchmark.is_mllm_model(model_name: str) -> bool |
Check if model name indicates a multimodal language model. | #L651-L657 |
vllm_mlx.benchmark.main |
function | vllm_mlx.benchmark.main() -> not annotated |
Run the benchmark. | #L1444-L1680 |
vllm_mlx.benchmark.MLLMBenchmarkResult |
class | vllm_mlx.benchmark.MLLMBenchmarkResult(resolution: str, width: int, height: int, pixels: int, time_seconds: float, tokens_generated: int, tokens_per_second: float, response_preview: str, memory_gb: float = 0.0, mlx_memory_gb: float = 0.0) |
Result from a single MLLM benchmark run. | #L661-L674 |
vllm_mlx.benchmark.print_mllm_summary |
function | vllm_mlx.benchmark.print_mllm_summary(results: list[MLLMBenchmarkResult], model_name: str) -> not annotated |
Print MLLM benchmark summary. | #L916-L975 |
vllm_mlx.benchmark.print_summary |
function | vllm_mlx.benchmark.print_summary(summary: BenchmarkSummary) -> not annotated |
Print a formatted summary of benchmark results using tabulate. | #L1349-L1441 |
vllm_mlx.benchmark.print_video_summary |
function | vllm_mlx.benchmark.print_video_summary(results: list[VideoBenchmarkResult], model_name: str) -> not annotated |
Print video benchmark summary. | #L1288-L1341 |
vllm_mlx.benchmark.reset_mlx_peak_memory |
function | vllm_mlx.benchmark.reset_mlx_peak_memory() -> not annotated |
Reset MLX peak memory counter. | #L83-L95 |
vllm_mlx.benchmark.resize_image |
function | vllm_mlx.benchmark.resize_image(img: Image.Image, width: int, height: int) -> Image.Image |
Resize image to specified dimensions. | #L687-L689 |
vllm_mlx.benchmark.ResourceMetrics |
class | vllm_mlx.benchmark.ResourceMetrics(process_memory_gb: float = 0.0, mlx_cache_gb: float = 0.0, mlx_peak_memory_gb: float = 0.0, system_memory_used_gb: float = 0.0, system_memory_total_gb: float = 0.0) |
Resource usage metrics during benchmark. | #L72-L80 |
vllm_mlx.benchmark.ResourceMonitor |
class | vllm_mlx.benchmark.ResourceMonitor() |
Monitor system resources during benchmark runs. | #L162-L213 |
vllm_mlx.benchmark.ResourceMonitor.__init__ |
method | vllm_mlx.benchmark.ResourceMonitor.__init__() -> not annotated |
Method `ResourceMonitor.__init__` updates `self.samples`, `self._start_time`, `self._start_memory`. | #L165-L168 |
vllm_mlx.benchmark.ResourceMonitor.get_summary |
method | vllm_mlx.benchmark.ResourceMonitor.get_summary() -> ResourceMetrics |
Get summary of all samples. | #L194-L213 |
vllm_mlx.benchmark.ResourceMonitor.sample |
method | vllm_mlx.benchmark.ResourceMonitor.sample() -> ResourceMetrics |
Take a resource sample. | #L178-L192 |
vllm_mlx.benchmark.ResourceMonitor.start |
method | vllm_mlx.benchmark.ResourceMonitor.start() -> not annotated |
Start monitoring. | #L170-L176 |
vllm_mlx.benchmark.run_benchmark |
function | vllm_mlx.benchmark.run_benchmark(model_name: str, num_prompts: int = 5, max_tokens: int = 256, temperature: float = 0.7, warmup_runs: int = 1) -> Optional[BenchmarkSummary] |
Run the full benchmark suite. | #L397-L610 |
vllm_mlx.benchmark.run_mllm_benchmark |
function | vllm_mlx.benchmark.run_mllm_benchmark(model_name: str, quick: bool = False, max_tokens: int = 256, warmup_runs: int = 1) -> list[MLLMBenchmarkResult] |
Run MLLM benchmark across multiple image resolutions. | #L803-L913 |
vllm_mlx.benchmark.run_video_benchmark |
function | vllm_mlx.benchmark.run_video_benchmark(model_name: str, video_url: str = None, video_path: str = None, quick: bool = False, max_tokens: int = 150, warmup_runs: int = 1) -> list[VideoBenchmarkResult] |
Run video benchmark across multiple frame configurations. | #L1165-L1285 |
vllm_mlx.benchmark.VideoBenchmarkResult |
class | vllm_mlx.benchmark.VideoBenchmarkResult(config_name: str, fps: float, max_frames: int, frames_extracted: int, video_duration: float, time_seconds: float, prompt_tokens: int, completion_tokens: int, tokens_per_second: float, response_preview: str, memory_gb: float = 0.0, mlx_memory_gb: float = 0.0) |
Result from a single video benchmark run. | #L984-L999 |
vllm_mlx.cli.bench_command |
function | vllm_mlx.cli.bench_command(args) -> not annotated |
Run benchmark. | #L506-L625 |
vllm_mlx.cli.bench_command.run_benchmark |
nested function | async vllm_mlx.cli.bench_command.run_benchmark() -> not annotated |
Nested Function `bench_command.run_benchmark` calls `print`, `load`, `SchedulerConfig`, `EngineConfig`; awaits asynchronous work. | #L520-L623 |
vllm_mlx.cli.bench_command.run_benchmark.get_output |
nested function | async vllm_mlx.cli.bench_command.run_benchmark.get_output(rid) -> not annotated |
Nested Function `bench_command.run_benchmark.get_output` calls `engine.stream_outputs`; has 2 explicit return paths. | #L597-L601 |
vllm_mlx.cli.bench_detok_command |
function | vllm_mlx.cli.bench_detok_command(args) -> not annotated |
Benchmark streaming detokenizer optimization. | #L628-L740 |
vllm_mlx.cli.bench_kv_cache_command |
function | vllm_mlx.cli.bench_kv_cache_command(args) -> not annotated |
Benchmark KV cache quantization memory savings and quality. | #L743-L886 |
vllm_mlx.cli.bench_serve_command |
function | vllm_mlx.cli.bench_serve_command(args) -> not annotated |
Run serving benchmark. | #L889-L990 |
vllm_mlx.cli.create_parser |
function | vllm_mlx.cli.create_parser() -> argparse.ArgumentParser |
Build the top-level CLI parser. | #L993-L2105 |
vllm_mlx.cli.download_command |
function | vllm_mlx.cli.download_command(args) -> not annotated |
Download a model to local cache without starting a server. | #L396-L410 |
vllm_mlx.cli.main |
function | vllm_mlx.cli.main() -> not annotated |
Parse the command line and dispatch to the selected vllm-mlx command. | #L2112-L2134 |
vllm_mlx.cli.model_command |
function | vllm_mlx.cli.model_command(args) -> not annotated |
Run model lifecycle helper commands. | #L413-L503 |
vllm_mlx.cli.serve_command |
function | vllm_mlx.cli.serve_command(args) -> not annotated |
Start the OpenAI-compatible server. | #L22-L393 |
vllm_mlx.cli_arg_types.make_json_object_arg_parser |
function | vllm_mlx.cli_arg_types.make_json_object_arg_parser(option_name: str) -> Callable[[str], dict[str, Any]] |
Create an argparse type parser for JSON object options. | #L25-L31 |
vllm_mlx.cli_arg_types.make_json_object_arg_parser._parser |
nested function | vllm_mlx.cli_arg_types.make_json_object_arg_parser._parser(value: str) -> dict[str, Any] |
Nested Function `make_json_object_arg_parser._parser` calls `parse_json_object_arg`; returns `parse_json_object_arg(value, option_name)`. | #L28-L29 |
vllm_mlx.cli_arg_types.make_positive_int_arg_parser |
function | vllm_mlx.cli_arg_types.make_positive_int_arg_parser(option_name: str) -> Callable[[str], int] |
Create an argparse type parser for positive integer options. | #L56-L62 |
vllm_mlx.cli_arg_types.make_positive_int_arg_parser._parser |
nested function | vllm_mlx.cli_arg_types.make_positive_int_arg_parser._parser(value: str) -> int |
Nested Function `make_positive_int_arg_parser._parser` calls `parse_positive_int_arg`; returns `parse_positive_int_arg(value, option_name)`. | #L59-L60 |
vllm_mlx.cli_arg_types.parse_json_object_arg |
function | vllm_mlx.cli_arg_types.parse_json_object_arg(value: str, option_name: str) -> dict[str, Any] |
Parse and validate that an option value is a JSON object. | #L10-L22 |
vllm_mlx.cli_arg_types.parse_positive_int_arg |
function | vllm_mlx.cli_arg_types.parse_positive_int_arg(value: str, option_name: str) -> int |
Parse and validate that an option value is a positive integer. | #L45-L53 |
vllm_mlx.cli_arg_types.positive_int_arg |
function | vllm_mlx.cli_arg_types.positive_int_arg(value: str) -> int |
Parse an argparse integer that must be greater than zero. | #L34-L42 |
vllm_mlx.constrained.cache._build_regular_tokens_list |
function | vllm_mlx.constrained.cache._build_regular_tokens_list(tokenizer: Any, vocab_size: int) -> list[tuple[int, str, bool]] |
Enumerate the regular (non-special) tokens in the vocabulary and produce the ``(token_id, decoded_with_leading_space_marker, is_word_start)`` tuples required by ``TokenEnforcerTokenizerData``. | #L56-L95 |
vllm_mlx.constrained.cache._decode_function |
function | vllm_mlx.constrained.cache._decode_function(tokenizer: Any, tokens: list[int]) -> str |
Function `_decode_function` calls `tokenizer.decode`, `isinstance`, `decoded.rstrip`; has 2 explicit return paths. | #L128-L133 |
vllm_mlx.constrained.cache._get_eos_token_id |
function | vllm_mlx.constrained.cache._get_eos_token_id(tokenizer: Any) -> int | list[int] |
Function `_get_eos_token_id` calls `getattr`, `isinstance`, `list`; has 3 explicit return paths. | #L98-L111 |
vllm_mlx.constrained.cache._get_vocab_size |
function | vllm_mlx.constrained.cache._get_vocab_size(tokenizer: Any) -> int |
Function `_get_vocab_size` calls `getattr`, `isinstance`, `len`, `callable`; can raise `ValueError`; has 3 explicit return paths. | #L114-L125 |
vllm_mlx.constrained.cache._resolve_inner_tokenizer |
function | vllm_mlx.constrained.cache._resolve_inner_tokenizer(tokenizer: Any) -> Any |
VLM processors wrap the actual tokenizer under ``processor.tokenizer``. | #L26-L53 |
vllm_mlx.constrained.cache.clear_cache |
function | vllm_mlx.constrained.cache.clear_cache() -> None |
Drop the cache (mainly for tests). | #L183-L186 |
vllm_mlx.constrained.cache.get_tokenizer_data |
function | vllm_mlx.constrained.cache.get_tokenizer_data(tokenizer: Any) -> Any | None |
Return a cached ``TokenEnforcerTokenizerData`` for ``tokenizer``. | #L136-L180 |
vllm_mlx.constrained.json_schema_processor._canonical_schema_key |
function | vllm_mlx.constrained.json_schema_processor._canonical_schema_key(schema: dict | None) -> str |
Function `_canonical_schema_key` calls `json.dumps(schema, sort_keys=True, separators=(',', ':')).encode`, `json.dumps`, `hashlib.sha256(blob).hexdigest`, `hashlib.sha256`; has 2 explicit return paths. | #L50-L54 |
vllm_mlx.constrained.json_schema_processor._collect_property_names |
function | vllm_mlx.constrained.json_schema_processor._collect_property_names(schema: dict | None) -> set[str] |
Collect all property names declared anywhere in *schema*. | #L241-L247 |
vllm_mlx.constrained.json_schema_processor._complete_json_eos_logits |
function | vllm_mlx.constrained.json_schema_processor._complete_json_eos_logits(eos_set: set[int], suffix: list[int], logits: mx.array, is_complete_json, build_allow_mask) -> mx.array | None |
Function `_complete_json_eos_logits` calls `is_complete_json`, `_eos_logits`; has 2 explicit return paths. | #L267-L276 |
vllm_mlx.constrained.json_schema_processor._eos_logits |
function | vllm_mlx.constrained.json_schema_processor._eos_logits(eos_set: set[int], logits: mx.array, build_allow_mask) -> mx.array | None |
Function `_eos_logits` calls `build_allow_mask`, `sorted`; has 2 explicit return paths. | #L279-L290 |
vllm_mlx.constrained.json_schema_processor._eos_logits_or_original |
function | vllm_mlx.constrained.json_schema_processor._eos_logits_or_original(eos_set: set[int], logits: mx.array, build_allow_mask) -> mx.array |
Function `_eos_logits_or_original` calls `_eos_logits`; returns `logits if masked is None else masked`. | #L293-L299 |
vllm_mlx.constrained.json_schema_processor._force_no_additional_properties |
function | vllm_mlx.constrained.json_schema_processor._force_no_additional_properties(schema: dict) -> dict |
Return a deep copy of *schema* with ``additionalProperties: false`` injected into every object-type sub-schema that declares ``properties``. | #L210-L224 |
vllm_mlx.constrained.json_schema_processor._get_or_build_parser |
function | vllm_mlx.constrained.json_schema_processor._get_or_build_parser(schema: dict | None) -> tuple[dict, Any] |
Return (parser_schema, JsonSchemaParser) for ``schema``, memoised. | #L57-L73 |
vllm_mlx.constrained.json_schema_processor._inject_no_additional_props |
function | vllm_mlx.constrained.json_schema_processor._inject_no_additional_props(node: Any) -> None |
Recursively inject ``additionalProperties: false`` into *node*. | #L227-L238 |
vllm_mlx.constrained.json_schema_processor._simplify_schema |
function | vllm_mlx.constrained.json_schema_processor._simplify_schema(schema: dict) -> dict |
Pre-process a JSON Schema for ``lm-format-enforcer`` compatibility. | #L97-L207 |
vllm_mlx.constrained.json_schema_processor._simplify_schema._resolve |
nested function | vllm_mlx.constrained.json_schema_processor._simplify_schema._resolve(node: Any, depth: int = 0) -> Any |
Nested Function `_simplify_schema._resolve` calls `isinstance`, `ref.split`, `len`, `resolving.add`; has 3 explicit return paths. | #L121-L205 |
vllm_mlx.constrained.json_schema_processor._walk_properties |
function | vllm_mlx.constrained.json_schema_processor._walk_properties(node: Any, names: set[str]) -> None |
Function `_walk_properties` calls `isinstance`, `node.get`, `names.update`, `props.keys`; returns `None`. | #L250-L264 |
vllm_mlx.constrained.json_schema_processor.is_available |
function | vllm_mlx.constrained.json_schema_processor.is_available() -> bool |
Return ``True`` iff ``lm-format-enforcer`` is importable. | #L76-L82 |
vllm_mlx.constrained.json_schema_processor.JSONSchemaLogitsProcessor |
class | vllm_mlx.constrained.json_schema_processor.JSONSchemaLogitsProcessor(schema: dict | None, tokenizer: Any) |
Logits processor that constrains generation to valid JSON. | #L302-L924 |
vllm_mlx.constrained.json_schema_processor.JSONSchemaLogitsProcessor.__call__ |
method | vllm_mlx.constrained.json_schema_processor.JSONSchemaLogitsProcessor.__call__(tokens: mx.array, logits: mx.array) -> mx.array |
Apply the allowed-tokens mask to ``logits``. | #L814-L910 |
vllm_mlx.constrained.json_schema_processor.JSONSchemaLogitsProcessor.__init__ |
method | vllm_mlx.constrained.json_schema_processor.JSONSchemaLogitsProcessor.__init__(schema: dict | None, tokenizer: Any) -> None |
Method `JSONSchemaLogitsProcessor.__init__` updates `self._tokenizer`, `self._schema`, `self._tok_data`, `self._disabled`; calls `is_available`, `LMFormatEnforcerNotAvailableError`, `get_tokenizer_data`, `_get_or_build_parser`; can raise `LMFormatEnforcerNotAvailableError`. | #L317-L414 |
vllm_mlx.constrained.json_schema_processor.JSONSchemaLogitsProcessor._build_allow_mask |
method | vllm_mlx.constrained.json_schema_processor.JSONSchemaLogitsProcessor._build_allow_mask(allowed: list[int], vocab_size: int) -> mx.array |
Build a 1-D mask of length ``vocab_size`` where allowed positions are ``0`` and disallowed positions are ``-inf``. | #L795-L810 |
vllm_mlx.constrained.json_schema_processor.JSONSchemaLogitsProcessor._decode_suffix |
method | vllm_mlx.constrained.json_schema_processor.JSONSchemaLogitsProcessor._decode_suffix(suffix: list[int]) -> str | None |
Decode suffix tokens to text. | #L444-L493 |
vllm_mlx.constrained.json_schema_processor.JSONSchemaLogitsProcessor._decode_token_cached |
method | vllm_mlx.constrained.json_schema_processor.JSONSchemaLogitsProcessor._decode_token_cached(tok_id: int) -> str | None |
Return the decoded text for a single token (cached). | #L428-L442 |
vllm_mlx.constrained.json_schema_processor.JSONSchemaLogitsProcessor._filter_at_key_context |
method | vllm_mlx.constrained.json_schema_processor.JSONSchemaLogitsProcessor._filter_at_key_context(context: str, suffix: list[int], allowed: list[int]) -> list[int] |
Apply schema-aware filtering when in key-related context. | #L645-L662 |
vllm_mlx.constrained.json_schema_processor.JSONSchemaLogitsProcessor._filter_in_key_tokens |
method | vllm_mlx.constrained.json_schema_processor.JSONSchemaLogitsProcessor._filter_in_key_tokens(suffix: list[int], allowed: list[int]) -> list[int] |
Filter tokens when we're inside an open key string. | #L719-L758 |
vllm_mlx.constrained.json_schema_processor.JSONSchemaLogitsProcessor._filter_key_start_tokens |
method | vllm_mlx.constrained.json_schema_processor.JSONSchemaLogitsProcessor._filter_key_start_tokens(suffix: list[int], allowed: list[int]) -> list[int] |
Filter tokens at key-start position. | #L664-L717 |
vllm_mlx.constrained.json_schema_processor.JSONSchemaLogitsProcessor._filter_nonprogress_whitespace_tokens |
method | vllm_mlx.constrained.json_schema_processor.JSONSchemaLogitsProcessor._filter_nonprogress_whitespace_tokens(suffix: list[int], allowed: list[int]) -> list[int] |
Stop constrained JSON from spending a long run on pure whitespace. | #L764-L793 |
vllm_mlx.constrained.json_schema_processor.JSONSchemaLogitsProcessor._get_json_context |
method | vllm_mlx.constrained.json_schema_processor.JSONSchemaLogitsProcessor._get_json_context(suffix: list[int]) -> str |
Determine the JSON structural context of the current suffix. | #L522-L643 |
vllm_mlx.constrained.json_schema_processor.JSONSchemaLogitsProcessor._is_valid_key_prefix |
method | vllm_mlx.constrained.json_schema_processor.JSONSchemaLogitsProcessor._is_valid_key_prefix(prefix: str) -> bool |
Return True if *prefix* is a prefix of at least one valid key name. | #L760-L762 |
vllm_mlx.constrained.json_schema_processor.JSONSchemaLogitsProcessor._suffix |
method | vllm_mlx.constrained.json_schema_processor.JSONSchemaLogitsProcessor._suffix(tokens_list: list[int]) -> list[int] |
Return the slice of ``tokens`` that corresponds to generated output. | #L418-L426 |
vllm_mlx.constrained.json_schema_processor.JSONSchemaLogitsProcessor._suffix_is_complete_json |
method | vllm_mlx.constrained.json_schema_processor.JSONSchemaLogitsProcessor._suffix_is_complete_json(suffix: list[int]) -> bool |
Return True if the decoded ``suffix`` parses as a complete JSON value. | #L495-L520 |
vllm_mlx.constrained.json_schema_processor.JSONSchemaLogitsProcessor.schema |
method | vllm_mlx.constrained.json_schema_processor.JSONSchemaLogitsProcessor.schema() -> dict | None |
Return the normalized JSON Schema enforced for this request. | #L915-L918 |
vllm_mlx.constrained.json_schema_processor.JSONSchemaLogitsProcessor.vocab_size |
method | vllm_mlx.constrained.json_schema_processor.JSONSchemaLogitsProcessor.vocab_size() -> int |
Return the tokenizer vocabulary size used to construct masks. | #L921-L924 |
vllm_mlx.constrained.json_schema_processor.LMFormatEnforcerNotAvailableError |
class | vllm_mlx.constrained.json_schema_processor.LMFormatEnforcerNotAvailableError() |
Raised when ``lm-format-enforcer`` is required but not installed. | #L33-L34 |
vllm_mlx.constrained.thinking_processor.BoundedSuffixMatcher |
class | vllm_mlx.constrained.thinking_processor.BoundedSuffixMatcher(target_ids: list[int]) |
Detect a target token sequence in a stream using a rolling suffix buffer. | #L16-L48 |
vllm_mlx.constrained.thinking_processor.BoundedSuffixMatcher.__init__ |
method | vllm_mlx.constrained.thinking_processor.BoundedSuffixMatcher.__init__(target_ids: list[int]) -> None |
Method `BoundedSuffixMatcher.__init__` updates `self.target`, `self._max_len`, `self._buf`; calls `ValueError`, `tuple`, `len`, `deque`; can raise `ValueError`. | #L25-L30 |
vllm_mlx.constrained.thinking_processor.BoundedSuffixMatcher.feed |
method | vllm_mlx.constrained.thinking_processor.BoundedSuffixMatcher.feed(token_id: int) -> bool |
Feed one token. | #L32-L35 |
vllm_mlx.constrained.thinking_processor.BoundedSuffixMatcher.reset |
method | vllm_mlx.constrained.thinking_processor.BoundedSuffixMatcher.reset() -> None |
Clear the buffer. | #L37-L39 |
vllm_mlx.constrained.thinking_processor.BoundedSuffixMatcher.restore |
method | vllm_mlx.constrained.thinking_processor.BoundedSuffixMatcher.restore(state: tuple[int, ...]) -> None |
Restore the suffix buffer from a previous snapshot. | #L45-L48 |
vllm_mlx.constrained.thinking_processor.BoundedSuffixMatcher.snapshot |
method | vllm_mlx.constrained.thinking_processor.BoundedSuffixMatcher.snapshot() -> tuple[int, ...] |
Return a serializable copy of the current suffix buffer. | #L41-L43 |
vllm_mlx.constrained.thinking_processor.Phase |
class | vllm_mlx.constrained.thinking_processor.Phase() |
Thinking lifecycle phases. | #L51-L57 |
vllm_mlx.constrained.thinking_processor.ThinkingAwareLogitsProcessor |
class | vllm_mlx.constrained.thinking_processor.ThinkingAwareLogitsProcessor(start_token_ids: list[int], end_token_ids: list[int], thinking_token_budget: int, inner: Callable[[mx.array, mx.array], mx.array] | None = None, vocab_size: int = 152064, prompt_has_think_tag: bool = False, no_final_content_token_limit: int | None = None) |
Unified logits processor for thinking-model lifecycle management. | #L60-L287 |
vllm_mlx.constrained.thinking_processor.ThinkingAwareLogitsProcessor.__call__ |
method | vllm_mlx.constrained.thinking_processor.ThinkingAwareLogitsProcessor.__call__(tokens: mx.array, logits: mx.array) -> mx.array |
Method `ThinkingAwareLogitsProcessor.__call__` calls `self._force_transition`, `self._call_inner`, `self._sync_to_tokens`; has 3 explicit return paths. | #L152-L170 |
vllm_mlx.constrained.thinking_processor.ThinkingAwareLogitsProcessor.__init__ |
method | vllm_mlx.constrained.thinking_processor.ThinkingAwareLogitsProcessor.__init__(start_token_ids: list[int], end_token_ids: list[int], thinking_token_budget: int, inner: Callable[[mx.array, mx.array], mx.array] | None = None, vocab_size: int = 152064, prompt_has_think_tag: bool = False, no_final_content_token_limit: int | None = None) -> None |
Method `ThinkingAwareLogitsProcessor.__init__` updates `self._start_matcher`, `self._end_matcher`, `self._end_token_ids`, `self._content_phase_mask_ids`; calls `BoundedSuffixMatcher`, `list`, `tuple`, `dict.fromkeys`. | #L92-L129 |
vllm_mlx.constrained.thinking_processor.ThinkingAwareLogitsProcessor._advance_with_token |
method | vllm_mlx.constrained.thinking_processor.ThinkingAwareLogitsProcessor._advance_with_token(token_id: int) -> None |
Method `ThinkingAwareLogitsProcessor._advance_with_token` updates `self._state`, `self._transition_index`, `self._thinking_tokens`, `self.watchdog_was_enforced`; calls `self._start_matcher.feed`, `self._end_matcher.feed`, `len`, `self._end_matcher.reset`; returns `None`. | #L254-L287 |
vllm_mlx.constrained.thinking_processor.ThinkingAwareLogitsProcessor._call_inner |
method | vllm_mlx.constrained.thinking_processor.ThinkingAwareLogitsProcessor._call_inner(tokens: mx.array, logits: mx.array) -> mx.array |
Delegate to inner processor if present. | #L184-L188 |
vllm_mlx.constrained.thinking_processor.ThinkingAwareLogitsProcessor._force_transition |
method | vllm_mlx.constrained.thinking_processor.ThinkingAwareLogitsProcessor._force_transition(logits: mx.array) -> mx.array |
Force the next token in the reasoning end sequence. | #L172-L182 |
vllm_mlx.constrained.thinking_processor.ThinkingAwareLogitsProcessor._mask_content_phase_control_tokens |
method | vllm_mlx.constrained.thinking_processor.ThinkingAwareLogitsProcessor._mask_content_phase_control_tokens(logits: mx.array) -> mx.array |
Prevent reserved think-tag starts from leaking into final content. | #L190-L197 |
vllm_mlx.constrained.thinking_processor.ThinkingAwareLogitsProcessor._restore_snapshot |
method | vllm_mlx.constrained.thinking_processor.ThinkingAwareLogitsProcessor._restore_snapshot(processed_len: int) -> None |
Method `ThinkingAwareLogitsProcessor._restore_snapshot` updates `self._state`, `self._thinking_tokens`, `self._transition_index`, `self.watchdog_was_enforced`; calls `min`, `len`, `self._start_matcher.restore`, `self._end_matcher.restore`. | #L211-L229 |
vllm_mlx.constrained.thinking_processor.ThinkingAwareLogitsProcessor._snapshot_state |
method | vllm_mlx.constrained.thinking_processor.ThinkingAwareLogitsProcessor._snapshot_state() -> tuple[Phase, int, int, tuple[int, ...], tuple[int, ...], bool] |
Method `ThinkingAwareLogitsProcessor._snapshot_state` calls `self._start_matcher.snapshot`, `self._end_matcher.snapshot`; returns `(self._state, self._thinking_tokens, self._transition_index, self._start_matcher.snapshot(), self._end_matcher.snapshot…`. | #L199-L209 |
vllm_mlx.constrained.thinking_processor.ThinkingAwareLogitsProcessor._sync_to_tokens |
method | vllm_mlx.constrained.thinking_processor.ThinkingAwareLogitsProcessor._sync_to_tokens(tokens: mx.array) -> None |
Method `ThinkingAwareLogitsProcessor._sync_to_tokens` updates `self._processed_len`; calls `int`, `tokens.tolist`, `min`, `self._restore_snapshot`; returns `None`. | #L231-L252 |
vllm_mlx.constrained.thinking_processor.ThinkingAwareLogitsProcessor.is_retired |
method | vllm_mlx.constrained.thinking_processor.ThinkingAwareLogitsProcessor.is_retired() -> bool |
True when the processor is in CONTENT with no inner constraint. | #L144-L150 |
vllm_mlx.constrained.thinking_processor.ThinkingAwareLogitsProcessor.state |
method | vllm_mlx.constrained.thinking_processor.ThinkingAwareLogitsProcessor.state() -> Phase |
Return the current reasoning lifecycle phase. | #L132-L135 |
vllm_mlx.constrained.thinking_processor.ThinkingAwareLogitsProcessor.thinking_tokens |
method | vllm_mlx.constrained.thinking_processor.ThinkingAwareLogitsProcessor.thinking_tokens() -> int |
Return the number of generated tokens counted as reasoning. | #L138-L141 |
vllm_mlx.embedding.EmbeddingEngine |
class | vllm_mlx.embedding.EmbeddingEngine(model_name: str) |
Wrapper around mlx-embeddings for text embedding generation. | #L19-L131 |
vllm_mlx.embedding.EmbeddingEngine.__init__ |
method | vllm_mlx.embedding.EmbeddingEngine.__init__(model_name: str) -> not annotated |
Method `EmbeddingEngine.__init__` updates `self.model_name`, `self._model`, `self._tokenizer`, `self._max_length`. | #L27-L31 |
vllm_mlx.embedding.EmbeddingEngine._ensure_loaded |
method | vllm_mlx.embedding.EmbeddingEngine._ensure_loaded() -> None |
Method `EmbeddingEngine._ensure_loaded` calls `self.load`. | #L49-L51 |
vllm_mlx.embedding.EmbeddingEngine._resolve_max_length |
method | vllm_mlx.embedding.EmbeddingEngine._resolve_max_length() -> int |
Tokenizer truncation length from the model config (cached). | #L53-L60 |
vllm_mlx.embedding.EmbeddingEngine.count_tokens |
method | vllm_mlx.embedding.EmbeddingEngine.count_tokens(texts: str | list[str]) -> int |
Approximate token count for usage reporting. | #L111-L131 |
vllm_mlx.embedding.EmbeddingEngine.embed |
method | vllm_mlx.embedding.EmbeddingEngine.embed(texts: str | list[str]) -> list[list[float]] |
Generate embeddings for one or more texts. | #L62-L109 |
vllm_mlx.embedding.EmbeddingEngine.is_loaded |
method | vllm_mlx.embedding.EmbeddingEngine.is_loaded() -> bool |
Return whether the embedding model has been loaded. | #L34-L37 |
vllm_mlx.embedding.EmbeddingEngine.load |
method | vllm_mlx.embedding.EmbeddingEngine.load() -> None |
Load the embedding model and tokenizer. | #L39-L47 |
vllm_mlx.endpoint_model_policies._reject_unknown_audio_model |
function | vllm_mlx.endpoint_model_policies._reject_unknown_audio_model(endpoint: str, requested_model: str, supported_aliases: dict[str, str]) -> None |
Function `_reject_unknown_audio_model` calls `', '.join`, `sorted`, `HTTPException`; can raise `HTTPException`. | #L66-L79 |
vllm_mlx.endpoint_model_policies._reject_unknown_embedding_model |
function | vllm_mlx.endpoint_model_policies._reject_unknown_embedding_model(requested_model: str) -> None |
Function `_reject_unknown_embedding_model` calls `', '.join`, `sorted`, `HTTPException`; can raise `HTTPException`. | #L53-L63 |
vllm_mlx.endpoint_model_policies._with_identity_aliases |
function | vllm_mlx.endpoint_model_policies._with_identity_aliases(model_map: dict[str, str]) -> dict[str, str] |
Function `_with_identity_aliases` calls `dict`, `model_map.values`; returns `expanded`. | #L42-L46 |
vllm_mlx.endpoint_model_policies.resolve_embedding_model_name |
function | vllm_mlx.endpoint_model_policies.resolve_embedding_model_name(requested_model: str, *, locked_model: str | None = None) -> str |
Resolve the embedding model for a request or raise HTTP 400. | #L82-L104 |
vllm_mlx.endpoint_model_policies.resolve_stt_model_name |
function | vllm_mlx.endpoint_model_policies.resolve_stt_model_name(requested_model: str) -> str |
Resolve an STT request model alias or configured model ID. | #L107-L111 |
vllm_mlx.endpoint_model_policies.resolve_tts_model_name |
function | vllm_mlx.endpoint_model_policies.resolve_tts_model_name(requested_model: str) -> str |
Resolve a TTS request model alias or configured model ID. | #L114-L118 |
vllm_mlx.engine.__getattr__ |
function | vllm_mlx.engine.__getattr__(name: str) -> not annotated |
Function `__getattr__` calls `AttributeError`; can raise `AttributeError`; has 3 explicit return paths. | #L34-L54 |
vllm_mlx.engine.base.BaseEngine |
class | vllm_mlx.engine.base.BaseEngine() |
Abstract base class for inference engines. | #L101-L288 |
vllm_mlx.engine.base.BaseEngine.abort_request |
method | async vllm_mlx.engine.base.BaseEngine.abort_request(request_id: str) -> bool |
Abort an active or queued request when the engine supports it. | #L286-L288 |
vllm_mlx.engine.base.BaseEngine.chat |
method | async vllm_mlx.engine.base.BaseEngine.chat(messages: list[dict[str, Any]], max_tokens: int = 256, temperature: float = 0.7, top_p: float = 0.9, tools: list[dict] | None = None, images: list[str] | None = None, videos: list[str] | None = None, **kwargs) -> GenerationOutput |
Chat completion (non-streaming). | #L215-L242 |
vllm_mlx.engine.base.BaseEngine.clear_runtime_caches |
method | vllm_mlx.engine.base.BaseEngine.clear_runtime_caches() -> dict[str, Any] | None |
Clear engine-managed runtime caches. | #L282-L284 |
vllm_mlx.engine.base.BaseEngine.generate |
method | async vllm_mlx.engine.base.BaseEngine.generate(prompt: str, max_tokens: int = 256, temperature: float = 0.7, top_p: float = 0.9, stop: list[str] | None = None, **kwargs) -> GenerationOutput |
Generate a complete response (non-streaming). | #L163-L186 |
vllm_mlx.engine.base.BaseEngine.get_cache_stats |
method | vllm_mlx.engine.base.BaseEngine.get_cache_stats() -> dict[str, Any] | None |
Get cache statistics. | #L278-L280 |
vllm_mlx.engine.base.BaseEngine.get_stats |
method | vllm_mlx.engine.base.BaseEngine.get_stats() -> dict[str, Any] |
Get engine statistics. | #L274-L276 |
vllm_mlx.engine.base.BaseEngine.is_mllm |
method | vllm_mlx.engine.base.BaseEngine.is_mllm() -> bool |
Check if this is a multimodal model. | #L117-L119 |
vllm_mlx.engine.base.BaseEngine.model_name |
method | vllm_mlx.engine.base.BaseEngine.model_name() -> str |
Get the model name. | #L111-L113 |
vllm_mlx.engine.base.BaseEngine.prepare_for_start |
method | vllm_mlx.engine.base.BaseEngine.prepare_for_start() -> None |
Run blocking startup work before async engine start. | #L143-L150 |
vllm_mlx.engine.base.BaseEngine.preserve_native_tool_format |
method | vllm_mlx.engine.base.BaseEngine.preserve_native_tool_format() -> bool |
Whether to preserve native tool message format. | #L128-L135 |
vllm_mlx.engine.base.BaseEngine.preserve_native_tool_format |
method | vllm_mlx.engine.base.BaseEngine.preserve_native_tool_format(value: bool) -> None |
Enable or disable preservation of model-native tool messages. | #L138-L141 |
vllm_mlx.engine.base.BaseEngine.start |
method | async vllm_mlx.engine.base.BaseEngine.start() -> None |
Start the engine (load model if not loaded). | #L153-L155 |
vllm_mlx.engine.base.BaseEngine.stop |
method | async vllm_mlx.engine.base.BaseEngine.stop() -> None |
Stop the engine and cleanup resources. | #L158-L160 |
vllm_mlx.engine.base.BaseEngine.stream_chat |
method | async vllm_mlx.engine.base.BaseEngine.stream_chat(messages: list[dict[str, Any]], max_tokens: int = 256, temperature: float = 0.7, top_p: float = 0.9, tools: list[dict] | None = None, images: list[str] | None = None, videos: list[str] | None = None, **kwargs) -> AsyncIterator[GenerationOutput] |
Stream chat completion token by token. | #L245-L272 |
vllm_mlx.engine.base.BaseEngine.stream_generate |
method | async vllm_mlx.engine.base.BaseEngine.stream_generate(prompt: str, max_tokens: int = 256, temperature: float = 0.7, top_p: float = 0.9, stop: list[str] | None = None, **kwargs) -> AsyncIterator[GenerationOutput] |
Stream generation token by token. | #L189-L212 |
vllm_mlx.engine.base.BaseEngine.tokenizer |
method | vllm_mlx.engine.base.BaseEngine.tokenizer() -> Any |
Get the tokenizer. | #L123-L125 |
vllm_mlx.engine.base.cleanup_startup_cancellation |
function | async vllm_mlx.engine.base.cleanup_startup_cancellation(cleanup: Callable[[], Awaitable[None]]) -> None |
Run startup cleanup without letting cleanup failures replace cancellation. | #L87-L98 |
vllm_mlx.engine.base.EngineBusy |
class | vllm_mlx.engine.base.EngineBusy() |
Raised when a serialized engine route is already serving a request. | #L40-L43 |
vllm_mlx.engine.base.GenerationOutput |
class | vllm_mlx.engine.base.GenerationOutput(text: str, tokens: list[int] = field(default_factory=list), prompt_tokens: int = 0, completion_tokens: int = 0, finish_reason: str | None = 'stop', mtp_drafts: int = 0, mtp_accepted: int = 0, new_text: str = '', finished: bool = True, mtp_drafts: int = 0, mtp_accepted: int = 0) |
Output from generation. | #L18-L37 |
vllm_mlx.engine.base.run_blocking_startup_work |
function | async vllm_mlx.engine.base.run_blocking_startup_work(work: Callable[[], Any]) -> None |
Run blocking startup work off-loop without leaking cancellation races. | #L70-L84 |
vllm_mlx.engine.base.suspend_cancellation |
function | vllm_mlx.engine.base.suspend_cancellation() -> not annotated |
Temporarily clear task cancellation so cleanup can finish deterministically. | #L47-L67 |
vllm_mlx.engine.batched._extract_media_from_messages |
function | vllm_mlx.engine.batched._extract_media_from_messages(messages: list[dict[str, Any]]) -> tuple |
Extract images, videos, and audio from OpenAI-format messages. | #L66-L137 |
vllm_mlx.engine.batched._normalize_tool_call_arguments_for_template |
function | vllm_mlx.engine.batched._normalize_tool_call_arguments_for_template(messages: list[dict]) -> list[dict] |
Normalize OpenAI tool-call replay for templates expecting mappings. | #L61-L63 |
vllm_mlx.engine.batched._resolve_metal_buffer_cache_limit |
function | vllm_mlx.engine.batched._resolve_metal_buffer_cache_limit(max_recommended: int, gpu_memory_utilization: float) -> tuple[int, str] |
Resolve the MLX retained-buffer cache cap for Metal startup. | #L35-L58 |
vllm_mlx.engine.batched.BatchedEngine |
class | vllm_mlx.engine.batched.BatchedEngine(model_name: str, trust_remote_code: bool = False, scheduler_config: Any | None = None, stream_interval: int = 1, force_mllm: bool = False, gpu_memory_utilization: float = 0.9) |
Batched engine for continuous batching. | #L178-L1231 |
vllm_mlx.engine.batched.BatchedEngine.__init__ |
method | vllm_mlx.engine.batched.BatchedEngine.__init__(model_name: str, trust_remote_code: bool = False, scheduler_config: Any | None = None, stream_interval: int = 1, force_mllm: bool = False, gpu_memory_utilization: float = 0.9) -> not annotated |
Initialize the batched engine. | #L189-L224 |
vllm_mlx.engine.batched.BatchedEngine._apply_chat_template |
method | vllm_mlx.engine.batched.BatchedEngine._apply_chat_template(messages: list[dict[str, Any]], tools: list[dict] | None = None, num_images: int = 0, num_audios: int = 0, chat_template_kwargs: dict[str, Any] | None = None, enable_thinking: bool | None = None) -> str |
Apply chat template to messages. | #L599-L687 |
vllm_mlx.engine.batched.BatchedEngine._compute_prefix_boundary |
method | vllm_mlx.engine.batched.BatchedEngine._compute_prefix_boundary(messages: list[dict[str, Any]], tools: list[dict] | None = None, chat_template_kwargs: dict[str, Any] | None = None) -> int |
Compute token count for the shared prefix across message variations. | #L986-L1046 |
vllm_mlx.engine.batched.BatchedEngine._configure_metal_memory_limits |
method | vllm_mlx.engine.batched.BatchedEngine._configure_metal_memory_limits() -> None |
Make MLX allocation failures graceful during startup. | #L513-L541 |
vllm_mlx.engine.batched.BatchedEngine._inject_mtp_mllm |
method | vllm_mlx.engine.batched.BatchedEngine._inject_mtp_mllm() -> None |
Inject MTP weights into the MLLM model's language_model. | #L431-L477 |
vllm_mlx.engine.batched.BatchedEngine._prepare_llm_model |
method | vllm_mlx.engine.batched.BatchedEngine._prepare_llm_model() -> None |
Load the LLM model/tokenizer before engine loop startup. | #L479-L511 |
vllm_mlx.engine.batched.BatchedEngine._prepare_mllm_messages |
method | vllm_mlx.engine.batched.BatchedEngine._prepare_mllm_messages(messages: list[dict[str, Any]]) -> list[dict[str, Any]] |
Convert OpenAI-style multimodal content to HuggingFace format. | #L690-L726 |
vllm_mlx.engine.batched.BatchedEngine._prepare_mllm_model |
method | vllm_mlx.engine.batched.BatchedEngine._prepare_mllm_model() -> None |
Load the MLLM model before scheduler startup. | #L289-L334 |
vllm_mlx.engine.batched.BatchedEngine._start_llm |
method | async vllm_mlx.engine.batched.BatchedEngine._start_llm() -> None |
Start the LLM engine with AsyncEngineCore. | #L543-L579 |
vllm_mlx.engine.batched.BatchedEngine._start_mllm |
method | async vllm_mlx.engine.batched.BatchedEngine._start_mllm() -> None |
Start the MLLM engine with MLLMScheduler (continuous batching). | #L336-L429 |
vllm_mlx.engine.batched.BatchedEngine._uses_default_prepare_for_start |
method | vllm_mlx.engine.batched.BatchedEngine._uses_default_prepare_for_start() -> bool |
Return True when prepare_for_start is the class implementation. | #L284-L287 |
vllm_mlx.engine.batched.BatchedEngine.abort_request |
method | async vllm_mlx.engine.batched.BatchedEngine.abort_request(request_id: str) -> bool |
Abort an active or queued batched request by request ID. | #L1190-L1199 |
vllm_mlx.engine.batched.BatchedEngine.chat |
method | async vllm_mlx.engine.batched.BatchedEngine.chat(messages: list[dict[str, Any]], max_tokens: int = 256, temperature: float = 0.7, top_p: float = 0.9, tools: list[dict] | None = None, images: list[str] | None = None, videos: list[str] | None = None, **kwargs) -> GenerationOutput |
Chat completion (non-streaming). | #L915-L984 |
vllm_mlx.engine.batched.BatchedEngine.clear_prefix_cache |
method | vllm_mlx.engine.batched.BatchedEngine.clear_prefix_cache() -> None |
Clear the in-memory prefix cache. | #L1222-L1231 |
vllm_mlx.engine.batched.BatchedEngine.clear_runtime_caches |
method | vllm_mlx.engine.batched.BatchedEngine.clear_runtime_caches() -> dict[str, Any] | None |
Clear engine-managed runtime caches. | #L1182-L1188 |
vllm_mlx.engine.batched.BatchedEngine.generate |
method | async vllm_mlx.engine.batched.BatchedEngine.generate(prompt: str, max_tokens: int = 256, temperature: float = 0.7, top_p: float = 0.9, stop: list[str] | None = None, images: list[str] | None = None, videos: list[str] | None = None, audio: list[str] | None = None, **kwargs) -> GenerationOutput |
Generate a complete response (non-streaming). | #L728-L817 |
vllm_mlx.engine.batched.BatchedEngine.get_cache_stats |
method | vllm_mlx.engine.batched.BatchedEngine.get_cache_stats() -> dict[str, Any] | None |
Get cache statistics. | #L1171-L1180 |
vllm_mlx.engine.batched.BatchedEngine.get_stats |
method | vllm_mlx.engine.batched.BatchedEngine.get_stats() -> dict[str, Any] |
Get engine statistics. | #L1129-L1169 |
vllm_mlx.engine.batched.BatchedEngine.is_mllm |
method | vllm_mlx.engine.batched.BatchedEngine.is_mllm() -> bool |
Check if this is a multimodal model. | #L232-L234 |
vllm_mlx.engine.batched.BatchedEngine.load_cache_from_disk |
method | vllm_mlx.engine.batched.BatchedEngine.load_cache_from_disk(cache_dir: str) -> int |
Load prefix cache from disk. | #L1211-L1220 |
vllm_mlx.engine.batched.BatchedEngine.model_name |
method | vllm_mlx.engine.batched.BatchedEngine.model_name() -> str |
Get the model name. | #L227-L229 |
vllm_mlx.engine.batched.BatchedEngine.prepare_for_start |
method | vllm_mlx.engine.batched.BatchedEngine.prepare_for_start() -> None |
Load heavyweight model state off the serving event loop. | #L243-L251 |
vllm_mlx.engine.batched.BatchedEngine.save_cache_to_disk |
method | vllm_mlx.engine.batched.BatchedEngine.save_cache_to_disk(cache_dir: str) -> bool |
Save prefix cache to disk for persistence across restarts. | #L1201-L1209 |
vllm_mlx.engine.batched.BatchedEngine.start |
method | async vllm_mlx.engine.batched.BatchedEngine.start() -> None |
Start the engine (load model if not loaded). | #L253-L282 |
vllm_mlx.engine.batched.BatchedEngine.stop |
method | async vllm_mlx.engine.batched.BatchedEngine.stop() -> None |
Stop the engine and cleanup resources. | #L581-L597 |
vllm_mlx.engine.batched.BatchedEngine.stream_chat |
method | async vllm_mlx.engine.batched.BatchedEngine.stream_chat(messages: list[dict[str, Any]], max_tokens: int = 256, temperature: float = 0.7, top_p: float = 0.9, tools: list[dict] | None = None, images: list[str] | None = None, videos: list[str] | None = None, **kwargs) -> AsyncIterator[GenerationOutput] |
Stream chat completion token by token. | #L1048-L1127 |
vllm_mlx.engine.batched.BatchedEngine.stream_generate |
method | async vllm_mlx.engine.batched.BatchedEngine.stream_generate(prompt: str, max_tokens: int = 256, temperature: float = 0.7, top_p: float = 0.9, stop: list[str] | None = None, images: list[str] | None = None, videos: list[str] | None = None, audio: list[str] | None = None, **kwargs) -> AsyncIterator[GenerationOutput] |
Stream generation token by token. | #L819-L913 |
vllm_mlx.engine.batched.BatchedEngine.tokenizer |
method | vllm_mlx.engine.batched.BatchedEngine.tokenizer() -> Any |
Get the tokenizer. | #L237-L241 |
vllm_mlx.engine.batched.MLLMModelWrapper |
class | vllm_mlx.engine.batched.MLLMModelWrapper(model) |
Wrapper for MLLM models to make them compatible with BatchGenerator. | #L140-L175 |
vllm_mlx.engine.batched.MLLMModelWrapper.__call__ |
method | vllm_mlx.engine.batched.MLLMModelWrapper.__call__(*args, **kwargs) -> not annotated |
Call the model and extract logits from LanguageModelOutput. | #L160-L171 |
vllm_mlx.engine.batched.MLLMModelWrapper.__getattr__ |
method | vllm_mlx.engine.batched.MLLMModelWrapper.__getattr__(name) -> not annotated |
Forward all other attributes to the wrapped model. | #L173-L175 |
vllm_mlx.engine.batched.MLLMModelWrapper.__init__ |
method | vllm_mlx.engine.batched.MLLMModelWrapper.__init__(model) -> not annotated |
Method `MLLMModelWrapper.__init__` updates `self._model`, `self._is_gemma3`; calls `hasattr`, `str(getattr(model, 'model_type', '')).lower`, `str`, `getattr`. | #L152-L158 |
vllm_mlx.engine.chat_template_safety._close_dangling_think_before_tool_call |
function | vllm_mlx.engine.chat_template_safety._close_dangling_think_before_tool_call(content: str) -> str |
Keep raw tool XML out of an unterminated ``<think>`` section. | #L8-L29 |
vllm_mlx.engine.chat_template_safety._message_to_dict |
function | vllm_mlx.engine.chat_template_safety._message_to_dict(message: Any) -> dict[str, Any] | Any |
Convert OpenAI message model objects without stringifying them. | #L32-L46 |
vllm_mlx.engine.chat_template_safety.normalize_messages_for_chat_template |
function | vllm_mlx.engine.chat_template_safety.normalize_messages_for_chat_template(messages: list[Any]) -> list[dict] |
Return a JSON-safe copy of messages for chat-template rendering. | #L49-L90 |
vllm_mlx.engine.simple._bind_worker_generation_streams |
function | vllm_mlx.engine.simple._bind_worker_generation_streams() -> None |
Rebind mlx generation streams inside the current worker thread. | #L48-L50 |
vllm_mlx.engine.simple._processors_can_retire |
function | vllm_mlx.engine.simple._processors_can_retire(processors: list[Any] | None) -> bool |
True when any processor advertises a retire-to-content transition. | #L100-L106 |
vllm_mlx.engine.simple._processors_retired |
function | vllm_mlx.engine.simple._processors_retired(processors: list[Any] | None) -> bool |
True when any retire-capable processor has entered its retired state. | #L109-L115 |
vllm_mlx.engine.simple._sample_with_processors |
function | vllm_mlx.engine.simple._sample_with_processors(tokens: mx.array | None, logits: mx.array, sampler: Any, logits_processors: list[Any] | None) -> tuple[mx.array, mx.array] |
Sample a token while honoring any active logits processors. | #L80-L97 |
vllm_mlx.engine.simple._seed_logits_processors |
function | vllm_mlx.engine.simple._seed_logits_processors(seed_tokens: mx.array | None, processors: list[Any] | None) -> list[Any] | None |
Wrap logits processors so continuation decode sees the full prompt. | #L53-L77 |
vllm_mlx.engine.simple._seed_logits_processors._wrap |
nested function | vllm_mlx.engine.simple._seed_logits_processors._wrap(processor) -> not annotated |
Nested Function `_seed_logits_processors._wrap` returns `_seeded`. | #L63-L75 |
vllm_mlx.engine.simple._seed_logits_processors._wrap._seeded |
nested function | vllm_mlx.engine.simple._seed_logits_processors._wrap._seeded(tokens, logits) -> not annotated |
Nested Function `_seed_logits_processors._wrap._seeded` calls `isinstance`, `mx.array`, `mx.concatenate`, `processor`; returns `processor(merged, logits)`. | #L64-L73 |
vllm_mlx.engine.simple._SpecPrefillCancelled |
class | vllm_mlx.engine.simple._SpecPrefillCancelled() |
Cooperative cancellation sentinel for blocking SpecPrefill workers. | #L118-L119 |
vllm_mlx.engine.simple.SimpleEngine |
class | vllm_mlx.engine.simple.SimpleEngine(model_name: str, trust_remote_code: bool = False, enable_cache: bool = True, force_mllm: bool = False, mtp: bool = False, mtp_num_draft_tokens: int = 1, prefill_step_size: int = 2048, specprefill_enabled: bool = False, specprefill_threshold: int = 8192, specprefill_keep_pct: float = 0.3, specprefill_backbone_pct: float = 0.0, specprefill_draft_model: str | None = None, max_kv_size: int = 0, mllm_draft_model: str | None = None, mllm_draft_kind: str | None = None, mllm_draft_block_size: int | None = None) |
Simple engine for direct model calls. | #L122-L2912 |
vllm_mlx.engine.simple.SimpleEngine.__init__ |
method | vllm_mlx.engine.simple.SimpleEngine.__init__(model_name: str, trust_remote_code: bool = False, enable_cache: bool = True, force_mllm: bool = False, mtp: bool = False, mtp_num_draft_tokens: int = 1, prefill_step_size: int = 2048, specprefill_enabled: bool = False, specprefill_threshold: int = 8192, specprefill_keep_pct: float = 0.3, specprefill_backbone_pct: float = 0.0, specprefill_draft_model: str | None = None, max_kv_size: int = 0, mllm_draft_model: str | None = None, mllm_draft_kind: str | None = None, mllm_draft_block_size: int | None = None) -> not annotated |
Initialize the simple engine. | #L130-L257 |
vllm_mlx.engine.simple.SimpleEngine._acquire_generation_slot |
method | async vllm_mlx.engine.simple.SimpleEngine._acquire_generation_slot(request_id: str) -> not annotated |
Admission control for SimpleEngine's serialized MLX route. | #L374-L398 |
vllm_mlx.engine.simple.SimpleEngine._cache_class_is_system_snapshot_safe |
method | vllm_mlx.engine.simple.SimpleEngine._cache_class_is_system_snapshot_safe(cache_entry: Any) -> bool |
Method `SimpleEngine._cache_class_is_system_snapshot_safe` calls `isinstance`, `type`; has 2 explicit return paths. | #L296-L303 |
vllm_mlx.engine.simple.SimpleEngine._clone_cache_state |
method | vllm_mlx.engine.simple.SimpleEngine._clone_cache_state(value: Any) -> Any |
Copy cache state containers without duplicating immutable MLX arrays. | #L260-L266 |
vllm_mlx.engine.simple.SimpleEngine._eval_cache_snapshot |
method | vllm_mlx.engine.simple.SimpleEngine._eval_cache_snapshot(snapshot: list[Any]) -> None |
Method `SimpleEngine._eval_cache_snapshot` calls `list`, `cls._iter_cache_state_arrays`, `mx.eval`. | #L290-L293 |
vllm_mlx.engine.simple.SimpleEngine._generation_lock_holder_summary |
method | vllm_mlx.engine.simple.SimpleEngine._generation_lock_holder_summary() -> str |
Method `SimpleEngine._generation_lock_holder_summary` calls `time.time`, `self._active_requests.items`, `info.get`, `round`; has 2 explicit return paths. | #L352-L371 |
vllm_mlx.engine.simple.SimpleEngine._iter_cache_state_arrays |
method | vllm_mlx.engine.simple.SimpleEngine._iter_cache_state_arrays(value: Any) -> not annotated |
Method `SimpleEngine._iter_cache_state_arrays` calls `isinstance`, `SimpleEngine._iter_cache_state_arrays`, `hasattr`; yields values incrementally. | #L282-L287 |
vllm_mlx.engine.simple.SimpleEngine._probe_system_kv_cache_support |
method | vllm_mlx.engine.simple.SimpleEngine._probe_system_kv_cache_support(model: Any, route: str) -> bool |
Method `SimpleEngine._probe_system_kv_cache_support` calls `make_prompt_cache`, `bool`, `all`, `cls._cache_class_is_system_snapshot_safe`; has 2 explicit return paths. | #L306-L331 |
vllm_mlx.engine.simple.SimpleEngine._restore_prompt_cache |
method | vllm_mlx.engine.simple.SimpleEngine._restore_prompt_cache(prompt_cache: list[Any], snapshot: list[Any]) -> None |
Restore cache states without letting decode mutate the saved snapshot. | #L274-L279 |
vllm_mlx.engine.simple.SimpleEngine._run_blocking_serialized |
method | async vllm_mlx.engine.simple.SimpleEngine._run_blocking_serialized(func, /, *args, request_id: str | None = None, on_cancel = None, **kwargs) -> not annotated |
Run a blocking MLX operation under the generation lock. | #L616-L666 |
vllm_mlx.engine.simple.SimpleEngine._run_blocking_serialized.run_bound |
nested function | vllm_mlx.engine.simple.SimpleEngine._run_blocking_serialized.run_bound() -> not annotated |
Nested Function `SimpleEngine._run_blocking_serialized.run_bound` calls `_bind_worker_generation_streams`, `func`; returns `func(*args, **kwargs)`. | #L644-L646 |
vllm_mlx.engine.simple.SimpleEngine._should_route_text_through_text_model |
method | vllm_mlx.engine.simple.SimpleEngine._should_route_text_through_text_model(*, mllm_draft_requested: bool = False) -> bool |
Return whether text-only MLLM requests may use mlx_lm TextModel. | #L610-L614 |
vllm_mlx.engine.simple.SimpleEngine._snapshot_prompt_cache |
method | vllm_mlx.engine.simple.SimpleEngine._snapshot_prompt_cache(prompt_cache: list[Any]) -> list[Any] |
Capture cache states without aliasing mutable state containers. | #L269-L271 |
vllm_mlx.engine.simple.SimpleEngine._stream_chat_impl |
method | async vllm_mlx.engine.simple.SimpleEngine._stream_chat_impl(messages: list[dict[str, Any]], max_tokens: int = 256, temperature: float = 0.7, top_p: float = 0.9, tools: list[dict] | None = None, images: list[str] | None = None, videos: list[str] | None = None, **kwargs) -> AsyncIterator[GenerationOutput] |
Stream chat completion token by token. | #L1173-L1794 |
vllm_mlx.engine.simple.SimpleEngine._stream_chat_impl._emit_done |
nested function | vllm_mlx.engine.simple.SimpleEngine._stream_chat_impl._emit_done() -> None |
Nested Function `SimpleEngine._stream_chat_impl._emit_done` calls `loop.call_soon_threadsafe`. | #L1614-L1615 |
vllm_mlx.engine.simple.SimpleEngine._stream_chat_impl._emit_error |
nested function | vllm_mlx.engine.simple.SimpleEngine._stream_chat_impl._emit_error(exc: BaseException) -> None |
Nested Function `SimpleEngine._stream_chat_impl._emit_error` calls `loop.call_soon_threadsafe`. | #L1617-L1618 |
vllm_mlx.engine.simple.SimpleEngine._stream_chat_impl._emit_response |
nested function | vllm_mlx.engine.simple.SimpleEngine._stream_chat_impl._emit_response(resp: Any) -> None |
Nested Function `SimpleEngine._stream_chat_impl._emit_response` calls `abort_event.is_set`, `loop.call_soon_threadsafe`; returns `None`. | #L1609-L1612 |
vllm_mlx.engine.simple.SimpleEngine._stream_chat_impl._produce_responses |
nested function | async vllm_mlx.engine.simple.SimpleEngine._stream_chat_impl._produce_responses() -> None |
Nested Function `SimpleEngine._stream_chat_impl._produce_responses` calls `self._run_blocking_serialized`, `_emit_error`, `_emit_done`; awaits asynchronous work. | #L1707-L1718 |
vllm_mlx.engine.simple.SimpleEngine._stream_chat_impl._run_with_cache |
nested function | vllm_mlx.engine.simple.SimpleEngine._stream_chat_impl._run_with_cache() -> None |
Nested Function `SimpleEngine._stream_chat_impl._run_with_cache` calls `make_sampler`, `make_prompt_cache`, `self._restore_prompt_cache`, `self._system_kv_cache.move_to_end`. | #L1620-L1705 |
vllm_mlx.engine.simple.SimpleEngine._stream_chat_impl._to_msg_dict |
nested function | vllm_mlx.engine.simple.SimpleEngine._stream_chat_impl._to_msg_dict(m: Any) -> dict[str, Any] |
Nested Function `SimpleEngine._stream_chat_impl._to_msg_dict` calls `isinstance`, `hasattr`, `m.model_dump`, `m.dict`; has 4 explicit return paths. | #L1499-L1509 |
vllm_mlx.engine.simple.SimpleEngine._stream_chat_impl._with_user |
nested function | vllm_mlx.engine.simple.SimpleEngine._stream_chat_impl._with_user(user_content: str) -> list[dict[str, Any]] |
Nested Function `SimpleEngine._stream_chat_impl._with_user` calls `dict`, `msgs[-1].get`; returns `msgs`. | #L1519-L1525 |
vllm_mlx.engine.simple.SimpleEngine._stream_chat_impl.mllm_call_kwargs |
nested function | vllm_mlx.engine.simple.SimpleEngine._stream_chat_impl.mllm_call_kwargs() -> dict |
Nested Function `SimpleEngine._stream_chat_impl.mllm_call_kwargs` calls `dict`; returns `local_kwargs`. | #L1236-L1242 |
vllm_mlx.engine.simple.SimpleEngine._stream_chat_impl.run_native_video |
nested function | vllm_mlx.engine.simple.SimpleEngine._stream_chat_impl.run_native_video() -> not annotated |
Nested Function `SimpleEngine._stream_chat_impl.run_native_video` calls `mllm_call_kwargs`, `list`, `self._model.stream_chat`; returns `list(self._model.stream_chat(messages=messages, max_tokens=max_tokens, temperature=temperature, tools=template_tools, *…`. | #L1299-L1309 |
vllm_mlx.engine.simple.SimpleEngine._stream_generate_impl |
method | async vllm_mlx.engine.simple.SimpleEngine._stream_generate_impl(prompt: str, max_tokens: int = 256, temperature: float = 0.7, top_p: float = 0.9, stop: list[str] | None = None, **kwargs) -> AsyncIterator[GenerationOutput] |
Stream generation token by token. | #L842-L1012 |
vllm_mlx.engine.simple.SimpleEngine._stream_generate_specprefill |
method | async vllm_mlx.engine.simple.SimpleEngine._stream_generate_specprefill(prompt: str, tokens: list[int], max_tokens: int, temperature: float, top_p: float, stop: list[str] | None = None, specprefill_keep_pct: float | None = None, specprefill_backbone_pct: float | None = None, **kwargs) -> AsyncIterator[GenerationOutput] |
SpecPrefill path for non-MTP models (Nemotron, GPT-OSS, etc). | #L1796-L2000 |
vllm_mlx.engine.simple.SimpleEngine._stream_generate_specprefill._cancel_check |
nested function | vllm_mlx.engine.simple.SimpleEngine._stream_generate_specprefill._cancel_check() -> None |
Nested Function `SimpleEngine._stream_generate_specprefill._cancel_check` calls `cancel_requested.is_set`, `_SpecPrefillCancelled`; can raise `_SpecPrefillCancelled`. | #L1824-L1826 |
vllm_mlx.engine.simple.SimpleEngine._stream_generate_specprefill._request_cancel |
nested function | vllm_mlx.engine.simple.SimpleEngine._stream_generate_specprefill._request_cancel() -> None |
Nested Function `SimpleEngine._stream_generate_specprefill._request_cancel` calls `cancel_requested.set`. | #L1821-L1822 |
vllm_mlx.engine.simple.SimpleEngine._stream_generate_specprefill._run_all |
nested function | vllm_mlx.engine.simple.SimpleEngine._stream_generate_specprefill._run_all() -> not annotated |
Nested Function `SimpleEngine._stream_generate_specprefill._run_all` calls `_run_specprefill`, `logger.error`, `_run_normal`; has 2 explicit return paths. | #L1828-L1835 |
vllm_mlx.engine.simple.SimpleEngine._stream_generate_specprefill._run_normal |
nested function | vllm_mlx.engine.simple.SimpleEngine._stream_generate_specprefill._run_normal() -> not annotated |
Fallback: normal generation without specprefill. | #L1941-L1962 |
vllm_mlx.engine.simple.SimpleEngine._stream_generate_specprefill._run_specprefill |
nested function | vllm_mlx.engine.simple.SimpleEngine._stream_generate_specprefill._run_specprefill() -> not annotated |
Score tokens, sparse prefill, generate autoregressively. | #L1837-L1939 |
vllm_mlx.engine.simple.SimpleEngine._stream_generate_text |
method | async vllm_mlx.engine.simple.SimpleEngine._stream_generate_text(messages: list[dict[str, Any]], max_tokens: int, temperature: float, top_p: float, tools: list | None = None, **kwargs) -> AsyncIterator[GenerationOutput] |
Text-only generation via mlx_lm TextModel. | #L2002-L2734 |
vllm_mlx.engine.simple.SimpleEngine._stream_generate_text._emit_done |
nested function | vllm_mlx.engine.simple.SimpleEngine._stream_generate_text._emit_done() -> None |
Nested Function `SimpleEngine._stream_generate_text._emit_done` calls `loop.call_soon_threadsafe`. | #L2277-L2278 |
vllm_mlx.engine.simple.SimpleEngine._stream_generate_text._emit_error |
nested function | vllm_mlx.engine.simple.SimpleEngine._stream_generate_text._emit_error(exc: BaseException) -> None |
Nested Function `SimpleEngine._stream_generate_text._emit_error` calls `loop.call_soon_threadsafe`. | #L2280-L2281 |
vllm_mlx.engine.simple.SimpleEngine._stream_generate_text._emit_response |
nested function | vllm_mlx.engine.simple.SimpleEngine._stream_generate_text._emit_response(resp: Any) -> None |
Nested Function `SimpleEngine._stream_generate_text._emit_response` calls `abort_event.is_set`, `loop.call_soon_threadsafe`; returns `None`. | #L2272-L2275 |
vllm_mlx.engine.simple.SimpleEngine._stream_generate_text._produce_responses |
nested function | async vllm_mlx.engine.simple.SimpleEngine._stream_generate_text._produce_responses() -> None |
Nested Function `SimpleEngine._stream_generate_text._produce_responses` calls `self._run_blocking_serialized`, `_emit_error`, `_emit_done`; awaits asynchronous work. | #L2666-L2677 |
vllm_mlx.engine.simple.SimpleEngine._stream_generate_text._resume_after_processor_retirement |
nested function | vllm_mlx.engine.simple.SimpleEngine._stream_generate_text._resume_after_processor_retirement(model, prompt_cache, prompt, remaining_tokens: int) -> None |
Nested Function `SimpleEngine._stream_generate_text._resume_after_processor_retirement` calls `dict`, `hasattr`, `model.make_mtp_cache`, `mlx_stream_generate`. | #L2293-L2320 |
vllm_mlx.engine.simple.SimpleEngine._stream_generate_text._run_all |
nested function | vllm_mlx.engine.simple.SimpleEngine._stream_generate_text._run_all() -> not annotated |
Nested Function `SimpleEngine._stream_generate_text._run_all` calls `_processors_can_retire`, `hasattr`, `logger.info`, `make_prompt_cache`; returns `None`. | #L2323-L2485 |
vllm_mlx.engine.simple.SimpleEngine._stream_generate_text._run_specprefill |
nested function | vllm_mlx.engine.simple.SimpleEngine._stream_generate_text._run_specprefill(model, bc, use_mtp) -> not annotated |
Score tokens, sparse prefill, then continue on the standard decode path. | #L2487-L2664 |
vllm_mlx.engine.simple.SimpleEngine._stream_generate_text._seed_from_last_response |
nested function | vllm_mlx.engine.simple.SimpleEngine._stream_generate_text._seed_from_last_response(prompt_cache, last_resp) -> not annotated |
Nested Function `SimpleEngine._stream_generate_text._seed_from_last_response` calls `getattr`, `cache_module.trim_prompt_cache`, `mx.array`, `self._text_tokenizer.encode`; has 2 explicit return paths. | #L2283-L2291 |
vllm_mlx.engine.simple.SimpleEngine._stream_generate_text.make_cache_with_snapshot |
nested function | vllm_mlx.engine.simple.SimpleEngine._stream_generate_text.make_cache_with_snapshot(text_model, system_kv_snapshot, _max_kv_size = self._max_kv_size) -> not annotated |
Nested Function `SimpleEngine._stream_generate_text.make_cache_with_snapshot` calls `make_prompt_cache`, `SimpleEngine._restore_prompt_cache`, `mx.array`; returns `(backbone_cache, prompt_to_send)`. | #L2159-L2176 |
vllm_mlx.engine.simple.SimpleEngine._track_request_stream |
method | async vllm_mlx.engine.simple.SimpleEngine._track_request_stream(source_gen: AsyncIterator[GenerationOutput], *, max_tokens: int = 0) -> AsyncIterator[GenerationOutput] |
Yield-through wrapper that records per-request live state and final ``prompt_tokens``/``completion_tokens`` counters. | #L732-L817 |
vllm_mlx.engine.simple.SimpleEngine._uses_default_prepare_for_start |
method | vllm_mlx.engine.simple.SimpleEngine._uses_default_prepare_for_start() -> bool |
Return True when prepare_for_start is the class implementation. | #L429-L432 |
vllm_mlx.engine.simple.SimpleEngine.chat |
method | async vllm_mlx.engine.simple.SimpleEngine.chat(messages: list[dict[str, Any]], max_tokens: int = 256, temperature: float = 0.7, top_p: float = 0.9, tools: list[dict] | None = None, images: list[str] | None = None, videos: list[str] | None = None, **kwargs) -> GenerationOutput |
Chat completion (non-streaming). | #L1014-L1144 |
vllm_mlx.engine.simple.SimpleEngine.chat.aggregate_stream_chat |
nested function | async vllm_mlx.engine.simple.SimpleEngine.chat.aggregate_stream_chat() -> GenerationOutput |
Nested Function `SimpleEngine.chat.aggregate_stream_chat` calls `GenerationOutput`, `self.stream_chat`, `clean_output_text`, `list`; returns `GenerationOutput(text=text, tokens=list(final_output.tokens), prompt_tokens=final_output.prompt_tokens, completion_toke…`. | #L1046-L1069 |
vllm_mlx.engine.simple.SimpleEngine.clear_runtime_caches |
method | vllm_mlx.engine.simple.SimpleEngine.clear_runtime_caches() -> dict[str, Any] | None |
Clear engine-managed runtime caches. | #L2880-L2912 |
vllm_mlx.engine.simple.SimpleEngine.generate |
method | async vllm_mlx.engine.simple.SimpleEngine.generate(prompt: str, max_tokens: int = 256, temperature: float = 0.7, top_p: float = 0.9, stop: list[str] | None = None, **kwargs) -> GenerationOutput |
Generate a complete response (non-streaming). | #L668-L730 |
vllm_mlx.engine.simple.SimpleEngine.get_cache_stats |
method | vllm_mlx.engine.simple.SimpleEngine.get_cache_stats() -> dict[str, Any] | None |
Get cache statistics for the system-prompt KV LRU plus, when the model is multimodal, the MLLM's own cache stats. | #L2860-L2878 |
vllm_mlx.engine.simple.SimpleEngine.get_stats |
method | vllm_mlx.engine.simple.SimpleEngine.get_stats() -> dict[str, Any] |
Get engine statistics. | #L2736-L2858 |
vllm_mlx.engine.simple.SimpleEngine.is_mllm |
method | vllm_mlx.engine.simple.SimpleEngine.is_mllm() -> bool |
Check if this is a multimodal model. | #L339-L341 |
vllm_mlx.engine.simple.SimpleEngine.model_name |
method | vllm_mlx.engine.simple.SimpleEngine.model_name() -> str |
Get the model name. | #L334-L336 |
vllm_mlx.engine.simple.SimpleEngine.prepare_for_start |
method | vllm_mlx.engine.simple.SimpleEngine.prepare_for_start() -> None |
Load the backing model off the serving event loop. | #L400-L427 |
vllm_mlx.engine.simple.SimpleEngine.start |
method | async vllm_mlx.engine.simple.SimpleEngine.start() -> None |
Start the engine (load model if not loaded). | #L434-L595 |
vllm_mlx.engine.simple.SimpleEngine.stop |
method | async vllm_mlx.engine.simple.SimpleEngine.stop() -> None |
Stop the engine and cleanup resources. | #L597-L608 |
vllm_mlx.engine.simple.SimpleEngine.stream_chat |
method | async vllm_mlx.engine.simple.SimpleEngine.stream_chat(messages: list[dict[str, Any]], max_tokens: int = 256, temperature: float = 0.7, top_p: float = 0.9, tools: list[dict] | None = None, images: list[str] | None = None, videos: list[str] | None = None, **kwargs) -> AsyncIterator[GenerationOutput] |
Public stream-chat wrapper with request stats tracking. | #L1146-L1171 |
vllm_mlx.engine.simple.SimpleEngine.stream_generate |
method | async vllm_mlx.engine.simple.SimpleEngine.stream_generate(prompt: str, max_tokens: int = 256, temperature: float = 0.7, top_p: float = 0.9, stop: list[str] | None = None, **kwargs) -> AsyncIterator[GenerationOutput] |
Public stream-generate wrapper with request stats tracking. | #L819-L840 |
vllm_mlx.engine.simple.SimpleEngine.tokenizer |
method | vllm_mlx.engine.simple.SimpleEngine.tokenizer() -> Any |
Get the tokenizer. | #L344-L350 |
vllm_mlx.engine_core._is_stream_thread_error |
function | vllm_mlx.engine_core._is_stream_thread_error(error: Exception) -> bool |
True when MLX reports stream ownership mismatch across threads. | #L33-L36 |
vllm_mlx.engine_core.AsyncEngineCore |
class | vllm_mlx.engine_core.AsyncEngineCore(model: Any, tokenizer: Any, config: Optional[EngineConfig] = None) |
Async context manager wrapper for EngineCore. | #L701-L794 |
vllm_mlx.engine_core.AsyncEngineCore.__aenter__ |
method | async vllm_mlx.engine_core.AsyncEngineCore.__aenter__() -> 'AsyncEngineCore' |
Method `AsyncEngineCore.__aenter__` calls `self.engine.start`; awaits asynchronous work; returns `self`. | #L720-L722 |
vllm_mlx.engine_core.AsyncEngineCore.__aexit__ |
method | async vllm_mlx.engine_core.AsyncEngineCore.__aexit__(*args) -> None |
Method `AsyncEngineCore.__aexit__` calls `self.engine.stop`; awaits asynchronous work. | #L724-L725 |
vllm_mlx.engine_core.AsyncEngineCore.__init__ |
method | vllm_mlx.engine_core.AsyncEngineCore.__init__(model: Any, tokenizer: Any, config: Optional[EngineConfig] = None) -> not annotated |
Method `AsyncEngineCore.__init__` updates `self.engine`; calls `EngineCore`. | #L712-L718 |
vllm_mlx.engine_core.AsyncEngineCore.abort_request |
method | async vllm_mlx.engine_core.AsyncEngineCore.abort_request(request_id: str) -> bool |
Abort a request. | #L750-L752 |
vllm_mlx.engine_core.AsyncEngineCore.add_request |
method | async vllm_mlx.engine_core.AsyncEngineCore.add_request(prompt: Union[str, List[int]], sampling_params: Optional[SamplingParams] = None, request_id: Optional[str] = None, **kwargs) -> str |
Add a request. | #L735-L748 |
vllm_mlx.engine_core.AsyncEngineCore.clear_runtime_caches |
method | vllm_mlx.engine_core.AsyncEngineCore.clear_runtime_caches() -> Dict[str, Any] | None |
Clear scheduler-managed runtime caches. | #L792-L794 |
vllm_mlx.engine_core.AsyncEngineCore.generate |
method | async vllm_mlx.engine_core.AsyncEngineCore.generate(prompt: Union[str, List[int]], sampling_params: Optional[SamplingParams] = None, **kwargs) -> RequestOutput |
Generate complete response. | #L763-L774 |
vllm_mlx.engine_core.AsyncEngineCore.get_cache_stats |
method | vllm_mlx.engine_core.AsyncEngineCore.get_cache_stats() -> Optional[Dict[str, Any]] |
Get prefix cache statistics. | #L780-L782 |
vllm_mlx.engine_core.AsyncEngineCore.get_stats |
method | vllm_mlx.engine_core.AsyncEngineCore.get_stats() -> Dict[str, Any] |
Get engine stats. | #L776-L778 |
vllm_mlx.engine_core.AsyncEngineCore.load_cache_from_disk |
method | vllm_mlx.engine_core.AsyncEngineCore.load_cache_from_disk(cache_dir: str) -> int |
Load prefix cache from disk. | #L788-L790 |
vllm_mlx.engine_core.AsyncEngineCore.save_cache_to_disk |
method | vllm_mlx.engine_core.AsyncEngineCore.save_cache_to_disk(cache_dir: str) -> bool |
Save prefix cache to disk. | #L784-L786 |
vllm_mlx.engine_core.AsyncEngineCore.start |
method | vllm_mlx.engine_core.AsyncEngineCore.start() -> None |
Start engine (creates task in current loop). | #L727-L729 |
vllm_mlx.engine_core.AsyncEngineCore.stop |
method | async vllm_mlx.engine_core.AsyncEngineCore.stop() -> None |
Stop the engine. | #L731-L733 |
vllm_mlx.engine_core.AsyncEngineCore.stream_outputs |
method | async vllm_mlx.engine_core.AsyncEngineCore.stream_outputs(request_id: str, timeout: Optional[float] = None) -> AsyncIterator[RequestOutput] |
Stream outputs. | #L754-L761 |
vllm_mlx.engine_core.EngineConfig |
class | vllm_mlx.engine_core.EngineConfig(model_name: str = '', scheduler_config: Optional[SchedulerConfig] = None, step_interval: float = 0.001, stream_interval: int = 1, gpu_memory_utilization: float = 0.9) |
Configuration for the engine. | #L40-L47 |
vllm_mlx.engine_core.EngineCore |
class | vllm_mlx.engine_core.EngineCore(model: Any, tokenizer: Any, config: Optional[EngineConfig] = None, engine_id: Optional[str] = None, force_model_ownership: bool = True) |
Core engine for vllm-mlx inference with continuous batching. | #L50-L698 |
vllm_mlx.engine_core.EngineCore.__del__ |
method | vllm_mlx.engine_core.EngineCore.__del__() -> not annotated |
Cleanup on destruction. | #L687-L693 |
vllm_mlx.engine_core.EngineCore.__init__ |
method | vllm_mlx.engine_core.EngineCore.__init__(model: Any, tokenizer: Any, config: Optional[EngineConfig] = None, engine_id: Optional[str] = None, force_model_ownership: bool = True) -> not annotated |
Initialize the engine. | #L58-L114 |
vllm_mlx.engine_core.EngineCore._cleanup_request |
method | vllm_mlx.engine_core.EngineCore._cleanup_request(request_id: str) -> None |
Clean up request tracking. | #L392-L399 |
vllm_mlx.engine_core.EngineCore._engine_loop |
method | async vllm_mlx.engine_core.EngineCore._engine_loop() -> None |
Main engine loop. | #L146-L334 |
vllm_mlx.engine_core.EngineCore._engine_loop._bind_model_streams_once |
nested function | vllm_mlx.engine_core.EngineCore._engine_loop._bind_model_streams_once() -> None |
Nested Function `EngineCore._engine_loop._bind_model_streams_once` calls `bind_generation_streams`. | #L166-L170 |
vllm_mlx.engine_core.EngineCore._engine_loop._bind_worker_streams_once |
nested function | vllm_mlx.engine_core.EngineCore._engine_loop._bind_worker_streams_once() -> None |
Nested Function `EngineCore._engine_loop._bind_worker_streams_once` calls `bind_generation_streams`. | #L160-L164 |
vllm_mlx.engine_core.EngineCore._engine_loop._clear_cache_on_worker |
nested function | vllm_mlx.engine_core.EngineCore._engine_loop._clear_cache_on_worker() -> None |
Nested Function `EngineCore._engine_loop._clear_cache_on_worker` calls `_bind_worker_streams_once`, `mx.clear_cache`. | #L217-L219 |
vllm_mlx.engine_core.EngineCore._engine_loop._close_batch_generator_on_worker |
nested function | vllm_mlx.engine_core.EngineCore._engine_loop._close_batch_generator_on_worker() -> None |
Nested Function `EngineCore._engine_loop._close_batch_generator_on_worker` calls `_bind_worker_streams_once`, `self.scheduler._close_batch_generator`. | #L221-L223 |
vllm_mlx.engine_core.EngineCore._engine_loop._recover_stream_thread_error_on_worker |
nested function | vllm_mlx.engine_core.EngineCore._engine_loop._recover_stream_thread_error_on_worker() -> None |
Nested Function `EngineCore._engine_loop._recover_stream_thread_error_on_worker` calls `_bind_worker_streams_once`, `self.scheduler._recover_from_cache_error`, `self.scheduler._reschedule_running_requests`. | #L212-L215 |
vllm_mlx.engine_core.EngineCore._engine_loop._step_on_model_thread |
nested function | vllm_mlx.engine_core.EngineCore._engine_loop._step_on_model_thread() -> not annotated |
Nested Function `EngineCore._engine_loop._step_on_model_thread` updates `self._steps_executed`; calls `_bind_model_streams_once`, `self.scheduler.step`, `mx.get_active_memory`, `mx.clear_cache`; returns `output`. | #L192-L210 |
vllm_mlx.engine_core.EngineCore._engine_loop._step_on_worker |
nested function | vllm_mlx.engine_core.EngineCore._engine_loop._step_on_worker() -> not annotated |
Nested Function `EngineCore._engine_loop._step_on_worker` updates `self._steps_executed`; calls `_bind_worker_streams_once`, `self.scheduler.step`, `mx.get_active_memory`, `mx.clear_cache`; returns `output`. | #L172-L190 |
vllm_mlx.engine_core.EngineCore._release_model |
method | vllm_mlx.engine_core.EngineCore._release_model() -> None |
Release model ownership. | #L647-L653 |
vllm_mlx.engine_core.EngineCore.abort_request |
method | async vllm_mlx.engine_core.EngineCore.abort_request(request_id: str) -> bool |
Abort a request. | #L386-L390 |
vllm_mlx.engine_core.EngineCore.add_request |
method | async vllm_mlx.engine_core.EngineCore.add_request(prompt: Union[str, List[int]], sampling_params: Optional[SamplingParams] = None, request_id: Optional[str] = None, images: Optional[List[Any]] = None, videos: Optional[List[Any]] = None, prefix_boundary: int = 0) -> str |
Add a request for processing. | #L336-L384 |
vllm_mlx.engine_core.EngineCore.clear_prefix_cache |
method | vllm_mlx.engine_core.EngineCore.clear_prefix_cache() -> None |
Clear the prefix cache (delegates to scheduler). | #L642-L645 |
vllm_mlx.engine_core.EngineCore.clear_runtime_caches |
method | vllm_mlx.engine_core.EngineCore.clear_runtime_caches() -> Dict[str, Any] | None |
Clear scheduler-managed runtime caches. | #L638-L640 |
vllm_mlx.engine_core.EngineCore.close |
method | vllm_mlx.engine_core.EngineCore.close() -> None |
Explicitly close the engine and release resources. | #L655-L685 |
vllm_mlx.engine_core.EngineCore.engine_id |
method | vllm_mlx.engine_core.EngineCore.engine_id() -> str |
Get the engine ID. | #L696-L698 |
vllm_mlx.engine_core.EngineCore.generate |
method | async vllm_mlx.engine_core.EngineCore.generate(prompt: Union[str, List[int]], sampling_params: Optional[SamplingParams] = None, request_id: Optional[str] = None, **kwargs) -> RequestOutput |
Generate a complete response (non-streaming). | #L490-L552 |
vllm_mlx.engine_core.EngineCore.generate_batch_sync |
method | vllm_mlx.engine_core.EngineCore.generate_batch_sync(prompts: List[Union[str, List[int]]], sampling_params: Optional[SamplingParams] = None) -> List[RequestOutput] |
Generate responses synchronously for maximum throughput. | #L554-L609 |
vllm_mlx.engine_core.EngineCore.get_cache_stats |
method | vllm_mlx.engine_core.EngineCore.get_cache_stats() -> Optional[Dict[str, Any]] |
Get prefix cache statistics. | #L626-L628 |
vllm_mlx.engine_core.EngineCore.get_stats |
method | vllm_mlx.engine_core.EngineCore.get_stats() -> Dict[str, Any] |
Get engine statistics. | #L611-L624 |
vllm_mlx.engine_core.EngineCore.is_running |
method | vllm_mlx.engine_core.EngineCore.is_running() -> bool |
Check if engine is running. | #L142-L144 |
vllm_mlx.engine_core.EngineCore.load_cache_from_disk |
method | vllm_mlx.engine_core.EngineCore.load_cache_from_disk(cache_dir: str) -> int |
Load prefix cache from disk. | #L634-L636 |
vllm_mlx.engine_core.EngineCore.save_cache_to_disk |
method | vllm_mlx.engine_core.EngineCore.save_cache_to_disk(cache_dir: str) -> bool |
Save prefix cache to disk. | #L630-L632 |
vllm_mlx.engine_core.EngineCore.start |
method | async vllm_mlx.engine_core.EngineCore.start() -> None |
Start the engine loop. | #L116-L124 |
vllm_mlx.engine_core.EngineCore.stop |
method | async vllm_mlx.engine_core.EngineCore.stop() -> None |
Stop the engine loop. | #L126-L140 |
vllm_mlx.engine_core.EngineCore.stream_outputs |
method | async vllm_mlx.engine_core.EngineCore.stream_outputs(request_id: str, timeout: Optional[float] = None) -> AsyncIterator[RequestOutput] |
Stream outputs for a request with low-latency non-blocking pattern. | #L401-L488 |
vllm_mlx.gradio_app.build_message_content |
function | vllm_mlx.gradio_app.build_message_content(text: str, files: list[str] | None = None) -> list | str |
Build OpenAI-compatible message content with text and optional files. | #L79-L108 |
vllm_mlx.gradio_app.create_chat_function |
function | vllm_mlx.gradio_app.create_chat_function(server_url: str, max_tokens: int, temperature: float, served_model_name: str = 'default') -> not annotated |
Create the chat function for Gradio ChatInterface. | #L111-L257 |
vllm_mlx.gradio_app.create_chat_function.chat |
nested function | vllm_mlx.gradio_app.create_chat_function.chat(message: dict, history: list) -> str |
Process a multimodal message and return response. | #L132-L255 |
vllm_mlx.gradio_app.encode_file_to_base64 |
function | vllm_mlx.gradio_app.encode_file_to_base64(file_path: str) -> tuple[str, str] |
Encode a file to base64 data URL. | #L33-L76 |
vllm_mlx.gradio_app.main |
function | vllm_mlx.gradio_app.main() -> not annotated |
Run the Gradio app. | #L260-L407 |
vllm_mlx.gradio_app.main.text_chat |
nested function | vllm_mlx.gradio_app.main.text_chat(message: str, history: list) -> str |
Process a text-only message. | #L331-L368 |
vllm_mlx.gradio_text_app.create_chat_function |
function | vllm_mlx.gradio_text_app.create_chat_function(server_url: str, max_tokens: int, temperature: float, served_model_name: str = 'default') -> not annotated |
Create the chat function for Gradio ChatInterface. | #L34-L108 |
vllm_mlx.gradio_text_app.create_chat_function.chat |
nested function | vllm_mlx.gradio_text_app.create_chat_function.chat(message: str, history: list) -> str |
Process a text message and return response. | #L53-L106 |
vllm_mlx.gradio_text_app.main |
function | vllm_mlx.gradio_text_app.main() -> not annotated |
Run the Gradio app. | #L111-L201 |
vllm_mlx.lifecycle.ModelSpec |
class | vllm_mlx.lifecycle.ModelSpec(model_key: str, model_name: str, use_batching: bool = False, scheduler_config: Any | None = None, stream_interval: int = 1, max_tokens: int = 32768, force_mllm: bool = False, mtp: bool = False, prefill_step_size: int = 2048, specprefill_enabled: bool = False, specprefill_threshold: int = 8192, specprefill_keep_pct: float = 0.3, specprefill_backbone_pct: float = 0.0, specprefill_draft_model: str | None = None) |
Immutable engine construction inputs for a resident model. | #L28-L44 |
vllm_mlx.lifecycle.ResidencyManager |
class | vllm_mlx.lifecycle.ResidencyManager(engine_factory: Callable[[ModelSpec], Awaitable[BaseEngine]], *, on_engine_loaded: Callable[[ModelSpec, BaseEngine], Awaitable[None] | None] | None = None, on_engine_unloading: Callable[[ModelSpec, BaseEngine], Awaitable[None] | None] | None = None, time_fn: Callable[[], float] | None = None, auto_unload_idle_seconds: float = 0) |
Single-flight lifecycle manager for resident models. | #L69-L493 |
vllm_mlx.lifecycle.ResidencyManager.__init__ |
method | vllm_mlx.lifecycle.ResidencyManager.__init__(engine_factory: Callable[[ModelSpec], Awaitable[BaseEngine]], *, on_engine_loaded: Callable[[ModelSpec, BaseEngine], Awaitable[None] | None] | None = None, on_engine_unloading: Callable[[ModelSpec, BaseEngine], Awaitable[None] | None] | None = None, time_fn: Callable[[], float] | None = None, auto_unload_idle_seconds: float = 0) -> None |
Method `ResidencyManager.__init__` updates `self._engine_factory`, `self._on_engine_loaded`, `self._on_engine_unloading`, `self._time_fn`; calls `__import__`, `asyncio.Lock`. | #L72-L91 |
vllm_mlx.lifecycle.ResidencyManager._cleanup_cancelled_load |
method | async vllm_mlx.lifecycle.ResidencyManager._cleanup_cancelled_load(resident: ResidentModel, engine: BaseEngine | None) -> None |
Stop a partially loaded engine and unwind resident state. | #L447-L465 |
vllm_mlx.lifecycle.ResidencyManager._load_engine |
method | async vllm_mlx.lifecycle.ResidencyManager._load_engine(resident: ResidentModel) -> BaseEngine |
Create and start a resident engine. | #L314-L354 |
vllm_mlx.lifecycle.ResidencyManager._prepare_engine_start |
method | async vllm_mlx.lifecycle.ResidencyManager._prepare_engine_start(resident: ResidentModel, engine: BaseEngine) -> None |
Run blocking startup work away from the serving event loop. | #L409-L445 |
vllm_mlx.lifecycle.ResidencyManager._release_load_waiter |
method | async vllm_mlx.lifecycle.ResidencyManager._release_load_waiter(model_key: str, task: asyncio.Task[BaseEngine]) -> None |
Drop one waiter from a shared load, canceling abandoned solo loads. | #L467-L493 |
vllm_mlx.lifecycle.ResidencyManager._resident |
method | vllm_mlx.lifecycle.ResidencyManager._resident(model_key: str) -> ResidentModel |
Method `ResidencyManager._resident` calls `KeyError`; can raise `KeyError`; returns `self._residents[model_key]`. | #L390-L394 |
vllm_mlx.lifecycle.ResidencyManager._run_hook |
method | async vllm_mlx.lifecycle.ResidencyManager._run_hook(hook: Callable[[ModelSpec, BaseEngine], Awaitable[None] | None] | None, spec: ModelSpec, engine: BaseEngine) -> None |
Method `ResidencyManager._run_hook` calls `hook`, `inspect.isawaitable`; awaits asynchronous work; returns `None`. | #L396-L407 |
vllm_mlx.lifecycle.ResidencyManager._unload_engine |
method | async vllm_mlx.lifecycle.ResidencyManager._unload_engine(resident: ResidentModel) -> bool |
Stop and drop a resident engine. | #L356-L388 |
vllm_mlx.lifecycle.ResidencyManager.acquire |
method | async vllm_mlx.lifecycle.ResidencyManager.acquire(model_key: str, *, count_activity: bool = True) -> BaseEngine |
Acquire a resident engine for request processing. | #L186-L206 |
vllm_mlx.lifecycle.ResidencyManager.ensure_loaded |
method | async vllm_mlx.lifecycle.ResidencyManager.ensure_loaded(model_key: str) -> BaseEngine |
Load and start a resident engine if needed. | #L132-L184 |
vllm_mlx.lifecycle.ResidencyManager.get_engine |
method | vllm_mlx.lifecycle.ResidencyManager.get_engine(model_key: str) -> BaseEngine | None |
Get the currently loaded engine, if any. | #L113-L115 |
vllm_mlx.lifecycle.ResidencyManager.get_status |
method | vllm_mlx.lifecycle.ResidencyManager.get_status(model_key: str) -> dict[str, Any] |
Return a serializable snapshot of resident state. | #L117-L130 |
vllm_mlx.lifecycle.ResidencyManager.register_model |
method | vllm_mlx.lifecycle.ResidencyManager.register_model(spec: ModelSpec) -> str |
Register a model spec, or replace a dormant resident entry. | #L93-L111 |
vllm_mlx.lifecycle.ResidencyManager.release |
method | async vllm_mlx.lifecycle.ResidencyManager.release(model_key: str, *, count_activity: bool = True) -> None |
Release a previously acquired resident engine. | #L208-L215 |
vllm_mlx.lifecycle.ResidencyManager.shutdown |
method | async vllm_mlx.lifecycle.ResidencyManager.shutdown() -> None |
Stop all loaded residents. | #L255-L312 |
vllm_mlx.lifecycle.ResidencyManager.unload_if_idle |
method | async vllm_mlx.lifecycle.ResidencyManager.unload_if_idle(model_key: str) -> bool |
Unload a resident engine if it has been idle past the threshold. | #L217-L253 |
vllm_mlx.lifecycle.ResidentModel |
class | vllm_mlx.lifecycle.ResidentModel(spec: ModelSpec, state: ResidentState = ResidentState.UNLOADED, engine: BaseEngine | None = None, active_requests: int = 0, last_used_at: float | None = None, loaded_at: float | None = None, last_error: str | None = None, estimated_memory_bytes: int | None = None, _load_waiters: int = field(default=0, repr=False), _load_waiter_task: asyncio.Task[BaseEngine] | None = field(default=None, repr=False), _prepare_task: asyncio.Task[None] | None = field(default=None, repr=False), _abandoned_loading_task: asyncio.Task[BaseEngine] | None = field(default=None, repr=False), _loading_task: asyncio.Task[BaseEngine] | None = field(default=None, repr=False), _unloading_task: asyncio.Task[bool] | None = field(default=None, repr=False)) |
Runtime state for a single resident model. | #L48-L66 |
vllm_mlx.lifecycle.ResidentState |
class | vllm_mlx.lifecycle.ResidentState() |
Runtime residency state for a configured model. | #L17-L24 |
vllm_mlx.mcp.client.MCPClient |
class | vllm_mlx.mcp.client.MCPClient(config: MCPServerConfig) |
Client for connecting to a single MCP server. | #L23-L328 |
vllm_mlx.mcp.client.MCPClient.__init__ |
method | vllm_mlx.mcp.client.MCPClient.__init__(config: MCPServerConfig) -> not annotated |
Initialize MCP client. | #L30-L45 |
vllm_mlx.mcp.client.MCPClient._connect_sse |
method | async vllm_mlx.mcp.client.MCPClient._connect_sse() -> not annotated |
Connect via SSE transport. | #L154-L170 |
vllm_mlx.mcp.client.MCPClient._connect_stdio |
method | async vllm_mlx.mcp.client.MCPClient._connect_stdio() -> not annotated |
Connect via stdio transport. | #L124-L152 |
vllm_mlx.mcp.client.MCPClient._discover_tools |
method | async vllm_mlx.mcp.client.MCPClient._discover_tools() -> not annotated |
Discover available tools from the server. | #L185-L208 |
vllm_mlx.mcp.client.MCPClient._extract_content |
method | vllm_mlx.mcp.client.MCPClient._extract_content(result) -> Any |
Extract content from MCP tool result. | #L303-L321 |
vllm_mlx.mcp.client.MCPClient._initialize_session |
method | async vllm_mlx.mcp.client.MCPClient._initialize_session() -> not annotated |
Initialize the MCP session. | #L172-L183 |
vllm_mlx.mcp.client.MCPClient.call_tool |
method | async vllm_mlx.mcp.client.MCPClient.call_tool(tool_name: str, arguments: Dict[str, Any], timeout: Optional[float] = None) -> MCPToolResult |
Call a tool on the MCP server. | #L237-L301 |
vllm_mlx.mcp.client.MCPClient.connect |
method | async vllm_mlx.mcp.client.MCPClient.connect() -> bool |
Connect to the MCP server. | #L78-L122 |
vllm_mlx.mcp.client.MCPClient.disconnect |
method | async vllm_mlx.mcp.client.MCPClient.disconnect() -> not annotated |
Disconnect from the MCP server. | #L210-L235 |
vllm_mlx.mcp.client.MCPClient.get_status |
method | vllm_mlx.mcp.client.MCPClient.get_status() -> MCPServerStatus |
Get server status. | #L67-L76 |
vllm_mlx.mcp.client.MCPClient.is_connected |
method | vllm_mlx.mcp.client.MCPClient.is_connected() -> bool |
Check if connected to server. | #L58-L60 |
vllm_mlx.mcp.client.MCPClient.name |
method | vllm_mlx.mcp.client.MCPClient.name() -> str |
Get server name. | #L48-L50 |
vllm_mlx.mcp.client.MCPClient.refresh_tools |
method | async vllm_mlx.mcp.client.MCPClient.refresh_tools() -> not annotated |
Refresh the list of available tools. | #L323-L328 |
vllm_mlx.mcp.client.MCPClient.state |
method | vllm_mlx.mcp.client.MCPClient.state() -> MCPServerState |
Get current connection state. | #L53-L55 |
vllm_mlx.mcp.client.MCPClient.tools |
method | vllm_mlx.mcp.client.MCPClient.tools() -> List[MCPTool] |
Get discovered tools. | #L63-L65 |
vllm_mlx.mcp.config._find_config_file |
function | vllm_mlx.mcp.config._find_config_file(explicit_path: Optional[Union[str, Path]] = None) -> Optional[Path] |
Find the config file to use. | #L73-L98 |
vllm_mlx.mcp.config.create_example_config |
function | vllm_mlx.mcp.config.create_example_config() -> str |
Create an example MCP configuration. | #L166-L199 |
vllm_mlx.mcp.config.load_mcp_config |
function | vllm_mlx.mcp.config.load_mcp_config(path: Optional[Union[str, Path]] = None) -> MCPConfig |
Load MCP configuration from file. | #L26-L70 |
vllm_mlx.mcp.config.validate_config |
function | vllm_mlx.mcp.config.validate_config(data: Dict[str, Any]) -> MCPConfig |
Validate and parse configuration dictionary. | #L101-L163 |
vllm_mlx.mcp.executor.execute_single_tool |
function | async vllm_mlx.mcp.executor.execute_single_tool(manager: MCPClientManager, tool_name: str, arguments: Dict[str, Any], timeout: Optional[float] = None) -> MCPToolResult |
Convenience function to execute a single tool. | #L482-L500 |
vllm_mlx.mcp.executor.ToolArgumentValidationError |
class | vllm_mlx.mcp.executor.ToolArgumentValidationError() |
Raised when tool arguments fail validation against schema. | #L22-L25 |
vllm_mlx.mcp.executor.ToolExecutor |
class | vllm_mlx.mcp.executor.ToolExecutor(manager: MCPClientManager, max_parallel: int = 5, default_timeout: Optional[float] = None, validate_arguments: bool = True, sandbox: Optional[ToolSandbox] = None) |
Handles execution of tool calls from model responses. | #L64-L479 |
vllm_mlx.mcp.executor.ToolExecutor.__init__ |
method | vllm_mlx.mcp.executor.ToolExecutor.__init__(manager: MCPClientManager, max_parallel: int = 5, default_timeout: Optional[float] = None, validate_arguments: bool = True, sandbox: Optional[ToolSandbox] = None) -> not annotated |
Initialize tool executor. | #L75-L97 |
vllm_mlx.mcp.executor.ToolExecutor._execute_parallel |
method | async vllm_mlx.mcp.executor.ToolExecutor._execute_parallel(tool_calls: List[Dict[str, Any]]) -> List[Tuple[MCPToolResult, str]] |
Execute tool calls in parallel with concurrency limit. | #L195-L305 |
vllm_mlx.mcp.executor.ToolExecutor._execute_parallel.execute_with_semaphore |
nested function | async vllm_mlx.mcp.executor.ToolExecutor._execute_parallel.execute_with_semaphore(tool_call: Dict[str, Any]) -> not annotated |
Nested Function `ToolExecutor._execute_parallel.execute_with_semaphore` calls `tool_call.get`, `func.get`, `isinstance`, `json.loads`; awaits asynchronous work; has 3 explicit return paths. | #L202-L281 |
vllm_mlx.mcp.executor.ToolExecutor._execute_sequential |
method | async vllm_mlx.mcp.executor.ToolExecutor._execute_sequential(tool_calls: List[Dict[str, Any]]) -> List[Tuple[MCPToolResult, str]] |
Execute tool calls sequentially. | #L307-L415 |
vllm_mlx.mcp.executor.ToolExecutor._get_server_for_tool |
method | vllm_mlx.mcp.executor.ToolExecutor._get_server_for_tool(full_name: str) -> str |
Extract server name from full tool name or find it. | #L185-L193 |
vllm_mlx.mcp.executor.ToolExecutor._get_tool_by_name |
method | vllm_mlx.mcp.executor.ToolExecutor._get_tool_by_name(full_name: str) -> Optional[MCPTool] |
Get a tool by its full name (server__tool or just tool). | #L122-L132 |
vllm_mlx.mcp.executor.ToolExecutor._tool_exists |
method | vllm_mlx.mcp.executor.ToolExecutor._tool_exists(full_name: str) -> bool |
Check if a tool exists in any connected server. | #L466-L479 |
vllm_mlx.mcp.executor.ToolExecutor._validate_sandbox |
method | vllm_mlx.mcp.executor.ToolExecutor._validate_sandbox(tool_name: str, server_name: str, arguments: Dict[str, Any]) -> Optional[str] |
Validate tool execution against sandbox policy. | #L167-L183 |
vllm_mlx.mcp.executor.ToolExecutor._validate_tool_call |
method | vllm_mlx.mcp.executor.ToolExecutor._validate_tool_call(tool_call: Dict[str, Any]) -> Optional[str] |
Validate a tool call's arguments against the tool's schema. | #L134-L165 |
vllm_mlx.mcp.executor.ToolExecutor.execute_and_format |
method | async vllm_mlx.mcp.executor.ToolExecutor.execute_and_format(tool_calls: List[Dict[str, Any]], parallel: bool = True) -> List[Dict[str, Any]] |
Execute tool calls and format results as messages. | #L417-L433 |
vllm_mlx.mcp.executor.ToolExecutor.execute_tool_calls |
method | async vllm_mlx.mcp.executor.ToolExecutor.execute_tool_calls(tool_calls: List[Dict[str, Any]], parallel: bool = True) -> List[Tuple[MCPToolResult, str]] |
Execute multiple tool calls. | #L99-L120 |
vllm_mlx.mcp.executor.ToolExecutor.extract_and_validate |
method | vllm_mlx.mcp.executor.ToolExecutor.extract_and_validate(response: Dict[str, Any]) -> Tuple[List[Dict[str, Any]], bool] |
Extract tool calls from response and validate them. | #L435-L464 |
vllm_mlx.mcp.executor.validate_tool_arguments |
function | vllm_mlx.mcp.executor.validate_tool_arguments(tool: MCPTool, arguments: Dict[str, Any], strict: bool = True) -> None |
Validate tool arguments against the tool's input schema. | #L28-L61 |
vllm_mlx.mcp.manager.MCPClientManager |
class | vllm_mlx.mcp.manager.MCPClientManager(config: MCPConfig) |
Manages multiple MCP server connections. | #L22-L301 |
vllm_mlx.mcp.manager.MCPClientManager.__init__ |
method | vllm_mlx.mcp.manager.MCPClientManager.__init__(config: MCPConfig) -> not annotated |
Initialize MCP Client Manager. | #L33-L47 |
vllm_mlx.mcp.manager.MCPClientManager._find_tool_server |
method | vllm_mlx.mcp.manager.MCPClientManager._find_tool_server(tool_name: str) -> Optional[str] |
Find which server has a tool by name. | #L258-L273 |
vllm_mlx.mcp.manager.MCPClientManager.execute_tool |
method | async vllm_mlx.mcp.manager.MCPClientManager.execute_tool(full_name: str, arguments: Dict[str, Any], timeout: Optional[float] = None) -> MCPToolResult |
Execute a tool by its full name (server__tool). | #L174-L232 |
vllm_mlx.mcp.manager.MCPClientManager.execute_tool_call |
method | async vllm_mlx.mcp.manager.MCPClientManager.execute_tool_call(tool_call: Dict[str, Any], timeout: Optional[float] = None) -> MCPToolResult |
Execute a tool call from OpenAI format. | #L234-L256 |
vllm_mlx.mcp.manager.MCPClientManager.get_all_tools |
method | vllm_mlx.mcp.manager.MCPClientManager.get_all_tools() -> List[MCPTool] |
Get all tools from all connected servers. | #L114-L125 |
vllm_mlx.mcp.manager.MCPClientManager.get_all_tools_openai |
method | vllm_mlx.mcp.manager.MCPClientManager.get_all_tools_openai() -> List[Dict[str, Any]] |
Get all tools in OpenAI function calling format. | #L127-L134 |
vllm_mlx.mcp.manager.MCPClientManager.get_client |
method | vllm_mlx.mcp.manager.MCPClientManager.get_client(server_name: str) -> Optional[MCPClient] |
Get client for a specific server. | #L162-L172 |
vllm_mlx.mcp.manager.MCPClientManager.get_merged_tools |
method | vllm_mlx.mcp.manager.MCPClientManager.get_merged_tools(user_tools: Optional[List[Dict[str, Any]]] = None) -> List[Dict[str, Any]] |
Get MCP tools merged with user-provided tools. | #L136-L151 |
vllm_mlx.mcp.manager.MCPClientManager.get_server_status |
method | vllm_mlx.mcp.manager.MCPClientManager.get_server_status() -> List[MCPServerStatus] |
Get status of all servers. | #L153-L160 |
vllm_mlx.mcp.manager.MCPClientManager.is_started |
method | vllm_mlx.mcp.manager.MCPClientManager.is_started() -> bool |
Check if manager has been started. | #L50-L52 |
vllm_mlx.mcp.manager.MCPClientManager.reconnect |
method | async vllm_mlx.mcp.manager.MCPClientManager.reconnect(server_name: Optional[str] = None) -> not annotated |
Reconnect to server(s). | #L285-L301 |
vllm_mlx.mcp.manager.MCPClientManager.refresh_tools |
method | async vllm_mlx.mcp.manager.MCPClientManager.refresh_tools() -> not annotated |
Refresh tools from all connected servers. | #L275-L283 |
vllm_mlx.mcp.manager.MCPClientManager.start |
method | async vllm_mlx.mcp.manager.MCPClientManager.start() -> not annotated |
Start the manager and connect to all enabled servers. | #L54-L96 |
vllm_mlx.mcp.manager.MCPClientManager.stop |
method | async vllm_mlx.mcp.manager.MCPClientManager.stop() -> not annotated |
Stop the manager and disconnect from all servers. | #L98-L112 |
vllm_mlx.mcp.security.get_sandbox |
function | vllm_mlx.mcp.security.get_sandbox() -> ToolSandbox |
Get the global tool sandbox instance. | #L841-L846 |
vllm_mlx.mcp.security.get_validator |
function | vllm_mlx.mcp.security.get_validator() -> MCPCommandValidator |
Get the global command validator instance. | #L434-L441 |
vllm_mlx.mcp.security.MCPCommandValidator |
class | vllm_mlx.mcp.security.MCPCommandValidator(allowed_commands: Optional[Set[str]] = None, allow_unsafe: bool = False, custom_whitelist: Optional[Set[str]] = None, check_path_exists: bool = True) |
Validates MCP server commands for security. | #L112-L427 |
vllm_mlx.mcp.security.MCPCommandValidator.__init__ |
method | vllm_mlx.mcp.security.MCPCommandValidator.__init__(allowed_commands: Optional[Set[str]] = None, allow_unsafe: bool = False, custom_whitelist: Optional[Set[str]] = None, check_path_exists: bool = True) -> not annotated |
Initialize the command validator. | #L120-L149 |
vllm_mlx.mcp.security.MCPCommandValidator._check_control_chars |
method | vllm_mlx.mcp.security.MCPCommandValidator._check_control_chars(value: str, context: str, server_name: str) -> None |
Block command separators carried via literal newlines. | #L151-L157 |
vllm_mlx.mcp.security.MCPCommandValidator._check_path_traversal |
method | vllm_mlx.mcp.security.MCPCommandValidator._check_path_traversal(value: str, context: str, server_name: str) -> None |
Block parent-directory traversal, including URL-encoded forms. | #L159-L194 |
vllm_mlx.mcp.security.MCPCommandValidator.validate_args |
method | vllm_mlx.mcp.security.MCPCommandValidator.validate_args(args: List[str], server_name: str) -> None |
Validate command arguments for dangerous patterns. | #L260-L286 |
vllm_mlx.mcp.security.MCPCommandValidator.validate_command |
method | vllm_mlx.mcp.security.MCPCommandValidator.validate_command(command: str, server_name: str) -> None |
Validate that a command is safe to execute. | #L196-L258 |
vllm_mlx.mcp.security.MCPCommandValidator.validate_command_args |
method | vllm_mlx.mcp.security.MCPCommandValidator.validate_command_args(command: str, args: List[str], server_name: str) -> None |
Validate command-specific argument combinations. | #L288-L330 |
vllm_mlx.mcp.security.MCPCommandValidator.validate_env |
method | vllm_mlx.mcp.security.MCPCommandValidator.validate_env(env: Optional[Dict[str, str]], server_name: str) -> None |
Validate environment variables for dangerous values. | #L332-L383 |
vllm_mlx.mcp.security.MCPCommandValidator.validate_url |
method | vllm_mlx.mcp.security.MCPCommandValidator.validate_url(url: str, server_name: str) -> None |
Validate SSE URL for security. | #L385-L427 |
vllm_mlx.mcp.security.MCPSecurityError |
class | vllm_mlx.mcp.security.MCPSecurityError() |
Raised when MCP security validation fails. | #L106-L109 |
vllm_mlx.mcp.security.set_sandbox |
function | vllm_mlx.mcp.security.set_sandbox(sandbox: ToolSandbox) -> None |
Set a custom global sandbox. | #L849-L852 |
vllm_mlx.mcp.security.set_validator |
function | vllm_mlx.mcp.security.set_validator(validator: MCPCommandValidator) -> None |
Set a custom global validator. | #L444-L447 |
vllm_mlx.mcp.security.ToolExecutionAudit |
class | vllm_mlx.mcp.security.ToolExecutionAudit(timestamp: float, tool_name: str, server_name: str, arguments: Dict[str, Any], success: bool, error_message: Optional[str] = None, execution_time_ms: Optional[float] = None) |
Record of a tool execution for audit purposes. | #L516-L525 |
vllm_mlx.mcp.security.ToolSandbox |
class | vllm_mlx.mcp.security.ToolSandbox(allowed_tools: Optional[Set[str]] = None, blocked_tools: Optional[Set[str]] = None, allowed_high_risk_tools: Optional[Set[str]] = None, blocked_arg_patterns: Optional[List[re.Pattern]] = None, max_calls_per_minute: int = 60, audit_callback: Optional[Callable[[ToolExecutionAudit], None]] = None, enabled: bool = True) |
Sandboxing controls for MCP tool execution. | #L528-L834 |
vllm_mlx.mcp.security.ToolSandbox.__init__ |
method | vllm_mlx.mcp.security.ToolSandbox.__init__(allowed_tools: Optional[Set[str]] = None, blocked_tools: Optional[Set[str]] = None, allowed_high_risk_tools: Optional[Set[str]] = None, blocked_arg_patterns: Optional[List[re.Pattern]] = None, max_calls_per_minute: int = 60, audit_callback: Optional[Callable[[ToolExecutionAudit], None]] = None, enabled: bool = True) -> not annotated |
Initialize tool sandbox. | #L539-L586 |
vllm_mlx.mcp.security.ToolSandbox._check_high_risk_tool |
method | vllm_mlx.mcp.security.ToolSandbox._check_high_risk_tool(tool_name: str, full_name: str) -> None |
Check if tool matches high-risk patterns. | #L644-L663 |
vllm_mlx.mcp.security.ToolSandbox._check_rate_limit |
method | vllm_mlx.mcp.security.ToolSandbox._check_rate_limit(full_name: str) -> None |
Check and enforce rate limit for tool calls. | #L688-L710 |
vllm_mlx.mcp.security.ToolSandbox._is_blocked |
method | vllm_mlx.mcp.security.ToolSandbox._is_blocked(tool_name: str, full_name: str) -> bool |
Check if tool is in blocklist. | #L636-L642 |
vllm_mlx.mcp.security.ToolSandbox._sanitize_arguments_for_log |
method | vllm_mlx.mcp.security.ToolSandbox._sanitize_arguments_for_log(arguments: Dict[str, Any]) -> Dict[str, Any] |
Sanitize arguments for logging (redact sensitive data). | #L774-L794 |
vllm_mlx.mcp.security.ToolSandbox._sanitize_arguments_for_log.sanitize |
nested function | vllm_mlx.mcp.security.ToolSandbox._sanitize_arguments_for_log.sanitize(obj: Any) -> Any |
Nested Function `ToolSandbox._sanitize_arguments_for_log.sanitize` calls `isinstance`, `any`, `k.lower`, `sanitize`; has 4 explicit return paths. | #L778-L792 |
vllm_mlx.mcp.security.ToolSandbox._validate_arguments |
method | vllm_mlx.mcp.security.ToolSandbox._validate_arguments(tool_name: str, arguments: Dict[str, Any]) -> None |
Validate tool arguments for dangerous patterns. | #L665-L686 |
vllm_mlx.mcp.security.ToolSandbox._validate_arguments.check_value |
nested function | vllm_mlx.mcp.security.ToolSandbox._validate_arguments.check_value(key: str, value: Any, path: str = '') -> None |
Nested Function `ToolSandbox._validate_arguments.check_value` calls `isinstance`, `pattern.search`, `MCPSecurityError`, `value.items`; can raise `MCPSecurityError`. | #L668-L683 |
vllm_mlx.mcp.security.ToolSandbox.clear_audit_log |
method | vllm_mlx.mcp.security.ToolSandbox.clear_audit_log() -> int |
Clear audit log and return number of entries cleared. | #L829-L834 |
vllm_mlx.mcp.security.ToolSandbox.get_audit_log |
method | vllm_mlx.mcp.security.ToolSandbox.get_audit_log(limit: int = 100, tool_filter: Optional[str] = None, server_filter: Optional[str] = None, errors_only: bool = False) -> List[ToolExecutionAudit] |
Get audit log entries. | #L796-L827 |
vllm_mlx.mcp.security.ToolSandbox.record_execution |
method | vllm_mlx.mcp.security.ToolSandbox.record_execution(tool_name: str, server_name: str, arguments: Dict[str, Any], success: bool, error_message: Optional[str] = None, execution_time_ms: Optional[float] = None) -> ToolExecutionAudit |
Record a tool execution for audit purposes. | #L712-L772 |
vllm_mlx.mcp.security.ToolSandbox.validate_tool_execution |
method | vllm_mlx.mcp.security.ToolSandbox.validate_tool_execution(tool_name: str, server_name: str, arguments: Dict[str, Any]) -> None |
Validate that a tool execution is allowed. | #L588-L634 |
vllm_mlx.mcp.security.validate_mcp_server_config |
function | vllm_mlx.mcp.security.validate_mcp_server_config(server_name: str, command: Optional[str] = None, args: Optional[List[str]] = None, env: Optional[Dict[str, str]] = None, url: Optional[str] = None) -> None |
Validate MCP server configuration for security. | #L450-L486 |
vllm_mlx.mcp.tools.extract_tool_calls |
function | vllm_mlx.mcp.tools.extract_tool_calls(response: Dict[str, Any]) -> List[Dict[str, Any]] |
Extract tool calls from model response. | #L146-L161 |
vllm_mlx.mcp.tools.format_tool_result |
function | vllm_mlx.mcp.tools.format_tool_result(result: MCPToolResult, tool_call_id: str) -> Dict[str, Any] |
Format tool result for inclusion in conversation messages. | #L87-L98 |
vllm_mlx.mcp.tools.format_tool_results |
function | vllm_mlx.mcp.tools.format_tool_results(results: List[Tuple[MCPToolResult, str]]) -> List[Dict[str, Any]] |
Format multiple tool results as messages. | #L101-L113 |
vllm_mlx.mcp.tools.has_tool_calls |
function | vllm_mlx.mcp.tools.has_tool_calls(response: Dict[str, Any]) -> bool |
Check if response contains tool calls. | #L164-L174 |
vllm_mlx.mcp.tools.mcp_tool_to_openai |
function | vllm_mlx.mcp.tools.mcp_tool_to_openai(tool: MCPTool) -> Dict[str, Any] |
Convert MCP tool schema to OpenAI function calling format. | #L12-L33 |
vllm_mlx.mcp.tools.mcp_tools_to_openai |
function | vllm_mlx.mcp.tools.mcp_tools_to_openai(tools: List[MCPTool]) -> List[Dict[str, Any]] |
Convert list of MCP tools to OpenAI format. | #L36-L46 |
vllm_mlx.mcp.tools.merge_tools |
function | vllm_mlx.mcp.tools.merge_tools(mcp_tools: List[MCPTool], user_tools: Optional[List[Dict[str, Any]]] = None) -> List[Dict[str, Any]] |
Merge MCP tools with user-provided tools. | #L116-L143 |
vllm_mlx.mcp.tools.openai_call_to_mcp |
function | vllm_mlx.mcp.tools.openai_call_to_mcp(tool_call: Dict[str, Any]) -> Tuple[str, str, Dict[str, Any]] |
Parse OpenAI tool call back to MCP format. | #L49-L84 |
vllm_mlx.mcp.types.MCPConfig |
class | vllm_mlx.mcp.types.MCPConfig(servers: Dict[str, MCPServerConfig] = field(default_factory=dict), max_tool_calls: int = 10, default_timeout: float = 30.0, allowed_high_risk_tools: Set[str] = field(default_factory=set)) |
Root configuration for MCP client. | #L82-L103 |
vllm_mlx.mcp.types.MCPConfig.from_dict |
method | vllm_mlx.mcp.types.MCPConfig.from_dict(data: Dict[str, Any]) -> 'MCPConfig' |
Create config from dictionary. | #L91-L103 |
vllm_mlx.mcp.types.MCPServerConfig |
class | vllm_mlx.mcp.types.MCPServerConfig(name: str, transport: MCPTransport = MCPTransport.STDIO, command: Optional[str] = None, args: Optional[List[str]] = None, env: Optional[Dict[str, str]] = None, url: Optional[str] = None, enabled: bool = True, timeout: float = 30.0) |
Configuration for a single MCP server. | #L28-L78 |
vllm_mlx.mcp.types.MCPServerConfig.__post_init__ |
method | vllm_mlx.mcp.types.MCPServerConfig.__post_init__() -> not annotated |
Validate configuration. | #L46-L63 |
vllm_mlx.mcp.types.MCPServerConfig._validate_security |
method | vllm_mlx.mcp.types.MCPServerConfig._validate_security() -> None |
Validate security of the configuration. | #L65-L78 |
vllm_mlx.mcp.types.MCPServerState |
class | vllm_mlx.mcp.types.MCPServerState() |
MCP server connection states. | #L18-L24 |
vllm_mlx.mcp.types.MCPServerStatus |
class | vllm_mlx.mcp.types.MCPServerStatus(name: str, state: MCPServerState, transport: MCPTransport, tools_count: int = 0, error: Optional[str] = None, last_connected: Optional[float] = None) |
Status of an MCP server connection. | #L160-L179 |
vllm_mlx.mcp.types.MCPServerStatus.to_dict |
method | vllm_mlx.mcp.types.MCPServerStatus.to_dict() -> Dict[str, Any] |
Convert to dictionary for API response. | #L170-L179 |
vllm_mlx.mcp.types.MCPTool |
class | vllm_mlx.mcp.types.MCPTool(server_name: str, name: str, description: str, input_schema: Dict[str, Any] = field(default_factory=dict)) |
Normalized tool representation from MCP server. | #L107-L129 |
vllm_mlx.mcp.types.MCPTool.full_name |
method | vllm_mlx.mcp.types.MCPTool.full_name() -> str |
Get namespaced tool name (server__tool). | #L116-L118 |
vllm_mlx.mcp.types.MCPTool.to_openai_format |
method | vllm_mlx.mcp.types.MCPTool.to_openai_format() -> Dict[str, Any] |
Convert to OpenAI function calling format. | #L120-L129 |
vllm_mlx.mcp.types.MCPToolResult |
class | vllm_mlx.mcp.types.MCPToolResult(tool_name: str, content: Any, is_error: bool = False, error_message: Optional[str] = None) |
Result from a tool execution. | #L133-L156 |
vllm_mlx.mcp.types.MCPToolResult.to_message |
method | vllm_mlx.mcp.types.MCPToolResult.to_message(tool_call_id: str) -> Dict[str, Any] |
Convert to OpenAI tool result message format. | #L141-L156 |
vllm_mlx.mcp.types.MCPTransport |
class | vllm_mlx.mcp.types.MCPTransport() |
Supported MCP transport types. | #L11-L15 |
vllm_mlx.memory_cache._array_memory |
function | vllm_mlx.memory_cache._array_memory(arr) -> int |
Estimate array memory from shape+dtype without triggering lazy eval. | #L66-L88 |
vllm_mlx.memory_cache._CacheEntry |
class | vllm_mlx.memory_cache._CacheEntry(tokens: tuple[int, ...], cache: list[Any], memory_bytes: int) |
Internal cache entry with memory tracking. | #L272-L287 |
vllm_mlx.memory_cache._CacheEntry.create |
method | vllm_mlx.memory_cache._CacheEntry.create(tokens: list[int], cache: list[Any]) -> _CacheEntry |
Create a cache entry with memory estimation. | #L280-L287 |
vllm_mlx.memory_cache._compute_model_fingerprint |
function | vllm_mlx.memory_cache._compute_model_fingerprint(model: Any) -> str |
Compute a fingerprint from model architecture for cache compatibility. | #L648-L682 |
vllm_mlx.memory_cache._dequantize_cache |
function | vllm_mlx.memory_cache._dequantize_cache(cache: list[Any]) -> list[Any] |
Dequantize _QuantizedCacheWrapper layers and copy non-quantized layers. | #L593-L645 |
vllm_mlx.memory_cache._get_available_memory |
function | vllm_mlx.memory_cache._get_available_memory() -> int |
Get available system memory in bytes. | #L47-L63 |
vllm_mlx.memory_cache._is_cache_layer_trimmable |
function | vllm_mlx.memory_cache._is_cache_layer_trimmable(layer_cache: Any) -> bool |
Return whether a cache layer can safely be rewound for partial reuse. | #L290-L314 |
vllm_mlx.memory_cache._needs_kv_trim |
function | vllm_mlx.memory_cache._needs_kv_trim(layer: Any) -> bool |
Check if a cache layer has oversized KV arrays (duck-typed, no MLX import). | #L484-L495 |
vllm_mlx.memory_cache._nested_array_memory |
function | vllm_mlx.memory_cache._nested_array_memory(value: Any) -> int |
Sum ``_array_memory`` over an arbitrarily nested state structure. | #L91-L105 |
vllm_mlx.memory_cache._quantize_cache |
function | vllm_mlx.memory_cache._quantize_cache(cache: list[Any], bits: int = 8, group_size: int = 64) -> list[Any] |
Quantize KV cache layers to reduce memory. | #L574-L590 |
vllm_mlx.memory_cache._QuantizedCacheWrapper |
class | vllm_mlx.memory_cache._QuantizedCacheWrapper(layer: Any, bits: int, group_size: int) |
Lightweight wrapper storing quantized KV arrays + original cache metadata. | #L541-L571 |
vllm_mlx.memory_cache._QuantizedCacheWrapper.__init__ |
method | vllm_mlx.memory_cache._QuantizedCacheWrapper.__init__(layer: Any, bits: int, group_size: int) -> not annotated |
Method `_QuantizedCacheWrapper.__init__` updates `self.keys`, `self.values`, `self.offset`, `self.bits`; calls `mx.quantize`, `type`, `hasattr`, `getattr`. | #L558-L571 |
vllm_mlx.memory_cache._trim_cache_offset |
function | vllm_mlx.memory_cache._trim_cache_offset(cache: list[Any], trim_by: int) -> list[Any] |
Create copies of cache layers with the last ``trim_by`` positions removed. | #L317-L481 |
vllm_mlx.memory_cache._trim_to_offset |
function | vllm_mlx.memory_cache._trim_to_offset(cache: list[Any]) -> list[Any] |
Trim KV arrays to their actual used size (offset) before storage. | #L498-L538 |
vllm_mlx.memory_cache.CacheStats |
class | vllm_mlx.memory_cache.CacheStats(hits: int = 0, misses: int = 0, evictions: int = 0, tokens_saved: int = 0, current_memory_bytes: int = 0, max_memory_bytes: int = 0, entry_count: int = 0) |
Statistics for cache performance monitoring. | #L229-L268 |
vllm_mlx.memory_cache.CacheStats.hit_rate |
method | vllm_mlx.memory_cache.CacheStats.hit_rate() -> float |
Return successful lookups divided by all completed lookups. | #L241-L245 |
vllm_mlx.memory_cache.CacheStats.memory_utilization |
method | vllm_mlx.memory_cache.CacheStats.memory_utilization() -> float |
Return the fraction of the configured memory budget in use. | #L248-L253 |
vllm_mlx.memory_cache.CacheStats.to_dict |
method | vllm_mlx.memory_cache.CacheStats.to_dict() -> dict[str, Any] |
Return rounded cache counters and memory values for APIs and logs. | #L255-L268 |
vllm_mlx.memory_cache.estimate_kv_cache_memory |
function | vllm_mlx.memory_cache.estimate_kv_cache_memory(cache: list[Any]) -> int |
Estimate memory usage of a KV cache in bytes. | #L108-L162 |
vllm_mlx.memory_cache.MemoryAwarePrefixCache |
class | vllm_mlx.memory_cache.MemoryAwarePrefixCache(model: Any, config: MemoryCacheConfig | None = None) |
Prefix cache with memory-based eviction. | #L685-L1463 |
vllm_mlx.memory_cache.MemoryAwarePrefixCache.__contains__ |
method | vllm_mlx.memory_cache.MemoryAwarePrefixCache.__contains__(tokens: list[int]) -> bool |
Check if tokens are cached. | #L1200-L1202 |
vllm_mlx.memory_cache.MemoryAwarePrefixCache.__init__ |
method | vllm_mlx.memory_cache.MemoryAwarePrefixCache.__init__(model: Any, config: MemoryCacheConfig | None = None) -> None |
Initialize the memory-aware prefix cache. | #L703-L746 |
vllm_mlx.memory_cache.MemoryAwarePrefixCache.__len__ |
method | vllm_mlx.memory_cache.MemoryAwarePrefixCache.__len__() -> int |
Return number of cached entries. | #L1196-L1198 |
vllm_mlx.memory_cache.MemoryAwarePrefixCache._evict_lru |
method | vllm_mlx.memory_cache.MemoryAwarePrefixCache._evict_lru() -> None |
Evict the least recently used entry. | #L1100-L1126 |
vllm_mlx.memory_cache.MemoryAwarePrefixCache._remove_from_sorted |
method | vllm_mlx.memory_cache.MemoryAwarePrefixCache._remove_from_sorted(key: tuple[int, ...]) -> None |
Remove a key from the sorted index using bisect for O(log N). | #L1094-L1098 |
vllm_mlx.memory_cache.MemoryAwarePrefixCache.check_ssd |
method | vllm_mlx.memory_cache.MemoryAwarePrefixCache.check_ssd(tokens: list[int]) -> dict | None |
Check if tokens have an SSD cache hit (without reading data). | #L1216-L1249 |
vllm_mlx.memory_cache.MemoryAwarePrefixCache.clear |
method | vllm_mlx.memory_cache.MemoryAwarePrefixCache.clear() -> None |
Clear all cached entries. | #L1149-L1156 |
vllm_mlx.memory_cache.MemoryAwarePrefixCache.fetch |
method | vllm_mlx.memory_cache.MemoryAwarePrefixCache.fetch(tokens: list[int]) -> tuple[list[Any] | None, list[int]] |
Find cached KV state for the given tokens. | #L748-L977 |
vllm_mlx.memory_cache.MemoryAwarePrefixCache.get_stats |
method | vllm_mlx.memory_cache.MemoryAwarePrefixCache.get_stats() -> dict[str, Any] |
Get cache statistics. | #L1158-L1160 |
vllm_mlx.memory_cache.MemoryAwarePrefixCache.load_from_disk |
method | vllm_mlx.memory_cache.MemoryAwarePrefixCache.load_from_disk(cache_dir: str) -> int |
Load cache entries from disk. | #L1348-L1463 |
vllm_mlx.memory_cache.MemoryAwarePrefixCache.memory_limit_mb |
method | vllm_mlx.memory_cache.MemoryAwarePrefixCache.memory_limit_mb() -> float |
Memory limit in MB. | #L1177-L1179 |
vllm_mlx.memory_cache.MemoryAwarePrefixCache.memory_usage_mb |
method | vllm_mlx.memory_cache.MemoryAwarePrefixCache.memory_usage_mb() -> float |
Current memory usage in MB. | #L1172-L1174 |
vllm_mlx.memory_cache.MemoryAwarePrefixCache.release_reserved_memory |
method | vllm_mlx.memory_cache.MemoryAwarePrefixCache.release_reserved_memory(nbytes: int) -> None |
Release memory previously reserved by try_reserve_memory(). | #L1190-L1194 |
vllm_mlx.memory_cache.MemoryAwarePrefixCache.remove |
method | vllm_mlx.memory_cache.MemoryAwarePrefixCache.remove(tokens: list[int]) -> bool |
Remove a specific cache entry. | #L1128-L1147 |
vllm_mlx.memory_cache.MemoryAwarePrefixCache.reset_stats |
method | vllm_mlx.memory_cache.MemoryAwarePrefixCache.reset_stats() -> None |
Reset statistics while preserving cache contents. | #L1162-L1169 |
vllm_mlx.memory_cache.MemoryAwarePrefixCache.save_to_disk |
method | vllm_mlx.memory_cache.MemoryAwarePrefixCache.save_to_disk(cache_dir: str) -> bool |
Save all cache entries to disk using mlx_lm's safetensors format. | #L1255-L1346 |
vllm_mlx.memory_cache.MemoryAwarePrefixCache.set_ssd_tier |
method | vllm_mlx.memory_cache.MemoryAwarePrefixCache.set_ssd_tier(ssd_tier) -> None |
Attach an SSD cache tier for eviction spilling. | #L1204-L1214 |
vllm_mlx.memory_cache.MemoryAwarePrefixCache.store |
method | vllm_mlx.memory_cache.MemoryAwarePrefixCache.store(tokens: list[int], cache: list[Any], evict_prefixes: bool = True) -> bool |
Store KV cache for future reuse. | #L979-L1092 |
vllm_mlx.memory_cache.MemoryAwarePrefixCache.try_reserve_memory |
method | vllm_mlx.memory_cache.MemoryAwarePrefixCache.try_reserve_memory(nbytes: int) -> bool |
Tentatively reserve cache memory for an upcoming promotion. | #L1181-L1188 |
vllm_mlx.memory_cache.MemoryCacheConfig |
class | vllm_mlx.memory_cache.MemoryCacheConfig(max_memory_mb: int | None = None, max_memory_percent: float = _DEFAULT_MEMORY_PERCENT, max_entries: int = 1000, enable_memory_tracking: bool = True, kv_quantize: bool = False, kv_bits: int = 8, kv_group_size: int = 64, kv_min_quantize_tokens: int = 256, min_prefix_tokens: int = 128) |
Configuration for memory-aware prefix cache. | #L166-L225 |
vllm_mlx.memory_cache.MemoryCacheConfig.__post_init__ |
method | vllm_mlx.memory_cache.MemoryCacheConfig.__post_init__() -> None |
Method `MemoryCacheConfig.__post_init__` calls `ValueError`; can raise `ValueError`. | #L192-L206 |
vllm_mlx.memory_cache.MemoryCacheConfig.compute_memory_limit |
method | vllm_mlx.memory_cache.MemoryCacheConfig.compute_memory_limit() -> int |
Compute the memory limit in bytes. | #L208-L225 |
vllm_mlx.metrics._bool_str |
function | vllm_mlx.metrics._bool_str(value: bool) -> str |
Function `_bool_str` returns `'true' if value else 'false'`. | #L17-L18 |
vllm_mlx.metrics._coerce_float |
function | vllm_mlx.metrics._coerce_float(value: Any, default: float = 0.0) -> float |
Function `_coerce_float` calls `float`; has 2 explicit return paths. | #L21-L27 |
vllm_mlx.metrics._coerce_int |
function | vllm_mlx.metrics._coerce_int(value: Any, default: int = 0) -> int |
Function `_coerce_int` calls `int`; has 2 explicit return paths. | #L30-L36 |
vllm_mlx.metrics.InferenceTracker |
class | vllm_mlx.metrics.InferenceTracker(collector: 'MetricsCollector | None', endpoint: str, stream: bool, start_time: float = field(default_factory=time.perf_counter), _finished: bool = False, _ttft_observed: bool = False) |
Request-scoped inference timing and token accounting. | #L40-L81 |
vllm_mlx.metrics.InferenceTracker.finish |
method | vllm_mlx.metrics.InferenceTracker.finish(*, result: str, prompt_tokens: int = 0, completion_tokens: int = 0) -> None |
Record terminal latency and token counts once for this request. | #L62-L81 |
vllm_mlx.metrics.InferenceTracker.observe_ttft |
method | vllm_mlx.metrics.InferenceTracker.observe_ttft() -> None |
Record time to first token once for this inference request. | #L50-L60 |
vllm_mlx.metrics.MetricsCollector |
class | vllm_mlx.metrics.MetricsCollector() |
Lazy Prometheus-backed metrics collector. | #L84-L529 |
vllm_mlx.metrics.MetricsCollector.__init__ |
method | vllm_mlx.metrics.MetricsCollector.__init__() -> None |
Method `MetricsCollector.__init__` updates `self._enabled`, `self._lock`, `self._prom`; calls `threading.Lock`. | #L87-L90 |
vllm_mlx.metrics.MetricsCollector._init_prometheus |
method | vllm_mlx.metrics.MetricsCollector._init_prometheus() -> None |
Method `MetricsCollector._init_prometheus` updates `self._prom`; calls `CollectorRegistry`, `Counter`, `Histogram`, `Gauge`. | #L107-L291 |
vllm_mlx.metrics.MetricsCollector._update_engine_gauges |
method | vllm_mlx.metrics.MetricsCollector._update_engine_gauges(*, engine: Any | None, mcp_manager: Any | None) -> None |
Method `MetricsCollector._update_engine_gauges` calls `engine.get_stats`, `self._prom['model_loaded'].set`, `stats.get`, `self._prom['engine_type'].labels(engine_type=engine_type).set`. | #L375-L507 |
vllm_mlx.metrics.MetricsCollector.configure |
method | vllm_mlx.metrics.MetricsCollector.configure(*, enabled: bool) -> None |
Enable or disable collection and lazily initialize Prometheus state. | #L98-L105 |
vllm_mlx.metrics.MetricsCollector.enabled |
method | vllm_mlx.metrics.MetricsCollector.enabled() -> bool |
Return whether metric collection is enabled. | #L93-L96 |
vllm_mlx.metrics.MetricsCollector.observe_http_finish |
method | vllm_mlx.metrics.MetricsCollector.observe_http_finish(*, method: str, path: str, status_code: int, duration: float) -> None |
Record an HTTP result and decrement its in-flight gauge. | #L307-L328 |
vllm_mlx.metrics.MetricsCollector.observe_http_start |
method | vllm_mlx.metrics.MetricsCollector.observe_http_start(*, method: str, path: str) -> None |
Increment the in-flight request gauge for a normalized route. | #L300-L305 |
vllm_mlx.metrics.MetricsCollector.observe_inference |
method | vllm_mlx.metrics.MetricsCollector.observe_inference(*, endpoint: str, stream: bool, result: str, duration: float, prompt_tokens: int, completion_tokens: int) -> None |
Record one terminal inference outcome, latency, and token totals. | #L330-L363 |
vllm_mlx.metrics.MetricsCollector.observe_ttft |
method | vllm_mlx.metrics.MetricsCollector.observe_ttft(*, endpoint: str, stream: bool, value: float) -> None |
Observe time to first token for a streaming or buffered request. | #L365-L373 |
vllm_mlx.metrics.MetricsCollector.render_metrics |
method | vllm_mlx.metrics.MetricsCollector.render_metrics(*, engine: Any | None, mcp_manager: Any | None) -> tuple[bytes, str] |
Refresh runtime gauges and render Prometheus exposition bytes. | #L509-L529 |
vllm_mlx.metrics.MetricsCollector.track_inference |
method | vllm_mlx.metrics.MetricsCollector.track_inference(endpoint: str, *, stream: bool) -> InferenceTracker |
Create request-scoped inference timing state for an endpoint. | #L293-L298 |
vllm_mlx.mllm_batch_generator._accept_sampled_draft |
function | vllm_mlx.mllm_batch_generator._accept_sampled_draft(target_logprob: float, draft_logprob: float, uniform_draw: float) -> bool |
Apply the exact min(1, p/q) stochastic speculative acceptance rule. | #L142-L149 |
vllm_mlx.mllm_batch_generator._cache_eval_tensors |
function | vllm_mlx.mllm_batch_generator._cache_eval_tensors(cache: List[Any]) -> List[Any] |
Return realized tensors that break lazy cache graphs between chunks. | #L160-L183 |
vllm_mlx.mllm_batch_generator._drop_retired_processors |
function | vllm_mlx.mllm_batch_generator._drop_retired_processors(processors: Optional[List[Callable]]) -> tuple[Optional[List[Callable]], int] |
Drop retire-capable processors that have completed their work. | #L60-L74 |
vllm_mlx.mllm_batch_generator._eval_prompt_cache |
function | vllm_mlx.mllm_batch_generator._eval_prompt_cache(cache: List[Any]) -> None |
Evaluate all cache tensors used by hybrid chunked prefill. | #L186-L190 |
vllm_mlx.mllm_batch_generator._left_pad_prompts |
function | vllm_mlx.mllm_batch_generator._left_pad_prompts(prompts: List[List[int]], max_length: Optional[int] = None) -> mx.array |
Left-pad prompts to uniform length. | #L439-L454 |
vllm_mlx.mllm_batch_generator._mark_mtp_attempts_on_primary_responses |
function | vllm_mlx.mllm_batch_generator._mark_mtp_attempts_on_primary_responses(responses: List['MLLMBatchResponse'], attempted_drafts_by_uid: Dict[int, int]) -> None |
Mark only responses from steps that actually attempted MTP drafts. | #L46-L57 |
vllm_mlx.mllm_batch_generator._processors_can_retire |
function | vllm_mlx.mllm_batch_generator._processors_can_retire(processors: Optional[List[Callable]]) -> bool |
True when any processor advertises a retire-to-content transition. | #L37-L43 |
vllm_mlx.mllm_batch_generator._request_uses_stochastic_sampling |
function | vllm_mlx.mllm_batch_generator._request_uses_stochastic_sampling(request: Any) -> bool |
Return whether a request needs sampler-aware speculative verification. | #L77-L92 |
vllm_mlx.mllm_batch_generator._residual_logprobs |
function | vllm_mlx.mllm_batch_generator._residual_logprobs(target_logprobs: mx.array, draft_logprobs: mx.array) -> mx.array |
Return the normalized residual max(target - draft, 0) distribution. | #L126-L139 |
vllm_mlx.mllm_batch_generator._sampling_logprobs |
function | vllm_mlx.mllm_batch_generator._sampling_logprobs(logits: mx.array, request: Any) -> mx.array |
Match mlx-lm's request sampler in log-probability space. | #L95-L123 |
vllm_mlx.mllm_batch_generator.install_chunked_prefill_mllm |
function | vllm_mlx.mllm_batch_generator.install_chunked_prefill_mllm(batch_gen: 'MLLMBatchGenerator', budget: int = 1024) -> None |
Install interleaved prefill/decode on an MLLMBatchGenerator. | #L2593-L3073 |
vllm_mlx.mllm_batch_generator.install_chunked_prefill_mllm._chunked_next |
nested function | vllm_mlx.mllm_batch_generator.install_chunked_prefill_mllm._chunked_next() -> List[MLLMBatchResponse] |
Interleaved prefill/decode: one prefill chunk + one gen step. | #L2715-L3058 |
vllm_mlx.mllm_batch_generator.install_chunked_prefill_mllm._generation_step |
nested function | vllm_mlx.mllm_batch_generator.install_chunked_prefill_mllm._generation_step() -> List[MLLMBatchResponse] |
Run one generation step for the active batch. | #L2623-L2713 |
vllm_mlx.mllm_batch_generator.install_chunked_prefill_mllm._patched_remove |
nested function | vllm_mlx.mllm_batch_generator.install_chunked_prefill_mllm._patched_remove(uids: List[int]) -> None |
Nested Function `install_chunked_prefill_mllm._patched_remove` calls `set`, `mx.clear_cache`, `_orig_remove`. | #L3063-L3068 |
vllm_mlx.mllm_batch_generator.install_mtp_mllm |
function | vllm_mlx.mllm_batch_generator.install_mtp_mllm(batch_gen: 'MLLMBatchGenerator', language_model: Any, num_draft_tokens: int = 1) -> None |
Install MTP (Multi-Token Prediction) on an MLLMBatchGenerator. | #L2045-L2590 |
vllm_mlx.mllm_batch_generator.install_mtp_mllm._get_mtp_stats |
nested function | vllm_mlx.mllm_batch_generator.install_mtp_mllm._get_mtp_stats() -> Dict[str, Any] |
Nested Function `install_mtp_mllm._get_mtp_stats` calls `dict`; returns `{'enabled': True, 'requested_draft_tokens': num_draft_tokens, 'effective_draft_tokens': 1, 'mode': 'request_local_sampl…`. | #L2089-L2110 |
vllm_mlx.mllm_batch_generator.install_mtp_mllm._mtp_next |
nested function | vllm_mlx.mllm_batch_generator.install_mtp_mllm._mtp_next() -> List[MLLMBatchResponse] |
Wrapper around _next that emits deferred MTP draft tokens. | #L2460-L2576 |
vllm_mlx.mllm_batch_generator.install_mtp_mllm._mtp_step |
nested function | vllm_mlx.mllm_batch_generator.install_mtp_mllm._mtp_step(input_tokens: mx.array, cache: List[Any], logits_processors: Optional[List[Optional[List[Callable]]]] = None, output_tokens: Optional[List[List[int]]] = None, samplers: Optional[List[Optional[Callable]]] = None) -> Tuple[mx.array, List[mx.array]] |
Extended _step with MTP always-advance strategy. | #L2114-L2455 |
vllm_mlx.mllm_batch_generator.MLLMBatch |
class | vllm_mlx.mllm_batch_generator.MLLMBatch(uids: List[int], request_ids: List[str], y: mx.array, logprobs: List[mx.array], max_tokens: List[int], num_tokens: List[int], cache: List[Any], requests: List[MLLMBatchRequest], logits_processors: Optional[List[Optional[List[Callable]]]] = None, samplers: Optional[List[Optional[Callable]]] = None) |
Represents an active batch of MLLM requests. | #L260-L392 |
vllm_mlx.mllm_batch_generator.MLLMBatch.__len__ |
method | vllm_mlx.mllm_batch_generator.MLLMBatch.__len__() -> int |
Method `MLLMBatch.__len__` calls `len`; returns `len(self.uids)`. | #L279-L280 |
vllm_mlx.mllm_batch_generator.MLLMBatch.extend |
method | vllm_mlx.mllm_batch_generator.MLLMBatch.extend(other: 'MLLMBatch') -> None |
Extend this batch with another batch. | #L308-L351 |
vllm_mlx.mllm_batch_generator.MLLMBatch.extract_cache |
method | vllm_mlx.mllm_batch_generator.MLLMBatch.extract_cache(idx: int) -> List[Any] |
Extract cache for a single request (for prefix caching). | #L353-L392 |
vllm_mlx.mllm_batch_generator.MLLMBatch.filter |
method | vllm_mlx.mllm_batch_generator.MLLMBatch.filter(keep_idx: List[int]) -> None |
Filter batch to keep only requests at specified indices. | #L282-L306 |
vllm_mlx.mllm_batch_generator.MLLMBatchGenerator |
class | vllm_mlx.mllm_batch_generator.MLLMBatchGenerator(model: nn.Module, processor: Any, mm_processor: Optional[MultimodalProcessor] = None, max_tokens: int = 256, stop_tokens: Optional[set] = None, sampler: Optional[Callable[[mx.array], mx.array]] = None, prefill_batch_size: int = 4, completion_batch_size: int = 16, prefill_step_size: int = 1024, enable_vision_cache: bool = True, vision_cache_size: int = 100, prefix_cache_config: Optional[MemoryCacheConfig] = None, max_kv_size: int = 0) |
Batch generator for Vision Language Models. | #L457-L2042 |
vllm_mlx.mllm_batch_generator.MLLMBatchGenerator.__del__ |
method | vllm_mlx.mllm_batch_generator.MLLMBatchGenerator.__del__() -> not annotated |
Method `MLLMBatchGenerator.__del__` calls `self.close`. | #L810-L814 |
vllm_mlx.mllm_batch_generator.MLLMBatchGenerator.__init__ |
method | vllm_mlx.mllm_batch_generator.MLLMBatchGenerator.__init__(model: nn.Module, processor: Any, mm_processor: Optional[MultimodalProcessor] = None, max_tokens: int = 256, stop_tokens: Optional[set] = None, sampler: Optional[Callable[[mx.array], mx.array]] = None, prefill_batch_size: int = 4, completion_batch_size: int = 16, prefill_step_size: int = 1024, enable_vision_cache: bool = True, vision_cache_size: int = 100, prefix_cache_config: Optional[MemoryCacheConfig] = None, max_kv_size: int = 0) -> not annotated |
Initialize MLLM batch generator. | #L484-L632 |
vllm_mlx.mllm_batch_generator.MLLMBatchGenerator._compute_think_suffix_len |
method | vllm_mlx.mllm_batch_generator.MLLMBatchGenerator._compute_think_suffix_len() -> int |
Compute how many extra tokens enable_thinking=True adds at the END. | #L699-L758 |
vllm_mlx.mllm_batch_generator.MLLMBatchGenerator._copy_prefix_cache |
method | vllm_mlx.mllm_batch_generator.MLLMBatchGenerator._copy_prefix_cache(cache_list) -> not annotated |
Create shallow copies of cache objects to prevent mutation of stored prefix cache. | #L1026-L1054 |
vllm_mlx.mllm_batch_generator.MLLMBatchGenerator._has_empty_rotating_cache |
method | vllm_mlx.mllm_batch_generator.MLLMBatchGenerator._has_empty_rotating_cache(cache_list) -> not annotated |
Check if any RotatingKVCache layer has no data (keys=None). | #L1057-L1069 |
vllm_mlx.mllm_batch_generator.MLLMBatchGenerator._maybe_store_prefix_cache |
method | vllm_mlx.mllm_batch_generator.MLLMBatchGenerator._maybe_store_prefix_cache(batch: MLLMBatch, end_indices: List[int]) -> None |
Store KV caches for finished text-only requests into prefix cache. | #L1986-L2014 |
vllm_mlx.mllm_batch_generator.MLLMBatchGenerator._next |
method | vllm_mlx.mllm_batch_generator.MLLMBatchGenerator._next() -> List[MLLMBatchResponse] |
Internal next() implementation. | #L1748-L1964 |
vllm_mlx.mllm_batch_generator.MLLMBatchGenerator._normalize_chat_template_for_prefix_cache |
method | vllm_mlx.mllm_batch_generator.MLLMBatchGenerator._normalize_chat_template_for_prefix_cache() -> None |
Patch chat template so historical assistant turns are prefix-stable. | #L634-L697 |
vllm_mlx.mllm_batch_generator.MLLMBatchGenerator._preprocess_request |
method | vllm_mlx.mllm_batch_generator.MLLMBatchGenerator._preprocess_request(request: MLLMBatchRequest) -> None |
Preprocess a single MLLM request (vision encoding). | #L872-L1023 |
vllm_mlx.mllm_batch_generator.MLLMBatchGenerator._process_prompts |
method | vllm_mlx.mllm_batch_generator.MLLMBatchGenerator._process_prompts(requests: List[MLLMBatchRequest]) -> MLLMBatch |
Process a batch of requests through vision encoding and initial prefill. | #L1260-L1682 |
vllm_mlx.mllm_batch_generator.MLLMBatchGenerator._process_prompts._sample_first_token |
nested function | vllm_mlx.mllm_batch_generator.MLLMBatchGenerator._process_prompts._sample_first_token(req: MLLMBatchRequest, logits: mx.array) -> not annotated |
Nested Function `MLLMBatchGenerator._process_prompts._sample_first_token` calls `logits_processors_by_request.get`, `mx.array`, `processor`, `mx.logsumexp`; returns `(sampled, logprobs)`. | #L1348-L1362 |
vllm_mlx.mllm_batch_generator.MLLMBatchGenerator._run_chunked_text_prefill |
method | vllm_mlx.mllm_batch_generator.MLLMBatchGenerator._run_chunked_text_prefill(request: MLLMBatchRequest, cache: List[Any]) -> mx.array |
Run prefill in chunks for text-only requests, reporting real progress. | #L1108-L1206 |
vllm_mlx.mllm_batch_generator.MLLMBatchGenerator._run_vision_encoding |
method | vllm_mlx.mllm_batch_generator.MLLMBatchGenerator._run_vision_encoding(request: MLLMBatchRequest, cache: Optional[List[Any]] = None) -> mx.array |
Run the initial VLM forward pass to encode vision and get first logits. | #L1208-L1258 |
vllm_mlx.mllm_batch_generator.MLLMBatchGenerator._step |
method | vllm_mlx.mllm_batch_generator.MLLMBatchGenerator._step(input_tokens: mx.array, cache: List[Any], logits_processors: Optional[List[Optional[List[Callable]]]] = None, output_tokens: Optional[List[List[int]]] = None, samplers: Optional[List[Optional[Callable]]] = None) -> Tuple[mx.array, List[mx.array]] |
Run one generation step through the language model. | #L1684-L1746 |
vllm_mlx.mllm_batch_generator.MLLMBatchGenerator._trim_rotating_caches |
method | vllm_mlx.mllm_batch_generator.MLLMBatchGenerator._trim_rotating_caches(cache_list) -> not annotated |
Trim RotatingKVCache buffers restored from prefix cache. | #L1072-L1106 |
vllm_mlx.mllm_batch_generator.MLLMBatchGenerator.abort_prefill |
method | vllm_mlx.mllm_batch_generator.MLLMBatchGenerator.abort_prefill(request_id: str) -> None |
Signal that a request's prefill should be aborted. | #L767-L775 |
vllm_mlx.mllm_batch_generator.MLLMBatchGenerator.close |
method | vllm_mlx.mllm_batch_generator.MLLMBatchGenerator.close() -> None |
Release resources and reset wired limit. | #L760-L765 |
vllm_mlx.mllm_batch_generator.MLLMBatchGenerator.get_prefill_progress |
method | vllm_mlx.mllm_batch_generator.MLLMBatchGenerator.get_prefill_progress(request_id: str) -> Optional[Tuple[int, int]] |
Return (processed_tokens, total_tokens) or None. | #L2016-L2018 |
vllm_mlx.mllm_batch_generator.MLLMBatchGenerator.get_prefix_cache_stats |
method | vllm_mlx.mllm_batch_generator.MLLMBatchGenerator.get_prefix_cache_stats() -> Dict[str, Any] |
Get KV prefix cache statistics. | #L2024-L2038 |
vllm_mlx.mllm_batch_generator.MLLMBatchGenerator.get_vision_cache_stats |
method | vllm_mlx.mllm_batch_generator.MLLMBatchGenerator.get_vision_cache_stats() -> Dict[str, Any] |
Get vision cache statistics. | #L2020-L2022 |
vllm_mlx.mllm_batch_generator.MLLMBatchGenerator.has_pending |
method | vllm_mlx.mllm_batch_generator.MLLMBatchGenerator.has_pending() -> bool |
Check if there are pending or active requests. | #L2040-L2042 |
vllm_mlx.mllm_batch_generator.MLLMBatchGenerator.insert |
method | vllm_mlx.mllm_batch_generator.MLLMBatchGenerator.insert(requests: List[MLLMBatchRequest]) -> List[int] |
Insert requests for batch processing. | #L816-L846 |
vllm_mlx.mllm_batch_generator.MLLMBatchGenerator.next |
method | vllm_mlx.mllm_batch_generator.MLLMBatchGenerator.next() -> List[MLLMBatchResponse] |
Generate next token for all requests in the batch. | #L1966-L1974 |
vllm_mlx.mllm_batch_generator.MLLMBatchGenerator.process_pending_removals |
method | vllm_mlx.mllm_batch_generator.MLLMBatchGenerator.process_pending_removals() -> None |
Remove any UIDs enqueued via :meth:`schedule_removal`. | #L791-L808 |
vllm_mlx.mllm_batch_generator.MLLMBatchGenerator.remove |
method | vllm_mlx.mllm_batch_generator.MLLMBatchGenerator.remove(uids: List[int]) -> None |
Remove requests from processing. | #L848-L870 |
vllm_mlx.mllm_batch_generator.MLLMBatchGenerator.schedule_removal |
method | vllm_mlx.mllm_batch_generator.MLLMBatchGenerator.schedule_removal(uids: List[int]) -> None |
Thread-safe deferred removal of UIDs from the batch. | #L777-L789 |
vllm_mlx.mllm_batch_generator.MLLMBatchGenerator.stats |
method | vllm_mlx.mllm_batch_generator.MLLMBatchGenerator.stats() -> MLLMBatchStats |
Get generation statistics. | #L1976-L1984 |
vllm_mlx.mllm_batch_generator.MLLMBatchRequest |
class | vllm_mlx.mllm_batch_generator.MLLMBatchRequest(uid: int, request_id: str, prompt: str, images: Optional[List[str]] = None, videos: Optional[List[str]] = None, audio: Optional[List[str]] = None, max_tokens: int = 256, temperature: float = 0.7, top_p: float = 0.9, top_k: int = 0, min_p: float = 0.0, presence_penalty: float = 0.0, repetition_penalty: float = 1.0, logits_processors: Optional[List[Callable]] = None, input_ids: Optional[mx.array] = None, pixel_values: Optional[mx.array] = None, attention_mask: Optional[mx.array] = None, image_grid_thw: Optional[mx.array] = None, extra_kwargs: Dict[str, Any] = field(default_factory=dict), is_text_only: bool = False, num_tokens: int = 0, output_tokens: List[int] = field(default_factory=list), vision_encoded: bool = False, cross_attention_states: Optional[Any] = None, encoder_outputs: Optional[Any] = None) |
Request data for MLLM batch processing. | #L194-L237 |
vllm_mlx.mllm_batch_generator.MLLMBatchResponse |
class | vllm_mlx.mllm_batch_generator.MLLMBatchResponse(uid: int, request_id: str, token: int, logprobs: mx.array, finish_reason: Optional[str] = None, prompt_cache: Optional[Callable[[], List[Any]]] = None, from_draft: bool = False, mtp_attempted: bool = False, mtp_attempted_count: int = 0) |
Response from a batch generation step. | #L241-L256 |
vllm_mlx.mllm_batch_generator.MLLMBatchStats |
class | vllm_mlx.mllm_batch_generator.MLLMBatchStats() |
Statistics for MLLM batch generation. | #L395-L436 |
vllm_mlx.mllm_batch_generator.MLLMBatchStats.__init__ |
method | vllm_mlx.mllm_batch_generator.MLLMBatchStats.__init__() -> not annotated |
Method `MLLMBatchStats.__init__` updates `self.prompt_tokens`, `self.prompt_time`, `self.generation_tokens`, `self.generation_time`. | #L398-L405 |
vllm_mlx.mllm_batch_generator.MLLMBatchStats.generation_tps |
method | vllm_mlx.mllm_batch_generator.MLLMBatchStats.generation_tps() -> float |
Return measured decode throughput in tokens per second. | #L416-L421 |
vllm_mlx.mllm_batch_generator.MLLMBatchStats.prompt_tps |
method | vllm_mlx.mllm_batch_generator.MLLMBatchStats.prompt_tps() -> float |
Return measured multimodal prompt throughput in tokens per second. | #L408-L413 |
vllm_mlx.mllm_batch_generator.MLLMBatchStats.to_dict |
method | vllm_mlx.mllm_batch_generator.MLLMBatchStats.to_dict() -> Dict[str, Any] |
Return token, timing, vision, and peak-memory statistics. | #L423-L436 |
vllm_mlx.mllm_batch_generator.PrefillAbortedError |
class | vllm_mlx.mllm_batch_generator.PrefillAbortedError(request_id: str) |
Raised when a prefill is aborted due to client disconnect. | #L152-L157 |
vllm_mlx.mllm_batch_generator.PrefillAbortedError.__init__ |
method | vllm_mlx.mllm_batch_generator.PrefillAbortedError.__init__(request_id: str) -> not annotated |
Method `PrefillAbortedError.__init__` updates `self.request_id`; calls `super().__init__`, `super`. | #L155-L157 |
vllm_mlx.mllm_cache.compute_image_hash |
function | vllm_mlx.mllm_cache.compute_image_hash(image_path: str) -> str |
Compute hash of image content for cache key. | #L136-L161 |
vllm_mlx.mllm_cache.compute_images_hash |
function | vllm_mlx.mllm_cache.compute_images_hash(images: list[str]) -> str |
Compute combined hash for multiple images. | #L164-L179 |
vllm_mlx.mllm_cache.MLLMCacheStats |
class | vllm_mlx.mllm_cache.MLLMCacheStats(hits: int = 0, misses: int = 0, partial_hits: int = 0, tokens_saved: int = 0, image_cache_hits: int = 0, vision_encoder_skips: int = 0, total_queries: int = 0, evictions: int = 0) |
Statistics for MLLM cache performance. | #L34-L65 |
vllm_mlx.mllm_cache.MLLMCacheStats.hit_rate |
method | vllm_mlx.mllm_cache.MLLMCacheStats.hit_rate() -> float |
Calculate cache hit rate. | #L47-L51 |
vllm_mlx.mllm_cache.MLLMCacheStats.to_dict |
method | vllm_mlx.mllm_cache.MLLMCacheStats.to_dict() -> dict |
Convert stats to dictionary. | #L53-L65 |
vllm_mlx.mllm_cache.MLLMPrefixCacheEntry |
class | vllm_mlx.mllm_cache.MLLMPrefixCacheEntry(image_hash: str, prompt_hash: str, vision_embeddings: Any = None, kv_cache: list[Any] = field(default_factory=list), token_ids: list[int] = field(default_factory=list), num_image_tokens: int = 0, num_text_tokens: int = 0, prompt_tokens: int = 0, created_at: float = field(default_factory=time.time), hit_count: int = 0, model_name: str = '') |
Enhanced cache entry storing vision embeddings, KV cache, and token IDs. | #L69-L133 |
vllm_mlx.mllm_cache.MLLMPrefixCacheEntry.get_prefix_match_length |
method | vllm_mlx.mllm_cache.MLLMPrefixCacheEntry.get_prefix_match_length(new_token_ids: list[int]) -> int |
Find how many tokens match between cached prefix and new input. | #L121-L133 |
vllm_mlx.mllm_cache.MLLMPrefixCacheEntry.memory_size |
method | vllm_mlx.mllm_cache.MLLMPrefixCacheEntry.memory_size() -> int |
Estimate memory usage in bytes. | #L105-L119 |
vllm_mlx.mllm_cache.MLLMPrefixCacheEntry.total_tokens |
method | vllm_mlx.mllm_cache.MLLMPrefixCacheEntry.total_tokens() -> int |
Return the number of token IDs represented by this cache entry. | #L99-L102 |
vllm_mlx.mllm_cache.MLLMPrefixCacheManager |
class | vllm_mlx.mllm_cache.MLLMPrefixCacheManager(max_entries: int = 50, max_memory_mb: int = 2048) |
LRU Cache manager for MLLM prefix states with vision embedding caching. | #L182-L448 |
vllm_mlx.mllm_cache.MLLMPrefixCacheManager.__init__ |
method | vllm_mlx.mllm_cache.MLLMPrefixCacheManager.__init__(max_entries: int = 50, max_memory_mb: int = 2048) -> not annotated |
Initialize MLLM prefix cache manager. | #L211-L227 |
vllm_mlx.mllm_cache.MLLMPrefixCacheManager.__len__ |
method | vllm_mlx.mllm_cache.MLLMPrefixCacheManager.__len__() -> int |
Return number of cached entries. | #L442-L444 |
vllm_mlx.mllm_cache.MLLMPrefixCacheManager.__repr__ |
method | vllm_mlx.mllm_cache.MLLMPrefixCacheManager.__repr__() -> str |
Method `MLLMPrefixCacheManager.__repr__` calls `len`; returns `f'<MLLMPrefixCacheManager entries={len(self)} memory={mem_mb:.1f}MB>'`. | #L446-L448 |
vllm_mlx.mllm_cache.MLLMPrefixCacheManager._evict_by_count |
method | vllm_mlx.mllm_cache.MLLMPrefixCacheManager._evict_by_count() -> None |
Evict entries until we're under max_size. | #L248-L255 |
vllm_mlx.mllm_cache.MLLMPrefixCacheManager._evict_by_memory |
method | vllm_mlx.mllm_cache.MLLMPrefixCacheManager._evict_by_memory(required_size: int) -> None |
Evict entries until we have enough memory. | #L239-L246 |
vllm_mlx.mllm_cache.MLLMPrefixCacheManager._make_cache_key |
method | vllm_mlx.mllm_cache.MLLMPrefixCacheManager._make_cache_key(images: list[str], prompt: str) -> str |
Create cache key from images and prompt. | #L229-L233 |
vllm_mlx.mllm_cache.MLLMPrefixCacheManager._make_image_only_key |
method | vllm_mlx.mllm_cache.MLLMPrefixCacheManager._make_image_only_key(images: list[str]) -> str |
Create cache key for image-only lookup (vision embedding reuse). | #L235-L237 |
vllm_mlx.mllm_cache.MLLMPrefixCacheManager.clear |
method | vllm_mlx.mllm_cache.MLLMPrefixCacheManager.clear() -> None |
Clear all cached entries and reset stats. | #L436-L440 |
vllm_mlx.mllm_cache.MLLMPrefixCacheManager.fetch |
method | vllm_mlx.mllm_cache.MLLMPrefixCacheManager.fetch(images: list[str], prompt: str, token_ids: list[int] | None = None) -> tuple[MLLMPrefixCacheEntry | None, int] |
Fetch cached prefix state with prefix matching. | #L257-L329 |
vllm_mlx.mllm_cache.MLLMPrefixCacheManager.fetch_cache |
method | vllm_mlx.mllm_cache.MLLMPrefixCacheManager.fetch_cache(images: list[str], prompt: str) -> tuple[list[Any] | None, bool] |
Legacy API: Fetch cached KV state for image+prompt combination. | #L331-L345 |
vllm_mlx.mllm_cache.MLLMPrefixCacheManager.get_stats |
method | vllm_mlx.mllm_cache.MLLMPrefixCacheManager.get_stats() -> dict[str, Any] |
Get cache statistics. | #L423-L430 |
vllm_mlx.mllm_cache.MLLMPrefixCacheManager.reset_stats |
method | vllm_mlx.mllm_cache.MLLMPrefixCacheManager.reset_stats() -> None |
Reset statistics counters. | #L432-L434 |
vllm_mlx.mllm_cache.MLLMPrefixCacheManager.store |
method | vllm_mlx.mllm_cache.MLLMPrefixCacheManager.store(images: list[str], prompt: str, vision_embeddings: Any, kv_cache: list[Any], token_ids: list[int], num_image_tokens: int = 0, model_name: str = '') -> None |
Store prefix state in cache. | #L347-L396 |
vllm_mlx.mllm_cache.MLLMPrefixCacheManager.store_cache |
method | vllm_mlx.mllm_cache.MLLMPrefixCacheManager.store_cache(images: list[str], prompt: str, cache: list[Any] | None, num_tokens: int = 0) -> None |
Legacy API: Store KV cache for future reuse. | #L398-L421 |
vllm_mlx.mllm_scheduler.MLLMRequest |
class | vllm_mlx.mllm_scheduler.MLLMRequest(request_id: str, prompt: str, images: Optional[List[str]] = None, videos: Optional[List[str]] = None, audio: Optional[List[str]] = None, sampling_params: SamplingParams = field(default_factory=SamplingParams), arrival_time: float = field(default_factory=time.time), batch_uid: Optional[int] = None, status: RequestStatus = RequestStatus.WAITING, output_text: str = '', output_tokens: List[int] = field(default_factory=list), finish_reason: Optional[str] = None, num_prompt_tokens: int = 0, num_output_tokens: int = 0, mtp_drafts: int = 0, mtp_accepted: int = 0, first_token_time: Optional[float] = None) |
Extended request for MLLM processing. | #L96-L127 |
vllm_mlx.mllm_scheduler.MLLMScheduler |
class | vllm_mlx.mllm_scheduler.MLLMScheduler(model: Any, processor: Any, config: Optional[MLLMSchedulerConfig] = None) |
Scheduler for Vision Language Model requests with continuous batching. | #L150-L1242 |
vllm_mlx.mllm_scheduler.MLLMScheduler.__init__ |
method | vllm_mlx.mllm_scheduler.MLLMScheduler.__init__(model: Any, processor: Any, config: Optional[MLLMSchedulerConfig] = None) -> not annotated |
Initialize MLLM scheduler. | #L183-L248 |
vllm_mlx.mllm_scheduler.MLLMScheduler._cleanup_finished |
method | vllm_mlx.mllm_scheduler.MLLMScheduler._cleanup_finished(finished_ids: Set[str]) -> None |
Clean up finished requests. | #L718-L744 |
vllm_mlx.mllm_scheduler.MLLMScheduler._ensure_batch_generator |
method | vllm_mlx.mllm_scheduler.MLLMScheduler._ensure_batch_generator() -> None |
Ensure batch generator exists. | #L292-L374 |
vllm_mlx.mllm_scheduler.MLLMScheduler._get_stop_tokens |
method | vllm_mlx.mllm_scheduler.MLLMScheduler._get_stop_tokens() -> Set[int] |
Get stop token IDs from tokenizer and generation_config.json. | #L250-L290 |
vllm_mlx.mllm_scheduler.MLLMScheduler._process_batch_responses |
method | vllm_mlx.mllm_scheduler.MLLMScheduler._process_batch_responses(responses: List[MLLMBatchResponse]) -> Tuple[List[RequestOutput], Set[str]] |
Process responses from batch generator. | #L599-L716 |
vllm_mlx.mllm_scheduler.MLLMScheduler._process_loop |
method | async vllm_mlx.mllm_scheduler.MLLMScheduler._process_loop() -> None |
Main async processing loop. | #L852-L947 |
vllm_mlx.mllm_scheduler.MLLMScheduler._process_loop._ensure_streams_bound |
nested function | vllm_mlx.mllm_scheduler.MLLMScheduler._process_loop._ensure_streams_bound() -> None |
Nested Function `MLLMScheduler._process_loop._ensure_streams_bound` calls `bind_generation_streams`. | #L867-L871 |
vllm_mlx.mllm_scheduler.MLLMScheduler._schedule_waiting |
method | vllm_mlx.mllm_scheduler.MLLMScheduler._schedule_waiting() -> List[MLLMRequest] |
Move requests from waiting queue to running. | #L546-L597 |
vllm_mlx.mllm_scheduler.MLLMScheduler.abort_request |
method | vllm_mlx.mllm_scheduler.MLLMScheduler.abort_request(request_id: str) -> bool |
Abort a request. | #L455-L532 |
vllm_mlx.mllm_scheduler.MLLMScheduler.add_request |
method | vllm_mlx.mllm_scheduler.MLLMScheduler.add_request(prompt: str, images: Optional[List[str]] = None, videos: Optional[List[str]] = None, audio: Optional[List[str]] = None, max_tokens: int = 256, temperature: float = 0.7, top_p: float = 0.9, request_id: Optional[str] = None, **kwargs) -> str |
Add a multimodal request to the scheduler (sync version). | #L378-L453 |
vllm_mlx.mllm_scheduler.MLLMScheduler.add_request_async |
method | async vllm_mlx.mllm_scheduler.MLLMScheduler.add_request_async(prompt: str, images: Optional[List[str]] = None, videos: Optional[List[str]] = None, audio: Optional[List[str]] = None, max_tokens: int = 256, temperature: float = 0.7, top_p: float = 0.9, **kwargs) -> str |
Add a multimodal request (async version with output queue). | #L949-L990 |
vllm_mlx.mllm_scheduler.MLLMScheduler.clear_runtime_caches |
method | vllm_mlx.mllm_scheduler.MLLMScheduler.clear_runtime_caches() -> Dict[str, bool] |
Clear runtime caches without resetting scheduler/request state. | #L1206-L1221 |
vllm_mlx.mllm_scheduler.MLLMScheduler.generate |
method | async vllm_mlx.mllm_scheduler.MLLMScheduler.generate(prompt: str, images: Optional[List[str]] = None, videos: Optional[List[str]] = None, audio: Optional[List[str]] = None, **kwargs) -> RequestOutput |
Generate complete output for a request (non-streaming). | #L1029-L1078 |
vllm_mlx.mllm_scheduler.MLLMScheduler.get_num_running |
method | vllm_mlx.mllm_scheduler.MLLMScheduler.get_num_running() -> int |
Get number of running requests. | #L542-L544 |
vllm_mlx.mllm_scheduler.MLLMScheduler.get_num_waiting |
method | vllm_mlx.mllm_scheduler.MLLMScheduler.get_num_waiting() -> int |
Get number of waiting requests. | #L538-L540 |
vllm_mlx.mllm_scheduler.MLLMScheduler.get_request |
method | vllm_mlx.mllm_scheduler.MLLMScheduler.get_request(request_id: str) -> Optional[MLLMRequest] |
Get a request by ID. | #L815-L817 |
vllm_mlx.mllm_scheduler.MLLMScheduler.get_running_requests_info |
method | vllm_mlx.mllm_scheduler.MLLMScheduler.get_running_requests_info() -> List[Dict[str, Any]] |
Per-request details for status endpoint. | #L1082-L1151 |
vllm_mlx.mllm_scheduler.MLLMScheduler.get_stats |
method | vllm_mlx.mllm_scheduler.MLLMScheduler.get_stats() -> Dict[str, Any] |
Get scheduler statistics. | #L1153-L1204 |
vllm_mlx.mllm_scheduler.MLLMScheduler.has_requests |
method | vllm_mlx.mllm_scheduler.MLLMScheduler.has_requests() -> bool |
Check if there are any pending or running requests. | #L534-L536 |
vllm_mlx.mllm_scheduler.MLLMScheduler.remove_finished_request |
method | vllm_mlx.mllm_scheduler.MLLMScheduler.remove_finished_request(request_id: str) -> Optional[MLLMRequest] |
Remove a finished request from tracking. | #L819-L821 |
vllm_mlx.mllm_scheduler.MLLMScheduler.reset |
method | vllm_mlx.mllm_scheduler.MLLMScheduler.reset() -> None |
Reset the scheduler state. | #L1223-L1242 |
vllm_mlx.mllm_scheduler.MLLMScheduler.start |
method | async vllm_mlx.mllm_scheduler.MLLMScheduler.start() -> None |
Start the async scheduler processing loop. | #L825-L834 |
vllm_mlx.mllm_scheduler.MLLMScheduler.step |
method | vllm_mlx.mllm_scheduler.MLLMScheduler.step() -> MLLMSchedulerOutput |
Execute one scheduling step. | #L746-L813 |
vllm_mlx.mllm_scheduler.MLLMScheduler.stop |
method | async vllm_mlx.mllm_scheduler.MLLMScheduler.stop() -> None |
Stop the scheduler. | #L836-L850 |
vllm_mlx.mllm_scheduler.MLLMScheduler.stream_outputs |
method | async vllm_mlx.mllm_scheduler.MLLMScheduler.stream_outputs(request_id: str) -> AsyncIterator[RequestOutput] |
Stream outputs for a request. | #L992-L1027 |
vllm_mlx.mllm_scheduler.MLLMSchedulerConfig |
class | vllm_mlx.mllm_scheduler.MLLMSchedulerConfig(max_num_seqs: int = 16, prefill_batch_size: int = 16, completion_batch_size: int = 16, prefill_step_size: int = 1024, enable_vision_cache: bool = True, vision_cache_size: int = 100, default_max_tokens: int = 256, default_video_fps: float = 2.0, cache_memory_mb: Optional[int] = None, max_video_frames: int = 128, enable_mtp: bool = False, mtp_num_draft_tokens: int = 1, enable_prefix_cache: bool = True, use_memory_aware_cache: bool = True, prefix_cache_memory_mb: Optional[int] = None, kv_cache_quantization: bool = False, kv_cache_quantization_bits: int = 8, kv_cache_quantization_group_size: int = 64, chunked_prefill_tokens: int = 0, max_kv_size: int = 0, ssd_cache_dir: Optional[str] = None, ssd_cache_max_gb: float = 10.0) |
Configuration for MLLM scheduler. | #L46-L92 |
vllm_mlx.mllm_scheduler.MLLMSchedulerOutput |
class | vllm_mlx.mllm_scheduler.MLLMSchedulerOutput(scheduled_request_ids: List[str] = field(default_factory=list), num_scheduled_tokens: int = 0, finished_request_ids: Set[str] = field(default_factory=set), outputs: List[RequestOutput] = field(default_factory=list), has_work: bool = False) |
Output from a scheduling step. | #L131-L147 |
vllm_mlx.mlx_streams.bind_generation_streams |
function | vllm_mlx.mlx_streams.bind_generation_streams(module_names: Iterable[str] = ('mlx_lm.generate', 'mlx_vlm.generate')) -> object |
Bind mlx-lm/mlx-vlm generation streams to the current thread. | #L15-L39 |
vllm_mlx.model_registry._clone_scheduler_config |
function | vllm_mlx.model_registry._clone_scheduler_config(config: SchedulerConfig | None) -> SchedulerConfig | None |
Clone a SchedulerConfig so per-model overrides do not mutate globals. | #L234-L238 |
vllm_mlx.model_registry._device_working_set_bytes |
function | vllm_mlx.model_registry._device_working_set_bytes() -> int | None |
Best-effort Metal recommended working-set size, or None when unavailable. | #L266-L282 |
vllm_mlx.model_registry._estimate_model_bytes_from_source |
function | vllm_mlx.model_registry._estimate_model_bytes_from_source(source: str) -> int |
Estimate model footprint from local artifact size when possible. | #L505-L521 |
vllm_mlx.model_registry._ModelOwnershipRegistry |
class | vllm_mlx.model_registry._ModelOwnershipRegistry() |
Process-local model ownership guard used by EngineCore. | #L42-L82 |
vllm_mlx.model_registry._ModelOwnershipRegistry.__init__ |
method | vllm_mlx.model_registry._ModelOwnershipRegistry.__init__() -> None |
Method `_ModelOwnershipRegistry.__init__` updates `self._owners`. | #L45-L46 |
vllm_mlx.model_registry._ModelOwnershipRegistry.acquire |
method | vllm_mlx.model_registry._ModelOwnershipRegistry.acquire(*, model: Any, engine: Any, engine_id: str, force: bool = True) -> None |
Method `_ModelOwnershipRegistry.acquire` calls `id`, `self._owners.get`, `ModelOwnershipError`; can raise `ModelOwnershipError`. | #L48-L63 |
vllm_mlx.model_registry._ModelOwnershipRegistry.get_stats |
method | vllm_mlx.model_registry._ModelOwnershipRegistry.get_stats() -> dict[str, Any] |
Method `_ModelOwnershipRegistry.get_stats` calls `len`; returns `{'total_entries': len(self._owners), 'active_owners': len(self._owners)}`. | #L78-L82 |
vllm_mlx.model_registry._ModelOwnershipRegistry.is_owned |
method | vllm_mlx.model_registry._ModelOwnershipRegistry.is_owned(model: Any) -> tuple[bool, str | None] |
Method `_ModelOwnershipRegistry.is_owned` calls `id`, `self._owners.get`; has 2 explicit return paths. | #L71-L76 |
vllm_mlx.model_registry._ModelOwnershipRegistry.release |
method | vllm_mlx.model_registry._ModelOwnershipRegistry.release(model: Any, engine_id: str) -> None |
Method `_ModelOwnershipRegistry.release` calls `id`, `self._owners.get`, `self._owners.pop`. | #L65-L69 |
vllm_mlx.model_registry._parse_memory_budget_bytes |
function | vllm_mlx.model_registry._parse_memory_budget_bytes(value: Any) -> int |
Parse a memory budget from bytes, MB, or GB. | #L241-L256 |
vllm_mlx.model_registry._safe_available_memory_bytes |
function | vllm_mlx.model_registry._safe_available_memory_bytes() -> int |
Best-effort available system memory. | #L259-L263 |
vllm_mlx.model_registry.build_memory_budget_report |
function | vllm_mlx.model_registry.build_memory_budget_report(manager_config: RegistryManagerConfig, registry: dict[str, RegisteredModel], defaults: RegistryServeDefaults, *, device_working_set_bytes: int | None = None) -> MemoryBudgetReport |
Reconcile the manager weight budget against the Metal allocation ceiling. | #L342-L421 |
vllm_mlx.model_registry.ContentionPolicy |
class | vllm_mlx.model_registry.ContentionPolicy(strategy: ContentionStrategy = 'wait_then_fail', wait_timeout_s: float | None = 30.0, preempt_after_s: float | None = None) |
Policy used when a new model cannot fit inside the memory budget. | #L129-L134 |
vllm_mlx.model_registry.get_registry |
function | vllm_mlx.model_registry.get_registry() -> _ModelOwnershipRegistry |
Return the global model ownership registry used by EngineCore. | #L88-L90 |
vllm_mlx.model_registry.load_registry_config |
function | vllm_mlx.model_registry.load_registry_config(config_path: str | os.PathLike[str], defaults: RegistryServeDefaults) -> tuple[RegistryManagerConfig, dict[str, RegisteredModel]] |
Load and validate the models registry YAML file. | #L524-L621 |
vllm_mlx.model_registry.LoadedModel |
class | vllm_mlx.model_registry.LoadedModel(config: ResolvedModelConfig, engine: BaseEngine, loaded_at: float = field(default_factory=time.time), last_used_at: float = field(default_factory=time.time), active_requests: int = 0, active_tasks: set[asyncio.Task[Any]] = field(default_factory=set), preempting: bool = False) |
Runtime state for a loaded engine. | #L188-L197 |
vllm_mlx.model_registry.log_memory_budget_report |
function | vllm_mlx.model_registry.log_memory_budget_report(report: MemoryBudgetReport) -> None |
Log the budget/ceiling reconciliation, warning when they conflict. | #L424-L502 |
vllm_mlx.model_registry.MemoryBudgetReport |
class | vllm_mlx.model_registry.MemoryBudgetReport(budget_bytes: int, device_working_set_bytes: int | None, gpu_memory_utilization: float | None, gpu_memory_utilization_source: str | None, per_engine_cache_limit_bytes: int | None, per_engine_cache_percent: float | None, continuous_batching_entries: int, total_entries: int) |
Reconciliation of the manager weight budget with the Metal ceiling. | #L286-L339 |
vllm_mlx.model_registry.MemoryBudgetReport.allocation_ceiling_bytes |
method | vllm_mlx.model_registry.MemoryBudgetReport.allocation_ceiling_bytes() -> int | None |
Metal soft allocation limit that will be installed at engine start. | #L313-L322 |
vllm_mlx.model_registry.MemoryBudgetReport.cache_limit_exceeds_ceiling |
method | vllm_mlx.model_registry.MemoryBudgetReport.cache_limit_exceeds_ceiling() -> bool |
True when one engine's prefix cache could alone fill the ceiling. | #L334-L339 |
vllm_mlx.model_registry.MemoryBudgetReport.exceeds_ceiling |
method | vllm_mlx.model_registry.MemoryBudgetReport.exceeds_ceiling() -> bool |
True when the weights budget alone cannot fit under the ceiling. | #L325-L331 |
vllm_mlx.model_registry.ModelLease |
class | vllm_mlx.model_registry.ModelLease(manager: 'ModelManager | None', model_name: str, engine: BaseEngine, release_cb: Callable[[], Awaitable[None]]) |
Active lease for a loaded model. | #L210-L231 |
vllm_mlx.model_registry.ModelLease.__aenter__ |
method | async vllm_mlx.model_registry.ModelLease.__aenter__() -> 'ModelLease' |
Method `ModelLease.__aenter__` returns `self`. | #L227-L228 |
vllm_mlx.model_registry.ModelLease.__aexit__ |
method | async vllm_mlx.model_registry.ModelLease.__aexit__(exc_type, exc, tb) -> None |
Method `ModelLease.__aexit__` calls `self.release`; awaits asynchronous work. | #L230-L231 |
vllm_mlx.model_registry.ModelLease.release |
method | async vllm_mlx.model_registry.ModelLease.release() -> None |
Release this lease once and allow the model to become evictable. | #L218-L225 |
vllm_mlx.model_registry.ModelManager |
class | vllm_mlx.model_registry.ModelManager(manager_config: RegistryManagerConfig, registry: dict[str, RegisteredModel], defaults: RegistryServeDefaults, *, engine_factory: EngineFactory | None = None) |
Registry-backed model manager with lazy load and memory-budget eviction. | #L624-L1201 |
vllm_mlx.model_registry.ModelManager.__init__ |
method | vllm_mlx.model_registry.ModelManager.__init__(manager_config: RegistryManagerConfig, registry: dict[str, RegisteredModel], defaults: RegistryServeDefaults, *, engine_factory: EngineFactory | None = None) -> None |
Method `ModelManager.__init__` updates `self._config`, `self._registry`, `self._defaults`, `self._engine_factory`; calls `asyncio.Condition`. | #L627-L643 |
vllm_mlx.model_registry.ModelManager._begin_unload_locked |
method | vllm_mlx.model_registry.ModelManager._begin_unload_locked(model_name: str) -> LoadedModel |
Method `ModelManager._begin_unload_locked` calls `self._loaded.pop`; returns `loaded`. | #L943-L946 |
vllm_mlx.model_registry.ModelManager._can_reserve_locked |
method | vllm_mlx.model_registry.ModelManager._can_reserve_locked(required_bytes: int) -> bool |
Method `ModelManager._can_reserve_locked` calls `self._committed_bytes_locked`; returns `self._committed_bytes_locked() + required_bytes <= self._config.memory_budget_bytes`. | #L1028-L1032 |
vllm_mlx.model_registry.ModelManager._claim_loaded_locked |
method | vllm_mlx.model_registry.ModelManager._claim_loaded_locked(model_name: str, *, loaded_override: LoadedModel | None = None) -> ModelLease | None |
Method `ModelManager._claim_loaded_locked` calls `self._loaded.get`, `time.time`, `asyncio.current_task`, `loaded.active_tasks.add`; has 2 explicit return paths. | #L844-L874 |
vllm_mlx.model_registry.ModelManager._claim_loaded_locked._release |
nested function | async vllm_mlx.model_registry.ModelManager._claim_loaded_locked._release() -> None |
Nested Function `ModelManager._claim_loaded_locked._release` calls `self.release`; awaits asynchronous work. | #L866-L867 |
vllm_mlx.model_registry.ModelManager._collect_idle_unloads_locked |
method | vllm_mlx.model_registry.ModelManager._collect_idle_unloads_locked(requested_model: str, required_bytes: int) -> list[LoadedModel] |
Method `ModelManager._collect_idle_unloads_locked` calls `self._committed_bytes_locked`, `sorted`, `self._loaded.items`, `selected.append`; returns `selected`. | #L948-L968 |
vllm_mlx.model_registry.ModelManager._committed_bytes_locked |
method | vllm_mlx.model_registry.ModelManager._committed_bytes_locked() -> int |
Method `ModelManager._committed_bytes_locked` calls `sum`, `self._loaded.values`, `self._loading.values`, `self._unloading.values`; returns `loaded_bytes + loading_bytes + unloading_bytes`. | #L1034-L1042 |
vllm_mlx.model_registry.ModelManager._execute_load |
method | async vllm_mlx.model_registry.ModelManager._execute_load(pending: PendingLoad) -> LoadedModel |
Instantiate a reserved model load outside the manager lock. | #L876-L913 |
vllm_mlx.model_registry.ModelManager._instantiate_model |
method | async vllm_mlx.model_registry.ModelManager._instantiate_model(entry: RegisteredModel, resolved_source: str) -> LoadedModel |
Method `ModelManager._instantiate_model` calls `self._resolve_model_config`, `self._engine_factory`, `BatchedEngine`, `SimpleEngine`; awaits asynchronous work; returns `LoadedModel(config=config, engine=engine)`. | #L1044-L1073 |
vllm_mlx.model_registry.ModelManager._maybe_preempt_locked |
method | vllm_mlx.model_registry.ModelManager._maybe_preempt_locked(*, model_name: str, required_bytes: int, start: float) -> set[asyncio.Task[Any]] |
Method `ModelManager._maybe_preempt_locked` calls `self._should_preempt_locked`, `set`, `self._committed_bytes_locked`, `sorted`; has 2 explicit return paths. | #L970-L1003 |
vllm_mlx.model_registry.ModelManager._remaining_wait_timeout |
method | vllm_mlx.model_registry.ModelManager._remaining_wait_timeout(start: float) -> float | None |
Method `ModelManager._remaining_wait_timeout` calls `max`, `time.monotonic`; has 2 explicit return paths. | #L1022-L1026 |
vllm_mlx.model_registry.ModelManager._reserve_load_locked |
method | vllm_mlx.model_registry.ModelManager._reserve_load_locked(model_name: str, required_bytes: int) -> PendingLoad |
Method `ModelManager._reserve_load_locked` calls `asyncio.get_running_loop().create_future`, `asyncio.get_running_loop`, `PendingLoad`; returns `pending`. | #L933-L941 |
vllm_mlx.model_registry.ModelManager._resolve_estimated_bytes |
method | vllm_mlx.model_registry.ModelManager._resolve_estimated_bytes(entry: RegisteredModel, resolved_source: str) -> int |
Method `ModelManager._resolve_estimated_bytes` calls `_estimate_model_bytes_from_source`, `Path`, `source_path.exists`, `ValueError`; can raise `ValueError`; has 3 explicit return paths. | #L1089-L1121 |
vllm_mlx.model_registry.ModelManager._resolve_model_config |
method | vllm_mlx.model_registry.ModelManager._resolve_model_config(entry: RegisteredModel, resolved_source: str) -> ResolvedModelConfig |
Method `ModelManager._resolve_model_config` calls `_clone_scheduler_config`, `self._resolve_estimated_bytes`, `ResolvedModelConfig`; returns `ResolvedModelConfig(entry=entry, resolved_source=resolved_source, continuous_batching=continuous_batching, force_mllm=f…`. | #L1123-L1201 |
vllm_mlx.model_registry.ModelManager._resolve_source |
method | async vllm_mlx.model_registry.ModelManager._resolve_source(entry: RegisteredModel) -> str |
Method `ModelManager._resolve_source` calls `asyncio.to_thread`; awaits asynchronous work; returns `await asyncio.to_thread(self._resolve_source_sync, entry)`. | #L1075-L1076 |
vllm_mlx.model_registry.ModelManager._resolve_source_sync |
method | vllm_mlx.model_registry.ModelManager._resolve_source_sync(entry: RegisteredModel) -> str |
Method `ModelManager._resolve_source_sync` calls `Path(source).exists`, `Path`, `ensure_model_downloaded`, `is_mllm_model`; has 2 explicit return paths. | #L1078-L1087 |
vllm_mlx.model_registry.ModelManager._run_unloads |
method | async vllm_mlx.model_registry.ModelManager._run_unloads(unloads: list[LoadedModel]) -> None |
Method `ModelManager._run_unloads` calls `loaded.engine.stop`, `self._unloading.pop`, `self._condition.notify_all`; awaits asynchronous work. | #L924-L931 |
vllm_mlx.model_registry.ModelManager._should_preempt_locked |
method | vllm_mlx.model_registry.ModelManager._should_preempt_locked(start: float) -> bool |
Method `ModelManager._should_preempt_locked` calls `time.monotonic`; has 3 explicit return paths. | #L1012-L1020 |
vllm_mlx.model_registry.ModelManager._should_wait_locked |
method | vllm_mlx.model_registry.ModelManager._should_wait_locked(start: float) -> bool |
Method `ModelManager._should_wait_locked` calls `self._remaining_wait_timeout`; has 2 explicit return paths. | #L1005-L1010 |
vllm_mlx.model_registry.ModelManager._wait_for_change |
method | async vllm_mlx.model_registry.ModelManager._wait_for_change(timeout: float | None) -> None |
Method `ModelManager._wait_for_change` calls `self._condition.wait`, `RuntimeError`, `asyncio.wait_for`; awaits asynchronous work; can raise `RuntimeError`; returns `None`. | #L915-L922 |
vllm_mlx.model_registry.ModelManager.acquire |
method | async vllm_mlx.model_registry.ModelManager.acquire(model_name: str) -> ModelLease |
Acquire a lease for a configured model. | #L741-L819 |
vllm_mlx.model_registry.ModelManager.has_model |
method | vllm_mlx.model_registry.ModelManager.has_model(model_name: str) -> bool |
Return whether a model name is present in the serving registry. | #L656-L659 |
vllm_mlx.model_registry.ModelManager.list_models |
method | vllm_mlx.model_registry.ModelManager.list_models() -> list[dict[str, Any]] |
Return registry state for /v1/models. | #L661-L699 |
vllm_mlx.model_registry.ModelManager.memory_budget_bytes |
method | vllm_mlx.model_registry.ModelManager.memory_budget_bytes() -> int |
Return the registry's configured resident-model memory budget. | #L646-L649 |
vllm_mlx.model_registry.ModelManager.preload |
method | async vllm_mlx.model_registry.ModelManager.preload() -> None |
Preload any entries marked preload=true. | #L701-L706 |
vllm_mlx.model_registry.ModelManager.registered_model_names |
method | vllm_mlx.model_registry.ModelManager.registered_model_names() -> list[str] |
Return sorted list of all registered model names. | #L652-L654 |
vllm_mlx.model_registry.ModelManager.release |
method | async vllm_mlx.model_registry.ModelManager.release(model_name: str) -> None |
Release a previously acquired model lease. | #L821-L842 |
vllm_mlx.model_registry.ModelManager.shutdown |
method | async vllm_mlx.model_registry.ModelManager.shutdown() -> None |
Stop and unload every loaded engine. | #L708-L739 |
vllm_mlx.model_registry.ModelOwnershipError |
class | vllm_mlx.model_registry.ModelOwnershipError() |
Raised when an EngineCore attempts to use a model already in use. | #L38-L39 |
vllm_mlx.model_registry.PendingLoad |
class | vllm_mlx.model_registry.PendingLoad(model_name: str, required_bytes: int, future: asyncio.Future[LoadedModel]) |
A reserved model load in progress. | #L201-L206 |
vllm_mlx.model_registry.RegisteredModel |
class | vllm_mlx.model_registry.RegisteredModel(name: str, source: str, preload: bool = False, continuous_batching: bool | None = None, force_mllm: bool | None = None, enable_mtp: bool | None = None, prefill_step_size: int | None = None, specprefill_enabled: bool | None = None, specprefill_threshold: int | None = None, specprefill_keep_pct: float | None = None, specprefill_backbone_pct: float | None = None, specprefill_draft_model: str | None = None, stream_interval: int | None = None, gpu_memory_utilization: float | None = None, estimated_memory_bytes: int | None = None) |
One configured model entry. | #L146-L163 |
vllm_mlx.model_registry.RegistryManagerConfig |
class | vllm_mlx.model_registry.RegistryManagerConfig(memory_budget_bytes: int, policy: ContentionPolicy) |
Global registry manager configuration. | #L138-L142 |
vllm_mlx.model_registry.RegistryServeDefaults |
class | vllm_mlx.model_registry.RegistryServeDefaults(continuous_batching: bool, force_mllm: bool, enable_mtp: bool, prefill_step_size: int, specprefill_enabled: bool, specprefill_threshold: int, specprefill_keep_pct: float, specprefill_backbone_pct: float, specprefill_draft_model: str | None, stream_interval: int, gpu_memory_utilization: float, scheduler_config: SchedulerConfig | None, max_tokens: int, download_config: DownloadConfig) |
Global serve defaults inherited by registry entries. | #L109-L125 |
vllm_mlx.model_registry.ResolvedModelConfig |
class | vllm_mlx.model_registry.ResolvedModelConfig(entry: RegisteredModel, resolved_source: str, continuous_batching: bool, force_mllm: bool, enable_mtp: bool, prefill_step_size: int, specprefill_enabled: bool, specprefill_threshold: int, specprefill_keep_pct: float, specprefill_backbone_pct: float, specprefill_draft_model: str | None, stream_interval: int, gpu_memory_utilization: float, scheduler_config: SchedulerConfig | None, estimated_memory_bytes: int) |
Effective configuration for a loaded model. | #L167-L184 |
vllm_mlx.model_runner.MLXModelRunner |
class | vllm_mlx.model_runner.MLXModelRunner(vllm_config: 'VllmConfig', enable_optimizations: bool = True) |
Model runner that uses mlx-lm for inference. | #L53-L476 |
vllm_mlx.model_runner.MLXModelRunner.__init__ |
method | vllm_mlx.model_runner.MLXModelRunner.__init__(vllm_config: 'VllmConfig', enable_optimizations: bool = True) -> not annotated |
Initialize MLX model runner. | #L69-L104 |
vllm_mlx.model_runner.MLXModelRunner.__repr__ |
method | vllm_mlx.model_runner.MLXModelRunner.__repr__() -> str |
Method `MLXModelRunner.__repr__` returns `f'<MLXModelRunner model={self.model_config.model} status={status} mode={opt_status}>'`. | #L473-L476 |
vllm_mlx.model_runner.MLXModelRunner._apply_optimizations |
method | vllm_mlx.model_runner.MLXModelRunner._apply_optimizations() -> None |
Apply low-level optimizations for maximum performance. | #L147-L168 |
vllm_mlx.model_runner.MLXModelRunner._continue_generation |
method | vllm_mlx.model_runner.MLXModelRunner._continue_generation(req_id: str) -> list[int] |
Continue generation for an existing request. | #L420-L428 |
vllm_mlx.model_runner.MLXModelRunner._create_default_sampler |
method | vllm_mlx.model_runner.MLXModelRunner._create_default_sampler() -> None |
Create default sampler for generation. | #L195-L205 |
vllm_mlx.model_runner.MLXModelRunner._generate_for_request |
method | vllm_mlx.model_runner.MLXModelRunner._generate_for_request(prompt_token_ids: list[int], sampling_params: Any, max_tokens: int = 1) -> list[int] |
Generate tokens for a single request. | #L364-L418 |
vllm_mlx.model_runner.MLXModelRunner._prefill_with_chunking |
method | vllm_mlx.model_runner.MLXModelRunner._prefill_with_chunking(input_ids: mx.array, cache: Optional[Any] = None) -> tuple[mx.array, Any] |
Process prompt with optimal chunking for L2 cache efficiency. | #L317-L362 |
vllm_mlx.model_runner.MLXModelRunner._prefill_with_chunking.get_optimal_prefill_size |
nested function | vllm_mlx.model_runner.MLXModelRunner._prefill_with_chunking.get_optimal_prefill_size(seq_len) -> not annotated |
Nested Function `MLXModelRunner._prefill_with_chunking.get_optimal_prefill_size` calls `min`; returns `min(512, seq_len)`. | #L339-L340 |
vllm_mlx.model_runner.MLXModelRunner._setup_compiled_forward |
method | vllm_mlx.model_runner.MLXModelRunner._setup_compiled_forward() -> None |
Setup compiled forward pass using mx.compile() for kernel fusion. | #L170-L193 |
vllm_mlx.model_runner.MLXModelRunner.decode_tokens |
method | vllm_mlx.model_runner.MLXModelRunner.decode_tokens(token_ids: list[int]) -> str |
Decode token IDs to text. | #L430-L434 |
vllm_mlx.model_runner.MLXModelRunner.execute_model |
method | vllm_mlx.model_runner.MLXModelRunner.execute_model(scheduler_output: 'SchedulerOutput') -> MLXModelRunnerOutput |
Execute model inference for scheduled requests. | #L265-L315 |
vllm_mlx.model_runner.MLXModelRunner.get_cache_block_size_bytes |
method | vllm_mlx.model_runner.MLXModelRunner.get_cache_block_size_bytes() -> int |
Calculate cache block size in bytes. | #L222-L240 |
vllm_mlx.model_runner.MLXModelRunner.get_kv_cache_spec |
method | vllm_mlx.model_runner.MLXModelRunner.get_kv_cache_spec() -> dict |
Get KV cache specification. | #L215-L220 |
vllm_mlx.model_runner.MLXModelRunner.get_model_info |
method | vllm_mlx.model_runner.MLXModelRunner.get_model_info() -> dict |
Get information about the loaded model and optimizations. | #L436-L471 |
vllm_mlx.model_runner.MLXModelRunner.initialize_cache |
method | vllm_mlx.model_runner.MLXModelRunner.initialize_cache(num_blocks: int) -> None |
Initialize KV cache. | #L207-L210 |
vllm_mlx.model_runner.MLXModelRunner.load_model |
method | vllm_mlx.model_runner.MLXModelRunner.load_model() -> None |
Load model using mlx-lm with optimizations. | #L106-L145 |
vllm_mlx.model_runner.MLXModelRunner.warm_up |
method | vllm_mlx.model_runner.MLXModelRunner.warm_up() -> None |
Warm up model with a test generation. | #L242-L263 |
vllm_mlx.model_runner.MLXModelRunnerOutput |
class | vllm_mlx.model_runner.MLXModelRunnerOutput(req_id_to_token_ids: dict[str, list[int]], req_id_to_logprobs: dict[str, list[dict]] | None = None, num_tokens_generated: int = 0, generation_time_s: float = 0.0) |
Output from MLX model runner, compatible with vLLM's ModelRunnerOutput. | #L37-L50 |
vllm_mlx.model_runner.SamplerOutput |
class | vllm_mlx.model_runner.SamplerOutput(token_ids: list[int], logprobs: list[dict] | None = None) |
Output from sampling. | #L29-L33 |
vllm_mlx.model_workflow._bytes_to_gb |
function | vllm_mlx.model_workflow._bytes_to_gb(size: int | float | None) -> float | None |
Function `_bytes_to_gb` calls `round`, `float`; has 2 explicit return paths. | #L105-L108 |
vllm_mlx.model_workflow._config_value |
function | vllm_mlx.model_workflow._config_value(config: dict[str, Any], key: str) -> Any |
Function `_config_value` calls `config.get`, `isinstance`, `text_config.get`; has 3 explicit return paths. | #L173-L179 |
vllm_mlx.model_workflow._conversion_command |
function | vllm_mlx.model_workflow._conversion_command(options: ConversionOptions) -> list[str] |
Function `_conversion_command` calls `command.append`, `command.extend`, `str`; returns `command`. | #L449-L474 |
vllm_mlx.model_workflow._drop_none |
function | vllm_mlx.model_workflow._drop_none(payload: dict[str, Any]) -> dict[str, Any] |
Function `_drop_none` calls `payload.items`; returns `{key: value for key, value in payload.items() if value is not None}`. | #L543-L544 |
vllm_mlx.model_workflow._estimate_fit |
function | vllm_mlx.model_workflow._estimate_fit(*, total_bytes: int | None, model_files_bytes: int | None, config: dict[str, Any]) -> dict[str, Any] |
Function `_estimate_fit` calls `_model_family(config).get`, `_model_family`, `isinstance`, `warnings.append`; returns `{'download_size_gb': _bytes_to_gb(total_bytes), 'model_file_size_gb': _bytes_to_gb(model_files_bytes), 'estimated_conve…`. | #L204-L231 |
vllm_mlx.model_workflow._existing_manifests |
function | vllm_mlx.model_workflow._existing_manifests(path: Path) -> dict[str, Any] |
Function `_existing_manifests` calls `manifest_path.exists`, `str`, `_read_json`; returns `manifests`. | #L528-L540 |
vllm_mlx.model_workflow._fast_transfer_env |
function | vllm_mlx.model_workflow._fast_transfer_env(requested: bool) -> tuple[dict[str, str], dict[str, Any]] |
Function `_fast_transfer_env` calls `find_spec`; has 3 explicit return paths. | #L282-L297 |
vllm_mlx.model_workflow._hf_config |
function | vllm_mlx.model_workflow._hf_config(model_id: str, *, revision: str | None, local_files_only: bool) -> dict[str, Any] |
Function `_hf_config` calls `hf_hub_download`, `_read_json`, `Path`; returns `_read_json(Path(config_path))`. | #L161-L170 |
vllm_mlx.model_workflow._hf_file_inventory |
function | vllm_mlx.model_workflow._hf_file_inventory(model_id: str, *, revision: str | None, local_files_only: bool) -> tuple[list[dict[str, Any]], int | None, str | None] |
Function `_hf_file_inventory` calls `HfApi().model_info`, `HfApi`, `getattr`, `int`; has 2 explicit return paths. | #L138-L158 |
vllm_mlx.model_workflow._is_mlx_quantization |
function | vllm_mlx.model_workflow._is_mlx_quantization(quant: Any) -> bool |
Return True only when *quant* looks like an mlx-lm quantization config. | #L252-L265 |
vllm_mlx.model_workflow._is_model_id |
function | vllm_mlx.model_workflow._is_model_id(value: str) -> bool |
Function `_is_model_id` calls `bool`, `_MODEL_ID_RE.fullmatch`; returns `bool(_MODEL_ID_RE.fullmatch(value))`. | #L278-L279 |
vllm_mlx.model_workflow._local_file_inventory |
function | vllm_mlx.model_workflow._local_file_inventory(path: Path) -> tuple[list[dict[str, Any]], int] |
Function `_local_file_inventory` calls `sorted`, `path.rglob`, `item.is_file`, `item.stat`; returns `(files, total)`. | #L123-L135 |
vllm_mlx.model_workflow._looks_like_mlx_name |
function | vllm_mlx.model_workflow._looks_like_mlx_name(model: str, *, source: str) -> bool |
Function `_looks_like_mlx_name` calls `model.lower`, `Path(model).name.lower`, `Path`, `name.startswith`; returns `name.startswith('mlx-community/') or '-mlx' in name or '_mlx' in name or name.endswith('mlx')`. | #L268-L275 |
vllm_mlx.model_workflow._model_family |
function | vllm_mlx.model_workflow._model_family(config: dict[str, Any]) -> dict[str, Any] |
Function `_model_family` calls `_config_value`, `isinstance`, `config.get`; returns `{'model_type': _config_value(config, 'model_type'), 'architectures': architectures, 'torch_dtype': _config_value(config…`. | #L182-L201 |
vllm_mlx.model_workflow._model_file_bytes |
function | vllm_mlx.model_workflow._model_file_bytes(files: list[dict[str, Any]]) -> int | None |
Function `_model_file_bytes` calls `str`, `entry.get`, `path.endswith`, `int`; has 2 explicit return paths. | #L234-L246 |
vllm_mlx.model_workflow._now_iso |
function | vllm_mlx.model_workflow._now_iso() -> str |
Function `_now_iso` calls `datetime.now(timezone.utc).isoformat`, `datetime.now`; returns `datetime.now(timezone.utc).isoformat()`. | #L101-L102 |
vllm_mlx.model_workflow._qualification_command |
function | vllm_mlx.model_workflow._qualification_command(options: QualificationOptions) -> list[str] |
Function `_qualification_command` calls `command.extend`, `str`; returns `command`. | #L606-L627 |
vllm_mlx.model_workflow._read_json |
function | vllm_mlx.model_workflow._read_json(path: Path) -> dict[str, Any] |
Function `_read_json` calls `json.loads`, `path.read_text`; has 2 explicit return paths. | #L111-L115 |
vllm_mlx.model_workflow._write_json |
function | vllm_mlx.model_workflow._write_json(path: Path, payload: dict[str, Any]) -> None |
Function `_write_json` calls `path.parent.mkdir`, `path.write_text`, `json.dumps`. | #L118-L120 |
vllm_mlx.model_workflow.acquire_model |
function | vllm_mlx.model_workflow.acquire_model(model_id: str, *, options: AcquisitionOptions | None = None) -> dict[str, Any] |
Download a model repository and write a finalized artifact manifest. | #L369-L446 |
vllm_mlx.model_workflow.AcquisitionOptions |
class | vllm_mlx.model_workflow.AcquisitionOptions(revision: str | None = None, target_dir: str | None = None, staging_dir: str | None = None, is_mllm: bool = False, fast_transfer: bool = True, local_files_only: bool = False) |
Options for Hugging Face model acquisition. | #L38-L46 |
vllm_mlx.model_workflow.ConversionOptions |
class | vllm_mlx.model_workflow.ConversionOptions(source_path: str, output_path: str, quantize: bool = False, q_bits: int | None = None, q_group_size: int | None = None, q_mode: str | None = None, quant_predicate: str | None = None, dtype: str | None = None, trust_remote_code: bool = False, dry_run: bool = False) |
Options for the mlx-lm conversion backend. | #L50-L62 |
vllm_mlx.model_workflow.convert_model |
function | vllm_mlx.model_workflow.convert_model(options: ConversionOptions) -> dict[str, Any] |
Run mlx-lm conversion and record the exact recipe. | #L477-L525 |
vllm_mlx.model_workflow.inspect_model |
function | vllm_mlx.model_workflow.inspect_model(model: str, *, revision: str | None = None, local_files_only: bool = False) -> dict[str, Any] |
Inspect a local model path or Hugging Face model id without loading weights. | #L300-L366 |
vllm_mlx.model_workflow.QualificationOptions |
class | vllm_mlx.model_workflow.QualificationOptions(model_id: str, server_url: str = 'http://127.0.0.1:8080', workload_path: str | None = None, output_path: str | None = None, result_path: str | None = None, repetitions: int | None = None, dry_run: bool = False, extra_args: list[str] | None = None) |
Options for creating or running a bench-serve qualification handoff. | #L88-L98 |
vllm_mlx.model_workflow.qualify_model |
function | vllm_mlx.model_workflow.qualify_model(options: QualificationOptions) -> dict[str, Any] |
Create or run a bench-serve qualification handoff. | #L630-L661 |
vllm_mlx.model_workflow.register_model |
function | vllm_mlx.model_workflow.register_model(options: RegistrationOptions) -> dict[str, Any] |
Write a portable registration manifest for a finalized local artifact. | #L547-L603 |
vllm_mlx.model_workflow.RegistrationOptions |
class | vllm_mlx.model_workflow.RegistrationOptions(artifact_path: str, model_id: str | None = None, served_model_name: str | None = None, preset_alias: str | None = None, output_path: str | None = None, mllm: bool | None = None, tool_call_parser: str | None = None, reasoning_parser: str | None = None, default_temperature: float | None = None, default_top_p: float | None = None, default_top_k: int | None = None, default_min_p: float | None = None, default_presence_penalty: float | None = None, default_repetition_penalty: float | None = None, chat_template_kwargs: dict[str, Any] | None = None, feature_flags: list[str] | None = None) |
Options for generating a portable model registration manifest. | #L66-L84 |
vllm_mlx.models.llm.GenerationOutput |
class | vllm_mlx.models.llm.GenerationOutput(text: str, tokens: list[int], finish_reason: str | None = None) |
Output from text generation. | #L21-L26 |
vllm_mlx.models.llm.MLXLanguageModel |
class | vllm_mlx.models.llm.MLXLanguageModel(model_name: str, tokenizer_name: str | None = None, trust_remote_code: bool = False, mtp: bool = False, mtp_num_draft_tokens: int = 1) |
Wrapper around mlx-lm for LLM inference. | #L40-L422 |
vllm_mlx.models.llm.MLXLanguageModel.__init__ |
method | vllm_mlx.models.llm.MLXLanguageModel.__init__(model_name: str, tokenizer_name: str | None = None, trust_remote_code: bool = False, mtp: bool = False, mtp_num_draft_tokens: int = 1) -> not annotated |
Initialize the MLX language model. | #L53-L79 |
vllm_mlx.models.llm.MLXLanguageModel.__repr__ |
method | vllm_mlx.models.llm.MLXLanguageModel.__repr__() -> str |
Method `MLXLanguageModel.__repr__` returns `f'<MLXLanguageModel model={self.model_name} status={status}>'`. | #L420-L422 |
vllm_mlx.models.llm.MLXLanguageModel._create_logits_processors |
method | vllm_mlx.models.llm.MLXLanguageModel._create_logits_processors(presence_penalty: float = 0.0, repetition_penalty: float = 1.0) -> not annotated |
Create logits processors for penalty-based sampling. | #L133-L147 |
vllm_mlx.models.llm.MLXLanguageModel._create_sampler |
method | vllm_mlx.models.llm.MLXLanguageModel._create_sampler(temperature: float = 0.7, top_p: float = 0.9, top_k: int = 0, min_p: float = 0.0) -> not annotated |
Create a sampler for text generation. | #L116-L131 |
vllm_mlx.models.llm.MLXLanguageModel.chat |
method | vllm_mlx.models.llm.MLXLanguageModel.chat(messages: list[dict], max_tokens: int = 256, temperature: float = 0.7, top_p: float = 0.9, tools: list | None = None, chat_template_kwargs: dict | None = None, **kwargs) -> GenerationOutput |
Generate a chat response. | #L327-L393 |
vllm_mlx.models.llm.MLXLanguageModel.generate |
method | vllm_mlx.models.llm.MLXLanguageModel.generate(prompt: str, max_tokens: int = 256, temperature: float = 0.7, top_p: float = 0.9, top_k: int = 0, min_p: float = 0.0, presence_penalty: float = 0.0, repetition_penalty: float = 1.0, stop: list[str] | None = None, logits_processors: list | None = None, **kwargs) -> GenerationOutput |
Generate text from a prompt. | #L149-L219 |
vllm_mlx.models.llm.MLXLanguageModel.get_model_info |
method | vllm_mlx.models.llm.MLXLanguageModel.get_model_info() -> dict |
Get information about the loaded model. | #L395-L418 |
vllm_mlx.models.llm.MLXLanguageModel.load |
method | vllm_mlx.models.llm.MLXLanguageModel.load() -> None |
Load the model and tokenizer. | #L81-L114 |
vllm_mlx.models.llm.MLXLanguageModel.stream_generate |
method | vllm_mlx.models.llm.MLXLanguageModel.stream_generate(prompt: Union[str, 'mx.array', list[int]], max_tokens: int = 256, temperature: float = 0.7, top_p: float = 0.9, top_k: int = 0, min_p: float = 0.0, presence_penalty: float = 0.0, repetition_penalty: float = 1.0, stop: list[str] | None = None, logits_processors: list | None = None, prompt_cache = None, **kwargs) -> Iterator[StreamingOutput] |
Stream text generation token by token. | #L221-L325 |
vllm_mlx.models.llm.StreamingOutput |
class | vllm_mlx.models.llm.StreamingOutput(text: str, token: int, finished: bool = False, finish_reason: str | None = None, prompt_tokens: int = 0) |
Streaming output chunk. | #L30-L37 |
vllm_mlx.models.mllm._append_ordered_mllm_content_part |
function | vllm_mlx.models.mllm._append_ordered_mllm_content_part(raw_item: object, *, built_parts: list[dict[str, str]], text_parts: list[str], all_image_urls: list[str], video_frame_count: int) -> int |
Function `_append_ordered_mllm_content_part` calls `_normalize_content_part`, `isinstance`, `_append_text_content_part`, `item.get`; has 2 explicit return paths. | #L185-L220 |
vllm_mlx.models.mllm._append_text_content_part |
function | vllm_mlx.models.mllm._append_text_content_part(built_parts: list[dict[str, str]], text_parts: list[str], text: str) -> None |
Function `_append_text_content_part` calls `built_parts.append`, `_text_content_part`, `text_parts.append`; returns `None`. | #L168-L174 |
vllm_mlx.models.mllm._build_mllm_chat_messages |
function | vllm_mlx.models.mllm._build_mllm_chat_messages(messages: list[dict], *, all_image_urls: list[str], video_frame_counts: dict[int, int]) -> list[dict] |
Build chat-template messages without reordering multimodal content parts. | #L271-L315 |
vllm_mlx.models.mllm._build_ordered_mllm_message_content |
function | vllm_mlx.models.mllm._build_ordered_mllm_message_content(content: object, *, role: str, all_image_urls: list[str], video_frame_count: int = 0) -> tuple[object, bool] |
Build template content while preserving OpenAI media/text part order. | #L223-L254 |
vllm_mlx.models.mllm._build_string_mllm_message_content |
function | vllm_mlx.models.mllm._build_string_mllm_message_content(content: str, role: str) -> tuple[object, bool] |
Function `_build_string_mllm_message_content` calls `_text_content_part`; has 3 explicit return paths. | #L177-L182 |
vllm_mlx.models.mllm._count_draft_tokens |
function | vllm_mlx.models.mllm._count_draft_tokens(draft_tokens) -> int |
Best-effort drafted-token count for an mlx-vlm drafter output. | #L387-L398 |
vllm_mlx.models.mllm._download_media |
function | vllm_mlx.models.mllm._download_media(url: str, media_type: str, ext_map: dict[str, str], default_ext: str, timeout: int, max_size: int) -> str |
Download media from URL, enforce size limits, and return a local temp path. | #L670-L748 |
vllm_mlx.models.mllm._extract_media_url |
function | vllm_mlx.models.mllm._extract_media_url(item: dict, item_type: str) -> str |
Function `_extract_media_url` calls `item.get`, `isinstance`, `media_value.get`; has 2 explicit return paths. | #L147-L161 |
vllm_mlx.models.mllm._install_draft_metrics_hooks |
function | vllm_mlx.models.mllm._install_draft_metrics_hooks(draft_model) -> None |
Record actual drafted token counts from mlx-vlm assistant drafters. | #L401-L428 |
vllm_mlx.models.mllm._install_draft_metrics_hooks.draft_block_with_metrics |
nested function | vllm_mlx.models.mllm._install_draft_metrics_hooks.draft_block_with_metrics(*args, **kwargs) -> not annotated |
Nested Function `_install_draft_metrics_hooks.draft_block_with_metrics` calls `draft_block`, `draft_model._vllm_mlx_draft_counts.append`, `_count_draft_tokens`; returns `draft_tokens`. | #L412-L415 |
vllm_mlx.models.mllm._install_draft_metrics_hooks.reset_with_metrics |
nested function | vllm_mlx.models.mllm._install_draft_metrics_hooks.reset_with_metrics(*args, **kwargs) -> not annotated |
Nested Function `_install_draft_metrics_hooks.reset_with_metrics` calls `reset`; returns `reset(*args, **kwargs)`. | #L422-L424 |
vllm_mlx.models.mllm._model_has_sound_encoder |
function | vllm_mlx.models.mllm._model_has_sound_encoder(model) -> bool |
Whether a loaded model exposes a usable sound encoder. | #L938-L947 |
vllm_mlx.models.mllm._normalize_content_part |
function | vllm_mlx.models.mllm._normalize_content_part(item: object) -> object |
Convert Pydantic content parts into plain Python objects. | #L138-L144 |
vllm_mlx.models.mllm._normalize_mllm_tool_calls |
function | vllm_mlx.models.mllm._normalize_mllm_tool_calls(tool_calls: list) -> list |
Normalize replayed assistant tool calls for chat templates. | #L257-L268 |
vllm_mlx.models.mllm._request_with_safe_redirects |
function | vllm_mlx.models.mllm._request_with_safe_redirects(method: str, url: str, *, timeout: int, headers: dict[str, str], stream: bool = False, max_redirects: int = 5) -> not annotated |
Issue a requests call while validating every redirect target. | #L522-L557 |
vllm_mlx.models.mllm._text_content_part |
function | vllm_mlx.models.mllm._text_content_part(text: str) -> dict[str, str] |
Function `_text_content_part` returns `{'type': 'text', 'text': text, 'content': text}`. | #L164-L165 |
vllm_mlx.models.mllm._validate_url_safety |
function | vllm_mlx.models.mllm._validate_url_safety(url: str) -> None |
Reject remote URLs that target local or private network resources. | #L483-L519 |
vllm_mlx.models.mllm._video_has_audio_track |
function | vllm_mlx.models.mllm._video_has_audio_track(video_path: str) -> bool |
Return True if ffprobe finds an audio stream in the video. | #L908-L935 |
vllm_mlx.models.mllm.ceil_by_factor |
function | vllm_mlx.models.mllm.ceil_by_factor(x: float, factor: int) -> int |
Ceiling to next multiple of factor. | #L1088-L1090 |
vllm_mlx.models.mllm.cleanup_all_temp_files |
function | vllm_mlx.models.mllm.cleanup_all_temp_files() -> int |
Clean up all tracked temporary files. | #L98-L100 |
vllm_mlx.models.mllm.cleanup_temp_file |
function | vllm_mlx.models.mllm.cleanup_temp_file(path: str) -> bool |
Clean up a specific temporary file. | #L93-L95 |
vllm_mlx.models.mllm.decode_base64_audio |
function | vllm_mlx.models.mllm.decode_base64_audio(base64_string: str, max_length: int = MAX_BASE64_AUDIO_LENGTH) -> str |
Decode base64 audio to temp file and return path. | #L810-L836 |
vllm_mlx.models.mllm.decode_base64_image |
function | vllm_mlx.models.mllm.decode_base64_image(base64_string: str, max_length: int = MAX_BASE64_IMAGE_LENGTH) -> bytes |
Decode base64 image to bytes. | #L453-L480 |
vllm_mlx.models.mllm.decode_base64_video |
function | vllm_mlx.models.mllm.decode_base64_video(base64_string: str, max_length: int = MAX_BASE64_VIDEO_LENGTH) -> str |
Decode base64 video to temp file and return path. | #L761-L807 |
vllm_mlx.models.mllm.download_audio |
function | vllm_mlx.models.mllm.download_audio(url: str, timeout: int = 120, max_size: int = MAX_AUDIO_SIZE) -> str |
Download audio from URL and return local path. | #L756-L758 |
vllm_mlx.models.mllm.download_image |
function | vllm_mlx.models.mllm.download_image(url: str, timeout: int = 30, max_size: int = MAX_IMAGE_SIZE) -> str |
Download image from URL and return local path. | #L560-L645 |
vllm_mlx.models.mllm.download_video |
function | vllm_mlx.models.mllm.download_video(url: str, timeout: int = 120, max_size: int = MAX_VIDEO_SIZE) -> str |
Download video from URL and return local path. | #L751-L753 |
vllm_mlx.models.mllm.extract_audio_from_video |
function | vllm_mlx.models.mllm.extract_audio_from_video(video_path: str) -> str | None |
Extract the audio track from a video file as 16 kHz mono WAV. | #L950-L1005 |
vllm_mlx.models.mllm.extract_video_frames_smart |
function | vllm_mlx.models.mllm.extract_video_frames_smart(video_path: str, fps: float = DEFAULT_FPS, max_frames: int = MAX_FRAMES, resize: tuple[int, int] | None = None) -> list[np.ndarray] |
Extract frames from video with smart sampling. | #L1123-L1187 |
vllm_mlx.models.mllm.FileSizeExceededError |
class | vllm_mlx.models.mllm.FileSizeExceededError() |
Raised when a downloaded file exceeds the size limit. | #L119-L122 |
vllm_mlx.models.mllm.floor_by_factor |
function | vllm_mlx.models.mllm.floor_by_factor(x: float, factor: int) -> int |
Floor to previous multiple of factor. | #L1093-L1095 |
vllm_mlx.models.mllm.is_base64_audio |
function | vllm_mlx.models.mllm.is_base64_audio(s: str) -> bool |
Check if string is base64-encoded audio data. | #L448-L450 |
vllm_mlx.models.mllm.is_base64_image |
function | vllm_mlx.models.mllm.is_base64_image(s: str) -> bool |
Check if string is base64-encoded image data. | #L431-L435 |
vllm_mlx.models.mllm.is_base64_video |
function | vllm_mlx.models.mllm.is_base64_video(s: str) -> bool |
Check if string is base64-encoded video data. | #L443-L445 |
vllm_mlx.models.mllm.is_url |
function | vllm_mlx.models.mllm.is_url(s: str) -> bool |
Check if string is a URL. | #L438-L440 |
vllm_mlx.models.mllm.load_gemma4_assistant_drafter |
function | vllm_mlx.models.mllm.load_gemma4_assistant_drafter(model_path: str) -> not annotated |
Load a Gemma 4 assistant drafter for mlx-vlm speculative decoding. | #L340-L381 |
vllm_mlx.models.mllm.MLLMOutput |
class | vllm_mlx.models.mllm.MLLMOutput(text: str, finish_reason: str | None = None, prompt_tokens: int = 0, completion_tokens: int = 0, mtp_drafts: int = 0, mtp_accepted: int = 0) |
Output from multimodal language model. | #L329-L337 |
vllm_mlx.models.mllm.MLXMultimodalLM |
class | vllm_mlx.models.mllm.MLXMultimodalLM(model_name: str, trust_remote_code: bool = False, enable_cache: bool = True, cache_size: int = 50, max_kv_size: int = 0, draft_model: str | None = None, draft_kind: str | None = None, draft_block_size: int | None = None) |
Wrapper around mlx-vlm for multimodal inference. | #L1207-L2938 |
vllm_mlx.models.mllm.MLXMultimodalLM.__init__ |
method | vllm_mlx.models.mllm.MLXMultimodalLM.__init__(model_name: str, trust_remote_code: bool = False, enable_cache: bool = True, cache_size: int = 50, max_kv_size: int = 0, draft_model: str | None = None, draft_kind: str | None = None, draft_block_size: int | None = None) -> not annotated |
Initialize the MLX multimodal language model. | #L1235-L1278 |
vllm_mlx.models.mllm.MLXMultimodalLM.__repr__ |
method | vllm_mlx.models.mllm.MLXMultimodalLM.__repr__() -> str |
Method `MLXMultimodalLM.__repr__` returns `f'<MLXMultimodalLM model={self.model_name} status={status}>'`. | #L2936-L2938 |
vllm_mlx.models.mllm.MLXMultimodalLM._collect_audio_inputs |
method | vllm_mlx.models.mllm.MLXMultimodalLM._collect_audio_inputs(messages: list[dict]) -> dict[int, list] |
Collect audio inputs from messages, keyed by message index. | #L1499-L1528 |
vllm_mlx.models.mllm.MLXMultimodalLM._collect_video_inputs |
method | vllm_mlx.models.mllm.MLXMultimodalLM._collect_video_inputs(messages: list[dict]) -> dict[int, list] |
Collect video inputs from messages, keyed by message index. | #L1465-L1497 |
vllm_mlx.models.mllm.MLXMultimodalLM._draft_generation_kwargs |
method | vllm_mlx.models.mllm.MLXMultimodalLM._draft_generation_kwargs(call_kwargs: dict | None = None) -> dict |
Return mlx-vlm drafter kwargs when the request explicitly opts in. | #L1334-L1355 |
vllm_mlx.models.mllm.MLXMultimodalLM._draft_metrics_since |
method | vllm_mlx.models.mllm.MLXMultimodalLM._draft_metrics_since(start_accept_lens: int) -> dict[str, int] |
Method `MLXMultimodalLM._draft_metrics_since` calls `list`, `getattr`, `len`, `int`; has 2 explicit return paths. | #L1366-L1395 |
vllm_mlx.models.mllm.MLXMultimodalLM._generate_native_video |
method | vllm_mlx.models.mllm.MLXMultimodalLM._generate_native_video(messages: list[dict], max_tokens: int = 256, temperature: float = 0.7, video_fps: float = DEFAULT_FPS, video_max_frames: int = MAX_FRAMES, tools: list | None = None, **kwargs) -> MLLMOutput |
Generate using native video pipeline (Qwen-family models). | #L1650-L1695 |
vllm_mlx.models.mllm.MLXMultimodalLM._load_draft_model |
method | vllm_mlx.models.mllm.MLXMultimodalLM._load_draft_model() -> not annotated |
Method `MLXMultimodalLM._load_draft_model` calls `load_gemma4_assistant_drafter`, `load`; has 2 explicit return paths. | #L1325-L1332 |
vllm_mlx.models.mllm.MLXMultimodalLM._prepare_audio |
method | vllm_mlx.models.mllm.MLXMultimodalLM._prepare_audio(audio_inputs: list) -> list[str] |
Process audio inputs and return local file paths. | #L1416-L1425 |
vllm_mlx.models.mllm.MLXMultimodalLM._prepare_images |
method | vllm_mlx.models.mllm.MLXMultimodalLM._prepare_images(images: list) -> list[str] |
Process remote/base64 image inputs into local temp file paths. | #L1405-L1414 |
vllm_mlx.models.mllm.MLXMultimodalLM._prepare_native_video_inputs |
method | vllm_mlx.models.mllm.MLXMultimodalLM._prepare_native_video_inputs(messages: list[dict], video_fps: float = DEFAULT_FPS, video_max_frames: int = MAX_FRAMES, tools: list | None = None) -> tuple[str, dict] |
Preprocess messages into prompt + generation kwargs for native video. | #L1530-L1648 |
vllm_mlx.models.mllm.MLXMultimodalLM._prepare_video |
method | vllm_mlx.models.mllm.MLXMultimodalLM._prepare_video(video_input: str | dict, fps: float = DEFAULT_FPS, max_frames: int = MAX_FRAMES, resolved_path: str | None = None) -> list[str] |
Process video input and extract frames. | #L1427-L1463 |
vllm_mlx.models.mllm.MLXMultimodalLM._reset_draft_metrics |
method | vllm_mlx.models.mllm.MLXMultimodalLM._reset_draft_metrics() -> int |
Method `MLXMultimodalLM._reset_draft_metrics` updates `self._draft_model.accept_lens`, `self._draft_model._vllm_mlx_draft_counts`; calls `hasattr`; returns `0`. | #L1357-L1364 |
vllm_mlx.models.mllm.MLXMultimodalLM._translate_messages_for_native_video |
method | vllm_mlx.models.mllm.MLXMultimodalLM._translate_messages_for_native_video(messages: list[dict], video_fps: float, video_max_frames: int) -> list[dict] |
Translate OpenAI API format messages to process_vision_info format. | #L1697-L1832 |
vllm_mlx.models.mllm.MLXMultimodalLM.answer_about_image |
method | vllm_mlx.models.mllm.MLXMultimodalLM.answer_about_image(image: str, question: str, max_tokens: int = 256, **kwargs) -> str |
Answer a question about an image. | #L2766-L2791 |
vllm_mlx.models.mllm.MLXMultimodalLM.chat |
method | vllm_mlx.models.mllm.MLXMultimodalLM.chat(messages: list[dict], max_tokens: int = 256, temperature: float = 0.7, **kwargs) -> MLLMOutput |
Chat with OpenAI-compatible message format. | #L2095-L2487 |
vllm_mlx.models.mllm.MLXMultimodalLM.clear_cache |
method | vllm_mlx.models.mllm.MLXMultimodalLM.clear_cache() -> None |
Clear the MLLM KV cache. | #L2848-L2852 |
vllm_mlx.models.mllm.MLXMultimodalLM.describe_image |
method | vllm_mlx.models.mllm.MLXMultimodalLM.describe_image(image: str, prompt: str = 'Describe this image in detail.', max_tokens: int = 512, **kwargs) -> str |
Convenience method to describe an image. | #L2739-L2764 |
vllm_mlx.models.mllm.MLXMultimodalLM.describe_video |
method | vllm_mlx.models.mllm.MLXMultimodalLM.describe_video(video: str | dict, prompt: str = 'Describe what happens in this video.', fps: float = 2.0, max_frames: int = 32, max_tokens: int = 512, **kwargs) -> str |
Describe a video using frame extraction. | #L2793-L2830 |
vllm_mlx.models.mllm.MLXMultimodalLM.generate |
method | vllm_mlx.models.mllm.MLXMultimodalLM.generate(prompt: str, images: list | None = None, videos: list | None = None, audio: list[str] | None = None, max_tokens: int = 256, temperature: float = 0.7, top_p: float = 0.9, video_fps: float = DEFAULT_FPS, video_max_frames: int = MAX_FRAMES, use_cache: bool = True, **kwargs) -> MLLMOutput |
Generate text from multimodal input. | #L1834-L2002 |
vllm_mlx.models.mllm.MLXMultimodalLM.get_cache_stats |
method | vllm_mlx.models.mllm.MLXMultimodalLM.get_cache_stats() -> dict |
Get MLLM cache statistics. | #L2832-L2846 |
vllm_mlx.models.mllm.MLXMultimodalLM.get_language_model |
method | vllm_mlx.models.mllm.MLXMultimodalLM.get_language_model() -> not annotated |
Extract the underlying language model for mlx_lm TextModel construction. | #L1397-L1399 |
vllm_mlx.models.mllm.MLXMultimodalLM.get_model_info |
method | vllm_mlx.models.mllm.MLXMultimodalLM.get_model_info() -> dict |
Get information about the loaded model. | #L2854-L2874 |
vllm_mlx.models.mllm.MLXMultimodalLM.get_tokenizer |
method | vllm_mlx.models.mllm.MLXMultimodalLM.get_tokenizer() -> not annotated |
Get the text tokenizer (not the multimodal processor). | #L1401-L1403 |
vllm_mlx.models.mllm.MLXMultimodalLM.is_mllm_model |
method | vllm_mlx.models.mllm.MLXMultimodalLM.is_mllm_model(model_name: str) -> bool |
Check if a model name indicates an MLLM model. | #L2900-L2934 |
vllm_mlx.models.mllm.MLXMultimodalLM.list_supported_model_families |
method | vllm_mlx.models.mllm.MLXMultimodalLM.list_supported_model_families() -> dict[str, str] |
List supported model families and their patterns. | #L2877-L2897 |
vllm_mlx.models.mllm.MLXMultimodalLM.load |
method | vllm_mlx.models.mllm.MLXMultimodalLM.load() -> None |
Load the model and processor. | #L1280-L1323 |
vllm_mlx.models.mllm.MLXMultimodalLM.stream_chat |
method | vllm_mlx.models.mllm.MLXMultimodalLM.stream_chat(messages: list[dict], max_tokens: int = 256, temperature: float = 0.7, **kwargs) -> Iterator[MLLMOutput] |
Stream chat with OpenAI-compatible message format. | #L2489-L2737 |
vllm_mlx.models.mllm.MLXMultimodalLM.stream_generate |
method | vllm_mlx.models.mllm.MLXMultimodalLM.stream_generate(prompt: str, images: list | None = None, videos: list[str] | None = None, audio: list[str] | None = None, max_tokens: int = 256, temperature: float = 0.7, video_fps: float = DEFAULT_FPS, **kwargs) -> Iterator[str] |
Stream text generation for multimodal input. | #L2004-L2093 |
vllm_mlx.models.mllm.MultimodalInput |
class | vllm_mlx.models.mllm.MultimodalInput(prompt: str, images: list[str] = field(default_factory=list), videos: list[str] = field(default_factory=list), audio: list[str] = field(default_factory=list)) |
Input for multimodal generation. | #L319-L325 |
vllm_mlx.models.mllm.process_audio_input |
function | vllm_mlx.models.mllm.process_audio_input(audio: str | dict) -> str |
Process audio input in various formats and return local path. | #L877-L905 |
vllm_mlx.models.mllm.process_image_input |
function | vllm_mlx.models.mllm.process_image_input(image: str | dict) -> str |
Process image input in various formats and return local path. | #L1051-L1080 |
vllm_mlx.models.mllm.process_video_input |
function | vllm_mlx.models.mllm.process_video_input(video: str | dict) -> str |
Process video input in various formats and return local path. | #L839-L874 |
vllm_mlx.models.mllm.round_by_factor |
function | vllm_mlx.models.mllm.round_by_factor(x: int, factor: int) -> int |
Round to nearest multiple of factor. | #L1083-L1085 |
vllm_mlx.models.mllm.save_base64_image |
function | vllm_mlx.models.mllm.save_base64_image(base64_string: str) -> str |
Save base64 image to temp file and return path. | #L1012-L1048 |
vllm_mlx.models.mllm.save_frames_to_temp |
function | vllm_mlx.models.mllm.save_frames_to_temp(frames: list[np.ndarray]) -> list[str] |
Save frame arrays to temporary files and return paths. | #L1190-L1204 |
vllm_mlx.models.mllm.smart_nframes |
function | vllm_mlx.models.mllm.smart_nframes(total_frames: int, video_fps: float, target_fps: float = DEFAULT_FPS, min_frames: int = MIN_FRAMES, max_frames: int = MAX_FRAMES) -> int |
Calculate optimal number of frames to extract from video. | #L1098-L1120 |
vllm_mlx.models.mllm.TempFileManager |
class | vllm_mlx.models.mllm.TempFileManager() |
Thread-safe manager for tracking and cleaning up temporary files. | #L41-L86 |
vllm_mlx.models.mllm.TempFileManager.__init__ |
method | vllm_mlx.models.mllm.TempFileManager.__init__() -> not annotated |
Method `TempFileManager.__init__` updates `self._files`, `self._lock`; calls `set`, `threading.Lock`, `atexit.register`. | #L44-L47 |
vllm_mlx.models.mllm.TempFileManager.cleanup |
method | vllm_mlx.models.mllm.TempFileManager.cleanup(path: str) -> bool |
Clean up a specific temp file. | #L55-L67 |
vllm_mlx.models.mllm.TempFileManager.cleanup_all |
method | vllm_mlx.models.mllm.TempFileManager.cleanup_all() -> int |
Clean up all tracked temp files. | #L69-L86 |
vllm_mlx.models.mllm.TempFileManager.register |
method | vllm_mlx.models.mllm.TempFileManager.register(path: str) -> str |
Register a temp file for tracking. | #L49-L53 |
vllm_mlx.models.mllm.UnsafeRemoteURLError |
class | vllm_mlx.models.mllm.UnsafeRemoteURLError(message: str, *, public_message: str = 'Remote media URL is not allowed') |
Raised when a remote media URL targets an unsafe destination. | #L125-L135 |
vllm_mlx.models.mllm.UnsafeRemoteURLError.__init__ |
method | vllm_mlx.models.mllm.UnsafeRemoteURLError.__init__(message: str, *, public_message: str = 'Remote media URL is not allowed') -> None |
Method `UnsafeRemoteURLError.__init__` updates `self.public_message`; calls `super().__init__`, `super`. | #L128-L135 |
vllm_mlx.multimodal_processor.MultimodalProcessor |
class | vllm_mlx.multimodal_processor.MultimodalProcessor(model: Any, processor: Any, config: Optional[Any] = None) |
Processor for preparing multimodal inputs for VLM batching. | #L52-L431 |
vllm_mlx.multimodal_processor.MultimodalProcessor.__init__ |
method | vllm_mlx.multimodal_processor.MultimodalProcessor.__init__(model: Any, processor: Any, config: Optional[Any] = None) -> not annotated |
Initialize the multimodal processor. | #L68-L94 |
vllm_mlx.multimodal_processor.MultimodalProcessor.batch_image_grid_thw |
method | vllm_mlx.multimodal_processor.MultimodalProcessor.batch_image_grid_thw(grid_thw_list: List[Optional[mx.array]]) -> Optional[mx.array] |
Batch multiple image_grid_thw tensors together. | #L257-L279 |
vllm_mlx.multimodal_processor.MultimodalProcessor.batch_pixel_values |
method | vllm_mlx.multimodal_processor.MultimodalProcessor.batch_pixel_values(pixel_values_list: List[Optional[mx.array]]) -> Optional[mx.array] |
Batch multiple pixel_values tensors together. | #L226-L255 |
vllm_mlx.multimodal_processor.MultimodalProcessor.compute_vision_hash |
method | vllm_mlx.multimodal_processor.MultimodalProcessor.compute_vision_hash(pixel_values: mx.array) -> str |
Compute a hash for pixel values for caching purposes. | #L411-L431 |
vllm_mlx.multimodal_processor.MultimodalProcessor.extract_vision_embeddings |
method | vllm_mlx.multimodal_processor.MultimodalProcessor.extract_vision_embeddings(pixel_values: mx.array, image_grid_thw: Optional[mx.array] = None) -> mx.array |
Extract vision embeddings from pixel values. | #L368-L409 |
vllm_mlx.multimodal_processor.MultimodalProcessor.prepare_for_batch |
method | vllm_mlx.multimodal_processor.MultimodalProcessor.prepare_for_batch(processed_inputs: List[ProcessedMultimodalInput]) -> Tuple[mx.array, Dict[str, Any], List[int]] |
Prepare multiple processed inputs for batch generation. | #L281-L366 |
vllm_mlx.multimodal_processor.MultimodalProcessor.process |
method | vllm_mlx.multimodal_processor.MultimodalProcessor.process(prompt: str, images: Optional[List[str]] = None, videos: Optional[List[str]] = None, video_fps: float = DEFAULT_FPS, video_max_frames: int = MAX_FRAMES, add_special_tokens: bool = True, **kwargs) -> ProcessedMultimodalInput |
Process multimodal inputs for batching. | #L96-L186 |
vllm_mlx.multimodal_processor.MultimodalProcessor.process_for_request |
method | vllm_mlx.multimodal_processor.MultimodalProcessor.process_for_request(prompt: str, images: Optional[List[str]] = None, videos: Optional[List[str]] = None, **kwargs) -> Dict[str, Any] |
Process inputs and return a dict suitable for Request fields. | #L188-L224 |
vllm_mlx.multimodal_processor.ProcessedMultimodalInput |
class | vllm_mlx.multimodal_processor.ProcessedMultimodalInput(input_ids: mx.array, pixel_values: Optional[mx.array] = None, attention_mask: Optional[mx.array] = None, image_grid_thw: Optional[mx.array] = None, num_images: int = 0, num_tokens: int = 0, extra_kwargs: Dict[str, Any] = field(default_factory=dict)) |
Container for processed multimodal inputs ready for batching. | #L29-L49 |
vllm_mlx.optimizations.benchmark_memory_bandwidth |
function | vllm_mlx.optimizations.benchmark_memory_bandwidth() -> dict |
Benchmark actual memory bandwidth achieved. | #L144-L174 |
vllm_mlx.optimizations.detect_hardware |
function | vllm_mlx.optimizations.detect_hardware() -> HardwareInfo |
Detect Apple Silicon hardware and return info. | #L97-L141 |
vllm_mlx.optimizations.get_optimization_status |
function | vllm_mlx.optimizations.get_optimization_status() -> dict |
Get current hardware and MLX status. | #L177-L209 |
vllm_mlx.optimizations.get_system_memory_gb |
function | vllm_mlx.optimizations.get_system_memory_gb() -> float |
Get actual system memory in GB. | #L68-L94 |
vllm_mlx.optimizations.HardwareInfo |
class | vllm_mlx.optimizations.HardwareInfo(chip_name: str, total_memory_gb: float, memory_bandwidth_gbs: float, gpu_cores: int) |
Hardware information for Apple Silicon. | #L34-L40 |
vllm_mlx.output_collector.RequestOutputCollector |
class | vllm_mlx.output_collector.RequestOutputCollector(aggregate: bool = True) |
Per-request output collector with smart buffering. | #L17-L170 |
vllm_mlx.output_collector.RequestOutputCollector.__init__ |
method | vllm_mlx.output_collector.RequestOutputCollector.__init__(aggregate: bool = True) -> not annotated |
Initialize the collector. | #L42-L53 |
vllm_mlx.output_collector.RequestOutputCollector._merge_outputs |
method | vllm_mlx.output_collector.RequestOutputCollector._merge_outputs(existing: RequestOutput, new: RequestOutput) -> RequestOutput |
Merge two outputs when producer gets ahead of consumer. | #L120-L152 |
vllm_mlx.output_collector.RequestOutputCollector.clear |
method | vllm_mlx.output_collector.RequestOutputCollector.clear() -> None |
Clear any pending output. | #L154-L161 |
vllm_mlx.output_collector.RequestOutputCollector.get |
method | async vllm_mlx.output_collector.RequestOutputCollector.get() -> RequestOutput |
Get output, blocking only if none available. | #L91-L118 |
vllm_mlx.output_collector.RequestOutputCollector.get_nowait |
method | vllm_mlx.output_collector.RequestOutputCollector.get_nowait() -> Optional[RequestOutput] |
Get output without blocking. | #L75-L89 |
vllm_mlx.output_collector.RequestOutputCollector.has_waiting_consumers |
method | vllm_mlx.output_collector.RequestOutputCollector.has_waiting_consumers() -> bool |
Check if any collector has waiting consumers. | #L164-L170 |
vllm_mlx.output_collector.RequestOutputCollector.put |
method | vllm_mlx.output_collector.RequestOutputCollector.put(output: RequestOutput) -> None |
Put an output into the collector (non-blocking). | #L55-L73 |
vllm_mlx.output_collector.RequestStreamState |
class | vllm_mlx.output_collector.RequestStreamState(stream_interval: int = 1, sent_tokens: int = 0) |
Tracks streaming state for a request. | #L174-L212 |
vllm_mlx.output_collector.RequestStreamState.mark_sent |
method | vllm_mlx.output_collector.RequestStreamState.mark_sent(total_tokens: int) -> None |
Update state after sending output. | #L205-L212 |
vllm_mlx.output_collector.RequestStreamState.should_send |
method | vllm_mlx.output_collector.RequestStreamState.should_send(total_tokens: int, finished: bool) -> bool |
Determine if output should be sent based on stream_interval. | #L185-L203 |
vllm_mlx.paged_cache.BlockHashToBlockMap |
class | vllm_mlx.paged_cache.BlockHashToBlockMap() |
Cache mapping block hashes to blocks for prefix caching. | #L345-L407 |
vllm_mlx.paged_cache.BlockHashToBlockMap.__init__ |
method | vllm_mlx.paged_cache.BlockHashToBlockMap.__init__() -> None |
Method `BlockHashToBlockMap.__init__` updates `self._cache`. | #L353-L354 |
vllm_mlx.paged_cache.BlockHashToBlockMap.__len__ |
method | vllm_mlx.paged_cache.BlockHashToBlockMap.__len__() -> int |
Method `BlockHashToBlockMap.__len__` calls `len`; returns `len(self._cache)`. | #L401-L402 |
vllm_mlx.paged_cache.BlockHashToBlockMap.clear |
method | vllm_mlx.paged_cache.BlockHashToBlockMap.clear() -> None |
Remove every block-hash mapping without mutating the blocks. | #L404-L407 |
vllm_mlx.paged_cache.BlockHashToBlockMap.get_block |
method | vllm_mlx.paged_cache.BlockHashToBlockMap.get_block(block_hash: BlockHash) -> Optional[CacheBlock] |
Get any block with the given hash. | #L356-L365 |
vllm_mlx.paged_cache.BlockHashToBlockMap.insert |
method | vllm_mlx.paged_cache.BlockHashToBlockMap.insert(block_hash: BlockHash, block: CacheBlock) -> None |
Insert a block into the cache. | #L367-L378 |
vllm_mlx.paged_cache.BlockHashToBlockMap.pop |
method | vllm_mlx.paged_cache.BlockHashToBlockMap.pop(block_hash: BlockHash, block_id: int) -> Optional[CacheBlock] |
Remove and return a specific block from the cache. | #L380-L399 |
vllm_mlx.paged_cache.BlockTable |
class | vllm_mlx.paged_cache.BlockTable(request_id: str, block_ids: List[int] = field(default_factory=list), num_tokens: int = 0) |
Per-request block table mapping logical to physical blocks. | #L416-L447 |
vllm_mlx.paged_cache.BlockTable.__len__ |
method | vllm_mlx.paged_cache.BlockTable.__len__() -> int |
Method `BlockTable.__len__` calls `len`; returns `len(self.block_ids)`. | #L438-L439 |
vllm_mlx.paged_cache.BlockTable.add_block |
method | vllm_mlx.paged_cache.BlockTable.add_block(block_id: int, num_tokens: int) -> None |
Add a block to the table. | #L433-L436 |
vllm_mlx.paged_cache.BlockTable.copy |
method | vllm_mlx.paged_cache.BlockTable.copy(new_request_id: str) -> 'BlockTable' |
Create a copy with new request ID. | #L441-L447 |
vllm_mlx.paged_cache.CacheBlock |
class | vllm_mlx.paged_cache.CacheBlock(block_id: int, ref_count: int = 0, block_hash: Optional[BlockHash] = None, prev_free_block: Optional['CacheBlock'] = None, next_free_block: Optional['CacheBlock'] = None, is_null: bool = False, cache_data: Optional[List[Tuple[Any, Any]]] = None, token_count: int = 0, hash_value: Optional[str] = None, last_access: float = field(default_factory=time.time)) |
KV cache block metadata following vLLM's design. | #L84-L146 |
vllm_mlx.paged_cache.CacheBlock.__repr__ |
method | vllm_mlx.paged_cache.CacheBlock.__repr__() -> str |
Method `CacheBlock.__repr__` returns `f'CacheBlock(id={self.block_id}, ref={self.ref_count}, tokens={self.token_count}, prev={prev_id}, next={next_id})'`. | #L140-L146 |
vllm_mlx.paged_cache.CacheBlock.is_full |
method | vllm_mlx.paged_cache.CacheBlock.is_full(block_size: int) -> bool |
Check if block is at capacity. | #L123-L125 |
vllm_mlx.paged_cache.CacheBlock.is_shared |
method | vllm_mlx.paged_cache.CacheBlock.is_shared() -> bool |
Check if block is shared (ref_count > 1). | #L127-L129 |
vllm_mlx.paged_cache.CacheBlock.reset_hash |
method | vllm_mlx.paged_cache.CacheBlock.reset_hash() -> None |
Reset block hash when evicted from cache. | #L131-L134 |
vllm_mlx.paged_cache.CacheBlock.touch |
method | vllm_mlx.paged_cache.CacheBlock.touch() -> None |
Update last access time. | #L136-L138 |
vllm_mlx.paged_cache.CacheStats |
class | vllm_mlx.paged_cache.CacheStats(total_blocks: int = 0, allocated_blocks: int = 0, free_blocks: int = 0, shared_blocks: int = 0, total_tokens_cached: int = 0, cache_hits: int = 0, cache_misses: int = 0, cow_copies: int = 0, evictions: int = 0) |
Statistics for cache monitoring. | #L456-L467 |
vllm_mlx.paged_cache.compute_block_hash |
function | vllm_mlx.paged_cache.compute_block_hash(parent_hash: Optional[BlockHash], token_ids: List[int], extra_keys: Optional[Tuple[Any, ...]] = None) -> BlockHash |
Compute hash for a block based on its content and parent block. | #L40-L75 |
vllm_mlx.paged_cache.FreeKVCacheBlockQueue |
class | vllm_mlx.paged_cache.FreeKVCacheBlockQueue(blocks: List[CacheBlock]) |
Doubly linked list of free blocks following vLLM's design. | #L158-L337 |
vllm_mlx.paged_cache.FreeKVCacheBlockQueue.__init__ |
method | vllm_mlx.paged_cache.FreeKVCacheBlockQueue.__init__(blocks: List[CacheBlock]) -> None |
Initialize queue with all blocks as free. | #L174-L201 |
vllm_mlx.paged_cache.FreeKVCacheBlockQueue.append |
method | vllm_mlx.paged_cache.FreeKVCacheBlockQueue.append(block: CacheBlock) -> None |
Append a block to the end (MRU position). | #L290-L305 |
vllm_mlx.paged_cache.FreeKVCacheBlockQueue.append_n |
method | vllm_mlx.paged_cache.FreeKVCacheBlockQueue.append_n(blocks: List[CacheBlock]) -> None |
Append multiple blocks to the end. | #L307-L328 |
vllm_mlx.paged_cache.FreeKVCacheBlockQueue.get_all_free_blocks |
method | vllm_mlx.paged_cache.FreeKVCacheBlockQueue.get_all_free_blocks() -> List[CacheBlock] |
Get all free blocks (for testing). | #L330-L337 |
vllm_mlx.paged_cache.FreeKVCacheBlockQueue.popleft |
method | vllm_mlx.paged_cache.FreeKVCacheBlockQueue.popleft() -> CacheBlock |
Pop and return the first (LRU) free block. | #L203-L225 |
vllm_mlx.paged_cache.FreeKVCacheBlockQueue.popleft_n |
method | vllm_mlx.paged_cache.FreeKVCacheBlockQueue.popleft_n(n: int) -> List[CacheBlock] |
Pop n blocks from the front. | #L227-L265 |
vllm_mlx.paged_cache.FreeKVCacheBlockQueue.remove |
method | vllm_mlx.paged_cache.FreeKVCacheBlockQueue.remove(block: CacheBlock) -> None |
Remove a block from the middle of the queue. | #L267-L288 |
vllm_mlx.paged_cache.PagedCacheManager |
class | vllm_mlx.paged_cache.PagedCacheManager(block_size: int = 64, max_blocks: int = 1000, enable_caching: bool = True) |
Paged KV cache manager following vLLM's BlockPool architecture. | #L475-L1197 |
vllm_mlx.paged_cache.PagedCacheManager.__init__ |
method | vllm_mlx.paged_cache.PagedCacheManager.__init__(block_size: int = 64, max_blocks: int = 1000, enable_caching: bool = True) -> not annotated |
Method `PagedCacheManager.__init__` updates `self.block_size`, `self.max_blocks`, `self.enable_caching`, `self.blocks`; calls `CacheBlock`, `range`, `FreeKVCacheBlockQueue`, `BlockHashToBlockMap`. | #L491-L540 |
vllm_mlx.paged_cache.PagedCacheManager._cow_copy_block |
method | vllm_mlx.paged_cache.PagedCacheManager._cow_copy_block(source_block: CacheBlock) -> Optional[CacheBlock] |
Create a copy of a block for COW. | #L1031-L1046 |
vllm_mlx.paged_cache.PagedCacheManager._maybe_evict_cached_block |
method | vllm_mlx.paged_cache.PagedCacheManager._maybe_evict_cached_block(block: CacheBlock) -> bool |
Evict a block from the hash cache if present. | #L608-L634 |
vllm_mlx.paged_cache.PagedCacheManager.add_block_to_table |
method | vllm_mlx.paged_cache.PagedCacheManager.add_block_to_table(table: BlockTable, block: CacheBlock, tokens_in_block: int) -> None |
Add a block to a block table. | #L936-L947 |
vllm_mlx.paged_cache.PagedCacheManager.allocate_block |
method | vllm_mlx.paged_cache.PagedCacheManager.allocate_block() -> Optional[CacheBlock] |
Allocate a new cache block. | #L546-L571 |
vllm_mlx.paged_cache.PagedCacheManager.allocate_blocks_for_tokens |
method | vllm_mlx.paged_cache.PagedCacheManager.allocate_blocks_for_tokens(num_tokens: int) -> List[CacheBlock] |
Allocate enough blocks to hold num_tokens. | #L1052-L1055 |
vllm_mlx.paged_cache.PagedCacheManager.cache_full_blocks |
method | vllm_mlx.paged_cache.PagedCacheManager.cache_full_blocks(blocks: List[CacheBlock], token_ids: List[int], num_cached_blocks: int, num_full_blocks: int) -> None |
Cache full blocks for prefix caching (vLLM style). | #L770-L824 |
vllm_mlx.paged_cache.PagedCacheManager.clear |
method | vllm_mlx.paged_cache.PagedCacheManager.clear() -> None |
Clear all cached data. | #L1173-L1197 |
vllm_mlx.paged_cache.PagedCacheManager.compute_block_hash |
method | vllm_mlx.paged_cache.PagedCacheManager.compute_block_hash(tokens: List[int]) -> str |
Compute legacy string hash for a sequence of tokens. | #L875-L878 |
vllm_mlx.paged_cache.PagedCacheManager.create_block_table |
method | vllm_mlx.paged_cache.PagedCacheManager.create_block_table(request_id: str) -> BlockTable |
Create a new block table for a request. | #L909-L914 |
vllm_mlx.paged_cache.PagedCacheManager.decrement_ref |
method | vllm_mlx.paged_cache.PagedCacheManager.decrement_ref(block_id: int) -> bool |
Decrement reference count (alias for free_block). | #L741-L743 |
vllm_mlx.paged_cache.PagedCacheManager.delete_block_table |
method | vllm_mlx.paged_cache.PagedCacheManager.delete_block_table(request_id: str) -> None |
Delete block table and free associated blocks. | #L928-L934 |
vllm_mlx.paged_cache.PagedCacheManager.evict_lru_blocks |
method | vllm_mlx.paged_cache.PagedCacheManager.evict_lru_blocks(num_blocks: int) -> int |
Evict least recently used blocks. | #L1061-L1085 |
vllm_mlx.paged_cache.PagedCacheManager.find_cached_block |
method | vllm_mlx.paged_cache.PagedCacheManager.find_cached_block(tokens: List[int]) -> Optional[CacheBlock] |
Find a cached block matching the given tokens (legacy method). | #L880-L896 |
vllm_mlx.paged_cache.PagedCacheManager.find_shared_prefix |
method | vllm_mlx.paged_cache.PagedCacheManager.find_shared_prefix(tokens: List[int]) -> Tuple[List[int], List[int]] |
Find shared prefix blocks for a token sequence. | #L953-L974 |
vllm_mlx.paged_cache.PagedCacheManager.fork_block_table |
method | vllm_mlx.paged_cache.PagedCacheManager.fork_block_table(source_table: BlockTable, new_request_id: str) -> BlockTable |
Fork a block table for a new request (COW). | #L976-L997 |
vllm_mlx.paged_cache.PagedCacheManager.free_block |
method | vllm_mlx.paged_cache.PagedCacheManager.free_block(block_id: int) -> bool |
Free a cache block (decrements ref_count, frees if 0). | #L636-L667 |
vllm_mlx.paged_cache.PagedCacheManager.free_blocks |
method | vllm_mlx.paged_cache.PagedCacheManager.free_blocks(blocks: Iterable[CacheBlock]) -> None |
Free multiple blocks (vLLM style). | #L669-L696 |
vllm_mlx.paged_cache.PagedCacheManager.free_blocks |
method | vllm_mlx.paged_cache.PagedCacheManager.free_blocks() -> int |
Number of free blocks available. | #L1103-L1105 |
vllm_mlx.paged_cache.PagedCacheManager.get_block_table |
method | vllm_mlx.paged_cache.PagedCacheManager.get_block_table(request_id: str) -> Optional[BlockTable] |
Get block table for a request. | #L916-L919 |
vllm_mlx.paged_cache.PagedCacheManager.get_blocks_for_generation |
method | vllm_mlx.paged_cache.PagedCacheManager.get_blocks_for_generation(table: BlockTable) -> Tuple[List[CacheBlock], bool] |
Get blocks for generation, applying COW if needed. | #L999-L1029 |
vllm_mlx.paged_cache.PagedCacheManager.get_cached_block |
method | vllm_mlx.paged_cache.PagedCacheManager.get_cached_block(block_hash: BlockHash) -> Optional[CacheBlock] |
Get a cached block by its hash (vLLM style). | #L749-L768 |
vllm_mlx.paged_cache.PagedCacheManager.get_computed_blocks |
method | vllm_mlx.paged_cache.PagedCacheManager.get_computed_blocks(token_ids: List[int]) -> Tuple[List[CacheBlock], int] |
Find cached blocks for a token prefix (vLLM style). | #L826-L868 |
vllm_mlx.paged_cache.PagedCacheManager.get_memory_usage |
method | vllm_mlx.paged_cache.PagedCacheManager.get_memory_usage() -> Dict[str, Any] |
Get memory usage information. | #L1124-L1141 |
vllm_mlx.paged_cache.PagedCacheManager.get_new_blocks |
method | vllm_mlx.paged_cache.PagedCacheManager.get_new_blocks(num_blocks: int) -> List[CacheBlock] |
Allocate multiple blocks at once (vLLM style). | #L573-L606 |
vllm_mlx.paged_cache.PagedCacheManager.get_or_create_block_table |
method | vllm_mlx.paged_cache.PagedCacheManager.get_or_create_block_table(request_id: str) -> BlockTable |
Get or create block table for a request. | #L921-L926 |
vllm_mlx.paged_cache.PagedCacheManager.get_stats |
method | vllm_mlx.paged_cache.PagedCacheManager.get_stats() -> CacheStats |
Get current cache statistics. | #L1115-L1122 |
vllm_mlx.paged_cache.PagedCacheManager.handle_memory_pressure |
method | vllm_mlx.paged_cache.PagedCacheManager.handle_memory_pressure(requested_blocks: int) -> bool |
Handle memory pressure by evicting blocks. | #L1087-L1096 |
vllm_mlx.paged_cache.PagedCacheManager.increment_ref |
method | vllm_mlx.paged_cache.PagedCacheManager.increment_ref(block_id: int) -> bool |
Increment reference count for a block. | #L726-L739 |
vllm_mlx.paged_cache.PagedCacheManager.register_block_hash |
method | vllm_mlx.paged_cache.PagedCacheManager.register_block_hash(block: CacheBlock, tokens: List[int]) -> None |
Register a block's hash for deduplication (legacy method). | #L898-L903 |
vllm_mlx.paged_cache.PagedCacheManager.reset_prefix_cache |
method | vllm_mlx.paged_cache.PagedCacheManager.reset_prefix_cache() -> bool |
Reset the prefix cache. | #L1151-L1171 |
vllm_mlx.paged_cache.PagedCacheManager.reset_stats |
method | vllm_mlx.paged_cache.PagedCacheManager.reset_stats() -> None |
Reset statistics counters. | #L1143-L1149 |
vllm_mlx.paged_cache.PagedCacheManager.touch |
method | vllm_mlx.paged_cache.PagedCacheManager.touch(blocks: Iterable[CacheBlock]) -> None |
Touch blocks to prevent eviction (cache hit, vLLM style). | #L698-L720 |
vllm_mlx.paged_cache.PagedCacheManager.usage |
method | vllm_mlx.paged_cache.PagedCacheManager.usage() -> float |
Cache usage ratio (0.0 to 1.0). | #L1108-L1113 |
vllm_mlx.patches.gemma4_mllm.patch_gemma4_attention_for_batching |
function | vllm_mlx.patches.gemma4_mllm.patch_gemma4_attention_for_batching() -> bool |
Patch Gemma 4 Attention.__call__ to trim oversized masks. | #L28-L98 |
vllm_mlx.patches.gemma4_mllm.patch_gemma4_attention_for_batching._patched_call |
nested function | vllm_mlx.patches.gemma4_mllm.patch_gemma4_attention_for_batching._patched_call(x: mx.array, mask: Optional[mx.array] = None, cache: Optional[Any] = None, shared_kv: Optional[tuple] = None, offset: Optional[Any] = None) -> Any |
Nested Function `patch_gemma4_attention_for_batching._patched_call` calls `self.q_proj(x).reshape`, `self.q_proj`, `self.q_norm`, `self.k_proj(x).reshape`; returns `(self.o_proj(output), (keys, values), offset)`. | #L45-L93 |
vllm_mlx.patches.glm4v_moe_mllm.patch_glm4v_moe_for_batching |
function | vllm_mlx.patches.glm4v_moe_mllm.patch_glm4v_moe_for_batching() -> bool |
Monkey-patch GLM4VModel.__call__ to handle BatchKVCache offset. | #L31-L89 |
vllm_mlx.patches.glm4v_moe_mllm.patch_glm4v_moe_for_batching._patched_call |
nested function | vllm_mlx.patches.glm4v_moe_mllm.patch_glm4v_moe_for_batching._patched_call(inputs: mx.array, inputs_embeds: Optional[mx.array] = None, cache: Optional[Any] = None, mask: Optional[mx.array] = None, position_ids: Optional[mx.array] = None) -> mx.array |
Nested Function `patch_glm4v_moe_for_batching._patched_call` calls `self.embed_tokens`, `inputs_embeds.astype`, `isinstance`, `int`; returns `self.norm(h)`. | #L50-L84 |
vllm_mlx.patches.qwen3_5_mllm._apply_rotary |
function | vllm_mlx.patches.qwen3_5_mllm._apply_rotary(attention, queries: mx.array, keys: mx.array, values: mx.array, position_ids: mx.array, position_embeddings: Optional[tuple[mx.array, mx.array]], apply_multimodal_rotary_pos_emb) -> tuple[mx.array, mx.array] |
Function `_apply_rotary` calls `apply_multimodal_rotary_pos_emb`, `hasattr`, `attention.rotary_emb.apply_rotary`, `attention.rotary_emb`; has 2 explicit return paths. | #L79-L101 |
vllm_mlx.patches.qwen3_5_mllm._cache_offset_to_int |
function | vllm_mlx.patches.qwen3_5_mllm._cache_offset_to_int(cache) -> int |
Extract cache offset as int, handling BatchKVCache mx.array offset. | #L33-L42 |
vllm_mlx.patches.qwen3_5_mllm._default_target_verify_left_padded_attention |
function | vllm_mlx.patches.qwen3_5_mllm._default_target_verify_left_padded_attention(*args, **kwargs) -> not annotated |
Function `_default_target_verify_left_padded_attention` returns `None`. | #L49-L50 |
vllm_mlx.patches.qwen3_5_mllm._default_target_verify_linears |
function | vllm_mlx.patches.qwen3_5_mllm._default_target_verify_linears(linears, x, target_verify: bool) -> not annotated |
Function `_default_target_verify_linears` calls `tuple`, `linear`; returns `tuple((linear(x) for linear in linears))`. | #L45-L46 |
vllm_mlx.patches.qwen3_5_mllm._kv_seq_len |
function | vllm_mlx.patches.qwen3_5_mllm._kv_seq_len(keys: mx.array, cache: Optional[Any], offset: int) -> int |
Function `_kv_seq_len` returns `length + offset + 1 if cache is not None else length`. | #L74-L76 |
vllm_mlx.patches.qwen3_5_mllm._maybe_target_verify_attention |
function | vllm_mlx.patches.qwen3_5_mllm._maybe_target_verify_attention(queries: mx.array, keys: mx.array, values: mx.array, *, cache: Optional[Any], mask: Optional[mx.array], scale: float, target_verify: bool, length: int, left_padded_decode: bool, target_verify_left_padded_attention) -> Optional[mx.array] |
Function `_maybe_target_verify_attention` calls `target_verify_left_padded_attention`; has 2 explicit return paths. | #L119-L141 |
vllm_mlx.patches.qwen3_5_mllm._normalize_position_inputs |
function | vllm_mlx.patches.qwen3_5_mllm._normalize_position_inputs(position_ids: Optional[mx.array], position_embeddings: Optional[tuple[mx.array, mx.array]], length: int) -> tuple[Optional[mx.array], Optional[tuple[mx.array, mx.array]]] |
Function `_normalize_position_inputs` calls `logger.debug`; has 2 explicit return paths. | #L53-L65 |
vllm_mlx.patches.qwen3_5_mllm._position_ids_for_offset |
function | vllm_mlx.patches.qwen3_5_mllm._position_ids_for_offset(offset: int, length: int) -> mx.array |
Function `_position_ids_for_offset` calls `mx.arange`, `mx.expand_dims`, `mx.tile`; returns `mx.tile(position_ids, (3, 1, 1))`. | #L68-L71 |
vllm_mlx.patches.qwen3_5_mllm._slice_attention_mask |
function | vllm_mlx.patches.qwen3_5_mllm._slice_attention_mask(mask: Optional[mx.array], cache: Optional[Any], kv_seq_len: int, length: int) -> Optional[mx.array] |
Function `_slice_attention_mask` calls `isinstance`, `hasattr`, `int`, `kv_seq_len.max().item`; has 2 explicit return paths. | #L104-L116 |
vllm_mlx.patches.qwen3_5_mllm.patch_qwen35_attention_for_batching |
function | vllm_mlx.patches.qwen3_5_mllm.patch_qwen35_attention_for_batching() -> bool |
Monkey-patch Qwen3_5Attention.__call__ to handle BatchKVCache. | #L144-L266 |
vllm_mlx.patches.qwen3_5_mllm.patch_qwen35_attention_for_batching._patched_call |
nested function | vllm_mlx.patches.qwen3_5_mllm.patch_qwen35_attention_for_batching._patched_call(x: mx.array, mask: Optional[mx.array] = None, cache: Optional[Any] = None, position_ids: Optional[mx.array] = None, position_embeddings: Optional[tuple[mx.array, mx.array]] = None, target_verify: bool = False) -> mx.array |
Nested Function `patch_qwen35_attention_for_batching._patched_call` calls `target_verify_linears`, `mx.split`, `q_proj_output.reshape`, `gate.reshape`; returns `self.o_proj(output * mx.sigmoid(gate))`. | #L174-L261 |
vllm_mlx.patches.qwen3_5_mtp._apply_qwen_mtp_rmsnorm_offset_fixups |
function | vllm_mlx.patches.qwen3_5_mtp._apply_qwen_mtp_rmsnorm_offset_fixups(mtp_weights: dict) -> int |
Apply Qwen raw-offset RMSNorm fixups without double-shifting MLX weights. | #L80-L90 |
vllm_mlx.patches.qwen3_5_mtp._fixup_moe_mtp |
function | vllm_mlx.patches.qwen3_5_mtp._fixup_moe_mtp(mtp, inner_model, loaded_keys: set, mx) -> None |
Fix missing weights in MoE MTP module. | #L93-L157 |
vllm_mlx.patches.qwen3_5_mtp._is_qwen_mtp_rmsnorm_weight |
function | vllm_mlx.patches.qwen3_5_mtp._is_qwen_mtp_rmsnorm_weight(key: str, weight) -> bool |
Return True for MTP RMSNorm weights that use Qwen's offset convention. | #L73-L77 |
vllm_mlx.patches.qwen3_5_mtp._resolve_qwen_mtp_hidden_state_mode |
function | vllm_mlx.patches.qwen3_5_mtp._resolve_qwen_mtp_hidden_state_mode(config: dict) -> str |
Resolve the checkpoint's MTP hidden-state contract safely. | #L52-L65 |
vllm_mlx.patches.qwen3_5_mtp._select_qwen_mtp_hidden_state |
function | vllm_mlx.patches.qwen3_5_mtp._select_qwen_mtp_hidden_state(mode: str, hidden_states, normed) -> not annotated |
Select the representation expected by the checkpoint's MTP head. | #L68-L70 |
vllm_mlx.patches.qwen3_5_mtp._strip_mtp_key_prefix |
function | vllm_mlx.patches.qwen3_5_mtp._strip_mtp_key_prefix(key: str) -> str | None |
Return an MTP-relative key for supported standalone shard layouts. | #L31-L36 |
vllm_mlx.patches.qwen3_5_mtp.inject_mtp_support |
function | vllm_mlx.patches.qwen3_5_mtp.inject_mtp_support(model: Any, model_path, config: dict) -> bool |
Inject MTP module into a loaded Qwen3.5 model. | #L160-L447 |
vllm_mlx.patches.qwen3_5_mtp.inject_mtp_support._MTPModule |
nested class | vllm_mlx.patches.qwen3_5_mtp.inject_mtp_support._MTPModule(args, n_layers) |
Nested Class `inject_mtp_support._MTPModule` derives from `nn.Module` and declares 1 direct member(s). | #L239-L252 |
vllm_mlx.patches.qwen3_5_mtp.inject_mtp_support._MTPModule.__init__ |
nested function | vllm_mlx.patches.qwen3_5_mtp.inject_mtp_support._MTPModule.__init__(args, n_layers) -> not annotated |
Nested Function `inject_mtp_support._MTPModule.__init__` updates `self.pre_fc_norm_hidden`, `self.pre_fc_norm_embedding`, `self.fc`, `self.layers`; calls `super().__init__`, `super`, `nn.RMSNorm`, `nn.Linear`. | #L240-L252 |
vllm_mlx.patches.qwen3_5_mtp.inject_mtp_support._Qwen3_5MTP |
nested class | vllm_mlx.patches.qwen3_5_mtp.inject_mtp_support._Qwen3_5MTP() |
Qwen3.5 with MTP support (injected at runtime). | #L368-L438 |
vllm_mlx.patches.qwen3_5_mtp.inject_mtp_support._Qwen3_5MTP.__call__ |
nested function | vllm_mlx.patches.qwen3_5_mtp.inject_mtp_support._Qwen3_5MTP.__call__(inputs, cache = None, return_hidden: bool = False, input_embeddings = None, **kwargs) -> not annotated |
Nested Function `inject_mtp_support._Qwen3_5MTP.__call__` calls `inner.embed_tokens`, `len`, `create_attention_mask`, `create_ssm_mask`; has 2 explicit return paths. | #L371-L408 |
vllm_mlx.patches.qwen3_5_mtp.inject_mtp_support._Qwen3_5MTP.make_mtp_cache |
nested function | vllm_mlx.patches.qwen3_5_mtp.inject_mtp_support._Qwen3_5MTP.make_mtp_cache() -> not annotated |
Create KV cache for MTP layers. | #L434-L438 |
vllm_mlx.patches.qwen3_5_mtp.inject_mtp_support._Qwen3_5MTP.mtp_forward |
nested function | vllm_mlx.patches.qwen3_5_mtp.inject_mtp_support._Qwen3_5MTP.mtp_forward(hidden_states, next_token_ids, cache = None, mtp_cache = None) -> not annotated |
Run MTP head: predict token n+2 from hidden states + token n+1. | #L410-L432 |
vllm_mlx.patches.qwen3_5_mtp.validate_mtp_support |
function | vllm_mlx.patches.qwen3_5_mtp.validate_mtp_support(model: Any) -> bool |
Validate that a loaded model has working MTP support. | #L450-L512 |
vllm_mlx.patches.qwen3_next_mtp.inject_mtp_support |
function | vllm_mlx.patches.qwen3_next_mtp.inject_mtp_support(model: Any, model_path, config: dict) -> bool |
Inject MTP module into a loaded Qwen3-Next model. | #L27-L181 |
vllm_mlx.patches.qwen3_next_mtp.inject_mtp_support._mtp_quant_pred |
nested function | vllm_mlx.patches.qwen3_next_mtp.inject_mtp_support._mtp_quant_pred(path, module) -> not annotated |
Nested Function `inject_mtp_support._mtp_quant_pred` calls `isinstance`, `path.endswith`; has 3 explicit return paths. | #L93-L103 |
vllm_mlx.patches.qwen3_next_mtp.inject_mtp_support._MTPModule |
nested class | vllm_mlx.patches.qwen3_next_mtp.inject_mtp_support._MTPModule(args, n_layers) |
Nested Class `inject_mtp_support._MTPModule` derives from `nn.Module` and declares 1 direct member(s). | #L68-L83 |
vllm_mlx.patches.qwen3_next_mtp.inject_mtp_support._MTPModule.__init__ |
nested function | vllm_mlx.patches.qwen3_next_mtp.inject_mtp_support._MTPModule.__init__(args, n_layers) -> not annotated |
Nested Function `inject_mtp_support._MTPModule.__init__` updates `self.pre_fc_norm_hidden`, `self.pre_fc_norm_embedding`, `self.fc`, `self.layers`; calls `super().__init__`, `super`, `nn.RMSNorm`, `nn.Linear`. | #L69-L83 |
vllm_mlx.patches.qwen3_next_mtp.inject_mtp_support._Qwen3NextMTP |
nested class | vllm_mlx.patches.qwen3_next_mtp.inject_mtp_support._Qwen3NextMTP() |
Qwen3-Next with MTP support (injected at runtime). | #L125-L177 |
vllm_mlx.patches.qwen3_next_mtp.inject_mtp_support._Qwen3NextMTP.__call__ |
nested function | vllm_mlx.patches.qwen3_next_mtp.inject_mtp_support._Qwen3NextMTP.__call__(inputs, cache = None, return_hidden: bool = False) -> not annotated |
Nested Function `inject_mtp_support._Qwen3NextMTP.__call__` calls `inner.embed_tokens`, `len`, `create_attention_mask`, `create_ssm_mask`; has 2 explicit return paths. | #L128-L150 |
vllm_mlx.patches.qwen3_next_mtp.inject_mtp_support._Qwen3NextMTP.make_mtp_cache |
nested function | vllm_mlx.patches.qwen3_next_mtp.inject_mtp_support._Qwen3NextMTP.make_mtp_cache() -> not annotated |
Create KV cache for MTP layers. | #L173-L177 |
vllm_mlx.patches.qwen3_next_mtp.inject_mtp_support._Qwen3NextMTP.mtp_forward |
nested function | vllm_mlx.patches.qwen3_next_mtp.inject_mtp_support._Qwen3NextMTP.mtp_forward(hidden_states, next_token_ids, cache = None, mtp_cache = None) -> not annotated |
Run MTP head: predict token n+2 from hidden states + token n+1. | #L152-L171 |
vllm_mlx.patches.qwen3_next_mtp.validate_mtp_support |
function | vllm_mlx.patches.qwen3_next_mtp.validate_mtp_support(model: Any) -> bool |
Validate that a loaded model has working MTP support. | #L184-L261 |
vllm_mlx.plugin.get_mlx_device_info |
function | vllm_mlx.plugin.get_mlx_device_info() -> dict |
Get information about the MLX device. | #L83-L155 |
vllm_mlx.plugin.is_mlx_available |
function | vllm_mlx.plugin.is_mlx_available() -> bool |
Check if MLX platform can be used. | #L73-L80 |
vllm_mlx.plugin.mlx_platform_plugin |
function | vllm_mlx.plugin.mlx_platform_plugin() -> str | None |
Platform plugin entry point for vLLM. | #L17-L70 |
vllm_mlx.prefix_cache.BlockAwarePrefixCache |
class | vllm_mlx.prefix_cache.BlockAwarePrefixCache(model: Any, paged_cache_manager: PagedCacheManager) |
Prefix cache that uses PagedCacheManager for block-based storage. | #L372-L1039 |
vllm_mlx.prefix_cache.BlockAwarePrefixCache.__init__ |
method | vllm_mlx.prefix_cache.BlockAwarePrefixCache.__init__(model: Any, paged_cache_manager: PagedCacheManager) -> not annotated |
Initialize block-aware prefix cache. | #L399-L426 |
vllm_mlx.prefix_cache.BlockAwarePrefixCache.__len__ |
method | vllm_mlx.prefix_cache.BlockAwarePrefixCache.__len__() -> int |
Return number of active request entries. | #L1037-L1039 |
vllm_mlx.prefix_cache.BlockAwarePrefixCache._cache_state_seq_axis |
method | vllm_mlx.prefix_cache.BlockAwarePrefixCache._cache_state_seq_axis(state: Any) -> Optional[int] |
Return the sequence axis for cache states that support block concat. | #L704-L725 |
vllm_mlx.prefix_cache.BlockAwarePrefixCache._concat_cache_states |
method | vllm_mlx.prefix_cache.BlockAwarePrefixCache._concat_cache_states(states: List[Tuple[Any, ...] | List[Any]], seq_axis: int) -> Optional[Tuple[Any, ...] | List[Any]] |
Concatenate state fragments for a sequence-backed cache layer. | #L753-L768 |
vllm_mlx.prefix_cache.BlockAwarePrefixCache._extract_block_tensor_slice |
method | vllm_mlx.prefix_cache.BlockAwarePrefixCache._extract_block_tensor_slice(cache_data: List[Dict[str, Any]], start_idx: int, end_idx: int, total_tokens: int) -> Optional[List[Optional[Dict[str, Any]]]] |
Extract per-layer cache data for a single block. | #L630-L702 |
vllm_mlx.prefix_cache.BlockAwarePrefixCache._find_best_prefix_match |
method | vllm_mlx.prefix_cache.BlockAwarePrefixCache._find_best_prefix_match(tokens: List[int]) -> Optional[Tuple[List[int], List[int]]] |
Find best matching prefix in the index. | #L969-L992 |
vllm_mlx.prefix_cache.BlockAwarePrefixCache._slice_concat_cache_state |
method | vllm_mlx.prefix_cache.BlockAwarePrefixCache._slice_concat_cache_state(state: Tuple[Any, ...] | List[Any], start_idx: int, end_idx: int) -> Tuple[Any, ...] | List[Any] |
Slice a sequence-backed cache state across the token axis. | #L727-L751 |
vllm_mlx.prefix_cache.BlockAwarePrefixCache._slice_concat_cache_state._slice_tensor |
nested function | vllm_mlx.prefix_cache.BlockAwarePrefixCache._slice_concat_cache_state._slice_tensor(tensor: Any) -> Any |
Nested Function `BlockAwarePrefixCache._slice_concat_cache_state._slice_tensor` calls `slice`, `len`, `tuple`; returns `tensor[tuple(slices)]`. | #L745-L748 |
vllm_mlx.prefix_cache.BlockAwarePrefixCache._update_prefix_index |
method | vllm_mlx.prefix_cache.BlockAwarePrefixCache._update_prefix_index(tokens: List[int], block_ids: List[int]) -> None |
Update prefix index with new token sequence. | #L994-L1005 |
vllm_mlx.prefix_cache.BlockAwarePrefixCache.clear |
method | vllm_mlx.prefix_cache.BlockAwarePrefixCache.clear() -> None |
Clear all cached data. | #L1030-L1035 |
vllm_mlx.prefix_cache.BlockAwarePrefixCache.fetch_cache |
method | vllm_mlx.prefix_cache.BlockAwarePrefixCache.fetch_cache(request_id: str, tokens: List[int]) -> Tuple[Optional[BlockTable], List[int]] |
Find cached prefix blocks for the given tokens. | #L428-L502 |
vllm_mlx.prefix_cache.BlockAwarePrefixCache.fork_cache |
method | vllm_mlx.prefix_cache.BlockAwarePrefixCache.fork_cache(source_request_id: str, new_request_id: str) -> Optional[BlockTable] |
Fork cache from one request to another (COW). | #L813-L847 |
vllm_mlx.prefix_cache.BlockAwarePrefixCache.get_cache_for_generation |
method | vllm_mlx.prefix_cache.BlockAwarePrefixCache.get_cache_for_generation(request_id: str) -> Tuple[Optional[List[Any]], bool] |
Get cache data for generation, applying COW if needed. | #L770-L799 |
vllm_mlx.prefix_cache.BlockAwarePrefixCache.get_stats |
method | vllm_mlx.prefix_cache.BlockAwarePrefixCache.get_stats() -> Dict[str, Any] |
Get cache statistics. | #L1007-L1021 |
vllm_mlx.prefix_cache.BlockAwarePrefixCache.reconstruct_cache |
method | vllm_mlx.prefix_cache.BlockAwarePrefixCache.reconstruct_cache(block_table: BlockTable) -> Optional[List[Any]] |
Reconstruct cache objects from stored block tensor data. | #L849-L967 |
vllm_mlx.prefix_cache.BlockAwarePrefixCache.release_cache |
method | vllm_mlx.prefix_cache.BlockAwarePrefixCache.release_cache(request_id: str) -> None |
Release cache blocks for a completed request. | #L801-L811 |
vllm_mlx.prefix_cache.BlockAwarePrefixCache.reset_stats |
method | vllm_mlx.prefix_cache.BlockAwarePrefixCache.reset_stats() -> None |
Reset statistics. | #L1023-L1028 |
vllm_mlx.prefix_cache.BlockAwarePrefixCache.store_cache |
method | vllm_mlx.prefix_cache.BlockAwarePrefixCache.store_cache(request_id: str, tokens: List[int], cache_data: List[Any]) -> Optional[BlockTable] |
Store computed cache for future reuse. | #L504-L628 |
vllm_mlx.prefix_cache.BlockCacheEntry |
class | vllm_mlx.prefix_cache.BlockCacheEntry(block_table: BlockTable, cache_data: List[Any], last_access: float) |
Entry mapping a token sequence to cache blocks. | #L364-L369 |
vllm_mlx.prefix_cache.CacheEntry |
class | vllm_mlx.prefix_cache.CacheEntry(prompt_cache: List[Any], count: int) |
Entry in the prefix cache. | #L33-L37 |
vllm_mlx.prefix_cache.PrefixCacheManager |
class | vllm_mlx.prefix_cache.PrefixCacheManager(model: Any, max_entries: int = 100) |
Manages prefix caching for vllm-mlx using a trie-based LRU cache. | #L69-L355 |
vllm_mlx.prefix_cache.PrefixCacheManager.__init__ |
method | vllm_mlx.prefix_cache.PrefixCacheManager.__init__(model: Any, max_entries: int = 100) -> not annotated |
Initialize the prefix cache manager. | #L94-L115 |
vllm_mlx.prefix_cache.PrefixCacheManager.__len__ |
method | vllm_mlx.prefix_cache.PrefixCacheManager.__len__() -> int |
Return number of cached entries. | #L353-L355 |
vllm_mlx.prefix_cache.PrefixCacheManager._can_trim_cache |
method | vllm_mlx.prefix_cache.PrefixCacheManager._can_trim_cache(prompt_cache: List[Any]) -> bool |
Check if cache can be trimmed. | #L316-L330 |
vllm_mlx.prefix_cache.PrefixCacheManager._delete_cache |
method | vllm_mlx.prefix_cache.PrefixCacheManager._delete_cache(model_key: Any, tokens: List[int]) -> None |
Delete cache entry and clean up empty trie branches. | #L290-L314 |
vllm_mlx.prefix_cache.PrefixCacheManager._evict_lru |
method | vllm_mlx.prefix_cache.PrefixCacheManager._evict_lru() -> None |
Evict least recently used entry — O(1) popitem from OrderedDict. | #L281-L288 |
vllm_mlx.prefix_cache.PrefixCacheManager._get_cache_entry |
method | vllm_mlx.prefix_cache.PrefixCacheManager._get_cache_entry(tokens: List[int]) -> Optional[CacheEntry] |
Get cache entry for given tokens. | #L260-L271 |
vllm_mlx.prefix_cache.PrefixCacheManager._search |
method | vllm_mlx.prefix_cache.PrefixCacheManager._search(tokens: List[int]) -> Tuple[Optional[List[int]], Optional[List[int]], Optional[List[int]], int] |
Search for cached prefix matching tokens. | #L117-L164 |
vllm_mlx.prefix_cache.PrefixCacheManager._touch_lru |
method | vllm_mlx.prefix_cache.PrefixCacheManager._touch_lru(tokens_tuple: tuple) -> None |
Move entry to most-recently-used position — O(1) with OrderedDict. | #L273-L279 |
vllm_mlx.prefix_cache.PrefixCacheManager._trim_cache |
method | vllm_mlx.prefix_cache.PrefixCacheManager._trim_cache(prompt_cache: List[Any], num_tokens: int) -> List[Any] |
Trim cache by removing num_tokens from the end. | #L332-L337 |
vllm_mlx.prefix_cache.PrefixCacheManager.clear |
method | vllm_mlx.prefix_cache.PrefixCacheManager.clear() -> None |
Clear all cached entries. | #L347-L351 |
vllm_mlx.prefix_cache.PrefixCacheManager.fetch_cache |
method | vllm_mlx.prefix_cache.PrefixCacheManager.fetch_cache(tokens: List[int]) -> Tuple[Optional[List[Any]], List[int]] |
Find cached prefix for the given tokens. | #L166-L221 |
vllm_mlx.prefix_cache.PrefixCacheManager.get_stats |
method | vllm_mlx.prefix_cache.PrefixCacheManager.get_stats() -> Dict[str, Any] |
Get cache statistics. | #L339-L341 |
vllm_mlx.prefix_cache.PrefixCacheManager.reset_stats |
method | vllm_mlx.prefix_cache.PrefixCacheManager.reset_stats() -> None |
Reset statistics. | #L343-L345 |
vllm_mlx.prefix_cache.PrefixCacheManager.store_cache |
method | vllm_mlx.prefix_cache.PrefixCacheManager.store_cache(tokens: List[int], prompt_cache: List[Any]) -> None |
Store computed cache for future reuse. | #L223-L258 |
vllm_mlx.prefix_cache.PrefixCacheStats |
class | vllm_mlx.prefix_cache.PrefixCacheStats(hits: int = 0, misses: int = 0, tokens_saved: int = 0, total_queries: int = 0, evictions: int = 0) |
Statistics for prefix cache performance. | #L41-L66 |
vllm_mlx.prefix_cache.PrefixCacheStats.hit_rate |
method | vllm_mlx.prefix_cache.PrefixCacheStats.hit_rate() -> float |
Calculate cache hit rate. | #L51-L55 |
vllm_mlx.prefix_cache.PrefixCacheStats.to_dict |
method | vllm_mlx.prefix_cache.PrefixCacheStats.to_dict() -> Dict[str, Any] |
Convert stats to dictionary. | #L57-L66 |
vllm_mlx.prompt_warmup._build_strict_prefix_string |
function | vllm_mlx.prompt_warmup._build_strict_prefix_string(tokenizer: Any, messages: list[dict[str, Any]], enable_thinking: bool = True) -> str | None |
Build a STRING prefix that is a prefix of any real request's rendered chat template for the same system and empty chat history. | #L94-L176 |
vllm_mlx.prompt_warmup._build_strict_prefix_string._with_user |
nested function | vllm_mlx.prompt_warmup._build_strict_prefix_string._with_user(user_content: str) -> list[dict[str, Any]] |
Nested Function `_build_strict_prefix_string._with_user` calls `dict`, `msgs[-1].get`; returns `msgs`. | #L121-L127 |
vllm_mlx.prompt_warmup._ensure_user_terminator |
function | vllm_mlx.prompt_warmup._ensure_user_terminator(messages: list[dict[str, Any]]) -> list[dict[str, Any]] |
Ensure the message list ends with a user message. | #L79-L91 |
vllm_mlx.prompt_warmup.load_warmup_file |
function | vllm_mlx.prompt_warmup.load_warmup_file(path: str) -> list[list[dict[str, Any]]] |
Load and validate a warm-up prompts JSON file. | #L41-L76 |
vllm_mlx.prompt_warmup.warm_prefix_cache |
function | async vllm_mlx.prompt_warmup.warm_prefix_cache(engine: Any, prompts: list[list[dict[str, Any]]], *, max_tokens: int = 1) -> dict[str, Any] |
Run each prompt through the engine to populate the prefix cache. | #L179-L275 |
vllm_mlx.prompt_warmup.warm_prefix_cache._one_chat |
nested function | async vllm_mlx.prompt_warmup.warm_prefix_cache._one_chat(idx: int, messages: list[dict[str, Any]]) -> tuple[int, int, str | None] |
Nested Function `warm_prefix_cache._one_chat` calls `_ensure_user_terminator`, `engine.stream_chat`, `int`, `type`; has 3 explicit return paths. | #L237-L253 |
vllm_mlx.prompt_warmup.warm_prefix_cache._one_strict |
nested function | async vllm_mlx.prompt_warmup.warm_prefix_cache._one_strict(idx: int, messages: list[dict[str, Any]]) -> tuple[int, int, str | None] |
Nested Function `warm_prefix_cache._one_strict` calls `_build_strict_prefix_string`, `_one_chat`, `engine.stream_generate`, `int`; awaits asynchronous work; has 3 explicit return paths. | #L217-L235 |
vllm_mlx.reasoning._register_builtin_parsers |
function | vllm_mlx.reasoning._register_builtin_parsers() -> not annotated |
Register built-in parsers. | #L76-L94 |
vllm_mlx.reasoning.base.DeltaMessage |
class | vllm_mlx.reasoning.base.DeltaMessage(role: str | None = None, content: str | None = None, reasoning: str | None = None) |
Delta message for streaming reasoning output. | #L15-L33 |
vllm_mlx.reasoning.base.DeltaMessage.reasoning_content |
method | vllm_mlx.reasoning.base.DeltaMessage.reasoning_content() -> str | None |
Deprecated: use reasoning instead. | #L31-L33 |
vllm_mlx.reasoning.base.ReasoningParser |
class | vllm_mlx.reasoning.base.ReasoningParser(tokenizer: Any | None = None) |
Abstract base class for reasoning content extraction. | #L36-L126 |
vllm_mlx.reasoning.base.ReasoningParser.__init__ |
method | vllm_mlx.reasoning.base.ReasoningParser.__init__(tokenizer: Any | None = None) -> not annotated |
Initialize parser with optional tokenizer. | #L49-L57 |
vllm_mlx.reasoning.base.ReasoningParser.extract_reasoning |
method | vllm_mlx.reasoning.base.ReasoningParser.extract_reasoning(model_output: str) -> tuple[str | None, str | None] |
Extract reasoning content from complete model output. | #L60-L74 |
vllm_mlx.reasoning.base.ReasoningParser.extract_reasoning_streaming |
method | vllm_mlx.reasoning.base.ReasoningParser.extract_reasoning_streaming(previous_text: str, current_text: str, delta_text: str) -> DeltaMessage | None |
Extract reasoning from streaming delta. | #L77-L100 |
vllm_mlx.reasoning.base.ReasoningParser.finalize_stream |
method | vllm_mlx.reasoning.base.ReasoningParser.finalize_stream() -> DeltaMessage | None |
Finalize streaming state at end of stream. | #L112-L126 |
vllm_mlx.reasoning.base.ReasoningParser.reset_state |
method | vllm_mlx.reasoning.base.ReasoningParser.reset_state() -> not annotated |
Reset any internal state for a new request. | #L102-L110 |
vllm_mlx.reasoning.deepseek_r1_parser.DeepSeekR1ReasoningParser |
class | vllm_mlx.reasoning.deepseek_r1_parser.DeepSeekR1ReasoningParser() |
Reasoning parser for DeepSeek-R1 model. | #L14-L114 |
vllm_mlx.reasoning.deepseek_r1_parser.DeepSeekR1ReasoningParser.end_token |
method | vllm_mlx.reasoning.deepseek_r1_parser.DeepSeekR1ReasoningParser.end_token() -> str |
Return the marker that closes a DeepSeek reasoning span. | #L38-L41 |
vllm_mlx.reasoning.deepseek_r1_parser.DeepSeekR1ReasoningParser.extract_reasoning |
method | vllm_mlx.reasoning.deepseek_r1_parser.DeepSeekR1ReasoningParser.extract_reasoning(model_output: str) -> tuple[str | None, str | None] |
Extract reasoning from DeepSeek-R1 output. | #L43-L67 |
vllm_mlx.reasoning.deepseek_r1_parser.DeepSeekR1ReasoningParser.extract_reasoning_streaming |
method | vllm_mlx.reasoning.deepseek_r1_parser.DeepSeekR1ReasoningParser.extract_reasoning_streaming(previous_text: str, current_text: str, delta_text: str) -> DeltaMessage | None |
Extract reasoning from streaming delta. | #L69-L114 |
vllm_mlx.reasoning.deepseek_r1_parser.DeepSeekR1ReasoningParser.start_token |
method | vllm_mlx.reasoning.deepseek_r1_parser.DeepSeekR1ReasoningParser.start_token() -> str |
Return the marker that opens an explicit DeepSeek reasoning span. | #L32-L35 |
vllm_mlx.reasoning.gemma4_parser._strip_channel_name |
function | vllm_mlx.reasoning.gemma4_parser._strip_channel_name(text: str, prefix: str) -> str |
Strip channel name and leading whitespace/newline from text start. | #L46-L50 |
vllm_mlx.reasoning.gemma4_parser._strip_channel_tokens |
function | vllm_mlx.reasoning.gemma4_parser._strip_channel_tokens(text: str) -> str |
Remove all channel special tokens and bare channel names from text. | #L53-L82 |
vllm_mlx.reasoning.gemma4_parser.Gemma4ReasoningParser |
class | vllm_mlx.reasoning.gemma4_parser.Gemma4ReasoningParser(tokenizer = None) |
Reasoning parser for Gemma 4 models. | #L85-L386 |
vllm_mlx.reasoning.gemma4_parser.Gemma4ReasoningParser.__init__ |
method | vllm_mlx.reasoning.gemma4_parser.Gemma4ReasoningParser.__init__(tokenizer = None) -> not annotated |
Method `Gemma4ReasoningParser.__init__` updates `self._pending`, `self._content_seen`; calls `super().__init__`, `super`. | #L126-L133 |
vllm_mlx.reasoning.gemma4_parser.Gemma4ReasoningParser._extract_from_safe_text |
method | vllm_mlx.reasoning.gemma4_parser.Gemma4ReasoningParser._extract_from_safe_text(previous_text: str, current_text: str, delta_text: str) -> DeltaMessage | None |
Parse safe (non-buffered) text. | #L293-L386 |
vllm_mlx.reasoning.gemma4_parser.Gemma4ReasoningParser._strip_channel_tokens_from_delta |
method | vllm_mlx.reasoning.gemma4_parser.Gemma4ReasoningParser._strip_channel_tokens_from_delta(msg: DeltaMessage | None) -> DeltaMessage | None |
Strip channel special tokens from content and reasoning in a delta. | #L275-L291 |
vllm_mlx.reasoning.gemma4_parser.Gemma4ReasoningParser._trailing_partial_marker_len |
method | vllm_mlx.reasoning.gemma4_parser.Gemma4ReasoningParser._trailing_partial_marker_len(text: str) -> int |
Return length of trailing substring of `text` that is a proper prefix of any transition marker (<channel|>, <|channel>response, <|channel>). | #L142-L166 |
vllm_mlx.reasoning.gemma4_parser.Gemma4ReasoningParser.end_token |
method | vllm_mlx.reasoning.gemma4_parser.Gemma4ReasoningParser.end_token() -> str |
Return Gemma's marker for entering the response channel. | #L121-L124 |
vllm_mlx.reasoning.gemma4_parser.Gemma4ReasoningParser.extract_reasoning |
method | vllm_mlx.reasoning.gemma4_parser.Gemma4ReasoningParser.extract_reasoning(model_output: str) -> tuple[str | None, str | None] |
Extract reasoning from complete output. | #L185-L233 |
vllm_mlx.reasoning.gemma4_parser.Gemma4ReasoningParser.extract_reasoning_streaming |
method | vllm_mlx.reasoning.gemma4_parser.Gemma4ReasoningParser.extract_reasoning_streaming(previous_text: str, current_text: str, delta_text: str) -> DeltaMessage | None |
Extract reasoning from streaming delta. | #L235-L272 |
vllm_mlx.reasoning.gemma4_parser.Gemma4ReasoningParser.finalize_stream |
method | vllm_mlx.reasoning.gemma4_parser.Gemma4ReasoningParser.finalize_stream() -> DeltaMessage | None |
Flush any buffered partial marker at the end of stream. | #L168-L183 |
vllm_mlx.reasoning.gemma4_parser.Gemma4ReasoningParser.reset_state |
method | vllm_mlx.reasoning.gemma4_parser.Gemma4ReasoningParser.reset_state() -> not annotated |
Reset base parsing state and buffered Gemma channel markers. | #L135-L140 |
vllm_mlx.reasoning.gemma4_parser.Gemma4ReasoningParser.start_token |
method | vllm_mlx.reasoning.gemma4_parser.Gemma4ReasoningParser.start_token() -> str |
Return Gemma's marker for entering the thought channel. | #L115-L118 |
vllm_mlx.reasoning.get_parser |
function | vllm_mlx.reasoning.get_parser(name: str) -> type[ReasoningParser] |
Get a reasoning parser class by name. | #L45-L63 |
vllm_mlx.reasoning.glm4_parser.Glm4ReasoningParser |
class | vllm_mlx.reasoning.glm4_parser.Glm4ReasoningParser() |
Reasoning parser for GLM-4 models. | #L27-L113 |
vllm_mlx.reasoning.glm4_parser.Glm4ReasoningParser.end_token |
method | vllm_mlx.reasoning.glm4_parser.Glm4ReasoningParser.end_token() -> str |
Return the marker that closes a GLM reasoning span. | #L56-L59 |
vllm_mlx.reasoning.glm4_parser.Glm4ReasoningParser.extract_reasoning |
method | vllm_mlx.reasoning.glm4_parser.Glm4ReasoningParser.extract_reasoning(model_output: str) -> tuple[str | None, str | None] |
Strip GLM box markers and split complete reasoning from content. | #L61-L68 |
vllm_mlx.reasoning.glm4_parser.Glm4ReasoningParser.extract_reasoning_streaming |
method | vllm_mlx.reasoning.glm4_parser.Glm4ReasoningParser.extract_reasoning_streaming(previous_text: str, current_text: str, delta_text: str) -> DeltaMessage | None |
Extract reasoning from streaming delta. | #L70-L113 |
vllm_mlx.reasoning.glm4_parser.Glm4ReasoningParser.start_token |
method | vllm_mlx.reasoning.glm4_parser.Glm4ReasoningParser.start_token() -> str |
Return the marker that opens a GLM reasoning span. | #L50-L53 |
vllm_mlx.reasoning.gpt_oss_parser._extract_channel |
function | vllm_mlx.reasoning.gpt_oss_parser._extract_channel(text: str, channel_name: str) -> str | None |
Extract content from a named channel. | #L33-L55 |
vllm_mlx.reasoning.gpt_oss_parser.GptOssReasoningParser |
class | vllm_mlx.reasoning.gpt_oss_parser.GptOssReasoningParser() |
Reasoning parser for GPT-OSS models. | #L58-L214 |
vllm_mlx.reasoning.gpt_oss_parser.GptOssReasoningParser._detect_phase |
method | vllm_mlx.reasoning.gpt_oss_parser.GptOssReasoningParser._detect_phase(text: str) -> str |
Detect current streaming phase from accumulated text. | #L164-L187 |
vllm_mlx.reasoning.gpt_oss_parser.GptOssReasoningParser._extract_content_after_marker_in_delta |
method | vllm_mlx.reasoning.gpt_oss_parser.GptOssReasoningParser._extract_content_after_marker_in_delta(current_text: str, phase: str) -> str | None |
When phase changes, extract only the content after the phase marker that falls within the current accumulated text's tail. | #L190-L209 |
vllm_mlx.reasoning.gpt_oss_parser.GptOssReasoningParser._strip_return |
method | vllm_mlx.reasoning.gpt_oss_parser.GptOssReasoningParser._strip_return(text: str) -> str |
Strip <|return|> from text. | #L212-L214 |
vllm_mlx.reasoning.gpt_oss_parser.GptOssReasoningParser.extract_reasoning |
method | vllm_mlx.reasoning.gpt_oss_parser.GptOssReasoningParser.extract_reasoning(model_output: str) -> tuple[str | None, str | None] |
Extract reasoning and content from complete model output. | #L72-L106 |
vllm_mlx.reasoning.gpt_oss_parser.GptOssReasoningParser.extract_reasoning_streaming |
method | vllm_mlx.reasoning.gpt_oss_parser.GptOssReasoningParser.extract_reasoning_streaming(previous_text: str, current_text: str, delta_text: str) -> DeltaMessage | None |
Extract reasoning from streaming delta. | #L108-L161 |
vllm_mlx.reasoning.harmony_parser.HarmonyReasoningParser |
class | vllm_mlx.reasoning.harmony_parser.HarmonyReasoningParser(tokenizer = None) |
Reasoning parser for GPT-OSS models using Harmony format. | #L35-L157 |
vllm_mlx.reasoning.harmony_parser.HarmonyReasoningParser.__init__ |
method | vllm_mlx.reasoning.harmony_parser.HarmonyReasoningParser.__init__(tokenizer = None) -> not annotated |
Method `HarmonyReasoningParser.__init__` updates `self._current_channel`, `self._in_message`; calls `super().__init__`, `super`. | #L49-L52 |
vllm_mlx.reasoning.harmony_parser.HarmonyReasoningParser.extract_reasoning |
method | vllm_mlx.reasoning.harmony_parser.HarmonyReasoningParser.extract_reasoning(model_output: str) -> tuple[str | None, str | None] |
Extract reasoning from complete Harmony output. | #L54-L78 |
vllm_mlx.reasoning.harmony_parser.HarmonyReasoningParser.extract_reasoning_streaming |
method | vllm_mlx.reasoning.harmony_parser.HarmonyReasoningParser.extract_reasoning_streaming(previous_text: str, current_text: str, delta_text: str) -> DeltaMessage | None |
Extract reasoning from streaming Harmony output. | #L80-L152 |
vllm_mlx.reasoning.harmony_parser.HarmonyReasoningParser.reset_state |
method | vllm_mlx.reasoning.harmony_parser.HarmonyReasoningParser.reset_state() -> not annotated |
Reset streaming state for a new request. | #L154-L157 |
vllm_mlx.reasoning.list_parsers |
function | vllm_mlx.reasoning.list_parsers() -> list[str] |
List available parser names. | #L66-L73 |
vllm_mlx.reasoning.mistral_parser.MistralReasoningParser |
class | vllm_mlx.reasoning.mistral_parser.MistralReasoningParser() |
Reasoning parser for Mistral/Ministral reasoning models. | #L19-L72 |
vllm_mlx.reasoning.mistral_parser.MistralReasoningParser.end_token |
method | vllm_mlx.reasoning.mistral_parser.MistralReasoningParser.end_token() -> str |
Return the Mistral reasoning closing marker. | #L46-L49 |
vllm_mlx.reasoning.mistral_parser.MistralReasoningParser.extract_reasoning |
method | vllm_mlx.reasoning.mistral_parser.MistralReasoningParser.extract_reasoning(model_output: str) -> tuple[str | None, str | None] |
Extract reasoning from Mistral/Ministral output. | #L51-L72 |
vllm_mlx.reasoning.mistral_parser.MistralReasoningParser.start_token |
method | vllm_mlx.reasoning.mistral_parser.MistralReasoningParser.start_token() -> str |
Return the Mistral reasoning opening marker. | #L40-L43 |
vllm_mlx.reasoning.poolside_v1_parser.PoolsideV1ReasoningParser |
class | vllm_mlx.reasoning.poolside_v1_parser.PoolsideV1ReasoningParser() |
Parse Laguna's template-injected ``<think>`` reasoning boundary. | #L7-L13 |
vllm_mlx.reasoning.qwen3_parser.Qwen3ReasoningParser |
class | vllm_mlx.reasoning.qwen3_parser.Qwen3ReasoningParser() |
Reasoning parser for Qwen3 models. | #L15-L68 |
vllm_mlx.reasoning.qwen3_parser.Qwen3ReasoningParser.end_token |
method | vllm_mlx.reasoning.qwen3_parser.Qwen3ReasoningParser.end_token() -> str |
Return the Qwen3 reasoning closing marker. | #L42-L45 |
vllm_mlx.reasoning.qwen3_parser.Qwen3ReasoningParser.extract_reasoning |
method | vllm_mlx.reasoning.qwen3_parser.Qwen3ReasoningParser.extract_reasoning(model_output: str) -> tuple[str | None, str | None] |
Extract reasoning from Qwen3 output. | #L47-L68 |
vllm_mlx.reasoning.qwen3_parser.Qwen3ReasoningParser.start_token |
method | vllm_mlx.reasoning.qwen3_parser.Qwen3ReasoningParser.start_token() -> str |
Return the Qwen3 reasoning opening marker. | #L36-L39 |
vllm_mlx.reasoning.register_parser |
function | vllm_mlx.reasoning.register_parser(name: str, parser_class: type[ReasoningParser]) -> None |
Register a reasoning parser. | #L34-L42 |
vllm_mlx.reasoning.think_parser.BaseThinkingReasoningParser |
class | vllm_mlx.reasoning.think_parser.BaseThinkingReasoningParser(tokenizer = None) |
Base parser for models using <think>...</think> style tags. | #L29-L462 |
vllm_mlx.reasoning.think_parser.BaseThinkingReasoningParser.__init__ |
method | vllm_mlx.reasoning.think_parser.BaseThinkingReasoningParser.__init__(tokenizer = None) -> not annotated |
Method `BaseThinkingReasoningParser.__init__` updates `self._phase`, `self._content_started`, `self._content_buffer`, `self._in_tool_call`; calls `super().__init__`, `super`. | #L63-L71 |
vllm_mlx.reasoning.think_parser.BaseThinkingReasoningParser._content_delta |
method | vllm_mlx.reasoning.think_parser.BaseThinkingReasoningParser._content_delta(delta_text: str) -> DeltaMessage | None |
Emit content after consuming repeated leading think blocks. | #L278-L325 |
vllm_mlx.reasoning.think_parser.BaseThinkingReasoningParser._extract_complete_reasoning |
method | vllm_mlx.reasoning.think_parser.BaseThinkingReasoningParser._extract_complete_reasoning(text: str) -> tuple[str | None, str | None] |
Split complete output into leading reasoning spans and final content. | #L226-L260 |
vllm_mlx.reasoning.think_parser.BaseThinkingReasoningParser._promote_tool_calls |
method | vllm_mlx.reasoning.think_parser.BaseThinkingReasoningParser._promote_tool_calls(reasoning: str | None, content: str | None) -> tuple[str | None, str | None] |
Method `BaseThinkingReasoningParser._promote_tool_calls` calls `cls._TOOL_CALL_CLOSED_RE.sub`, `cls._TOOL_CALL_UNCLOSED_RE.search`, `unclosed_match.group`, `unclosed_match.start`; has 2 explicit return paths. | #L409-L462 |
vllm_mlx.reasoning.think_parser.BaseThinkingReasoningParser._promote_tool_calls._collect_closed |
nested function | vllm_mlx.reasoning.think_parser.BaseThinkingReasoningParser._promote_tool_calls._collect_closed(match) -> not annotated |
Nested Function `BaseThinkingReasoningParser._promote_tool_calls._collect_closed` calls `closed.append`, `match.group`; returns `''`. | #L419-L421 |
vllm_mlx.reasoning.think_parser.BaseThinkingReasoningParser._thinking_tool_call |
method | vllm_mlx.reasoning.think_parser.BaseThinkingReasoningParser._thinking_tool_call(previous_text: str, current_text: str, delta_text: str) -> DeltaMessage | None |
Handle streaming while inside a <tool_call> during thinking phase. | #L327-L396 |
vllm_mlx.reasoning.think_parser.BaseThinkingReasoningParser._transition_to_content |
method | vllm_mlx.reasoning.think_parser.BaseThinkingReasoningParser._transition_to_content(reasoning: str | None, content: str | None) -> DeltaMessage | None |
Return a delta while suppressing leading post-transition think blocks. | #L262-L276 |
vllm_mlx.reasoning.think_parser.BaseThinkingReasoningParser.end_token |
method | vllm_mlx.reasoning.think_parser.BaseThinkingReasoningParser.end_token() -> str |
The token/tag that ends reasoning content (e.g., '</think>'). | #L55-L56 |
vllm_mlx.reasoning.think_parser.BaseThinkingReasoningParser.extract_reasoning |
method | vllm_mlx.reasoning.think_parser.BaseThinkingReasoningParser.extract_reasoning(model_output: str) -> tuple[str | None, str | None] |
Extract reasoning from complete output. | #L81-L110 |
vllm_mlx.reasoning.think_parser.BaseThinkingReasoningParser.extract_reasoning_streaming |
method | vllm_mlx.reasoning.think_parser.BaseThinkingReasoningParser.extract_reasoning_streaming(previous_text: str, current_text: str, delta_text: str) -> DeltaMessage | None |
Extract reasoning from a streaming delta using state-machine tracking. | #L112-L224 |
vllm_mlx.reasoning.think_parser.BaseThinkingReasoningParser.finalize_stream |
method | vllm_mlx.reasoning.think_parser.BaseThinkingReasoningParser.finalize_stream() -> DeltaMessage | None |
Flush any buffered tool call text at end of stream. | #L398-L406 |
vllm_mlx.reasoning.think_parser.BaseThinkingReasoningParser.reset_state |
method | vllm_mlx.reasoning.think_parser.BaseThinkingReasoningParser.reset_state() -> not annotated |
Reset state machine for a new streaming request. | #L73-L79 |
vllm_mlx.reasoning.think_parser.BaseThinkingReasoningParser.start_token |
method | vllm_mlx.reasoning.think_parser.BaseThinkingReasoningParser.start_token() -> str |
The token/tag that starts reasoning content (e.g., '<think>'). | #L50-L51 |
vllm_mlx.request.Request |
class | vllm_mlx.request.Request(request_id: str, prompt: Union[str, List[int]], sampling_params: SamplingParams, arrival_time: float = field(default_factory=time.time), priority: int = 0, prompt_token_ids: Optional[List[int]] = None, num_prompt_tokens: int = 0, status: RequestStatus = RequestStatus.WAITING, num_computed_tokens: int = 0, output_token_ids: List[int] = field(default_factory=list), output_text: str = '', batch_uid: Optional[int] = None, prompt_cache: Optional[List[Any]] = None, cached_tokens: int = 0, remaining_tokens: Optional[List[int]] = None, prefix_boundary: int = 0, block_table: Optional['BlockTable'] = None, shared_prefix_blocks: int = 0, images: Optional[List[Any]] = None, videos: Optional[List[Any]] = None, pixel_values: Optional[Any] = None, image_grid_thw: Optional[Any] = None, attention_mask: Optional[Any] = None, multimodal_kwargs: Optional[Dict[str, Any]] = None, is_multimodal: bool = False, finish_reason: Optional[str] = None, first_token_time: Optional[float] = None, cache_hit_type: Optional[str] = None) |
Represents a single inference request in the scheduling system. | #L77-L192 |
vllm_mlx.request.Request.__eq__ |
method | vllm_mlx.request.Request.__eq__(other: object) -> bool |
Method `Request.__eq__` calls `isinstance`; has 2 explicit return paths. | #L189-L192 |
vllm_mlx.request.Request.__hash__ |
method | vllm_mlx.request.Request.__hash__() -> int |
Method `Request.__hash__` calls `hash`; returns `hash(self.request_id)`. | #L186-L187 |
vllm_mlx.request.Request.__lt__ |
method | vllm_mlx.request.Request.__lt__(other: 'Request') -> bool |
Compare requests for priority queue ordering. | #L180-L184 |
vllm_mlx.request.Request.append_output_token |
method | vllm_mlx.request.Request.append_output_token(token_id: int) -> None |
Append a generated token to the output. | #L170-L173 |
vllm_mlx.request.Request.get_finish_reason |
method | vllm_mlx.request.Request.get_finish_reason() -> Optional[str] |
Get the finish reason if finished. | #L164-L168 |
vllm_mlx.request.Request.is_finished |
method | vllm_mlx.request.Request.is_finished() -> bool |
Check if request has finished. | #L160-L162 |
vllm_mlx.request.Request.max_tokens |
method | vllm_mlx.request.Request.max_tokens() -> int |
Maximum output tokens for this request. | #L156-L158 |
vllm_mlx.request.Request.num_output_tokens |
method | vllm_mlx.request.Request.num_output_tokens() -> int |
Number of output tokens generated so far. | #L146-L148 |
vllm_mlx.request.Request.num_tokens |
method | vllm_mlx.request.Request.num_tokens() -> int |
Total number of tokens (prompt + output). | #L151-L153 |
vllm_mlx.request.Request.set_finished |
method | vllm_mlx.request.Request.set_finished(status: RequestStatus, reason: Optional[str] = None) -> None |
Mark the request as finished. | #L175-L178 |
vllm_mlx.request.RequestOutput |
class | vllm_mlx.request.RequestOutput(request_id: str, new_token_ids: List[int] = field(default_factory=list), new_text: str = '', output_token_ids: List[int] = field(default_factory=list), output_text: str = '', finished: bool = False, finish_reason: Optional[str] = None, prompt_tokens: int = 0, completion_tokens: int = 0, mtp_drafts: int = 0, mtp_accepted: int = 0) |
Output for a single request after a generation step. | #L196-L227 |
vllm_mlx.request.RequestOutput.usage |
method | vllm_mlx.request.RequestOutput.usage() -> Dict[str, int] |
Return usage statistics compatible with OpenAI API. | #L221-L227 |
vllm_mlx.request.RequestStatus |
class | vllm_mlx.request.RequestStatus() |
Status of a request in the scheduling system. | #L18-L48 |
vllm_mlx.request.RequestStatus.get_finish_reason |
method | vllm_mlx.request.RequestStatus.get_finish_reason(status: 'RequestStatus') -> Optional[str] |
Get the finish reason string for a finished status. | #L40-L48 |
vllm_mlx.request.RequestStatus.is_finished |
method | vllm_mlx.request.RequestStatus.is_finished(status: 'RequestStatus') -> bool |
Check if the status indicates a finished request. | #L35-L37 |
vllm_mlx.request.SamplingParams |
class | vllm_mlx.request.SamplingParams(max_tokens: int = 256, temperature: float = 0.7, top_p: float = 0.9, top_k: int = 0, min_p: float = 0.0, presence_penalty: float = 0.0, repetition_penalty: float = 1.0, stop: Optional[List[str]] = None, stop_token_ids: Optional[List[int]] = None, logits_processors: Optional[List[Callable]] = None) |
Sampling parameters for text generation. | #L52-L73 |
vllm_mlx.request.SamplingParams.__post_init__ |
method | vllm_mlx.request.SamplingParams.__post_init__() -> not annotated |
Method `SamplingParams.__post_init__` updates `self.stop`, `self.stop_token_ids`. | #L69-L73 |
vllm_mlx.rerank._build_classifier_model |
function | vllm_mlx.rerank._build_classifier_model(model_type, config, weights, num_labels) -> not annotated |
Build an MLX sequence classification model from config and weights. | #L341-L349 |
vllm_mlx.rerank._ClassifierOutput |
class | vllm_mlx.rerank._ClassifierOutput(logits: mx.array) |
Simple container for classifier output logits. | #L394-L398 |
vllm_mlx.rerank._ClassifierOutput.__init__ |
method | vllm_mlx.rerank._ClassifierOutput.__init__(logits: mx.array) -> not annotated |
Method `_ClassifierOutput.__init__` updates `self.logits`. | #L397-L398 |
vllm_mlx.rerank._MLXClassifierWrapper |
class | vllm_mlx.rerank._MLXClassifierWrapper(config: dict, weights: dict, num_labels: int) |
Minimal MLX wrapper for sequence classification models. | #L352-L391 |
vllm_mlx.rerank._MLXClassifierWrapper.__call__ |
method | vllm_mlx.rerank._MLXClassifierWrapper.__call__(input_ids: mx.array, attention_mask: mx.array = None) -> not annotated |
Forward pass through the classifier. | #L371-L391 |
vllm_mlx.rerank._MLXClassifierWrapper.__init__ |
method | vllm_mlx.rerank._MLXClassifierWrapper.__init__(config: dict, weights: dict, num_labels: int) -> not annotated |
Method `_MLXClassifierWrapper.__init__` updates `self.config`, `self.weights`, `self.num_labels`, `self._params`; calls `list`, `weights.values`. | #L361-L365 |
vllm_mlx.rerank._MLXClassifierWrapper.parameters |
method | vllm_mlx.rerank._MLXClassifierWrapper.parameters() -> not annotated |
Return model parameters for mx.eval. | #L367-L369 |
vllm_mlx.rerank.get_adapter |
function | vllm_mlx.rerank.get_adapter(model_name: str) -> RerankAdapter |
Return the appropriate adapter for a model. | #L124-L133 |
vllm_mlx.rerank.RerankAdapter |
class | vllm_mlx.rerank.RerankAdapter() |
Per-family adapter for reranker models. | #L29-L80 |
vllm_mlx.rerank.RerankAdapter.extract_score |
method | vllm_mlx.rerank.RerankAdapter.extract_score(logits) -> float |
Extract a raw relevance score from model output logits. | #L57-L67 |
vllm_mlx.rerank.RerankAdapter.normalize |
method | vllm_mlx.rerank.RerankAdapter.normalize(raw_score: float) -> float |
Normalize a raw score to [0, 1] range. | #L70-L80 |
vllm_mlx.rerank.RerankAdapter.tokenize_pair |
method | vllm_mlx.rerank.RerankAdapter.tokenize_pair(tokenizer, query: str, document: str, max_length: int) -> dict |
Tokenize a (query, document) pair for the cross-encoder. | #L39-L54 |
vllm_mlx.rerank.RerankEngine |
class | vllm_mlx.rerank.RerankEngine(model_name: str, token_budget: int = 4096, max_concurrency: int = 1) |
Reranker engine for cross-encoder sequence classification models. | #L136-L338 |
vllm_mlx.rerank.RerankEngine.__init__ |
method | vllm_mlx.rerank.RerankEngine.__init__(model_name: str, token_budget: int = 4096, max_concurrency: int = 1) -> not annotated |
Method `RerankEngine.__init__` updates `self.model_name`, `self.token_budget`, `self.max_concurrency`, `self._semaphore`; calls `asyncio.Semaphore`. | #L147-L159 |
vllm_mlx.rerank.RerankEngine._ensure_loaded |
method | vllm_mlx.rerank.RerankEngine._ensure_loaded() -> None |
Method `RerankEngine._ensure_loaded` calls `self.load`. | #L232-L234 |
vllm_mlx.rerank.RerankEngine._load_mlx_model |
method | vllm_mlx.rerank.RerankEngine._load_mlx_model(model_name: str) -> not annotated |
Load an MLX cross-encoder model from HuggingFace Hub. | #L187-L230 |
vllm_mlx.rerank.RerankEngine.is_loaded |
method | vllm_mlx.rerank.RerankEngine.is_loaded() -> bool |
Return whether the reranking model has been loaded. | #L162-L165 |
vllm_mlx.rerank.RerankEngine.load |
method | vllm_mlx.rerank.RerankEngine.load() -> None |
Load the cross-encoder model and tokenizer. | #L167-L184 |
vllm_mlx.rerank.RerankEngine.score_pairs |
method | vllm_mlx.rerank.RerankEngine.score_pairs(query: str, documents: list[str]) -> tuple[list[float], int] |
Score each (query, document) pair and return normalized relevance scores. | #L236-L338 |
vllm_mlx.rerank.SigmoidAdapter |
class | vllm_mlx.rerank.SigmoidAdapter() |
Default adapter for single-logit sigmoid rerankers. | #L83-L111 |
vllm_mlx.rerank.SigmoidAdapter.extract_score |
method | vllm_mlx.rerank.SigmoidAdapter.extract_score(logits) -> float |
Extract the first logit as the relevance score. | #L105-L107 |
vllm_mlx.rerank.SigmoidAdapter.normalize |
method | vllm_mlx.rerank.SigmoidAdapter.normalize(raw_score: float) -> float |
Apply sigmoid normalization. | #L109-L111 |
vllm_mlx.rerank.SigmoidAdapter.tokenize_pair |
method | vllm_mlx.rerank.SigmoidAdapter.tokenize_pair(tokenizer, query: str, document: str, max_length: int) -> dict |
Tokenize as a sentence pair (query, document). | #L92-L103 |
vllm_mlx.rerank_forward._apply_hidden_activation |
function | vllm_mlx.rerank_forward._apply_hidden_activation(x: mx.array, config: dict) -> mx.array |
Apply the configured encoder hidden activation. | #L240-L265 |
vllm_mlx.rerank_forward._classification_head_forward |
function | vllm_mlx.rerank_forward._classification_head_forward(pooled: mx.array, weights: dict[str, mx.array]) -> mx.array |
Run BERT flat or XLM-RoBERTa two-layer sequence-classification head. | #L108-L122 |
vllm_mlx.rerank_forward._detect_prefix |
function | vllm_mlx.rerank_forward._detect_prefix(weights: dict) -> str |
Detect the model weight prefix (bert, roberta, xlm-roberta). | #L125-L135 |
vllm_mlx.rerank_forward._encoder_layer |
function | vllm_mlx.rerank_forward._encoder_layer(hidden: mx.array, ext_mask: mx.array | None, weights: dict, prefix: str, num_heads: int, head_dim: int, eps: float, config: dict) -> mx.array |
Run one BERT encoder layer (self-attention + FFN). | #L145-L217 |
vllm_mlx.rerank_forward._gelu |
function | vllm_mlx.rerank_forward._gelu(x: mx.array) -> mx.array |
GELU activation (exact form). | #L220-L222 |
vllm_mlx.rerank_forward._gelu_new |
function | vllm_mlx.rerank_forward._gelu_new(x: mx.array) -> mx.array |
BERT GELU approximation used by transformers gelu_new. | #L225-L227 |
vllm_mlx.rerank_forward._layer_norm |
function | vllm_mlx.rerank_forward._layer_norm(x: mx.array, weight: mx.array, bias: mx.array, eps: float) -> mx.array |
Apply layer normalization. | #L138-L142 |
vllm_mlx.rerank_forward._position_ids_for_config |
function | vllm_mlx.rerank_forward._position_ids_for_config(config: dict, input_ids: mx.array, attention_mask: mx.array | None) -> mx.array |
Build BERT or RoBERTa-family absolute position IDs. | #L88-L105 |
vllm_mlx.rerank_forward._relu |
function | vllm_mlx.rerank_forward._relu(x: mx.array) -> mx.array |
ReLU activation. | #L230-L232 |
vllm_mlx.rerank_forward._silu |
function | vllm_mlx.rerank_forward._silu(x: mx.array) -> mx.array |
SiLU/swish activation. | #L235-L237 |
vllm_mlx.rerank_forward.classifier_forward |
function | vllm_mlx.rerank_forward.classifier_forward(input_ids: mx.array, attention_mask: mx.array, weights: dict[str, mx.array], config: dict) -> mx.array |
Run a BERT-family classifier forward pass on MLX. | #L16-L85 |
vllm_mlx.scheduler._configure_chunked_prefill |
function | vllm_mlx.scheduler._configure_chunked_prefill(scheduler: 'Scheduler', batch_gen: 'BatchGenerator', budget: int, prompt_cache_save) -> None |
Enable the matching legacy or native mlx-lm chunked-prefill API. | #L722-L777 |
vllm_mlx.scheduler._install_chunked_prefill |
function | vllm_mlx.scheduler._install_chunked_prefill(batch_gen: 'BatchGenerator', budget: int, mid_prefill_save = None, prompt_cache_save = None, pending_abort_ids: Optional[Set[str]] = None, uid_to_request_id: Optional[Dict[int, str]] = None, requests: Optional[Dict[str, Any]] = None) -> None |
Monkey-patch a BatchGenerator instance so that large prefills are broken into chunks of at most *budget* tokens each. | #L190-L697 |
vllm_mlx.scheduler._install_chunked_prefill._batch_cls |
nested class | vllm_mlx.scheduler._install_chunked_prefill._batch_cls(uids: List[int], y: Any, logprobs: List[Any], max_tokens: List[int], num_tokens: List[int], cache: List[Any], samplers: List[Any], logits_processors: List[Any], tokens: List[Any]) |
Nested Class `_install_chunked_prefill._batch_cls` declares 4 direct member(s). | #L233-L273 |
vllm_mlx.scheduler._install_chunked_prefill._batch_cls.__len__ |
nested function | vllm_mlx.scheduler._install_chunked_prefill._batch_cls.__len__() -> not annotated |
Nested Function `_install_chunked_prefill._batch_cls.__len__` calls `len`; returns `len(self.uids)`. | #L244-L245 |
vllm_mlx.scheduler._install_chunked_prefill._batch_cls.extend |
nested function | vllm_mlx.scheduler._install_chunked_prefill._batch_cls.extend(other) -> not annotated |
Nested Function `_install_chunked_prefill._batch_cls.extend` updates `self.y`; calls `self.uids.extend`, `mx.concatenate`, `self.logprobs.extend`, `self.num_tokens.extend`. | #L260-L270 |
vllm_mlx.scheduler._install_chunked_prefill._batch_cls.extract_cache |
nested function | vllm_mlx.scheduler._install_chunked_prefill._batch_cls.extract_cache(idx) -> not annotated |
Nested Function `_install_chunked_prefill._batch_cls.extract_cache` calls `c.extract`; returns `[c.extract(idx) for c in self.cache]`. | #L272-L273 |
vllm_mlx.scheduler._install_chunked_prefill._batch_cls.filter |
nested function | vllm_mlx.scheduler._install_chunked_prefill._batch_cls.filter(keep_idx: List[int]) -> not annotated |
Nested Function `_install_chunked_prefill._batch_cls.filter` updates `self.uids`, `self.logprobs`, `self.max_tokens`, `self.num_tokens`; calls `mx.array`, `c.filter`. | #L247-L258 |
vllm_mlx.scheduler._install_chunked_prefill._chunked_next |
nested function | vllm_mlx.scheduler._install_chunked_prefill._chunked_next() -> not annotated |
Replacement for _next() that chunks large prefills. | #L362-L678 |
vllm_mlx.scheduler._install_chunked_prefill._generation_step |
nested function | vllm_mlx.scheduler._install_chunked_prefill._generation_step() -> not annotated |
Run one generation step on the active batch. | #L303-L360 |
vllm_mlx.scheduler._install_chunked_prefill._lazy_extract_cache |
nested function | vllm_mlx.scheduler._install_chunked_prefill._lazy_extract_cache(cache, idx) -> not annotated |
Nested Function `_install_chunked_prefill._lazy_extract_cache` calls `c.extract`; returns `(c.extract(idx) for c in cache)`. | #L225-L226 |
vllm_mlx.scheduler._install_chunked_prefill._patched_process_prompts |
nested function | vllm_mlx.scheduler._install_chunked_prefill._patched_process_prompts(prompts, _self = batch_gen) -> not annotated |
Nested Function `_install_chunked_prefill._patched_process_prompts` calls `_orig_process_prompts`, `enumerate`, `prompt_cache_save`, `batch.extract_cache`; returns `batch`. | #L291-L299 |
vllm_mlx.scheduler._install_chunked_prefill._patched_remove |
nested function | vllm_mlx.scheduler._install_chunked_prefill._patched_remove(uids_to_remove, _self = batch_gen) -> not annotated |
Clear partial state if aborted request is being prefilled. | #L680-L691 |
vllm_mlx.scheduler._install_mtp |
function | vllm_mlx.scheduler._install_mtp(batch_gen: 'BatchGenerator', model: Any, num_draft_tokens: int = 1, optimistic: bool = False, stats_state: Optional['_MTPStatsState'] = None) -> None |
Monkey-patch a BatchGenerator to use MTP (Multi-Token Prediction) with always-advance strategy for hybrid MambaCache + KVCache. | #L780-L1262 |
vllm_mlx.scheduler._install_mtp._get_mtp_stats |
nested function | vllm_mlx.scheduler._install_mtp._get_mtp_stats() -> Dict[str, Any] |
Nested Function `_install_mtp._get_mtp_stats` calls `dict`; returns `{'enabled': True, 'requested_draft_tokens': num_draft_tokens, 'effective_draft_tokens': 1, 'mode': 'always_advance_opti…`. | #L823-L845 |
vllm_mlx.scheduler._install_mtp._mtp_bypass_reasons |
nested function | vllm_mlx.scheduler._install_mtp._mtp_bypass_reasons(input_tokens, prompt_cache) -> not annotated |
Nested Function `_install_mtp._mtp_bypass_reasons` calls `reasons.append`; returns `reasons`. | #L849-L857 |
vllm_mlx.scheduler._install_mtp._mtp_next |
nested function | vllm_mlx.scheduler._install_mtp._mtp_next() -> not annotated |
Wrapper around _next that emits deferred MTP draft tokens. | #L1147-L1247 |
vllm_mlx.scheduler._install_mtp._mtp_step |
nested function | vllm_mlx.scheduler._install_mtp._mtp_step(input_tokens, prompt_cache, samplers, logits_processors, tokens) -> not annotated |
Extended _step with MTP always-advance strategy. | #L864-L1138 |
vllm_mlx.scheduler._install_mtp._record_mtp_bypass |
nested function | vllm_mlx.scheduler._install_mtp._record_mtp_bypass(reasons) -> None |
Nested Function `_install_mtp._record_mtp_bypass` contains no state mutation, call, raise, return, await, or yield. | #L859-L862 |
vllm_mlx.scheduler._install_prompt_cache_save |
function | vllm_mlx.scheduler._install_prompt_cache_save(batch_gen: 'BatchGenerator', prompt_cache_save) -> None |
Monkey-patch ``_process_prompts`` to capture prompt-only cache state. | #L163-L187 |
vllm_mlx.scheduler._install_prompt_cache_save._patched_process_prompts |
nested function | vllm_mlx.scheduler._install_prompt_cache_save._patched_process_prompts(prompts, _self = batch_gen) -> not annotated |
Nested Function `_install_prompt_cache_save._patched_process_prompts` calls `_orig_process_prompts`, `enumerate`, `prompt_cache_save`, `batch.extract_cache`; returns `batch`. | #L177-L185 |
vllm_mlx.scheduler._mtp_status_snapshot |
function | vllm_mlx.scheduler._mtp_status_snapshot(batch_generator) -> Dict[str, Any] |
Function `_mtp_status_snapshot` calls `getattr`, `callable`, `get_mtp_stats`; has 2 explicit return paths. | #L1265-L1269 |
vllm_mlx.scheduler._MTPStatsState |
class | vllm_mlx.scheduler._MTPStatsState(counters: Dict[str, int] = field(default_factory=lambda: {'attempted': 0, 'accepted': 0, 'rejected': 0, 'errors': 0}), bypass_counts: Dict[str, int] = field(default_factory=lambda: {'prefill': 0, 'no_active_batch': 0, 'cache_mismatch': 0}), lock: Any = field(default_factory=Lock)) |
Cumulative native-MTP counters shared across generator instances. | #L701-L719 |
vllm_mlx.scheduler._normalize_logits_processors |
function | vllm_mlx.scheduler._normalize_logits_processors(logits_processors) -> not annotated |
Normalize empty per-sequence processor slots to lists. | #L46-L50 |
vllm_mlx.scheduler._sanitize_batch_generator_logits_processors |
function | vllm_mlx.scheduler._sanitize_batch_generator_logits_processors(batch_generator) -> None |
Sanitize stale BatchGenerator processor state before decode. | #L53-L65 |
vllm_mlx.scheduler.Scheduler |
class | vllm_mlx.scheduler.Scheduler(model: Any, tokenizer: Any, config: Optional[SchedulerConfig] = None) |
Scheduler for continuous batching using mlx-lm BatchGenerator. | #L1272-L3518 |
vllm_mlx.scheduler.Scheduler.__init__ |
method | vllm_mlx.scheduler.Scheduler.__init__(model: Any, tokenizer: Any, config: Optional[SchedulerConfig] = None) -> not annotated |
Initialize the scheduler. | #L1286-L1402 |
vllm_mlx.scheduler.Scheduler._cache_coverage |
method | vllm_mlx.scheduler.Scheduler._cache_coverage(cache: Any) -> int | None |
How many tokens the live cache actually holds. | #L2383-L2409 |
vllm_mlx.scheduler.Scheduler._cache_coverage._offset_of |
nested function | vllm_mlx.scheduler.Scheduler._cache_coverage._offset_of(layer: Any) -> int | None |
Nested Function `Scheduler._cache_coverage._offset_of` calls `getattr`, `isinstance`, `_offset_of`; has 3 explicit return paths. | #L2393-L2403 |
vllm_mlx.scheduler.Scheduler._cache_key_for_snapshot |
method | vllm_mlx.scheduler.Scheduler._cache_key_for_snapshot(request: Any, response: Any, raw_cache: Any) -> list[int] | None |
Key the entry by the tokens the cache covers, not by the prompt. | #L2411-L2468 |
vllm_mlx.scheduler.Scheduler._cleanup_detokenizer |
method | vllm_mlx.scheduler.Scheduler._cleanup_detokenizer(request_id: str) -> None |
Remove the streaming detokenizer for a finished request. | #L1433-L1435 |
vllm_mlx.scheduler.Scheduler._cleanup_finished |
method | vllm_mlx.scheduler.Scheduler._cleanup_finished(finished_ids: Set[str]) -> None |
Clean up finished requests and store caches for reuse. | #L2712-L2865 |
vllm_mlx.scheduler.Scheduler._close_batch_generator |
method | vllm_mlx.scheduler.Scheduler._close_batch_generator() -> None |
Properly close BatchGenerator to restore wired_limit. | #L1657-L1665 |
vllm_mlx.scheduler.Scheduler._copy_cache_state |
method | vllm_mlx.scheduler.Scheduler._copy_cache_state(value: Any) -> Any |
Deep-copy a cache ``state`` payload. | #L2279-L2295 |
vllm_mlx.scheduler.Scheduler._create_batch_generator |
method | vllm_mlx.scheduler.Scheduler._create_batch_generator(sampling_params: SamplingParams) -> BatchGenerator |
Create a BatchGenerator with the given sampling parameters. | #L1457-L1539 |
vllm_mlx.scheduler.Scheduler._create_batch_generator._prefill_progress |
nested function | vllm_mlx.scheduler.Scheduler._create_batch_generator._prefill_progress(progress_list) -> not annotated |
Log prefill progress for each uid chunk. | #L1472-L1479 |
vllm_mlx.scheduler.Scheduler._decode_tokens |
method | vllm_mlx.scheduler.Scheduler._decode_tokens(token_ids: List[int]) -> str |
Decode token IDs to text, handling both tokenizers and processors. | #L1420-L1424 |
vllm_mlx.scheduler.Scheduler._do_abort_request |
method | vllm_mlx.scheduler.Scheduler._do_abort_request(request_id: str) -> bool |
Actually abort a request. | #L2022-L2087 |
vllm_mlx.scheduler.Scheduler._ensure_batch_generator |
method | vllm_mlx.scheduler.Scheduler._ensure_batch_generator(sampling_params: SamplingParams) -> None |
Ensure BatchGenerator exists with compatible settings. | #L1667-L1709 |
vllm_mlx.scheduler.Scheduler._extract_cache_for_uid |
method | vllm_mlx.scheduler.Scheduler._extract_cache_for_uid(uid: int) -> Any |
Pull one sequence's cache out of the live BatchGenerator batch. | #L2319-L2336 |
vllm_mlx.scheduler.Scheduler._extract_cache_states |
method | vllm_mlx.scheduler.Scheduler._extract_cache_states(raw_cache: List[Any]) -> List[Dict[str, Any]] |
Extract actual tensor state from each layer cache. | #L1771-L1806 |
vllm_mlx.scheduler.Scheduler._get_actual_tokenizer |
method | vllm_mlx.scheduler.Scheduler._get_actual_tokenizer(tokenizer: Any) -> Any |
Get the actual tokenizer from a processor or tokenizer. | #L1404-L1418 |
vllm_mlx.scheduler.Scheduler._get_detokenizer |
method | vllm_mlx.scheduler.Scheduler._get_detokenizer(request_id: str) -> Any |
Get or create a streaming detokenizer for a request. | #L1426-L1431 |
vllm_mlx.scheduler.Scheduler._get_stop_tokens |
method | vllm_mlx.scheduler.Scheduler._get_stop_tokens() -> Set[int] |
Get stop token IDs from tokenizer or processor. | #L1437-L1455 |
vllm_mlx.scheduler.Scheduler._is_cache_corruption_error |
method | vllm_mlx.scheduler.Scheduler._is_cache_corruption_error(error: Exception) -> bool |
Check if an error indicates cache corruption. | #L2867-L2870 |
vllm_mlx.scheduler.Scheduler._is_stream_thread_error |
method | vllm_mlx.scheduler.Scheduler._is_stream_thread_error(error: Exception) -> bool |
Check if an error indicates MLX stream/thread ownership mismatch. | #L2872-L2875 |
vllm_mlx.scheduler.Scheduler._make_mid_prefill_save_callback |
method | vllm_mlx.scheduler.Scheduler._make_mid_prefill_save_callback(save_interval: int) -> not annotated |
Create a callback for saving intermediate KV cache during chunked prefill. | #L1587-L1655 |
vllm_mlx.scheduler.Scheduler._make_mid_prefill_save_callback._mid_prefill_save |
nested function | vllm_mlx.scheduler.Scheduler._make_mid_prefill_save_callback._mid_prefill_save(uid, processed_tokens, prompt_cache) -> not annotated |
Nested Function `Scheduler._make_mid_prefill_save_callback._mid_prefill_save` calls `self.uid_to_request_id.get`, `self.requests.get`, `getattr`, `self._extract_cache_states`; returns `None`. | #L1598-L1653 |
vllm_mlx.scheduler.Scheduler._make_prompt_cache_save_callback |
method | vllm_mlx.scheduler.Scheduler._make_prompt_cache_save_callback() -> not annotated |
Create a callback that stores prompt-only KV/Mamba cache. | #L1541-L1585 |
vllm_mlx.scheduler.Scheduler._make_prompt_cache_save_callback._prompt_cache_save |
nested function | vllm_mlx.scheduler.Scheduler._make_prompt_cache_save_callback._prompt_cache_save(uid, extracted_cache) -> not annotated |
Nested Function `Scheduler._make_prompt_cache_save_callback._prompt_cache_save` calls `self.uid_to_request_id.get`, `self.requests.get`, `list`, `_trim_cache_offset`; returns `None`. | #L1554-L1583 |
vllm_mlx.scheduler.Scheduler._make_snapshot_destination |
method | vllm_mlx.scheduler.Scheduler._make_snapshot_destination(live_cache: Any) -> Any |
Build a destination cache with the same topology as the live one. | #L2338-L2380 |
vllm_mlx.scheduler.Scheduler._make_snapshot_destination._mirror |
nested function | vllm_mlx.scheduler.Scheduler._make_snapshot_destination._mirror(layer: Any) -> Any |
Nested Function `Scheduler._make_snapshot_destination._mirror` calls `getattr`, `_mirror`, `copy.copy`, `type(children)`; has 2 explicit return paths. | #L2360-L2370 |
vllm_mlx.scheduler.Scheduler._process_batch_responses |
method | vllm_mlx.scheduler.Scheduler._process_batch_responses(responses: List[Any]) -> Tuple[List[RequestOutput], Set[str]] |
Process responses from BatchGenerator. | #L2583-L2710 |
vllm_mlx.scheduler.Scheduler._process_pending_aborts |
method | vllm_mlx.scheduler.Scheduler._process_pending_aborts() -> None |
Drain and process pending abort requests. | #L2016-L2020 |
vllm_mlx.scheduler.Scheduler._prompt_output_entry_is_useless |
method | vllm_mlx.scheduler.Scheduler._prompt_output_entry_is_useless(cache: Any) -> bool |
Would a prompt+output entry built from this cache ever be reusable? | #L2303-L2317 |
vllm_mlx.scheduler.Scheduler._reconstruct_cache_from_states |
method | vllm_mlx.scheduler.Scheduler._reconstruct_cache_from_states(extracted_states: List[Dict[str, Any]]) -> Optional[List[Any]] |
Reconstruct cache objects from extracted cache states. | #L1808-L1872 |
vllm_mlx.scheduler.Scheduler._reconstruct_ssd_layers |
method | vllm_mlx.scheduler.Scheduler._reconstruct_ssd_layers(layer_dicts: list[dict]) -> list | None |
Reconstruct cache objects from deserialized layer dicts. | #L3462-L3518 |
vllm_mlx.scheduler.Scheduler._reconstruct_ssd_layers._mx_dtype_from_name |
nested function | vllm_mlx.scheduler.Scheduler._reconstruct_ssd_layers._mx_dtype_from_name(name: str) -> not annotated |
Nested Function `Scheduler._reconstruct_ssd_layers._mx_dtype_from_name` calls `getattr`; returns `getattr(mx, name, None)`. | #L3473-L3474 |
vllm_mlx.scheduler.Scheduler._recover_from_cache_error |
method | vllm_mlx.scheduler.Scheduler._recover_from_cache_error() -> None |
Recover from cache corruption error. | #L2877-L2895 |
vllm_mlx.scheduler.Scheduler._recover_from_generation_error |
method | vllm_mlx.scheduler.Scheduler._recover_from_generation_error() -> Set[str] |
Recover from fatal generation error (OOM, Metal crash). | #L2897-L2933 |
vllm_mlx.scheduler.Scheduler._reschedule_running_requests |
method | vllm_mlx.scheduler.Scheduler._reschedule_running_requests() -> None |
Move running requests back to waiting queue for retry. | #L2935-L2951 |
vllm_mlx.scheduler.Scheduler._schedule_waiting |
method | vllm_mlx.scheduler.Scheduler._schedule_waiting() -> List[Request] |
Move requests from waiting queue to running. | #L2101-L2276 |
vllm_mlx.scheduler.Scheduler._store_prompt_only_cache |
method | vllm_mlx.scheduler.Scheduler._store_prompt_only_cache(request: Any, response: Any) -> None |
Store the post-prefill cache under the prompt tokens alone. | #L2470-L2581 |
vllm_mlx.scheduler.Scheduler._try_promote_ssd_pending |
method | vllm_mlx.scheduler.Scheduler._try_promote_ssd_pending() -> None |
Attempt synchronous SSD promotion for waiting requests tagged ssd_pending. | #L3315-L3395 |
vllm_mlx.scheduler.Scheduler._validate_cache |
method | vllm_mlx.scheduler.Scheduler._validate_cache(cache: Any) -> bool |
Validate that a cache object is usable. | #L1711-L1769 |
vllm_mlx.scheduler.Scheduler.abort_request |
method | vllm_mlx.scheduler.Scheduler.abort_request(request_id: str) -> bool |
Queue request for abort. | #L1999-L2014 |
vllm_mlx.scheduler.Scheduler.add_request |
method | vllm_mlx.scheduler.Scheduler.add_request(request: Request) -> None |
Add a new request to the scheduler. | #L1874-L1997 |
vllm_mlx.scheduler.Scheduler.clear_prefix_cache |
method | vllm_mlx.scheduler.Scheduler.clear_prefix_cache() -> None |
Clear the in-memory prefix cache (keeps disk cache untouched). | #L3296-L3306 |
vllm_mlx.scheduler.Scheduler.clear_runtime_caches |
method | vllm_mlx.scheduler.Scheduler.clear_runtime_caches() -> Dict[str, bool] |
Clear prefix-cache state without resetting scheduler/request state. | #L3205-L3221 |
vllm_mlx.scheduler.Scheduler.close_ssd_tier |
method | vllm_mlx.scheduler.Scheduler.close_ssd_tier() -> None |
Shut down the SSD cache tier if present. | #L3308-L3313 |
vllm_mlx.scheduler.Scheduler.deep_reset |
method | vllm_mlx.scheduler.Scheduler.deep_reset() -> None |
Deep reset that clears ALL cache state including model-level caches. | #L3248-L3276 |
vllm_mlx.scheduler.Scheduler.get_cache_stats |
method | vllm_mlx.scheduler.Scheduler.get_cache_stats() -> Optional[Dict[str, Any]] |
Get cache statistics. | #L3195-L3203 |
vllm_mlx.scheduler.Scheduler.get_num_running |
method | vllm_mlx.scheduler.Scheduler.get_num_running() -> int |
Get number of running requests. | #L2097-L2099 |
vllm_mlx.scheduler.Scheduler.get_num_waiting |
method | vllm_mlx.scheduler.Scheduler.get_num_waiting() -> int |
Get number of waiting requests. | #L2093-L2095 |
vllm_mlx.scheduler.Scheduler.get_request |
method | vllm_mlx.scheduler.Scheduler.get_request(request_id: str) -> Optional[Request] |
Get a request by ID. | #L3091-L3093 |
vllm_mlx.scheduler.Scheduler.get_running_requests_info |
method | vllm_mlx.scheduler.Scheduler.get_running_requests_info() -> List[Dict[str, Any]] |
Per-request details for status endpoint. | #L3099-L3165 |
vllm_mlx.scheduler.Scheduler.get_stats |
method | vllm_mlx.scheduler.Scheduler.get_stats() -> Dict[str, Any] |
Get scheduler statistics. | #L3167-L3193 |
vllm_mlx.scheduler.Scheduler.has_requests |
method | vllm_mlx.scheduler.Scheduler.has_requests() -> bool |
Check if there are any pending or running requests. | #L2089-L2091 |
vllm_mlx.scheduler.Scheduler.load_cache_from_disk |
method | vllm_mlx.scheduler.Scheduler.load_cache_from_disk(cache_dir: str) -> int |
Load prefix cache from disk. | #L3289-L3294 |
vllm_mlx.scheduler.Scheduler.promote_from_ssd |
method | async vllm_mlx.scheduler.Scheduler.promote_from_ssd(request) -> bool |
Promote a cold-tier cache entry for a request (async version). | #L3397-L3460 |
vllm_mlx.scheduler.Scheduler.promote_from_ssd.release_budget |
nested function | vllm_mlx.scheduler.Scheduler.promote_from_ssd.release_budget(nbytes: int) -> None |
Release tentatively reserved budget on failure. | #L3418-L3421 |
vllm_mlx.scheduler.Scheduler.promote_from_ssd.reserve_budget |
nested function | vllm_mlx.scheduler.Scheduler.promote_from_ssd.reserve_budget(nbytes: int) -> bool |
Tentatively reserve RAM budget for promotion. | #L3412-L3416 |
vllm_mlx.scheduler.Scheduler.remove_finished_request |
method | vllm_mlx.scheduler.Scheduler.remove_finished_request(request_id: str) -> Optional[Request] |
Remove a finished request from tracking. | #L3095-L3097 |
vllm_mlx.scheduler.Scheduler.reset |
method | vllm_mlx.scheduler.Scheduler.reset() -> None |
Reset the scheduler state. | #L3223-L3246 |
vllm_mlx.scheduler.Scheduler.save_cache_to_disk |
method | vllm_mlx.scheduler.Scheduler.save_cache_to_disk(cache_dir: str) -> bool |
Save prefix cache to disk for persistence across restarts. | #L3282-L3287 |
vllm_mlx.scheduler.Scheduler.step |
method | vllm_mlx.scheduler.Scheduler.step(max_retries: int = 1) -> SchedulerOutput |
Execute one scheduling step with automatic error recovery. | #L2953-L3089 |
vllm_mlx.scheduler.SchedulerConfig |
class | vllm_mlx.scheduler.SchedulerConfig(max_num_seqs: int = 256, max_num_batched_tokens: int = 8192, policy: SchedulingPolicy = SchedulingPolicy.FCFS, prefill_batch_size: int = 8, completion_batch_size: int = 32, prefill_step_size: int = 2048, mllm_prefill_step_size: Optional[int] = None, enable_prefix_cache: bool = True, prefix_cache_size: int = 100, use_memory_aware_cache: bool = True, cache_memory_mb: Optional[int] = None, cache_memory_percent: float = 0.2, kv_cache_quantization: bool = False, kv_cache_quantization_bits: int = 8, kv_cache_quantization_group_size: int = 64, kv_cache_min_quantize_tokens: int = 256, use_paged_cache: bool = False, paged_cache_block_size: int = 64, max_cache_blocks: int = 1000, chunked_prefill_tokens: int = 0, mid_prefill_save_interval: int = 8192, ssd_cache_dir: Optional[str] = None, ssd_cache_max_gb: float = 10.0, max_kv_size: int = 0, enable_mtp: bool = False, mtp_num_draft_tokens: int = 1, mtp_optimistic: bool = False) |
Configuration for the scheduler. | #L76-L140 |
vllm_mlx.scheduler.SchedulerConfig.__post_init__ |
method | vllm_mlx.scheduler.SchedulerConfig.__post_init__() -> None |
Method `SchedulerConfig.__post_init__` calls `ValueError`; can raise `ValueError`. | #L138-L140 |
vllm_mlx.scheduler.SchedulerOutput |
class | vllm_mlx.scheduler.SchedulerOutput(scheduled_request_ids: List[str] = field(default_factory=list), num_scheduled_tokens: int = 0, finished_request_ids: Set[str] = field(default_factory=set), outputs: List[RequestOutput] = field(default_factory=list), has_work: bool = False) |
Output from a scheduling step. | #L144-L160 |
vllm_mlx.scheduler.SchedulingPolicy |
class | vllm_mlx.scheduler.SchedulingPolicy() |
Scheduling policy for request ordering. | #L68-L72 |
vllm_mlx.server._acquire_default_engine |
function | async vllm_mlx.server._acquire_default_engine(*, count_activity: bool = True) -> BaseEngine |
Acquire the default engine, auto-loading via the residency manager if needed. | #L1436-L1451 |
vllm_mlx.server._acquire_default_engine_for_request |
function | async vllm_mlx.server._acquire_default_engine_for_request(raw_request: Request, *, total_timeout: float, deadline: float, count_activity: bool = True, model: str | None = None) -> BaseEngine | None |
Acquire the engine for a request, using the model registry when active. | #L4661-L4719 |
vllm_mlx.server._acquire_default_engine_for_request._registry_acquire |
nested function | async vllm_mlx.server._acquire_default_engine_for_request._registry_acquire() -> not annotated |
Nested Function `_acquire_default_engine_for_request._registry_acquire` calls `_acquire_request_model`, `id`; awaits asynchronous work; returns `ctx.engine`. | #L4681-L4685 |
vllm_mlx.server._acquire_default_engine_for_request._registry_cleanup |
nested function | async vllm_mlx.server._acquire_default_engine_for_request._registry_cleanup(_result) -> not annotated |
Nested Function `_acquire_default_engine_for_request._registry_cleanup` calls `_active_request_contexts.pop`, `id`, `ctx.release`; awaits asynchronous work. | #L4687-L4690 |
vllm_mlx.server._acquire_request_model |
function | async vllm_mlx.server._acquire_request_model(request_model: str) -> RequestModelContext |
Acquire the model/engine that should serve this request. | #L1070-L1094 |
vllm_mlx.server._activate_engine |
function | vllm_mlx.server._activate_engine(engine: BaseEngine | None) -> BaseEngine | None |
Set the global engine pointer and refresh parser-sensitive state. | #L1365-L1375 |
vllm_mlx.server._apply_forced_tool_choice |
function | vllm_mlx.server._apply_forced_tool_choice(tool_choice, tools, messages, chat_kwargs = None) -> not annotated |
Apply forced tool_choice by filtering tools and injecting instructions. | #L5233-L5279 |
vllm_mlx.server._apply_response_format_or_raise |
function | vllm_mlx.server._apply_response_format_or_raise(text: str, response_format: object, *, ensure_ascii: bool = False) -> str |
Return validated JSON content or fail before returning a success response. | #L1933-L1952 |
vllm_mlx.server._attach_logit_bias_processor |
function | vllm_mlx.server._attach_logit_bias_processor(chat_kwargs: dict, logit_bias: dict[str, float] | None) -> not annotated |
Function `_attach_logit_bias_processor` calls `make_logits_processors`, `_coerce_logit_bias`, `chat_kwargs.get`, `list`; returns `None`. | #L733-L744 |
vllm_mlx.server._attach_response_format_logits_processor |
function | vllm_mlx.server._attach_response_format_logits_processor(chat_kwargs: dict, json_logits_processor: object) -> object |
Attach response_format constraints and keep thinking disabled. | #L700-L717 |
vllm_mlx.server._build_engine |
function | vllm_mlx.server._build_engine(spec: ModelSpec) -> BaseEngine |
Construct an engine instance from a model spec without starting it. | #L1293-L1323 |
vllm_mlx.server._build_reasoning_parser |
function | vllm_mlx.server._build_reasoning_parser(engine: BaseEngine | None = None) -> not annotated |
Create a fresh reasoning parser instance for a single request/stream. | #L1126-L1140 |
vllm_mlx.server._build_response_object |
function | vllm_mlx.server._build_response_object(request: ResponsesRequest, output_items: list[ResponseMessageItem | ResponseReasoningItem | ResponseFunctionCallItem], prompt_tokens: int, completion_tokens: int, finish_reason: str | None, response_id: str | None = None) -> ResponseObject |
Build a full Responses API object. | #L2328-L2367 |
vllm_mlx.server._build_responses_output_items |
function | vllm_mlx.server._build_responses_output_items(text: str | None, reasoning: str | None, tool_calls: list[ToolCall] | None) -> list[ResponseMessageItem | ResponseReasoningItem | ResponseFunctionCallItem] |
Convert parsed assistant output into Responses API output items. | #L2256-L2293 |
vllm_mlx.server._build_thinking_processor |
function | vllm_mlx.server._build_thinking_processor(engine: BaseEngine, thinking_token_budget: int, *, inner: object | None = None, prompt_has_think_tag: bool = True) -> object | None |
Build a ThinkingAwareLogitsProcessor if the tokenizer has think tokens. | #L500-L554 |
vllm_mlx.server._build_tool_parser |
function | vllm_mlx.server._build_tool_parser(engine: BaseEngine | None) -> not annotated |
Create a fresh tool parser instance for a single request/stream. | #L1109-L1123 |
vllm_mlx.server._coerce_logit_bias |
function | vllm_mlx.server._coerce_logit_bias(logit_bias: dict[str, float]) -> dict[int, float] |
Function `_coerce_logit_bias` calls `logit_bias.items`, `int`, `float`, `HTTPException`; can raise `HTTPException`; returns `coerced`. | #L720-L730 |
vllm_mlx.server._coerce_tool_arguments |
function | vllm_mlx.server._coerce_tool_arguments(arguments_json: str, tool_name: str, tools: list[dict] | None) -> str |
Coerce tool call arguments to match the tool schema. | #L1752-L1796 |
vllm_mlx.server._convert_anthropic_stop_reason |
function | vllm_mlx.server._convert_anthropic_stop_reason(openai_reason: str | None) -> str |
Convert OpenAI finish_reason to Anthropic stop_reason. | #L5327-L5335 |
vllm_mlx.server._detect_harmony_rendering |
function | vllm_mlx.server._detect_harmony_rendering() -> bool |
Detect whether the harmony rendering path should handle prompt building. | #L2986-L3019 |
vllm_mlx.server._detect_native_tool_support |
function | vllm_mlx.server._detect_native_tool_support() -> bool |
Detect if the active tool parser supports native tool format. | #L2954-L2983 |
vllm_mlx.server._disconnect_guard |
function | async vllm_mlx.server._disconnect_guard(generator: AsyncIterator[str], raw_request: Request, poll_interval: float = 0.5, heartbeat_interval: float = 5.0, cleanup = None, timeout: float | None = None) -> AsyncIterator[str] |
Wrap streaming generator to abort on client disconnect. | #L4377-L4546 |
vllm_mlx.server._disconnect_guard._deferred_generator_close |
nested function | async vllm_mlx.server._disconnect_guard._deferred_generator_close() -> not annotated |
Nested Function `_disconnect_guard._deferred_generator_close` calls `asyncio.sleep`, `_gen_to_close.aclose`, `logger.debug`, `type`; awaits asynchronous work. | #L4528-L4536 |
vllm_mlx.server._disconnect_guard._elapsed |
nested function | vllm_mlx.server._disconnect_guard._elapsed() -> not annotated |
Nested Function `_disconnect_guard._elapsed` calls `_time.monotonic`; returns `f'{_time.monotonic() - _t0:.1f}s'`. | #L4407-L4408 |
vllm_mlx.server._disconnect_guard._wait_disconnect |
nested function | async vllm_mlx.server._disconnect_guard._wait_disconnect() -> not annotated |
Nested Function `_disconnect_guard._wait_disconnect` calls `asyncio.sleep`, `_is_client_disconnected`, `logger.info`, `_elapsed`; awaits asynchronous work; returns `None`. | #L4417-L4429 |
vllm_mlx.server._emit_content_pieces |
function | vllm_mlx.server._emit_content_pieces(pieces: list[tuple[str, str]], current_block_type: str | None, block_index: int) -> tuple[list[str], str | None, int] |
Emit Anthropic SSE events for content pieces from the think router. | #L5669-L5719 |
vllm_mlx.server._engine_factory |
function | async vllm_mlx.server._engine_factory(spec: ModelSpec) -> BaseEngine |
Async engine factory used by the residency manager. | #L1326-L1328 |
vllm_mlx.server._ensure_sse_terminal |
function | async vllm_mlx.server._ensure_sse_terminal(generator: AsyncIterator[str], terminal_frame: str) -> AsyncIterator[str] |
Guarantee that *terminal_frame* is emitted exactly once at the end of *generator*, even if the generator raises mid-stream. | #L4275-L4296 |
vllm_mlx.server._extract_reasoning_and_tool_calls |
function | vllm_mlx.server._extract_reasoning_and_tool_calls(output_text: str, request: ChatCompletionRequest | None = None, *, allow_reasoning: bool = True, engine: BaseEngine | None = None) -> tuple[str | None, str | None, list[ToolCall] | None] |
Extract reasoning first, then parse tool calls from the cleaned content. | #L2895-L2951 |
vllm_mlx.server._extract_streaming_tool_delta |
function | vllm_mlx.server._extract_streaming_tool_delta(parser, previous_text: str, delta_text: str, request_context: dict) -> tuple[str, dict | None] |
Parse one request-local streaming delta and return new accumulated text. | #L3074-L3088 |
vllm_mlx.server._find_uvicorn_cycle |
function | vllm_mlx.server._find_uvicorn_cycle(obj, depth = 0, visited = None) -> not annotated |
Walk through middleware wrappers to find uvicorn's RequestResponseCycle. | #L4299-L4346 |
vllm_mlx.server._generation_metadata |
function | vllm_mlx.server._generation_metadata(thinking_processor: object | None) -> GenerationMetadata | None |
Function `_generation_metadata` calls `GenerationMetadata`, `getattr`, `bool`; has 2 explicit return paths. | #L571-L583 |
vllm_mlx.server._get_cache_dir |
function | vllm_mlx.server._get_cache_dir() -> str |
Get cache persistence directory based on actual model path. | #L1274-L1290 |
vllm_mlx.server._get_engine_tokenizer |
function | vllm_mlx.server._get_engine_tokenizer(engine: BaseEngine | None) -> object | None |
Return tokenizer-like parser state from the active engine. | #L1821-L1828 |
vllm_mlx.server._get_engine_tokenizer |
function | vllm_mlx.server._get_engine_tokenizer(engine) -> object | None |
Return the tokenizer backing ``engine``, if exposed. | #L5175-L5187 |
vllm_mlx.server._get_forced_tool_name |
function | vllm_mlx.server._get_forced_tool_name(tool_choice) -> str | None |
Extract forced tool name from tool_choice, if any. | #L5217-L5230 |
vllm_mlx.server._get_idle_unload_event |
function | vllm_mlx.server._get_idle_unload_event() -> asyncio.Event |
Return the idle-unload gate event, creating it on first use. | #L1206-L1217 |
vllm_mlx.server._get_lifecycle_status |
function | vllm_mlx.server._get_lifecycle_status() -> dict | None |
Get lifecycle status for the default resident if lifecycle is enabled. | #L1391-L1395 |
vllm_mlx.server._get_or_init_tool_parser |
function | vllm_mlx.server._get_or_init_tool_parser(engine: BaseEngine | None = None) -> not annotated |
Return the cached tool parser, initializing it from the given engine. | #L1831-L1841 |
vllm_mlx.server._get_streaming_tool_parser |
function | vllm_mlx.server._get_streaming_tool_parser(request: ChatCompletionRequest | None, engine: BaseEngine | None = None) -> not annotated |
Get a streaming-capable tool parser for this request. | #L3034-L3071 |
vllm_mlx.server._inject_json_instruction |
function | vllm_mlx.server._inject_json_instruction(messages: list, instruction: str) -> list |
Inject JSON instruction into messages. | #L5290-L5319 |
vllm_mlx.server._invalidate_tool_parser_cache |
function | vllm_mlx.server._invalidate_tool_parser_cache(reason: str | None = None) -> None |
Drop cached parser state when the serving tokenizer changes. | #L1220-L1229 |
vllm_mlx.server._is_client_disconnected |
function | vllm_mlx.server._is_client_disconnected(raw_request: Request) -> bool |
Reliable client disconnect check. | #L4349-L4374 |
vllm_mlx.server._iter_remote_media_urls |
function | vllm_mlx.server._iter_remote_media_urls(messages: list[Message | dict]) -> not annotated |
Yield remote media URLs from OpenAI-style multimodal message content. | #L401-L429 |
vllm_mlx.server._lifecycle_loop |
function | async vllm_mlx.server._lifecycle_loop() -> None |
Background idle-unload loop for the default resident. | #L1413-L1433 |
vllm_mlx.server._list_available_model_names |
function | vllm_mlx.server._list_available_model_names() -> list[str] |
Function `_list_available_model_names` has 2 explicit return paths. | #L1059-L1062 |
vllm_mlx.server._load_prefix_cache_from_disk |
function | vllm_mlx.server._load_prefix_cache_from_disk(engine: BaseEngine | None = None) -> None |
Load prefix cache from disk during startup. | #L1232-L1250 |
vllm_mlx.server._log_and_raise_internal_error |
function | vllm_mlx.server._log_and_raise_internal_error(log_prefix: str, exc: Exception, detail: str) -> None |
Log a sanitized exception string and raise a generic 500 response. | #L1025-L1028 |
vllm_mlx.server._make_keepalive_http_protocol |
function | vllm_mlx.server._make_keepalive_http_protocol(idle = 10, interval = 5, count = 3) -> not annotated |
Create a uvicorn HTTP protocol class with aggressive TCP keepalive. | #L6554-L6589 |
vllm_mlx.server._make_keepalive_http_protocol._KeepaliveProtocol |
nested class | vllm_mlx.server._make_keepalive_http_protocol._KeepaliveProtocol() |
Nested Class `_make_keepalive_http_protocol._KeepaliveProtocol` derives from `_Base` and declares 1 direct member(s). | #L6567-L6587 |
vllm_mlx.server._make_keepalive_http_protocol._KeepaliveProtocol.connection_made |
nested function | vllm_mlx.server._make_keepalive_http_protocol._KeepaliveProtocol.connection_made(transport) -> not annotated |
Nested Function `_make_keepalive_http_protocol._KeepaliveProtocol.connection_made` calls `super().connection_made`, `super`, `transport.get_extra_info`, `sock.setsockopt`; returns `None`. | #L6568-L6587 |
vllm_mlx.server._make_release_cleanup |
function | vllm_mlx.server._make_release_cleanup(raw_request: Request | None) -> not annotated |
Return a cleanup callable suitable for ``_disconnect_guard``. | #L4740-L4752 |
vllm_mlx.server._make_release_cleanup._cleanup |
nested function | async vllm_mlx.server._make_release_cleanup._cleanup() -> not annotated |
Nested Function `_make_release_cleanup._cleanup` calls `_active_request_contexts.pop`, `id`, `ctx.release`, `_release_default_engine`; awaits asynchronous work. | #L4744-L4749 |
vllm_mlx.server._metrics_middleware |
function | async vllm_mlx.server._metrics_middleware(request: Request, call_next) -> not annotated |
Capture generic HTTP request metrics when enabled. | #L1630-L1659 |
vllm_mlx.server._metrics_path_for_request |
function | vllm_mlx.server._metrics_path_for_request(request: Request) -> str |
Prefer route templates over raw URLs to keep metrics cardinality bounded. | #L1613-L1626 |
vllm_mlx.server._metrics_result_from_status |
function | vllm_mlx.server._metrics_result_from_status(status_code: int) -> str |
Map HTTP-ish status codes to low-cardinality inference results. | #L1602-L1610 |
vllm_mlx.server._new_response_item_id |
function | vllm_mlx.server._new_response_item_id(prefix: str) -> str |
Generate stable OpenAI-style item ids. | #L1984-L1986 |
vllm_mlx.server._normalize_messages |
function | vllm_mlx.server._normalize_messages(messages: list[dict]) -> list[dict] |
Normalize message roles and merge consecutive same-role messages. | #L5117-L5172 |
vllm_mlx.server._parse_streaming_tool_content |
function | vllm_mlx.server._parse_streaming_tool_content(parser, accumulated_text: str, delta_text: str, request_context: dict) -> tuple[str, dict | None, bool] |
Function `_parse_streaming_tool_content` calls `_extract_streaming_tool_delta`; returns `(accumulated_text, result, suppress)`. | #L3103-L3116 |
vllm_mlx.server._parse_tool_calls_with_parser |
function | vllm_mlx.server._parse_tool_calls_with_parser(output_text: str, request: ChatCompletionRequest | None = None, engine: BaseEngine | None = None) -> tuple[str, list | None] |
Parse tool calls from model output using the configured parser. | #L1844-L1930 |
vllm_mlx.server._persist_engine_state |
function | async vllm_mlx.server._persist_engine_state(spec: ModelSpec, engine: BaseEngine) -> None |
Persist engine-local state before an idle unload or shutdown unload. | #L1359-L1362 |
vllm_mlx.server._prepare_anthropic_endpoint_invocation |
function | vllm_mlx.server._prepare_anthropic_endpoint_invocation(engine: BaseEngine, openai_request: ChatCompletionRequest, effective_max_tokens: int) -> PreparedChatInvocation |
Prepare Anthropic invocation and convert URL-safety errors to 400s. | #L5338-L5351 |
vllm_mlx.server._prepare_anthropic_invocation |
function | vllm_mlx.server._prepare_anthropic_invocation(engine: BaseEngine, openai_request: ChatCompletionRequest, effective_max_tokens: int) -> PreparedChatInvocation |
Precompute messages, kwargs, and decoding constraints for Anthropic API. | #L858-L910 |
vllm_mlx.server._prepare_chat_completion_invocation |
function | vllm_mlx.server._prepare_chat_completion_invocation(engine: BaseEngine, request: ChatCompletionRequest, effective_max_tokens: int) -> PreparedChatInvocation |
Precompute messages, kwargs, and decoding constraints for chat completions. | #L747-L855 |
vllm_mlx.server._prepare_chat_messages |
function | vllm_mlx.server._prepare_chat_messages(engine: BaseEngine, request_messages: list[Message | dict]) -> tuple[list[dict], list, list, list, bool] |
Normalize messages and collect media once for both stream/non-stream paths. | #L314-L398 |
vllm_mlx.server._prepare_json_logits_processor |
function | vllm_mlx.server._prepare_json_logits_processor(engine: BaseEngine, messages: list[dict], response_format: object | None, *, tools: list | None, tool_choice: object | None, log_context: str | None = None, thinking_model: bool = False) -> tuple[list[dict], object | None] |
Inject response_format instruction and build constrained decoding processor. | #L447-L497 |
vllm_mlx.server._prepare_openai_stream_reasoning_state |
function | vllm_mlx.server._prepare_openai_stream_reasoning_state(engine: BaseEngine, request: ChatCompletionRequest, chat_kwargs: dict[str, object]) -> tuple[object | None, bool] |
Return request-local reasoning state and the legacy Nemotron marker state. | #L1159-L1171 |
vllm_mlx.server._prepare_responses_request |
function | vllm_mlx.server._prepare_responses_request(request: ResponsesRequest, *, validate_remote_media: bool = True) -> tuple[BaseEngine, ChatCompletionRequest, list[dict], dict] |
Prepare a Responses request for execution on the chat engine. | #L2370-L2414 |
vllm_mlx.server._prepare_streaming_reasoning_parser |
function | vllm_mlx.server._prepare_streaming_reasoning_parser(engine: BaseEngine, request: ChatCompletionRequest | ResponsesRequest | None, chat_kwargs: dict[str, object], *, allowed: bool = True) -> not annotated |
Build and reset request-local reasoning state when thinking is enabled. | #L1143-L1156 |
vllm_mlx.server._prepare_streaming_responses_request |
function | vllm_mlx.server._prepare_streaming_responses_request(request: ResponsesRequest) -> tuple[BaseEngine, ChatCompletionRequest, list[dict], dict] |
Prepare a streaming Responses request after eager URL validation. | #L2417-L2421 |
vllm_mlx.server._promote_streaming_response_format_delta |
function | vllm_mlx.server._promote_streaming_response_format_delta(content: str | None, reasoning: str | None, request: ChatCompletionRequest) -> tuple[str | None, str | None] |
Keep response_format JSON on the streaming content channel. | #L1963-L1981 |
vllm_mlx.server._public_lifecycle_status |
function | vllm_mlx.server._public_lifecycle_status(lifecycle: dict | None) -> dict | None |
Return residency status safe for unauthenticated public endpoints. | #L1398-L1410 |
vllm_mlx.server._raise_engine_busy |
function | vllm_mlx.server._raise_engine_busy(exc: EngineBusy) -> None |
Translate serialized-engine admission failures into retryable HTTP 503. | #L1031-L1039 |
vllm_mlx.server._raise_remote_media_http_error |
function | vllm_mlx.server._raise_remote_media_http_error(exc: UnsafeRemoteURLError) -> None |
Log internal URL-safety detail while returning a generic client error. | #L438-L444 |
vllm_mlx.server._release_default_engine |
function | async vllm_mlx.server._release_default_engine(*, count_activity: bool = True) -> None |
Release the default engine after request processing. | #L1454-L1463 |
vllm_mlx.server._release_engine_for_request |
function | async vllm_mlx.server._release_engine_for_request(raw_request: Request | None, *, count_activity: bool = True) -> None |
Release the engine acquired for this request. | #L4722-L4737 |
vllm_mlx.server._remaining_request_timeout |
function | vllm_mlx.server._remaining_request_timeout(total_timeout: float, deadline: float) -> float |
Compute remaining request budget or raise the standard timeout error. | #L4647-L4655 |
vllm_mlx.server._request_tool_definitions |
function | vllm_mlx.server._request_tool_definitions(request: ChatCompletionRequest) -> list | None |
Return the request tool schema once for streaming argument coercion. | #L1174-L1178 |
vllm_mlx.server._resolve_chat_template_kwargs |
function | vllm_mlx.server._resolve_chat_template_kwargs(request_value: dict[str, object] | None) -> dict[str, object] |
Resolve chat template kwargs: request > server default > empty dict. | #L291-L300 |
vllm_mlx.server._resolve_min_p |
function | vllm_mlx.server._resolve_min_p(request_value: float | None) -> float |
Resolve min_p: request > CLI default > fallback. | #L252-L258 |
vllm_mlx.server._resolve_no_final_content_token_limit |
function | vllm_mlx.server._resolve_no_final_content_token_limit() -> int | None |
Function `_resolve_no_final_content_token_limit` calls `os.environ.get`, `raw.strip`, `int`, `logger.warning`; has 2 explicit return paths. | #L557-L568 |
vllm_mlx.server._resolve_presence_penalty |
function | vllm_mlx.server._resolve_presence_penalty(request_value: float | None) -> float |
Resolve presence_penalty: request > CLI default > fallback. | #L261-L267 |
vllm_mlx.server._resolve_repetition_penalty |
function | vllm_mlx.server._resolve_repetition_penalty(request_value: float | None) -> float |
Resolve repetition_penalty: request > CLI default > fallback. | #L270-L276 |
vllm_mlx.server._resolve_request_max_tokens |
function | vllm_mlx.server._resolve_request_max_tokens(requested_value: int | None) -> int |
Resolve and validate a request's max_tokens budget. | #L279-L288 |
vllm_mlx.server._resolve_temperature |
function | vllm_mlx.server._resolve_temperature(request_value: float | None) -> float |
Resolve temperature: request > CLI default > fallback. | #L225-L231 |
vllm_mlx.server._resolve_top_k |
function | vllm_mlx.server._resolve_top_k(request_value: int | None) -> int |
Resolve top_k: request > CLI default > fallback. | #L243-L249 |
vllm_mlx.server._resolve_top_p |
function | vllm_mlx.server._resolve_top_p(request_value: float | None) -> float |
Resolve top_p: request > CLI default > fallback. | #L234-L240 |
vllm_mlx.server._response_content_to_text |
function | vllm_mlx.server._response_content_to_text(content) -> str |
Normalize Responses API content items into plain text. | #L1989-L2006 |
vllm_mlx.server._response_format_type |
function | vllm_mlx.server._response_format_type(response_format: object | None) -> str | None |
Function `_response_format_type` calls `isinstance`, `response_format.get`, `getattr`; has 3 explicit return paths. | #L1955-L1960 |
vllm_mlx.server._response_model_name |
function | vllm_mlx.server._response_model_name(request_model: str) -> str |
Return the response model field for single-model or registry mode. | #L1065-L1067 |
vllm_mlx.server._response_output_items_to_chat_messages |
function | vllm_mlx.server._response_output_items_to_chat_messages(output_items: list) -> list[dict] |
Persist assistant output in chat-completions form for previous_response_id. | #L2296-L2325 |
vllm_mlx.server._responses_input_to_chat_messages |
function | vllm_mlx.server._responses_input_to_chat_messages(request: ResponsesRequest) -> list[dict] |
Convert Responses API input items into chat-completions-style messages. | #L2052-L2170 |
vllm_mlx.server._responses_request_to_chat_request |
function | vllm_mlx.server._responses_request_to_chat_request(request: ResponsesRequest) -> ChatCompletionRequest |
Build a ChatCompletionRequest from a ResponsesRequest. | #L2203-L2253 |
vllm_mlx.server._responses_request_to_new_persisted_messages |
function | vllm_mlx.server._responses_request_to_new_persisted_messages(request: ResponsesRequest) -> list[dict] |
Persist only the current request's replayable input items. | #L2173-L2181 |
vllm_mlx.server._responses_request_to_persisted_messages |
function | vllm_mlx.server._responses_request_to_persisted_messages(request: ResponsesRequest) -> list[dict] |
Persist replayable history for chained previous_response_id requests. | #L2184-L2200 |
vllm_mlx.server._responses_sse_event |
function | vllm_mlx.server._responses_sse_event(event_type: str, payload: BaseModel | dict) -> str |
Encode a Responses API SSE event. | #L2871-L2878 |
vllm_mlx.server._responses_tools_to_chat_tools |
function | vllm_mlx.server._responses_tools_to_chat_tools(tools: list[ResponseFunctionTool | dict]) -> tuple[list[dict] | None, list[str]] |
Convert supported Responses tools and report unsupported tool types. | #L2009-L2049 |
vllm_mlx.server._restore_engine_state |
function | async vllm_mlx.server._restore_engine_state(spec: ModelSpec, engine: BaseEngine) -> None |
Restore engine-local state, such as prefix cache, after a cold load. | #L1353-L1356 |
vllm_mlx.server._run_blocking_engine_cache_io |
function | async vllm_mlx.server._run_blocking_engine_cache_io(io_fn, engine: BaseEngine) -> None |
Run blocking cache persistence off the event loop. | #L1331-L1350 |
vllm_mlx.server._run_responses_request |
function | async vllm_mlx.server._run_responses_request(request: ResponsesRequest, raw_request: Request) -> tuple[ResponseObject | None, list[dict]] |
Execute a Responses API request against the backend chat engine. | #L2424-L2477 |
vllm_mlx.server._sanitize_log_text |
function | vllm_mlx.server._sanitize_log_text(value: object, limit: int | None = None) -> str |
Escape control characters before logging untrusted text. | #L1000-L1022 |
vllm_mlx.server._save_prefix_cache_to_disk |
function | vllm_mlx.server._save_prefix_cache_to_disk(engine: BaseEngine | None = None) -> None |
Save prefix cache to disk during shutdown. | #L1253-L1271 |
vllm_mlx.server._start_request_budget |
function | vllm_mlx.server._start_request_budget(timeout: float | None) -> tuple[float, float] |
Return the total timeout and absolute deadline for a request. | #L4641-L4644 |
vllm_mlx.server._stream_anthropic_messages |
function | async vllm_mlx.server._stream_anthropic_messages(engine: BaseEngine, openai_request: ChatCompletionRequest, anthropic_request: AnthropicRequest, prepared: PreparedChatInvocation, metrics_tracker = None) -> AsyncIterator[str] |
Stream Anthropic Messages API SSE events. | #L5722-L5995 |
vllm_mlx.server._stream_request_metadata |
function | vllm_mlx.server._stream_request_metadata(request: ChatCompletionRequest) -> tuple[dict, list | None, bool] |
Function `_stream_request_metadata` calls `request.model_dump(include={'tools'}).get`, `request.model_dump`, `bool`; returns `({'tools': tools or []}, tools, include_usage)`. | #L3091-L3100 |
vllm_mlx.server._stream_responses_request |
function | async vllm_mlx.server._stream_responses_request(request: ResponsesRequest) -> AsyncIterator[str] |
Execute a Responses API request and stream SSE events incrementally. | #L2480-L2868 |
vllm_mlx.server._stream_responses_request._start_reasoning_item |
nested function | vllm_mlx.server._stream_responses_request._start_reasoning_item() -> list[str] |
Nested Function `_stream_responses_request._start_reasoning_item` calls `_new_response_item_id`, `events.append`, `_responses_sse_event`, `ResponseOutputItemAddedEvent`; returns `events`. | #L2563-L2598 |
vllm_mlx.server._stream_responses_request._start_text_item |
nested function | vllm_mlx.server._stream_responses_request._start_text_item() -> list[str] |
Nested Function `_stream_responses_request._start_text_item` calls `_new_response_item_id`, `events.append`, `_responses_sse_event`, `ResponseOutputItemAddedEvent`; returns `events`. | #L2525-L2561 |
vllm_mlx.server._stream_with_model_context |
function | async vllm_mlx.server._stream_with_model_context(context: RequestModelContext, stream: AsyncIterator[str]) -> AsyncIterator[str] |
Ensure model leases survive for the full streaming response. | #L1097-L1106 |
vllm_mlx.server._streaming_json_fence_stripper |
function | vllm_mlx.server._streaming_json_fence_stripper(request: ChatCompletionRequest) -> StreamingJsonFenceStripper | None |
Create a fence stripper only for JSON-constrained streaming responses. | #L1181-L1191 |
vllm_mlx.server._streaming_tool_markup_possible |
function | vllm_mlx.server._streaming_tool_markup_possible(text: str) -> bool |
Heuristic marker check to avoid parser work on ordinary text chunks. | #L3119-L3125 |
vllm_mlx.server._streaming_tool_markup_possible_after_delta |
function | vllm_mlx.server._streaming_tool_markup_possible_after_delta(accumulated_text: str, delta_text: str) -> bool |
Check only the boundary window needed to detect newly appearing tool markup. | #L3128-L3143 |
vllm_mlx.server._strip_backslash_before_unicode |
function | vllm_mlx.server._strip_backslash_before_unicode(obj: object) -> object |
Remove spurious backslashes before non-ASCII chars in JSON string values. | #L983-L997 |
vllm_mlx.server._strip_harmony_analysis_blocks |
function | vllm_mlx.server._strip_harmony_analysis_blocks(text: str) -> str |
Remove harmony analysis-channel blocks (and their content) so reasoning text is never handed to the tool parser, while commentary/final text is preserved. | #L2888-L2892 |
vllm_mlx.server._sync_engine_from_residency |
function | vllm_mlx.server._sync_engine_from_residency() -> BaseEngine | None |
Sync the global engine pointer from the residency manager state. | #L1378-L1388 |
vllm_mlx.server._thinking_disabled |
function | vllm_mlx.server._thinking_disabled(request, chat_kwargs: dict | None = None) -> bool |
Return True iff thinking is explicitly disabled for this request. | #L934-L950 |
vllm_mlx.server._ThinkingAwareLogitsProcessor |
class | vllm_mlx.server._ThinkingAwareLogitsProcessor(inner, prompt_has_think_tag: bool = False) |
Wrap a ``JSONSchemaLogitsProcessor`` so JSON constraining only activates after the model emits ``</think>``, letting it reason freely first. | #L586-L697 |
vllm_mlx.server._ThinkingAwareLogitsProcessor.__call__ |
method | vllm_mlx.server._ThinkingAwareLogitsProcessor.__call__(tokens, logits) -> not annotated |
Method `_ThinkingAwareLogitsProcessor.__call__` updates `self._base_prompt_len`, `self._in_thinking`, `self._waiting_for_json`, `self._json_scan_offset`; calls `self._inner`, `hasattr`, `tokens.tolist`, `list`; has 3 explicit return paths. | #L637-L688 |
vllm_mlx.server._ThinkingAwareLogitsProcessor.__init__ |
method | vllm_mlx.server._ThinkingAwareLogitsProcessor.__init__(inner, prompt_has_think_tag: bool = False) -> not annotated |
Method `_ThinkingAwareLogitsProcessor.__init__` updates `self._inner`, `self._active`, `self._in_thinking`, `self._waiting_for_json`. | #L597-L608 |
vllm_mlx.server._ThinkingAwareLogitsProcessor._disabled |
method | vllm_mlx.server._ThinkingAwareLogitsProcessor._disabled() -> not annotated |
Method `_ThinkingAwareLogitsProcessor._disabled` returns `self._inner._disabled`. | #L696-L697 |
vllm_mlx.server._ThinkingAwareLogitsProcessor._scan_for_json_start |
method | vllm_mlx.server._ThinkingAwareLogitsProcessor._scan_for_json_start(tokens_list, tokens, logits) -> not annotated |
Scan generated tokens for the first ``{`` or ``[``. | #L610-L635 |
vllm_mlx.server._ThinkingAwareLogitsProcessor.schema |
method | vllm_mlx.server._ThinkingAwareLogitsProcessor.schema() -> not annotated |
Method `_ThinkingAwareLogitsProcessor.schema` returns `self._inner.schema`. | #L692-L693 |
vllm_mlx.server._tool_choice_disabled |
function | vllm_mlx.server._tool_choice_disabled(request: ChatCompletionRequest | None) -> bool |
Return True when tool_choice explicitly disables tool calling. | #L3022-L3031 |
vllm_mlx.server._tool_name |
function | vllm_mlx.server._tool_name(tool: dict) -> str | None |
Extract function name from a tool definition dict. | #L5282-L5287 |
vllm_mlx.server._validate_model_name |
function | vllm_mlx.server._validate_model_name(request_model: str) -> None |
Validate that the request model name matches the served model. | #L1799-L1818 |
vllm_mlx.server._validate_remote_media_urls |
function | vllm_mlx.server._validate_remote_media_urls(messages: list[Message | dict]) -> None |
Validate remote media URLs during request preparation. | #L432-L435 |
vllm_mlx.server._wait_with_disconnect |
function | async vllm_mlx.server._wait_with_disconnect(coro, raw_request: Request, timeout: float, poll_interval: float = 0.5, timeout_detail_seconds: float | None = None, cleanup_result = None) -> not annotated |
Run a coroutine with both timeout and client disconnect detection. | #L4549-L4638 |
vllm_mlx.server._wait_with_disconnect._wait_disconnect |
nested function | async vllm_mlx.server._wait_with_disconnect._wait_disconnect() -> not annotated |
Nested Function `_wait_with_disconnect._wait_disconnect` calls `asyncio.sleep`, `_is_client_disconnected`, `logger.info`, `_time.monotonic`; awaits asynchronous work; returns `None`. | #L4569-L4581 |
vllm_mlx.server.cache_stats |
function | async vllm_mlx.server.cache_stats() -> not annotated |
Get cache statistics for debugging and monitoring. | #L3601-L3627 |
vllm_mlx.server.cancel_request |
function | async vllm_mlx.server.cancel_request(request_id: str) -> not annotated |
Cancel an active or queued request. | #L3720-L3747 |
vllm_mlx.server.check_rate_limit |
function | async vllm_mlx.server.check_rate_limit(request: Request) -> not annotated |
Rate limiting dependency. | #L1707-L1720 |
vllm_mlx.server.clear_cache |
function | async vllm_mlx.server.clear_cache() -> not annotated |
Clear all caches. | #L3631-L3659 |
vllm_mlx.server.clear_prefix_cache |
function | async vllm_mlx.server.clear_prefix_cache() -> not annotated |
Clear the text prefix cache used for KV reuse in continuous batching. | #L3663-L3713 |
vllm_mlx.server.clear_prefix_cache._rewarm |
nested function | async vllm_mlx.server.clear_prefix_cache._rewarm() -> not annotated |
Nested Function `clear_prefix_cache._rewarm` calls `load_warmup_file`, `warm_prefix_cache`, `logger.info`, `logger.warning`; awaits asynchronous work. | #L3688-L3707 |
vllm_mlx.server.count_anthropic_tokens |
function | async vllm_mlx.server.count_anthropic_tokens(request: Request) -> not annotated |
Count tokens for an Anthropic Messages API request. | #L5585-L5666 |
vllm_mlx.server.create_anthropic_message |
function | async vllm_mlx.server.create_anthropic_message(request: Request) -> not annotated |
Anthropic Messages API endpoint. | #L5357-L5578 |
vllm_mlx.server.create_chat_completion |
function | async vllm_mlx.server.create_chat_completion(request: ChatCompletionRequest, raw_request: Request) -> not annotated |
Create a chat completion (supports multimodal content for VLM models). | #L4916-L5114 |
vllm_mlx.server.create_completion |
function | async vllm_mlx.server.create_completion(request: CompletionRequest, raw_request: Request) -> not annotated |
Create a text completion. | #L4763-L4909 |
vllm_mlx.server.create_embeddings |
function | async vllm_mlx.server.create_embeddings(request: EmbeddingRequest) -> EmbeddingResponse |
Create embeddings for the given input text(s). | #L3787-L3908 |
vllm_mlx.server.create_parser |
function | vllm_mlx.server.create_parser() -> argparse.ArgumentParser |
Create the standalone server CLI parser. | #L6711-L6912 |
vllm_mlx.server.create_response |
function | async vllm_mlx.server.create_response(request: ResponsesRequest, raw_request: Request) -> not annotated |
Create a Responses API response. | #L5194-L5214 |
vllm_mlx.server.create_speech |
function | async vllm_mlx.server.create_speech(model: str = 'kokoro', input: str = '', voice: str = 'af_heart', speed: float = 1.0, response_format: str = 'wav') -> not annotated |
Generate speech from text (OpenAI TTS API compatible). | #L4200-L4254 |
vllm_mlx.server.create_transcription |
function | async vllm_mlx.server.create_transcription(file: UploadFile, model: str = 'whisper-large-v3', language: str | None = None, response_format: str = 'json') -> not annotated |
Transcribe audio to text (OpenAI Whisper API compatible). | #L4130-L4196 |
vllm_mlx.server.delete_request |
function | async vllm_mlx.server.delete_request(request_id: str) -> not annotated |
OpenAI-style alias for cancelling an active or queued request. | #L3754-L3756 |
vllm_mlx.server.execute_mcp_tool |
function | async vllm_mlx.server.execute_mcp_tool(request: MCPExecuteRequest) -> MCPExecuteResponse |
Execute an MCP tool. | #L4088-L4117 |
vllm_mlx.server.get_engine |
function | vllm_mlx.server.get_engine() -> BaseEngine |
Get the loaded engine, raising error if not loaded. | #L1745-L1749 |
vllm_mlx.server.get_usage |
function | vllm_mlx.server.get_usage(output: GenerationOutput) -> Usage |
Extract usage metrics from GenerationOutput. | #L3460-L3472 |
vllm_mlx.server.health |
function | async vllm_mlx.server.health() -> not annotated |
Health check endpoint. | #L3489-L3544 |
vllm_mlx.server.init_mcp |
function | async vllm_mlx.server.init_mcp(config_path: str) -> not annotated |
Initialize MCP manager from config file. | #L6520-L6546 |
vllm_mlx.server.lifespan |
function | async vllm_mlx.server.lifespan(app: FastAPI) -> not annotated |
FastAPI lifespan for startup/shutdown events. | #L1466-L1589 |
vllm_mlx.server.list_mcp_servers |
function | async vllm_mlx.server.list_mcp_servers() -> MCPServersResponse |
Get status of all MCP servers. | #L4067-L4084 |
vllm_mlx.server.list_mcp_tools |
function | async vllm_mlx.server.list_mcp_tools() -> MCPToolsResponse |
List all available MCP tools. | #L4047-L4063 |
vllm_mlx.server.list_models |
function | async vllm_mlx.server.list_models() -> ModelsResponse |
List available models. | #L3760-L3775 |
vllm_mlx.server.list_voices |
function | async vllm_mlx.server.list_voices(model: str = 'kokoro') -> not annotated |
List available voices for a TTS model. | #L4258-L4267 |
vllm_mlx.server.load_embedding_model |
function | vllm_mlx.server.load_embedding_model(model_name: str | None, *, lock: bool = False, reuse_existing: bool = True) -> None |
Load or reuse the embedding model engine when configured. | #L3146-L3171 |
vllm_mlx.server.load_model |
function | vllm_mlx.server.load_model(model_name: str, use_batching: bool = False, scheduler_config = None, stream_interval: int = 1, max_tokens: int = 32768, max_request_tokens: int = 32768, force_mllm: bool = False, gpu_memory_utilization: float = 0.9, served_model_name: str | None = None, trust_remote_code: bool = False, mtp: bool = False, prefill_step_size: int = 2048, specprefill_enabled: bool = False, specprefill_threshold: int = 8192, specprefill_keep_pct: float = 0.3, specprefill_backbone_pct: float = 0.0, specprefill_draft_model: str = None, mllm_draft_model: str | None = None, mllm_draft_kind: str | None = None, mllm_draft_block_size: int | None = None, warm_prompts_path: str | None = None, auto_unload_idle_seconds: float = 0.0, lazy_load_model: bool = False) -> not annotated |
Load a model (auto-detects MLLM vs LLM). | #L3202-L3431 |
vllm_mlx.server.load_model_registry |
function | vllm_mlx.server.load_model_registry(config_path: str, *, defaults: RegistryServeDefaults) -> None |
Load a registry-backed model manager from YAML configuration. | #L3434-L3457 |
vllm_mlx.server.load_reranker_model |
function | vllm_mlx.server.load_reranker_model(model_name: str | None, *, lock: bool = False, reuse_existing: bool = True) -> None |
Load or reuse the reranker model engine when configured. | #L3174-L3199 |
vllm_mlx.server.main |
function | vllm_mlx.server.main() -> not annotated |
Run the server. | #L6597-L6708 |
vllm_mlx.server.metrics |
function | async vllm_mlx.server.metrics() -> not annotated |
Prometheus scrape endpoint (disabled by default). | #L3476-L3485 |
vllm_mlx.server.PreparedChatInvocation |
class | vllm_mlx.server.PreparedChatInvocation(messages: list[dict], chat_kwargs: dict[str, object], response_format: object | None, json_logits_processor: object | None, thinking_processor: object | None = None) |
Fully prepared inputs for a single engine.chat/stream_chat call. | #L304-L311 |
vllm_mlx.server.RateLimiter |
class | vllm_mlx.server.RateLimiter(requests_per_minute: int = 60, enabled: bool = False) |
Simple in-memory rate limiter using sliding window. | #L1662-L1700 |
vllm_mlx.server.RateLimiter.__init__ |
method | vllm_mlx.server.RateLimiter.__init__(requests_per_minute: int = 60, enabled: bool = False) -> not annotated |
Method `RateLimiter.__init__` updates `self.requests_per_minute`, `self.enabled`, `self.window_size`, `self._requests`; calls `defaultdict`, `threading.Lock`. | #L1665-L1670 |
vllm_mlx.server.RateLimiter.is_allowed |
method | vllm_mlx.server.RateLimiter.is_allowed(client_id: str) -> tuple[bool, int] |
Check if request is allowed for client. | #L1672-L1700 |
vllm_mlx.server.RequestModelContext |
class | vllm_mlx.server.RequestModelContext(model_name: str, engine: BaseEngine, lease: ModelLease | None = None) |
Request-scoped engine/lease context. | #L1043-L1056 |
vllm_mlx.server.RequestModelContext.release |
method | async vllm_mlx.server.RequestModelContext.release() -> None |
Release the registry lease once, if this context owns one. | #L1050-L1056 |
vllm_mlx.server.rerank_documents |
function | async vllm_mlx.server.rerank_documents(request: RerankRequest) -> RerankResponse |
Rerank documents against a query using a cross-encoder model. | #L3920-L4038 |
vllm_mlx.server.status |
function | async vllm_mlx.server.status() -> not annotated |
Real-time status with per-request details for debugging and monitoring. | #L3548-L3597 |
vllm_mlx.server.stream_chat_completion |
function | async vllm_mlx.server.stream_chat_completion(engine: BaseEngine, messages: list, request: ChatCompletionRequest, metrics_tracker = None, **kwargs) -> AsyncIterator[str] |
Stream chat completion response. | #L6087-L6512 |
vllm_mlx.server.stream_completion |
function | async vllm_mlx.server.stream_completion(engine: BaseEngine, prompt: str, request: CompletionRequest, max_tokens: int, repetition_penalty: float | None = None, metrics_tracker = None) -> AsyncIterator[str] |
Stream completion response. | #L6003-L6084 |
vllm_mlx.server.verify_api_key |
function | async vllm_mlx.server.verify_api_key(credentials: HTTPAuthorizationCredentials = Depends(security)) -> not annotated |
Verify API key if authentication is enabled. | #L1723-L1742 |
vllm_mlx.specprefill._AttentionCapture |
class | vllm_mlx.specprefill._AttentionCapture(original, buf_idx, query_buffer, query_extractor = None) |
Wrapper that captures post-RoPE query vectors and delegates to original. | #L53-L73 |
vllm_mlx.specprefill._AttentionCapture.__call__ |
method | vllm_mlx.specprefill._AttentionCapture.__call__(x, mask = None, cache = None) -> not annotated |
Method `_AttentionCapture.__call__` calls `self._query_extractor`, `self._query_buffer[self._buf_idx].append`, `self._original`; returns `self._original(x, mask=mask, cache=cache)`. | #L67-L70 |
vllm_mlx.specprefill._AttentionCapture.__getattr__ |
method | vllm_mlx.specprefill._AttentionCapture.__getattr__(name) -> not annotated |
Method `_AttentionCapture.__getattr__` calls `getattr`; returns `getattr(self._original, name)`. | #L72-L73 |
vllm_mlx.specprefill._AttentionCapture.__init__ |
method | vllm_mlx.specprefill._AttentionCapture.__init__(original, buf_idx, query_buffer, query_extractor = None) -> not annotated |
Method `_AttentionCapture.__init__` updates `self._original`, `self._buf_idx`, `self._query_buffer`, `self._query_extractor`. | #L61-L65 |
vllm_mlx.specprefill._avg_pool1d |
function | vllm_mlx.specprefill._avg_pool1d(x, kernel_size) -> not annotated |
1D average pooling along last axis via prefix-sum. | #L207-L223 |
vllm_mlx.specprefill._build_layer_to_cache_map |
function | vllm_mlx.specprefill._build_layer_to_cache_map(model) -> not annotated |
Build mapping from model layer index to cache index. | #L666-L690 |
vllm_mlx.specprefill._compute_importance |
function | vllm_mlx.specprefill._compute_importance(query_buffer, attn_caches, n_prompt, n_attn_heads, n_kv_heads, pool_kernel = 13) -> not annotated |
Compute per-token importance from captured queries and cached keys. | #L226-L271 |
vllm_mlx.specprefill._find_attention_layers |
function | vllm_mlx.specprefill._find_attention_layers(model) -> not annotated |
Find all full-attention layers across architectures. | #L615-L630 |
vllm_mlx.specprefill._get_attn_module |
function | vllm_mlx.specprefill._get_attn_module(layer) -> not annotated |
Get the attention module from a layer (self_attn or mixer). | #L633-L639 |
vllm_mlx.specprefill._get_dims |
function | vllm_mlx.specprefill._get_dims(rope_module) -> not annotated |
Extract rotary dimensions from any RoPE variant. | #L598-L603 |
vllm_mlx.specprefill._get_pre_scale |
function | vllm_mlx.specprefill._get_pre_scale(rope_module) -> not annotated |
Extract pre-scale factor from custom RoPE variants (SuScaled, Yarn). | #L606-L612 |
vllm_mlx.specprefill._get_rope |
function | vllm_mlx.specprefill._get_rope(attn) -> not annotated |
Get the RoPE module from an attention layer, or None. | #L642-L647 |
vllm_mlx.specprefill._llama_extract_queries |
function | vllm_mlx.specprefill._llama_extract_queries(attn, x, cache = None) -> not annotated |
Extract post-RoPE queries from standard transformer attention. | #L95-L113 |
vllm_mlx.specprefill._lookahead_decode |
function | vllm_mlx.specprefill._lookahead_decode(model, first_logits, cache, n_steps, temp = 0.6, top_p = 0.95, cancel_check = None) -> not annotated |
Run n_steps autoregressive decode, returning generated token ids. | #L178-L204 |
vllm_mlx.specprefill._nemotron_h_extract_queries |
function | vllm_mlx.specprefill._nemotron_h_extract_queries(attn, x, cache = None) -> not annotated |
Extract queries from Nemotron-H attention (no RoPE, no gate, no q_norm). | #L116-L125 |
vllm_mlx.specprefill._OffsetAdjustedRoPE |
class | vllm_mlx.specprefill._OffsetAdjustedRoPE(original_rope, adjustment) |
Wraps a RoPE module to add a constant offset for decode after sparse prefill. | #L574-L590 |
vllm_mlx.specprefill._OffsetAdjustedRoPE.__call__ |
method | vllm_mlx.specprefill._OffsetAdjustedRoPE.__call__(x, offset = 0) -> not annotated |
Method `_OffsetAdjustedRoPE.__call__` calls `self._original`; returns `self._original(x, offset=offset + self._adjustment)`. | #L589-L590 |
vllm_mlx.specprefill._OffsetAdjustedRoPE.__init__ |
method | vllm_mlx.specprefill._OffsetAdjustedRoPE.__init__(original_rope, adjustment) -> not annotated |
Method `_OffsetAdjustedRoPE.__init__` updates `self._original`, `self._adjustment`. | #L585-L587 |
vllm_mlx.specprefill._patch_attention_for_capture |
function | vllm_mlx.specprefill._patch_attention_for_capture(model, query_buffer, query_extractor = None) -> not annotated |
Replace attention modules on full-attention layers with capture wrappers. | #L128-L149 |
vllm_mlx.specprefill._PositionMappedRoPE |
class | vllm_mlx.specprefill._PositionMappedRoPE(original_rope, all_positions, cache_start = 0) |
Wraps a RoPE module to apply rotation at non-contiguous positions. | #L536-L571 |
vllm_mlx.specprefill._PositionMappedRoPE.__call__ |
method | vllm_mlx.specprefill._PositionMappedRoPE.__call__(x, offset = 0) -> not annotated |
Method `_PositionMappedRoPE.__call__` calls `manual_rope_with_freqs`, `manual_rope`; has 2 explicit return paths. | #L563-L571 |
vllm_mlx.specprefill._PositionMappedRoPE.__init__ |
method | vllm_mlx.specprefill._PositionMappedRoPE.__init__(original_rope, all_positions, cache_start = 0) -> not annotated |
Method `_PositionMappedRoPE.__init__` updates `self._original`, `self._all_positions`, `self._cache_start`, `self._has_custom_freqs`; calls `hasattr`, `_get_dims`, `_get_pre_scale`. | #L547-L561 |
vllm_mlx.specprefill._prefill_draft |
function | vllm_mlx.specprefill._prefill_draft(model, tokens, cache, step_size = 2048, cancel_check = None) -> not annotated |
Prefill prompt tokens into cache. | #L158-L175 |
vllm_mlx.specprefill._qwen35_extract_queries |
function | vllm_mlx.specprefill._qwen35_extract_queries(attn, x, cache = None) -> not annotated |
Extract post-RoPE queries from Qwen3.5 attention (gate split + q_norm). | #L76-L92 |
vllm_mlx.specprefill._set_attn_module |
function | vllm_mlx.specprefill._set_attn_module(layer, module) -> not annotated |
Set the attention module on a layer (self_attn or mixer). | #L658-L663 |
vllm_mlx.specprefill._set_rope |
function | vllm_mlx.specprefill._set_rope(attn, rope_module) -> not annotated |
Set the RoPE module on an attention layer. | #L650-L655 |
vllm_mlx.specprefill._unpatch_attention_capture |
function | vllm_mlx.specprefill._unpatch_attention_capture(model, originals) -> not annotated |
Restore original attention modules after capture. | #L152-L155 |
vllm_mlx.specprefill.cleanup_rope |
function | vllm_mlx.specprefill.cleanup_rope(model) -> not annotated |
Restore original RoPE on all attention layers. | #L830-L845 |
vllm_mlx.specprefill.manual_rope |
function | vllm_mlx.specprefill.manual_rope(x, positions, dims, base = 10000.0, scale = 1.0) -> not annotated |
Apply RoPE at arbitrary (non-contiguous) positions. | #L480-L508 |
vllm_mlx.specprefill.manual_rope_with_freqs |
function | vllm_mlx.specprefill.manual_rope_with_freqs(x, positions, dims, freqs, pre_scale = 1.0) -> not annotated |
Apply RoPE at arbitrary positions using pre-computed frequencies. | #L511-L528 |
vllm_mlx.specprefill.score_tokens |
function | vllm_mlx.specprefill.score_tokens(model, tokens, n_lookahead = 8, pool_kernel = 13, temp = 0.6, top_p = 0.95, prefill_step_size = 2048, query_extractor = None, cancel_check = None) -> not annotated |
Score token importance using attention-based analysis on a draft model. | #L274-L396 |
vllm_mlx.specprefill.select_chunks |
function | vllm_mlx.specprefill.select_chunks(importance, keep_pct = 0.3, chunk_size = 32, backbone_pct = 0.0) -> not annotated |
Select top-k% token chunks by average importance. | #L399-L467 |
vllm_mlx.specprefill.select_chunks._selected_token_count |
nested function | vllm_mlx.specprefill.select_chunks._selected_token_count(chunks) -> not annotated |
Nested Function `select_chunks._selected_token_count` calls `min`; returns `total`. | #L437-L443 |
vllm_mlx.specprefill.sparse_prefill |
function | vllm_mlx.specprefill.sparse_prefill(model, tokens, selected_indices, cache, step_size = 2048, position_offset = 0, cancel_check = None) -> not annotated |
Prefill the model cache with selected tokens at their original positions. | #L698-L827 |
vllm_mlx.ssd_cache._blob_to_tokens |
function | vllm_mlx.ssd_cache._blob_to_tokens(blob: bytes) -> tuple[int, ...] |
Deserialize binary blob back to token tuple. | #L135-L139 |
vllm_mlx.ssd_cache._mx_to_numpy_safe |
function | vllm_mlx.ssd_cache._mx_to_numpy_safe(arr: Any) -> tuple[np.ndarray, str | None] |
mx.array → np.ndarray, upcasting numpy-unsupported dtypes (bf16) to fp32. | #L449-L467 |
vllm_mlx.ssd_cache._prefix_hash |
function | vllm_mlx.ssd_cache._prefix_hash(tokens: tuple[int, ...]) -> str |
Hash the bounded token prefix used to prefilter prefix lookups. | #L147-L149 |
vllm_mlx.ssd_cache._tokens_hash |
function | vllm_mlx.ssd_cache._tokens_hash(tokens: tuple[int, ...]) -> str |
Compute SHA-256 hex digest of a token sequence for use as primary key. | #L142-L144 |
vllm_mlx.ssd_cache._tokens_to_blob |
function | vllm_mlx.ssd_cache._tokens_to_blob(tokens: tuple[int, ...]) -> bytes |
Serialize token tuple to a compact binary blob for SQLite storage. | #L126-L132 |
vllm_mlx.ssd_cache.ArraysCacheSerializer |
class | vllm_mlx.ssd_cache.ArraysCacheSerializer() |
Serializer for ArraysCache (Mamba/linear attention) layers. | #L567-L627 |
vllm_mlx.ssd_cache.ArraysCacheSerializer.deserialize_layer |
method | vllm_mlx.ssd_cache.ArraysCacheSerializer.deserialize_layer(file_path: str, metadata: dict[str, Any]) -> dict |
Load arrays-cache state and any original dtype hints. | #L612-L627 |
vllm_mlx.ssd_cache.ArraysCacheSerializer.serialize_layer |
method | vllm_mlx.ssd_cache.ArraysCacheSerializer.serialize_layer(snapshot: dict[str, Any], layer_idx: int, file_path: str) -> dict[str, Any] |
Write arrays-cache state to safetensors and return its metadata. | #L589-L610 |
vllm_mlx.ssd_cache.ArraysCacheSerializer.snapshot_layer |
method | vllm_mlx.ssd_cache.ArraysCacheSerializer.snapshot_layer(layer: Any) -> dict[str, Any] |
Copy an arrays-cache state into NumPy-backed writer-thread data. | #L573-L587 |
vllm_mlx.ssd_cache.get_serializer_for_layer |
function | vllm_mlx.ssd_cache.get_serializer_for_layer(layer: Any) -> LayerSerializer |
Return the appropriate serializer for a cache layer. | #L630-L646 |
vllm_mlx.ssd_cache.KVCacheSerializer |
class | vllm_mlx.ssd_cache.KVCacheSerializer() |
Serializer for KVCache and RotatingKVCache layers. | #L470-L564 |
vllm_mlx.ssd_cache.KVCacheSerializer.deserialize_layer |
method | vllm_mlx.ssd_cache.KVCacheSerializer.deserialize_layer(file_path: str, metadata: dict[str, Any]) -> dict |
Load one KV layer as arrays plus cache reconstruction metadata. | #L544-L564 |
vllm_mlx.ssd_cache.KVCacheSerializer.serialize_layer |
method | vllm_mlx.ssd_cache.KVCacheSerializer.serialize_layer(snapshot: dict[str, Any], layer_idx: int, file_path: str) -> dict[str, Any] |
Write one KV layer to safetensors and return reconstruction metadata. | #L517-L542 |
vllm_mlx.ssd_cache.KVCacheSerializer.snapshot_layer |
method | vllm_mlx.ssd_cache.KVCacheSerializer.snapshot_layer(layer: Any) -> dict[str, Any] |
Copy a KV cache layer into NumPy-backed writer-thread data. | #L481-L515 |
vllm_mlx.ssd_cache.LayerSerializer |
class | vllm_mlx.ssd_cache.LayerSerializer() |
Interface for per-layer cache serialization. | #L419-L446 |
vllm_mlx.ssd_cache.LayerSerializer.deserialize_layer |
method | vllm_mlx.ssd_cache.LayerSerializer.deserialize_layer(file_path: str, metadata: dict[str, Any]) -> dict |
Read a layer back from disk. | #L444-L446 |
vllm_mlx.ssd_cache.LayerSerializer.serialize_layer |
method | vllm_mlx.ssd_cache.LayerSerializer.serialize_layer(snapshot: dict[str, Any], layer_idx: int, file_path: str) -> dict[str, Any] |
Writer-thread: persist a snapshot to safetensors at file_path. | #L434-L441 |
vllm_mlx.ssd_cache.LayerSerializer.snapshot_layer |
method | vllm_mlx.ssd_cache.LayerSerializer.snapshot_layer(layer: Any) -> dict[str, Any] |
Producer-thread CPU snapshot of an MLX-backed cache layer. | #L429-L431 |
vllm_mlx.ssd_cache.SSDCacheConfig |
class | vllm_mlx.ssd_cache.SSDCacheConfig(cache_dir: str | None = None, max_size_gb: float = 10.0, max_entries: int = 10000, file_permissions: int = 384, dir_permissions: int = 448, spill_queue_size: int = 64, retention_seconds: int | None = None) |
Configuration for SSD cache tier. | #L43-L78 |
vllm_mlx.ssd_cache.SSDCacheConfig.__post_init__ |
method | vllm_mlx.ssd_cache.SSDCacheConfig.__post_init__() -> None |
Method `SSDCacheConfig.__post_init__` calls `ValueError`; can raise `ValueError`. | #L65-L73 |
vllm_mlx.ssd_cache.SSDCacheConfig.max_size_bytes |
method | vllm_mlx.ssd_cache.SSDCacheConfig.max_size_bytes() -> int |
Maximum cache size in bytes. | #L76-L78 |
vllm_mlx.ssd_cache.SSDCacheStats |
class | vllm_mlx.ssd_cache.SSDCacheStats(spill_count: int = 0, spill_bytes: int = 0, ssd_hits: int = 0, ssd_misses: int = 0, reload_latency_sum: float = 0.0, reload_bytes: int = 0, promotion_failures: int = 0) |
Statistics for SSD cache tier — exposed from day one. | #L82-L123 |
vllm_mlx.ssd_cache.SSDCacheStats.to_dict |
method | vllm_mlx.ssd_cache.SSDCacheStats.to_dict() -> dict |
Return spill, lookup, reload, and promotion statistics. | #L103-L123 |
vllm_mlx.ssd_cache.SSDCacheTier |
class | vllm_mlx.ssd_cache.SSDCacheTier(config: SSDCacheConfig) |
Cold-tier disk cache for KV cache entries. | #L649-L1248 |
vllm_mlx.ssd_cache.SSDCacheTier.__init__ |
method | vllm_mlx.ssd_cache.SSDCacheTier.__init__(config: SSDCacheConfig) -> None |
Method `SSDCacheTier.__init__` updates `self._config`, `self._closed`, `self._writer_thread`, `self._cache_dir`; calls `ValueError`, `os.path.join`, `os.makedirs`, `SSDIndex`; can raise `ValueError`. | #L667-L705 |
vllm_mlx.ssd_cache.SSDCacheTier._enforce_capacity |
method | vllm_mlx.ssd_cache.SSDCacheTier._enforce_capacity() -> None |
Evict oldest SSD entries until within capacity limits. | #L1144-L1181 |
vllm_mlx.ssd_cache.SSDCacheTier._entry_hash |
method | vllm_mlx.ssd_cache.SSDCacheTier._entry_hash(tokens: tuple[int, ...]) -> str |
Compute deterministic hash for a token sequence. | #L708-L710 |
vllm_mlx.ssd_cache.SSDCacheTier._quarantine_entry |
method | vllm_mlx.ssd_cache.SSDCacheTier._quarantine_entry(tokens: tuple[int, ...], relative_path: str) -> None |
Move a corrupt entry to quarantine and remove from index. | #L1123-L1142 |
vllm_mlx.ssd_cache.SSDCacheTier._read_entry |
method | vllm_mlx.ssd_cache.SSDCacheTier._read_entry(tokens: tuple[int, ...], relative_path: str) -> list | None |
Read a cache entry from disk. | #L1078-L1121 |
vllm_mlx.ssd_cache.SSDCacheTier._write_entry |
method | vllm_mlx.ssd_cache.SSDCacheTier._write_entry(tokens_key: tuple[int, ...], layer_snapshots: list[tuple[LayerSerializer, dict[str, Any]]], memory_bytes: int) -> None |
Atomically persist one entry (writer thread; numpy-only input). | #L869-L944 |
vllm_mlx.ssd_cache.SSDCacheTier._writer_loop |
method | vllm_mlx.ssd_cache.SSDCacheTier._writer_loop() -> None |
Drain spill queue and persist entries. | #L727-L744 |
vllm_mlx.ssd_cache.SSDCacheTier.async_promote |
method | async vllm_mlx.ssd_cache.SSDCacheTier.async_promote(tokens: tuple[int, ...], reserve_budget_fn, release_budget_fn) -> list | None |
Promote an entry from SSD to RAM asynchronously. | #L970-L1076 |
vllm_mlx.ssd_cache.SSDCacheTier.close |
method | vllm_mlx.ssd_cache.SSDCacheTier.close() -> None |
Close the SSD cache tier and release resources. | #L1231-L1248 |
vllm_mlx.ssd_cache.SSDCacheTier.enqueue_spill |
method | vllm_mlx.ssd_cache.SSDCacheTier.enqueue_spill(tokens: tuple[int, ...], cache: list[Any], memory_bytes: int) -> bool |
Enqueue a cache entry for async spill to SSD. | #L746-L867 |
vllm_mlx.ssd_cache.SSDCacheTier.enqueue_spill._is_quantized_layer |
nested function | vllm_mlx.ssd_cache.SSDCacheTier.enqueue_spill._is_quantized_layer(layer) -> not annotated |
Nested Function `SSDCacheTier.enqueue_spill._is_quantized_layer` calls `isinstance`, `getattr`; has 2 explicit return paths. | #L772-L776 |
vllm_mlx.ssd_cache.SSDCacheTier.get_stats |
method | vllm_mlx.ssd_cache.SSDCacheTier.get_stats() -> dict |
Return current SSD cache statistics. | #L712-L714 |
vllm_mlx.ssd_cache.SSDCacheTier.lookup_ssd |
method | vllm_mlx.ssd_cache.SSDCacheTier.lookup_ssd(tokens: tuple[int, ...]) -> dict | None |
Synchronous check whether tokens exist in SSD tier. | #L946-L958 |
vllm_mlx.ssd_cache.SSDCacheTier.lookup_ssd_prefix |
method | vllm_mlx.ssd_cache.SSDCacheTier.lookup_ssd_prefix(tokens: tuple[int, ...]) -> dict | None |
Find the longest prefix match in the SSD tier. | #L960-L968 |
vllm_mlx.ssd_cache.SSDCacheTier.reconcile |
method | vllm_mlx.ssd_cache.SSDCacheTier.reconcile() -> int |
Reconcile index with files on disk. | #L1183-L1229 |
vllm_mlx.ssd_cache.SSDCacheTier.start_writer |
method | vllm_mlx.ssd_cache.SSDCacheTier.start_writer() -> None |
Start the background spill writer thread. | #L716-L725 |
vllm_mlx.ssd_cache.SSDIndex |
class | vllm_mlx.ssd_cache.SSDIndex(cache_dir: str) |
SQLite-backed index for SSD cache entries. | #L152-L405 |
vllm_mlx.ssd_cache.SSDIndex.__init__ |
method | vllm_mlx.ssd_cache.SSDIndex.__init__(cache_dir: str) -> None |
Method `SSDIndex.__init__` updates `self._cache_dir`, `self._db_lock`, `self._conn`, `self._conn.row_factory`; calls `threading.Lock`, `os.path.join`, `sqlite3.connect`, `self._conn.execute`. | #L165-L173 |
vllm_mlx.ssd_cache.SSDIndex._backfill_prefix_hashes |
method | vllm_mlx.ssd_cache.SSDIndex._backfill_prefix_hashes() -> None |
Method `SSDIndex._backfill_prefix_hashes` calls `self._conn.execute`, `cur.fetchall`, `_blob_to_tokens`, `_prefix_hash`. | #L220-L230 |
vllm_mlx.ssd_cache.SSDIndex._create_tables |
method | vllm_mlx.ssd_cache.SSDIndex._create_tables() -> None |
Method `SSDIndex._create_tables` calls `self._conn.executescript`, `self._ensure_column`, `self._conn.execute`, `cur.fetchone`. | #L175-L213 |
vllm_mlx.ssd_cache.SSDIndex._ensure_column |
method | vllm_mlx.ssd_cache.SSDIndex._ensure_column(table: str, column: str, definition: str) -> None |
Method `SSDIndex._ensure_column` calls `self._conn.execute`, `cur.fetchall`. | #L215-L218 |
vllm_mlx.ssd_cache.SSDIndex.all_entries |
method | vllm_mlx.ssd_cache.SSDIndex.all_entries() -> list[dict] |
Return all entries (for startup reconciliation). | #L381-L400 |
vllm_mlx.ssd_cache.SSDIndex.close |
method | vllm_mlx.ssd_cache.SSDIndex.close() -> None |
Close the SQLite connection. | #L402-L405 |
vllm_mlx.ssd_cache.SSDIndex.delete_entry |
method | vllm_mlx.ssd_cache.SSDIndex.delete_entry(tokens_key: tuple[int, ...]) -> None |
Delete an entry by token sequence. | #L326-L333 |
vllm_mlx.ssd_cache.SSDIndex.get_entry_count |
method | vllm_mlx.ssd_cache.SSDIndex.get_entry_count() -> int |
Get number of entries in the index. | #L365-L369 |
vllm_mlx.ssd_cache.SSDIndex.get_lru |
method | vllm_mlx.ssd_cache.SSDIndex.get_lru(limit: int = 10) -> list[dict] |
Get the least recently used entries, ordered oldest first. | #L335-L355 |
vllm_mlx.ssd_cache.SSDIndex.get_total_bytes |
method | vllm_mlx.ssd_cache.SSDIndex.get_total_bytes() -> int |
Get total memory_bytes across all entries. | #L357-L363 |
vllm_mlx.ssd_cache.SSDIndex.insert_entry |
method | vllm_mlx.ssd_cache.SSDIndex.insert_entry(tokens_key: tuple[int, ...], file_path: str, memory_bytes: int, num_tokens: int) -> None |
Insert or replace a cache entry in the index. | #L232-L263 |
vllm_mlx.ssd_cache.SSDIndex.lookup_exact |
method | vllm_mlx.ssd_cache.SSDIndex.lookup_exact(tokens_key: tuple[int, ...]) -> dict | None |
Look up an exact token sequence. | #L265-L280 |
vllm_mlx.ssd_cache.SSDIndex.lookup_prefix |
method | vllm_mlx.ssd_cache.SSDIndex.lookup_prefix(query_tokens: tuple[int, ...]) -> list[dict] |
Find entries whose token sequence is a prefix of query_tokens. | #L282-L324 |
vllm_mlx.ssd_cache.SSDIndex.touch |
method | vllm_mlx.ssd_cache.SSDIndex.touch(tokens_key: tuple[int, ...]) -> None |
Update accessed_at timestamp for an entry (marks as recently used). | #L371-L379 |
vllm_mlx.text_model_from_vlm._import_text_model_classes |
function | vllm_mlx.text_model_from_vlm._import_text_model_classes(model_type: str) -> not annotated |
Return ``(Model, ModelArgs)`` for a text config's ``model_type``. | #L41-L66 |
vllm_mlx.text_model_from_vlm._load_mtp_weights |
function | vllm_mlx.text_model_from_vlm._load_mtp_weights(model_path: Path) -> list[tuple[str, mx.array]] |
Load MTP weights from safetensors, stripping the language_model. | #L228-L272 |
vllm_mlx.text_model_from_vlm.build_text_model |
function | vllm_mlx.text_model_from_vlm.build_text_model(vlm_model: Any, model_path: str | Path) -> Any | None |
Build an mlx_lm TextModel from a vlm-loaded model's weights. | #L69-L225 |
vllm_mlx.text_model_from_vlm.build_text_model._class_predicate |
nested function | vllm_mlx.text_model_from_vlm.build_text_model._class_predicate(path, module) -> not annotated |
Nested Function `build_text_model._class_predicate` calls `hasattr`, `per_layer_overrides.items`, `key.endswith`; has 4 explicit return paths. | #L127-L137 |
vllm_mlx.tool_parsers.abstract_tool_parser.ExtractedToolCallInformation |
class | vllm_mlx.tool_parsers.abstract_tool_parser.ExtractedToolCallInformation(tools_called: bool, tool_calls: list[dict[str, Any]], content: str | None = None) |
Information extracted from model output about tool calls. | #L27-L37 |
vllm_mlx.tool_parsers.abstract_tool_parser.ToolParser |
class | vllm_mlx.tool_parsers.abstract_tool_parser.ToolParser(tokenizer: PreTrainedTokenizerBase | None = None) |
Abstract base class for tool call parsers. | #L40-L171 |
vllm_mlx.tool_parsers.abstract_tool_parser.ToolParser.__init__ |
method | vllm_mlx.tool_parsers.abstract_tool_parser.ToolParser.__init__(tokenizer: PreTrainedTokenizerBase | None = None) -> not annotated |
Initialize the tool parser. | #L103-L113 |
vllm_mlx.tool_parsers.abstract_tool_parser.ToolParser.extract_tool_calls |
method | vllm_mlx.tool_parsers.abstract_tool_parser.ToolParser.extract_tool_calls(model_output: str, request: dict[str, Any] | None = None) -> ExtractedToolCallInformation |
Extract tool calls from a complete model response. | #L123-L136 |
vllm_mlx.tool_parsers.abstract_tool_parser.ToolParser.extract_tool_calls_streaming |
method | vllm_mlx.tool_parsers.abstract_tool_parser.ToolParser.extract_tool_calls_streaming(previous_text: str, current_text: str, delta_text: str, previous_token_ids: Sequence[int] | None = None, current_token_ids: Sequence[int] | None = None, delta_token_ids: Sequence[int] | None = None, request: dict[str, Any] | None = None) -> dict[str, Any] | None |
Extract tool calls from streaming model output. | #L138-L166 |
vllm_mlx.tool_parsers.abstract_tool_parser.ToolParser.reset |
method | vllm_mlx.tool_parsers.abstract_tool_parser.ToolParser.reset() -> None |
Reset parser state for a new request. | #L168-L171 |
vllm_mlx.tool_parsers.abstract_tool_parser.ToolParser.strip_think_tags |
method | vllm_mlx.tool_parsers.abstract_tool_parser.ToolParser.strip_think_tags(text: str) -> str |
Strip think tags from text. | #L75-L101 |
vllm_mlx.tool_parsers.abstract_tool_parser.ToolParser.supports_native_format |
method | vllm_mlx.tool_parsers.abstract_tool_parser.ToolParser.supports_native_format() -> bool |
Check if this parser supports native tool message format. | #L60-L72 |
vllm_mlx.tool_parsers.abstract_tool_parser.ToolParser.vocab |
method | vllm_mlx.tool_parsers.abstract_tool_parser.ToolParser.vocab() -> dict[str, int] |
Get the tokenizer vocabulary. | #L116-L120 |
vllm_mlx.tool_parsers.abstract_tool_parser.ToolParserManager |
class | vllm_mlx.tool_parsers.abstract_tool_parser.ToolParserManager() |
Central registry for ToolParser implementations. | #L174-L286 |
vllm_mlx.tool_parsers.abstract_tool_parser.ToolParserManager._load_lazy_parser |
method | vllm_mlx.tool_parsers.abstract_tool_parser.ToolParserManager._load_lazy_parser(name: str) -> type[ToolParser] |
Import and register a lazily loaded parser. | #L210-L225 |
vllm_mlx.tool_parsers.abstract_tool_parser.ToolParserManager.get_tool_parser |
method | vllm_mlx.tool_parsers.abstract_tool_parser.ToolParserManager.get_tool_parser(name: str) -> type[ToolParser] |
Retrieve a registered ToolParser class by name. | #L185-L207 |
vllm_mlx.tool_parsers.abstract_tool_parser.ToolParserManager.list_registered |
method | vllm_mlx.tool_parsers.abstract_tool_parser.ToolParserManager.list_registered() -> list[str] |
Return names of all registered tool parsers. | #L284-L286 |
vllm_mlx.tool_parsers.abstract_tool_parser.ToolParserManager.register_lazy_module |
method | vllm_mlx.tool_parsers.abstract_tool_parser.ToolParserManager.register_lazy_module(name: str, module_path: str, class_name: str) -> None |
Register a lazy module mapping for deferred loading. | #L272-L281 |
vllm_mlx.tool_parsers.abstract_tool_parser.ToolParserManager.register_module |
method | vllm_mlx.tool_parsers.abstract_tool_parser.ToolParserManager.register_module(name: str | list[str], module: type[ToolParser] | None = None, force: bool = True) -> type[ToolParser] | None |
Register a ToolParser class. | #L228-L269 |
vllm_mlx.tool_parsers.abstract_tool_parser.ToolParserManager.register_module.decorator |
nested function | vllm_mlx.tool_parsers.abstract_tool_parser.ToolParserManager.register_module.decorator(parser_cls: type[ToolParser]) -> type[ToolParser] |
Nested Function `ToolParserManager.register_module.decorator` calls `KeyError`; can raise `KeyError`; returns `parser_cls`. | #L262-L267 |
vllm_mlx.tool_parsers.auto_tool_parser.AutoToolParser |
class | vllm_mlx.tool_parsers.auto_tool_parser.AutoToolParser() |
Auto-detecting tool call parser. | #L28-L414 |
vllm_mlx.tool_parsers.auto_tool_parser.AutoToolParser._parse_raw_json_tool_calls |
method | vllm_mlx.tool_parsers.auto_tool_parser.AutoToolParser._parse_raw_json_tool_calls(text: str) -> list[dict[str, Any]] |
Parse raw JSON tool calls from text. | #L270-L350 |
vllm_mlx.tool_parsers.auto_tool_parser.AutoToolParser.extract_tool_calls |
method | vllm_mlx.tool_parsers.auto_tool_parser.AutoToolParser.extract_tool_calls(model_output: str, request: dict[str, Any] | None = None) -> ExtractedToolCallInformation |
Extract tool calls by trying all known formats. | #L61-L268 |
vllm_mlx.tool_parsers.auto_tool_parser.AutoToolParser.extract_tool_calls_streaming |
method | vllm_mlx.tool_parsers.auto_tool_parser.AutoToolParser.extract_tool_calls_streaming(previous_text: str, current_text: str, delta_text: str, previous_token_ids: Sequence[int] | None = None, current_token_ids: Sequence[int] | None = None, delta_token_ids: Sequence[int] | None = None, request: dict[str, Any] | None = None) -> dict[str, Any] | None |
Extract tool calls from streaming model output. | #L352-L414 |
vllm_mlx.tool_parsers.auto_tool_parser.generate_tool_id |
function | vllm_mlx.tool_parsers.auto_tool_parser.generate_tool_id() -> str |
Generate a unique tool call ID. | #L22-L24 |
vllm_mlx.tool_parsers.deepseek_tool_parser.DeepSeekToolParser |
class | vllm_mlx.tool_parsers.deepseek_tool_parser.DeepSeekToolParser() |
Tool call parser for DeepSeek V3 and R1 models. | #L32-L170 |
vllm_mlx.tool_parsers.deepseek_tool_parser.DeepSeekToolParser.extract_tool_calls |
method | vllm_mlx.tool_parsers.deepseek_tool_parser.DeepSeekToolParser.extract_tool_calls(model_output: str, request: dict[str, Any] | None = None) -> ExtractedToolCallInformation |
Extract tool calls from DeepSeek model output. | #L69-L133 |
vllm_mlx.tool_parsers.deepseek_tool_parser.DeepSeekToolParser.extract_tool_calls_streaming |
method | vllm_mlx.tool_parsers.deepseek_tool_parser.DeepSeekToolParser.extract_tool_calls_streaming(previous_text: str, current_text: str, delta_text: str, previous_token_ids: Sequence[int] | None = None, current_token_ids: Sequence[int] | None = None, delta_token_ids: Sequence[int] | None = None, request: dict[str, Any] | None = None) -> dict[str, Any] | None |
Extract tool calls from streaming DeepSeek model output. | #L135-L170 |
vllm_mlx.tool_parsers.deepseek_tool_parser.generate_tool_id |
function | vllm_mlx.tool_parsers.deepseek_tool_parser.generate_tool_id() -> str |
Generate a unique tool call ID. | #L26-L28 |
vllm_mlx.tool_parsers.functionary_tool_parser.FunctionaryToolParser |
class | vllm_mlx.tool_parsers.functionary_tool_parser.FunctionaryToolParser() |
Tool call parser for MeetKai Functionary models. | #L28-L193 |
vllm_mlx.tool_parsers.functionary_tool_parser.FunctionaryToolParser.extract_tool_calls |
method | vllm_mlx.tool_parsers.functionary_tool_parser.FunctionaryToolParser.extract_tool_calls(model_output: str, request: dict[str, Any] | None = None) -> ExtractedToolCallInformation |
Extract tool calls from Functionary model output. | #L61-L153 |
vllm_mlx.tool_parsers.functionary_tool_parser.FunctionaryToolParser.extract_tool_calls_streaming |
method | vllm_mlx.tool_parsers.functionary_tool_parser.FunctionaryToolParser.extract_tool_calls_streaming(previous_text: str, current_text: str, delta_text: str, previous_token_ids: Sequence[int] | None = None, current_token_ids: Sequence[int] | None = None, delta_token_ids: Sequence[int] | None = None, request: dict[str, Any] | None = None) -> dict[str, Any] | None |
Extract tool calls from streaming Functionary model output. | #L155-L193 |
vllm_mlx.tool_parsers.functionary_tool_parser.generate_tool_id |
function | vllm_mlx.tool_parsers.functionary_tool_parser.generate_tool_id() -> str |
Generate a unique tool call ID. | #L22-L24 |
vllm_mlx.tool_parsers.gemma4_tool_parser._call_node_to_tool |
function | vllm_mlx.tool_parsers.gemma4_tool_parser._call_node_to_tool(call: ast.Call) -> tuple[str, dict[str, Any]] | None |
Map a Python `ast.Call` node to (function_name, kwargs_dict). | #L210-L233 |
vllm_mlx.tool_parsers.gemma4_tool_parser._find_balanced_brace |
function | vllm_mlx.tool_parsers.gemma4_tool_parser._find_balanced_brace(text: str, start: int) -> int |
Find the index of the closing } that balances the { at `start`. | #L92-L125 |
vllm_mlx.tool_parsers.gemma4_tool_parser._find_balanced_paren |
function | vllm_mlx.tool_parsers.gemma4_tool_parser._find_balanced_paren(text: str, start: int) -> int |
Find the index of the closing ) that balances the ( at `start`. | #L128-L159 |
vllm_mlx.tool_parsers.gemma4_tool_parser._gemma4_args_to_json |
function | vllm_mlx.tool_parsers.gemma4_tool_parser._gemma4_args_to_json(text: str) -> str |
Convert Gemma 4 tool call args to valid JSON. | #L171-L207 |
vllm_mlx.tool_parsers.gemma4_tool_parser._gemma4_args_to_json._capture |
nested function | vllm_mlx.tool_parsers.gemma4_tool_parser._gemma4_args_to_json._capture(m: re.Match) -> str |
Nested Function `_gemma4_args_to_json._capture` calls `strings.append`, `m.group`, `len`; returns `f'\x00{len(strings) - 1}\x00'`. | #L187-L189 |
vllm_mlx.tool_parsers.gemma4_tool_parser._gemma4_args_to_json._restore |
nested function | vllm_mlx.tool_parsers.gemma4_tool_parser._gemma4_args_to_json._restore(m: re.Match) -> str |
Nested Function `_gemma4_args_to_json._restore` calls `int`, `m.group`, `len`, `json.dumps`; returns `json.dumps(strings[idx]) if idx < len(strings) else m.group(0)`. | #L201-L203 |
vllm_mlx.tool_parsers.gemma4_tool_parser._parse_calls_from_code |
function | vllm_mlx.tool_parsers.gemma4_tool_parser._parse_calls_from_code(code: str) -> list[tuple[str, dict[str, Any]]] |
Parse every top-level `fn(...)` call statement in a code-fence body. | #L247-L259 |
vllm_mlx.tool_parsers.gemma4_tool_parser._parse_python_call |
function | vllm_mlx.tool_parsers.gemma4_tool_parser._parse_python_call(src: str) -> tuple[str, dict[str, Any]] | None |
Parse a single `fn(...)` Python call expression into (name, kwargs). | #L236-L244 |
vllm_mlx.tool_parsers.gemma4_tool_parser._quote_bare_value |
function | vllm_mlx.tool_parsers.gemma4_tool_parser._quote_bare_value(m: re.Match) -> str |
Substitution callback for _BARE_VALUE — quotes bare identifiers that are not JSON literals (true/false/null). | #L162-L168 |
vllm_mlx.tool_parsers.gemma4_tool_parser._strip_spans |
function | vllm_mlx.tool_parsers.gemma4_tool_parser._strip_spans(text: str, spans: list[tuple[int, int]]) -> str |
Remove the given [start, end) spans from `text` (handles overlaps). | #L262-L275 |
vllm_mlx.tool_parsers.gemma4_tool_parser.Gemma4ToolParser |
class | vllm_mlx.tool_parsers.gemma4_tool_parser.Gemma4ToolParser() |
Tool call parser for Gemma 4 models. | #L284-L513 |
vllm_mlx.tool_parsers.gemma4_tool_parser.Gemma4ToolParser._extract_canonical |
method | vllm_mlx.tool_parsers.gemma4_tool_parser.Gemma4ToolParser._extract_canonical(cleaned: str) -> tuple[list[dict[str, Any]], str | None] |
Parse the canonical <|tool_call>call:fn{...}<tool_call|> form. | #L326-L382 |
vllm_mlx.tool_parsers.gemma4_tool_parser.Gemma4ToolParser._extract_fallback |
method | vllm_mlx.tool_parsers.gemma4_tool_parser.Gemma4ToolParser._extract_fallback(cleaned: str) -> ExtractedToolCallInformation | None |
Parse the Python-style fallback forms (issue #80). | #L384-L463 |
vllm_mlx.tool_parsers.gemma4_tool_parser.Gemma4ToolParser._format_streaming |
method | vllm_mlx.tool_parsers.gemma4_tool_parser.Gemma4ToolParser._format_streaming(result: ExtractedToolCallInformation) -> dict[str, Any] |
Render extracted tool calls into the streaming delta shape. | #L465-L480 |
vllm_mlx.tool_parsers.gemma4_tool_parser.Gemma4ToolParser.extract_tool_calls |
method | vllm_mlx.tool_parsers.gemma4_tool_parser.Gemma4ToolParser.extract_tool_calls(model_output: str, request: dict[str, Any] | None = None) -> ExtractedToolCallInformation |
Extract tool calls from a complete Gemma 4 model response. | #L300-L324 |
vllm_mlx.tool_parsers.gemma4_tool_parser.Gemma4ToolParser.extract_tool_calls_streaming |
method | vllm_mlx.tool_parsers.gemma4_tool_parser.Gemma4ToolParser.extract_tool_calls_streaming(previous_text: str, current_text: str, delta_text: str, previous_token_ids: Sequence[int] | None = None, current_token_ids: Sequence[int] | None = None, delta_token_ids: Sequence[int] | None = None, request: dict[str, Any] | None = None) -> dict[str, Any] | None |
Extract tool calls from streaming Gemma 4 model output. | #L482-L513 |
vllm_mlx.tool_parsers.gemma4_tool_parser.generate_tool_id |
function | vllm_mlx.tool_parsers.gemma4_tool_parser.generate_tool_id() -> str |
Generate a unique tool call ID. | #L278-L280 |
vllm_mlx.tool_parsers.get_parser_stop_tokens |
function | vllm_mlx.tool_parsers.get_parser_stop_tokens(parser_name: str | None, user_stops: list[str] | None) -> list[str] |
Merge user-supplied stops with parser-declared extras (deduped). | #L68-L88 |
vllm_mlx.tool_parsers.glm47_tool_parser.generate_tool_id |
function | vllm_mlx.tool_parsers.glm47_tool_parser.generate_tool_id() -> str |
Generate a unique tool call ID. | #L22-L24 |
vllm_mlx.tool_parsers.glm47_tool_parser.Glm47ToolParser |
class | vllm_mlx.tool_parsers.glm47_tool_parser.Glm47ToolParser() |
Tool call parser for GLM-4.7 and GLM-4.7-Flash models. | #L28-L184 |
vllm_mlx.tool_parsers.glm47_tool_parser.Glm47ToolParser._deserialize |
method | vllm_mlx.tool_parsers.glm47_tool_parser.Glm47ToolParser._deserialize(value: str) -> Any |
Convert string value to appropriate Python type. | #L57-L67 |
vllm_mlx.tool_parsers.glm47_tool_parser.Glm47ToolParser._get_tool_names |
method | vllm_mlx.tool_parsers.glm47_tool_parser.Glm47ToolParser._get_tool_names(request: dict[str, Any] | None) -> set[str] |
Extract valid tool names from the request. | #L69-L77 |
vllm_mlx.tool_parsers.glm47_tool_parser.Glm47ToolParser.extract_tool_calls |
method | vllm_mlx.tool_parsers.glm47_tool_parser.Glm47ToolParser.extract_tool_calls(model_output: str, request: dict[str, Any] | None = None) -> ExtractedToolCallInformation |
Extract tool calls from a complete GLM-4.7 model response. | #L79-L137 |
vllm_mlx.tool_parsers.glm47_tool_parser.Glm47ToolParser.extract_tool_calls_streaming |
method | vllm_mlx.tool_parsers.glm47_tool_parser.Glm47ToolParser.extract_tool_calls_streaming(previous_text: str, current_text: str, delta_text: str, previous_token_ids: Sequence[int] | None = None, current_token_ids: Sequence[int] | None = None, delta_token_ids: Sequence[int] | None = None, request: dict[str, Any] | None = None) -> dict[str, Any] | None |
Extract tool calls from streaming GLM-4.7 model output. | #L139-L184 |
vllm_mlx.tool_parsers.granite_tool_parser.generate_tool_id |
function | vllm_mlx.tool_parsers.granite_tool_parser.generate_tool_id() -> str |
Generate a unique tool call ID. | #L22-L24 |
vllm_mlx.tool_parsers.granite_tool_parser.GraniteToolParser |
class | vllm_mlx.tool_parsers.granite_tool_parser.GraniteToolParser() |
Tool call parser for IBM Granite models. | #L28-L147 |
vllm_mlx.tool_parsers.granite_tool_parser.GraniteToolParser.extract_tool_calls |
method | vllm_mlx.tool_parsers.granite_tool_parser.GraniteToolParser.extract_tool_calls(model_output: str, request: dict[str, Any] | None = None) -> ExtractedToolCallInformation |
Extract tool calls from Granite model output. | #L47-L105 |
vllm_mlx.tool_parsers.granite_tool_parser.GraniteToolParser.extract_tool_calls_streaming |
method | vllm_mlx.tool_parsers.granite_tool_parser.GraniteToolParser.extract_tool_calls_streaming(previous_text: str, current_text: str, delta_text: str, previous_token_ids: Sequence[int] | None = None, current_token_ids: Sequence[int] | None = None, delta_token_ids: Sequence[int] | None = None, request: dict[str, Any] | None = None) -> dict[str, Any] | None |
Extract tool calls from streaming Granite model output. | #L107-L147 |
vllm_mlx.tool_parsers.harmony_tool_parser._generate_tool_id |
function | vllm_mlx.tool_parsers.harmony_tool_parser._generate_tool_id() -> str |
Generate a unique tool call ID. | #L32-L34 |
vllm_mlx.tool_parsers.harmony_tool_parser._is_control_token |
function | vllm_mlx.tool_parsers.harmony_tool_parser._is_control_token(text: str) -> bool |
Check if text is a Harmony control token. | #L243-L253 |
vllm_mlx.tool_parsers.harmony_tool_parser._strip_control_tokens |
function | vllm_mlx.tool_parsers.harmony_tool_parser._strip_control_tokens(text: str) -> str |
Remove Harmony control tokens from text. | #L222-L240 |
vllm_mlx.tool_parsers.harmony_tool_parser.HarmonyToolParser |
class | vllm_mlx.tool_parsers.harmony_tool_parser.HarmonyToolParser() |
Tool call parser for GPT-OSS models using Harmony format. | #L57-L219 |
vllm_mlx.tool_parsers.harmony_tool_parser.HarmonyToolParser.extract_tool_calls |
method | vllm_mlx.tool_parsers.harmony_tool_parser.HarmonyToolParser.extract_tool_calls(model_output: str, request: dict[str, Any] | None = None) -> ExtractedToolCallInformation |
Extract tool calls from a complete Harmony model response. | #L71-L140 |
vllm_mlx.tool_parsers.harmony_tool_parser.HarmonyToolParser.extract_tool_calls_streaming |
method | vllm_mlx.tool_parsers.harmony_tool_parser.HarmonyToolParser.extract_tool_calls_streaming(previous_text: str, current_text: str, delta_text: str, previous_token_ids: Sequence[int] | None = None, current_token_ids: Sequence[int] | None = None, delta_token_ids: Sequence[int] | None = None, request: dict[str, Any] | None = None) -> dict[str, Any] | None |
Extract tool calls from streaming Harmony model output. | #L142-L214 |
vllm_mlx.tool_parsers.harmony_tool_parser.HarmonyToolParser.reset |
method | vllm_mlx.tool_parsers.harmony_tool_parser.HarmonyToolParser.reset() -> None |
Reset parser state for a new request. | #L216-L219 |
vllm_mlx.tool_parsers.hermes_tool_parser._parse_param_value |
function | vllm_mlx.tool_parsers.hermes_tool_parser._parse_param_value(val: str) -> Any |
Parse a tool call parameter value, handling both JSON and Python literals. | #L27-L49 |
vllm_mlx.tool_parsers.hermes_tool_parser.generate_tool_id |
function | vllm_mlx.tool_parsers.hermes_tool_parser.generate_tool_id() -> str |
Generate a unique tool call ID. | #L22-L24 |
vllm_mlx.tool_parsers.hermes_tool_parser.HermesToolParser |
class | vllm_mlx.tool_parsers.hermes_tool_parser.HermesToolParser() |
Tool call parser for Hermes/Nous models. | #L53-L336 |
vllm_mlx.tool_parsers.hermes_tool_parser.HermesToolParser._format_streaming_tool_calls |
method | vllm_mlx.tool_parsers.hermes_tool_parser.HermesToolParser._format_streaming_tool_calls(tool_calls: list[dict], start_index: int = 0) -> dict[str, Any] |
Format tool calls for streaming response. | #L248-L265 |
vllm_mlx.tool_parsers.hermes_tool_parser.HermesToolParser.extract_tool_calls |
method | vllm_mlx.tool_parsers.hermes_tool_parser.HermesToolParser.extract_tool_calls(model_output: str, request: dict[str, Any] | None = None) -> ExtractedToolCallInformation |
Extract tool calls from a complete Hermes model response. | #L92-L245 |
vllm_mlx.tool_parsers.hermes_tool_parser.HermesToolParser.extract_tool_calls_streaming |
method | vllm_mlx.tool_parsers.hermes_tool_parser.HermesToolParser.extract_tool_calls_streaming(previous_text: str, current_text: str, delta_text: str, previous_token_ids: Sequence[int] | None = None, current_token_ids: Sequence[int] | None = None, delta_token_ids: Sequence[int] | None = None, request: dict[str, Any] | None = None) -> dict[str, Any] | None |
Extract tool calls from streaming Hermes model output. | #L267-L336 |
vllm_mlx.tool_parsers.kimi_tool_parser.generate_tool_id |
function | vllm_mlx.tool_parsers.kimi_tool_parser.generate_tool_id() -> str |
Generate a unique tool call ID. | #L23-L25 |
vllm_mlx.tool_parsers.kimi_tool_parser.KimiToolParser |
class | vllm_mlx.tool_parsers.kimi_tool_parser.KimiToolParser() |
Tool call parser for Kimi K2 and Moonshot models. | #L29-L160 |
vllm_mlx.tool_parsers.kimi_tool_parser.KimiToolParser._has_tool_section |
method | vllm_mlx.tool_parsers.kimi_tool_parser.KimiToolParser._has_tool_section(text: str) -> bool |
Check if text contains tool section markers. | #L59-L65 |
vllm_mlx.tool_parsers.kimi_tool_parser.KimiToolParser.extract_tool_calls |
method | vllm_mlx.tool_parsers.kimi_tool_parser.KimiToolParser.extract_tool_calls(model_output: str, request: dict[str, Any] | None = None) -> ExtractedToolCallInformation |
Extract tool calls from Kimi model output. | #L67-L124 |
vllm_mlx.tool_parsers.kimi_tool_parser.KimiToolParser.extract_tool_calls_streaming |
method | vllm_mlx.tool_parsers.kimi_tool_parser.KimiToolParser.extract_tool_calls_streaming(previous_text: str, current_text: str, delta_text: str, previous_token_ids: Sequence[int] | None = None, current_token_ids: Sequence[int] | None = None, delta_token_ids: Sequence[int] | None = None, request: dict[str, Any] | None = None) -> dict[str, Any] | None |
Extract tool calls from streaming Kimi model output. | #L126-L160 |
vllm_mlx.tool_parsers.llama_tool_parser.generate_tool_id |
function | vllm_mlx.tool_parsers.llama_tool_parser.generate_tool_id() -> str |
Generate a unique tool call ID. | #L22-L24 |
vllm_mlx.tool_parsers.llama_tool_parser.LlamaToolParser |
class | vllm_mlx.tool_parsers.llama_tool_parser.LlamaToolParser() |
Tool call parser for Llama models. | #L28-L128 |
vllm_mlx.tool_parsers.llama_tool_parser.LlamaToolParser.extract_tool_calls |
method | vllm_mlx.tool_parsers.llama_tool_parser.LlamaToolParser.extract_tool_calls(model_output: str, request: dict[str, Any] | None = None) -> ExtractedToolCallInformation |
Extract tool calls from a complete Llama model response. | #L44-L90 |
vllm_mlx.tool_parsers.llama_tool_parser.LlamaToolParser.extract_tool_calls_streaming |
method | vllm_mlx.tool_parsers.llama_tool_parser.LlamaToolParser.extract_tool_calls_streaming(previous_text: str, current_text: str, delta_text: str, previous_token_ids: Sequence[int] | None = None, current_token_ids: Sequence[int] | None = None, delta_token_ids: Sequence[int] | None = None, request: dict[str, Any] | None = None) -> dict[str, Any] | None |
Extract tool calls from streaming Llama model output. | #L92-L128 |
vllm_mlx.tool_parsers.minimax_tool_parser.generate_tool_id |
function | vllm_mlx.tool_parsers.minimax_tool_parser.generate_tool_id() -> str |
Return a short OpenAI-compatible identifier for a parsed tool call. | #L26-L29 |
vllm_mlx.tool_parsers.minimax_tool_parser.MiniMaxToolParser |
class | vllm_mlx.tool_parsers.minimax_tool_parser.MiniMaxToolParser() |
Parser for MiniMax-M2 tool call format. | #L33-L178 |
vllm_mlx.tool_parsers.minimax_tool_parser.MiniMaxToolParser._extract_invokes |
method | vllm_mlx.tool_parsers.minimax_tool_parser.MiniMaxToolParser._extract_invokes(text: str) -> list[dict[str, Any]] |
Extract tool calls from invoke elements, with or without wrapper. | #L54-L78 |
vllm_mlx.tool_parsers.minimax_tool_parser.MiniMaxToolParser._has_tool_end |
method | vllm_mlx.tool_parsers.minimax_tool_parser.MiniMaxToolParser._has_tool_end(current: str, previous: str) -> bool |
Check if a tool call block just completed. | #L129-L140 |
vllm_mlx.tool_parsers.minimax_tool_parser.MiniMaxToolParser._has_tool_start |
method | vllm_mlx.tool_parsers.minimax_tool_parser.MiniMaxToolParser._has_tool_start(text: str) -> bool |
Check if text contains the start of a tool call block. | #L123-L127 |
vllm_mlx.tool_parsers.minimax_tool_parser.MiniMaxToolParser.extract_tool_calls |
method | vllm_mlx.tool_parsers.minimax_tool_parser.MiniMaxToolParser.extract_tool_calls(model_output: str, request: dict[str, Any] | None = None) -> ExtractedToolCallInformation |
Extract wrapped or bare MiniMax invoke elements from complete output. | #L80-L121 |
vllm_mlx.tool_parsers.minimax_tool_parser.MiniMaxToolParser.extract_tool_calls_streaming |
method | vllm_mlx.tool_parsers.minimax_tool_parser.MiniMaxToolParser.extract_tool_calls_streaming(previous_text: str, current_text: str, delta_text: str, previous_token_ids: Sequence[int] | None = None, current_token_ids: Sequence[int] | None = None, delta_token_ids: Sequence[int] | None = None, request: dict[str, Any] | None = None) -> dict[str, Any] | None |
Emit content deltas or a completed MiniMax tool-call delta. | #L142-L178 |
vllm_mlx.tool_parsers.mistral_tool_parser._is_plain_tool_name |
function | vllm_mlx.tool_parsers.mistral_tool_parser._is_plain_tool_name(name: str) -> bool |
Return True for names that are safe to dispatch as function calls. | #L43-L45 |
vllm_mlx.tool_parsers.mistral_tool_parser.generate_mistral_tool_id |
function | vllm_mlx.tool_parsers.mistral_tool_parser.generate_mistral_tool_id() -> str |
Generate a random Mistral-compatible tool call ID. | #L34-L40 |
vllm_mlx.tool_parsers.mistral_tool_parser.MistralToolParser |
class | vllm_mlx.tool_parsers.mistral_tool_parser.MistralToolParser(tokenizer = None) |
Tool call parser for Mistral models. | #L49-L512 |
vllm_mlx.tool_parsers.mistral_tool_parser.MistralToolParser.__init__ |
method | vllm_mlx.tool_parsers.mistral_tool_parser.MistralToolParser.__init__(tokenizer = None) -> not annotated |
Method `MistralToolParser.__init__` updates `self.bot_token_id`, `self._args_started`, `self._args_in_string`, `self._args_escaped`; calls `super().__init__`, `super`, `self.vocab.get`. | #L68-L90 |
vllm_mlx.tool_parsers.mistral_tool_parser.MistralToolParser._parse_streaming_tool_delta |
method | vllm_mlx.tool_parsers.mistral_tool_parser.MistralToolParser._parse_streaming_tool_delta(text: str) -> dict[str, str] | None |
Parse a streaming delta for tool call information. | #L455-L512 |
vllm_mlx.tool_parsers.mistral_tool_parser.MistralToolParser._scan_args_for_new_call |
method | vllm_mlx.tool_parsers.mistral_tool_parser.MistralToolParser._scan_args_for_new_call(text: str) -> int |
Scan an argument delta, updating the persistent JSON string state, and return the position of the first [TOOL_CALLS] marker that sits outside a string (a new call), or -1 when there is none. | #L116-L146 |
vllm_mlx.tool_parsers.mistral_tool_parser.MistralToolParser._split_on_tool_call_markers |
method | vllm_mlx.tool_parsers.mistral_tool_parser.MistralToolParser._split_on_tool_call_markers(text: str) -> list[str] |
Split on [TOOL_CALLS] occurrences that are outside JSON strings. | #L148-L192 |
vllm_mlx.tool_parsers.mistral_tool_parser.MistralToolParser._start_new_tool_call |
method | vllm_mlx.tool_parsers.mistral_tool_parser.MistralToolParser._start_new_tool_call() -> None |
Begin a new streaming tool call: bump the index and reset the per-call name/arguments and id state. | #L104-L114 |
vllm_mlx.tool_parsers.mistral_tool_parser.MistralToolParser.extract_tool_calls |
method | vllm_mlx.tool_parsers.mistral_tool_parser.MistralToolParser.extract_tool_calls(model_output: str, request: dict[str, Any] | None = None) -> ExtractedToolCallInformation |
Extract tool calls from a complete Mistral model response. | #L194-L332 |
vllm_mlx.tool_parsers.mistral_tool_parser.MistralToolParser.extract_tool_calls_streaming |
method | vllm_mlx.tool_parsers.mistral_tool_parser.MistralToolParser.extract_tool_calls_streaming(previous_text: str, current_text: str, delta_text: str, previous_token_ids: Sequence[int] | None = None, current_token_ids: Sequence[int] | None = None, delta_token_ids: Sequence[int] | None = None, request: dict[str, Any] | None = None) -> dict[str, Any] | None |
Extract tool calls from streaming Mistral model output. | #L334-L453 |
vllm_mlx.tool_parsers.mistral_tool_parser.MistralToolParser.reset |
method | vllm_mlx.tool_parsers.mistral_tool_parser.MistralToolParser.reset() -> None |
Reset shared and Mistral-specific streaming tool-call state. | #L92-L102 |
vllm_mlx.tool_parsers.nemotron_tool_parser.generate_tool_id |
function | vllm_mlx.tool_parsers.nemotron_tool_parser.generate_tool_id() -> str |
Generate a unique tool call ID. | #L24-L26 |
vllm_mlx.tool_parsers.nemotron_tool_parser.NemotronToolParser |
class | vllm_mlx.tool_parsers.nemotron_tool_parser.NemotronToolParser() |
Tool call parser for NVIDIA Nemotron models. | #L30-L166 |
vllm_mlx.tool_parsers.nemotron_tool_parser.NemotronToolParser.extract_tool_calls |
method | vllm_mlx.tool_parsers.nemotron_tool_parser.NemotronToolParser.extract_tool_calls(model_output: str, request: dict[str, Any] | None = None) -> ExtractedToolCallInformation |
Extract tool calls from Nemotron model output. | #L55-L130 |
vllm_mlx.tool_parsers.nemotron_tool_parser.NemotronToolParser.extract_tool_calls_streaming |
method | vllm_mlx.tool_parsers.nemotron_tool_parser.NemotronToolParser.extract_tool_calls_streaming(previous_text: str, current_text: str, delta_text: str, previous_token_ids: Sequence[int] | None = None, current_token_ids: Sequence[int] | None = None, delta_token_ids: Sequence[int] | None = None, request: dict[str, Any] | None = None) -> dict[str, Any] | None |
Extract tool calls from streaming Nemotron model output. | #L132-L166 |
vllm_mlx.tool_parsers.poolside_v1_tool_parser._consume_stream_state |
function | vllm_mlx.tool_parsers.poolside_v1_tool_parser._consume_stream_state(parser, pending: dict[int, dict[str, Any]], valid_names: set[str], request: dict[str, Any] | None) -> tuple[bool, str] |
Function `_consume_stream_state` calls `parser._consume_text_before_tool`, `parser._consume_tool_name`, `parser._consume_string_value`, `parser._consume_pending_key`; has 5 explicit return paths. | #L16-L30 |
vllm_mlx.tool_parsers.poolside_v1_tool_parser.PoolsideV1ToolParser |
class | vllm_mlx.tool_parsers.poolside_v1_tool_parser.PoolsideV1ToolParser(tokenizer = None) |
Parse Laguna tool calls and stream schema-declared strings incrementally. | #L34-L362 |
vllm_mlx.tool_parsers.poolside_v1_tool_parser.PoolsideV1ToolParser.__init__ |
method | vllm_mlx.tool_parsers.poolside_v1_tool_parser.PoolsideV1ToolParser.__init__(tokenizer = None) -> not annotated |
Method `PoolsideV1ToolParser.__init__` calls `super().__init__`, `super`, `self.reset`. | #L45-L47 |
vllm_mlx.tool_parsers.poolside_v1_tool_parser.PoolsideV1ToolParser._argument_prefix |
method | vllm_mlx.tool_parsers.poolside_v1_tool_parser.PoolsideV1ToolParser._argument_prefix(key: str) -> str | None |
Method `PoolsideV1ToolParser._argument_prefix` calls `seen.add`, `json.dumps`; has 2 explicit return paths. | #L181-L188 |
vllm_mlx.tool_parsers.poolside_v1_tool_parser.PoolsideV1ToolParser._begin_tool_call |
method | vllm_mlx.tool_parsers.poolside_v1_tool_parser.PoolsideV1ToolParser._begin_tool_call() -> None |
Method `PoolsideV1ToolParser._begin_tool_call` updates `self.current_tool_id`, `self._in_tool_call`, `self._current_tool_name`, `self._pending_key`; calls `self._tool_ids.append`, `generate_tool_id`, `self._args_started.append`, `self._args_closed.append`. | #L140-L150 |
vllm_mlx.tool_parsers.poolside_v1_tool_parser.PoolsideV1ToolParser._close_arguments |
method | vllm_mlx.tool_parsers.poolside_v1_tool_parser.PoolsideV1ToolParser._close_arguments() -> str |
Method `PoolsideV1ToolParser._close_arguments` has 2 explicit return paths. | #L190-L194 |
vllm_mlx.tool_parsers.poolside_v1_tool_parser.PoolsideV1ToolParser._consume_pending_key |
method | vllm_mlx.tool_parsers.poolside_v1_tool_parser.PoolsideV1ToolParser._consume_pending_key(pending: dict[int, dict[str, Any]], request: dict[str, Any] | None) -> bool |
Consume the value for the currently buffered argument key. | #L269-L305 |
vllm_mlx.tool_parsers.poolside_v1_tool_parser.PoolsideV1ToolParser._consume_string_value |
method | vllm_mlx.tool_parsers.poolside_v1_tool_parser.PoolsideV1ToolParser._consume_string_value(pending: dict[int, dict[str, Any]]) -> bool |
Consume a string argument value, retaining incomplete suffixes. | #L249-L267 |
vllm_mlx.tool_parsers.poolside_v1_tool_parser.PoolsideV1ToolParser._consume_text_before_tool |
method | vllm_mlx.tool_parsers.poolside_v1_tool_parser.PoolsideV1ToolParser._consume_text_before_tool() -> tuple[bool, str] |
Consume plain text or enter the next ``<tool_call>`` state. | #L204-L214 |
vllm_mlx.tool_parsers.poolside_v1_tool_parser.PoolsideV1ToolParser._consume_tool_body |
method | vllm_mlx.tool_parsers.poolside_v1_tool_parser.PoolsideV1ToolParser._consume_tool_body(pending: dict[int, dict[str, Any]]) -> bool |
Consume an argument key or close the current tool call. | #L307-L328 |
vllm_mlx.tool_parsers.poolside_v1_tool_parser.PoolsideV1ToolParser._consume_tool_name |
method | vllm_mlx.tool_parsers.poolside_v1_tool_parser.PoolsideV1ToolParser._consume_tool_name(pending: dict[int, dict[str, Any]], valid_names: set[str]) -> bool |
Consume a tool name, or wait for enough input to identify it. | #L216-L247 |
vllm_mlx.tool_parsers.poolside_v1_tool_parser.PoolsideV1ToolParser._delta |
method | vllm_mlx.tool_parsers.poolside_v1_tool_parser.PoolsideV1ToolParser._delta(pending: dict[int, dict[str, Any]], *, name: str | None = None, arguments: str = '') -> None |
Method `PoolsideV1ToolParser._delta` calls `pending.setdefault`; returns `None`. | #L159-L179 |
vllm_mlx.tool_parsers.poolside_v1_tool_parser.PoolsideV1ToolParser._discard_through_tool_end |
method | vllm_mlx.tool_parsers.poolside_v1_tool_parser.PoolsideV1ToolParser._discard_through_tool_end() -> bool |
Method `PoolsideV1ToolParser._discard_through_tool_end` updates `self._buffer`; calls `self._buffer.find`, `len`, `self._finish_tool_call`; has 2 explicit return paths. | #L196-L202 |
vllm_mlx.tool_parsers.poolside_v1_tool_parser.PoolsideV1ToolParser._escape_string_content |
method | vllm_mlx.tool_parsers.poolside_v1_tool_parser.PoolsideV1ToolParser._escape_string_content(value: str) -> str |
Method `PoolsideV1ToolParser._escape_string_content` calls `json.dumps`; returns `json.dumps(value, ensure_ascii=False)[1:-1]`. | #L88-L89 |
vllm_mlx.tool_parsers.poolside_v1_tool_parser.PoolsideV1ToolParser._finish_tool_call |
method | vllm_mlx.tool_parsers.poolside_v1_tool_parser.PoolsideV1ToolParser._finish_tool_call() -> None |
Method `PoolsideV1ToolParser._finish_tool_call` updates `self._in_tool_call`, `self._current_tool_name`, `self._pending_key`, `self._streaming_string_value`. | #L152-L157 |
vllm_mlx.tool_parsers.poolside_v1_tool_parser.PoolsideV1ToolParser._hold_partial_suffix |
method | vllm_mlx.tool_parsers.poolside_v1_tool_parser.PoolsideV1ToolParser._hold_partial_suffix(buffer: str, marker: str) -> tuple[str, str] |
Method `PoolsideV1ToolParser._hold_partial_suffix` calls `range`, `min`, `len`, `buffer.endswith`; has 2 explicit return paths. | #L92-L96 |
vllm_mlx.tool_parsers.poolside_v1_tool_parser.PoolsideV1ToolParser._string_argument_names |
method | vllm_mlx.tool_parsers.poolside_v1_tool_parser.PoolsideV1ToolParser._string_argument_names(request: dict[str, Any] | None, tool_name: str) -> set[str] |
Method `PoolsideV1ToolParser._string_argument_names` calls `set`, `request.get`, `isinstance`, `tool.get`; has 2 explicit return paths. | #L65-L85 |
vllm_mlx.tool_parsers.poolside_v1_tool_parser.PoolsideV1ToolParser.extract_tool_calls |
method | vllm_mlx.tool_parsers.poolside_v1_tool_parser.PoolsideV1ToolParser.extract_tool_calls(model_output: str, request: dict[str, Any] | None = None) -> ExtractedToolCallInformation |
Extract complete Laguna tool blocks and preserve remaining content. | #L98-L138 |
vllm_mlx.tool_parsers.poolside_v1_tool_parser.PoolsideV1ToolParser.extract_tool_calls_streaming |
method | vllm_mlx.tool_parsers.poolside_v1_tool_parser.PoolsideV1ToolParser.extract_tool_calls_streaming(previous_text: str, current_text: str, delta_text: str, previous_token_ids: Sequence[int] | None = None, current_token_ids: Sequence[int] | None = None, delta_token_ids: Sequence[int] | None = None, request: dict[str, Any] | None = None) -> dict[str, Any] | None |
Incrementally emit Laguna content and schema-aware tool arguments. | #L330-L362 |
vllm_mlx.tool_parsers.poolside_v1_tool_parser.PoolsideV1ToolParser.reset |
method | vllm_mlx.tool_parsers.poolside_v1_tool_parser.PoolsideV1ToolParser.reset() -> None |
Reset Laguna parser buffers and per-call argument state. | #L49-L62 |
vllm_mlx.tool_parsers.qwen3_xml_tool_parser._FunctionDef |
class | vllm_mlx.tool_parsers.qwen3_xml_tool_parser._FunctionDef(d: dict) |
Wrap a function definition dict for attribute access. | #L85-L99 |
vllm_mlx.tool_parsers.qwen3_xml_tool_parser._FunctionDef.__init__ |
method | vllm_mlx.tool_parsers.qwen3_xml_tool_parser._FunctionDef.__init__(d: dict) -> not annotated |
Method `_FunctionDef.__init__` updates `self._d`. | #L90-L91 |
vllm_mlx.tool_parsers.qwen3_xml_tool_parser._FunctionDef.name |
method | vllm_mlx.tool_parsers.qwen3_xml_tool_parser._FunctionDef.name() -> str |
Method `_FunctionDef.name` calls `self._d.get`; returns `self._d.get('name', '')`. | #L94-L95 |
vllm_mlx.tool_parsers.qwen3_xml_tool_parser._FunctionDef.parameters |
method | vllm_mlx.tool_parsers.qwen3_xml_tool_parser._FunctionDef.parameters() -> dict |
Method `_FunctionDef.parameters` calls `self._d.get`; returns `self._d.get('parameters', {})`. | #L98-L99 |
vllm_mlx.tool_parsers.qwen3_xml_tool_parser._ToolDef |
class | vllm_mlx.tool_parsers.qwen3_xml_tool_parser._ToolDef(d: dict) |
Wrap a tool definition dict for attribute access. | #L102-L117 |
vllm_mlx.tool_parsers.qwen3_xml_tool_parser._ToolDef.__init__ |
method | vllm_mlx.tool_parsers.qwen3_xml_tool_parser._ToolDef.__init__(d: dict) -> not annotated |
Method `_ToolDef.__init__` updates `self._d`, `self._func`; calls `_FunctionDef`, `d.get`. | #L107-L109 |
vllm_mlx.tool_parsers.qwen3_xml_tool_parser._ToolDef.function |
method | vllm_mlx.tool_parsers.qwen3_xml_tool_parser._ToolDef.function() -> _FunctionDef |
Method `_ToolDef.function` returns `self._func`. | #L116-L117 |
vllm_mlx.tool_parsers.qwen3_xml_tool_parser._ToolDef.type |
method | vllm_mlx.tool_parsers.qwen3_xml_tool_parser._ToolDef.type() -> str |
Method `_ToolDef.type` calls `self._d.get`; returns `self._d.get('type', 'function')`. | #L112-L113 |
vllm_mlx.tool_parsers.qwen3_xml_tool_parser.DeltaFunctionCall |
class | vllm_mlx.tool_parsers.qwen3_xml_tool_parser.DeltaFunctionCall(name: Optional[str] = None, arguments: str = '') |
Incremental function name and argument payload used by the XML parser. | #L54-L58 |
vllm_mlx.tool_parsers.qwen3_xml_tool_parser.DeltaMessage |
class | vllm_mlx.tool_parsers.qwen3_xml_tool_parser.DeltaMessage(content: Optional[str] = None, tool_calls: Optional[list[DeltaToolCall]] = None, role: Optional[str] = None, reasoning_content: Optional[str] = None) |
Incremental content, reasoning, and tool calls from the parser shim. | #L72-L78 |
vllm_mlx.tool_parsers.qwen3_xml_tool_parser.DeltaToolCall |
class | vllm_mlx.tool_parsers.qwen3_xml_tool_parser.DeltaToolCall(index: int = 0, id: Optional[str] = None, type: str = 'function', function: Optional[DeltaFunctionCall] = None) |
Incremental indexed tool call produced by the XML parser shim. | #L62-L68 |
vllm_mlx.tool_parsers.qwen3_xml_tool_parser.Qwen3XMLToolParser |
class | vllm_mlx.tool_parsers.qwen3_xml_tool_parser.Qwen3XMLToolParser(tokenizer = None) |
XML tool call parser for Qwen 3.5 models, adapted for vllm-mlx. | #L1442-L1559 |
vllm_mlx.tool_parsers.qwen3_xml_tool_parser.Qwen3XMLToolParser.__init__ |
method | vllm_mlx.tool_parsers.qwen3_xml_tool_parser.Qwen3XMLToolParser.__init__(tokenizer = None) -> not annotated |
Method `Qwen3XMLToolParser.__init__` updates `self._xml_parser`; calls `super().__init__`, `super`, `StreamingXMLToolCallParser`, `logger.info`. | #L1454-L1460 |
vllm_mlx.tool_parsers.qwen3_xml_tool_parser.Qwen3XMLToolParser._wrap_tools |
method | vllm_mlx.tool_parsers.qwen3_xml_tool_parser.Qwen3XMLToolParser._wrap_tools(request: dict[str, Any] | None) -> list[_ToolDef] | None |
Convert tool definition dicts to _ToolDef wrappers for attribute access. | #L1463-L1467 |
vllm_mlx.tool_parsers.qwen3_xml_tool_parser.Qwen3XMLToolParser.extract_tool_calls |
method | vllm_mlx.tool_parsers.qwen3_xml_tool_parser.Qwen3XMLToolParser.extract_tool_calls(model_output: str, request: dict[str, Any] | None = None) -> ExtractedToolCallInformation |
Extract tool calls from complete Qwen 3.5 output. | #L1469-L1507 |
vllm_mlx.tool_parsers.qwen3_xml_tool_parser.Qwen3XMLToolParser.extract_tool_calls_streaming |
method | vllm_mlx.tool_parsers.qwen3_xml_tool_parser.Qwen3XMLToolParser.extract_tool_calls_streaming(previous_text: str, current_text: str, delta_text: str, previous_token_ids: Sequence[int] | None = None, current_token_ids: Sequence[int] | None = None, delta_token_ids: Sequence[int] | None = None, request: dict[str, Any] | None = None) -> dict[str, Any] | None |
Extract tool calls from streaming Qwen 3.5 output. | #L1509-L1559 |
vllm_mlx.tool_parsers.qwen3_xml_tool_parser.StreamingXMLToolCallParser |
class | vllm_mlx.tool_parsers.qwen3_xml_tool_parser.StreamingXMLToolCallParser() |
Streaming XML parser for Qwen 3.5 ``<tool_call>`` format. | #L126-L1427 |
vllm_mlx.tool_parsers.qwen3_xml_tool_parser.StreamingXMLToolCallParser.__init__ |
method | vllm_mlx.tool_parsers.qwen3_xml_tool_parser.StreamingXMLToolCallParser.__init__() -> not annotated |
Method `StreamingXMLToolCallParser.__init__` updates `self.tools`, `self.tool_call_start_token`, `self.tool_call_end_token`, `self.function_start_token`; calls `self.reset_streaming_state`. | #L146-L156 |
vllm_mlx.tool_parsers.qwen3_xml_tool_parser.StreamingXMLToolCallParser._abandon_pending_implicit_tool_call |
method | vllm_mlx.tool_parsers.qwen3_xml_tool_parser.StreamingXMLToolCallParser._abandon_pending_implicit_tool_call() -> tuple[str, str] |
Roll back a deferred bare-<function=> auto-open: prose followed. | #L902-L928 |
vllm_mlx.tool_parsers.qwen3_xml_tool_parser.StreamingXMLToolCallParser._auto_close_open_parameter_if_needed |
method | vllm_mlx.tool_parsers.qwen3_xml_tool_parser.StreamingXMLToolCallParser._auto_close_open_parameter_if_needed(incoming_tag: Optional[str] = None) -> not annotated |
Before starting to process new elements, if there are unclosed tags from before, automatically complete their endings to the parser. | #L754-L777 |
vllm_mlx.tool_parsers.qwen3_xml_tool_parser.StreamingXMLToolCallParser._char_data |
method | vllm_mlx.tool_parsers.qwen3_xml_tool_parser.StreamingXMLToolCallParser._char_data(data: str) -> not annotated |
Handle XML character data events | #L930-L1025 |
vllm_mlx.tool_parsers.qwen3_xml_tool_parser.StreamingXMLToolCallParser._convert_for_json_streaming |
method | vllm_mlx.tool_parsers.qwen3_xml_tool_parser.StreamingXMLToolCallParser._convert_for_json_streaming(converted_value: Any, param_type: str) -> str |
Convert converted_value based on whether it's empty and if type is string Args: converted_value: Converted value param_type: Parameter type Returns: Converted string for streaming output | #L1373-L1395 |
vllm_mlx.tool_parsers.qwen3_xml_tool_parser.StreamingXMLToolCallParser._convert_param_value |
method | vllm_mlx.tool_parsers.qwen3_xml_tool_parser.StreamingXMLToolCallParser._convert_param_value(param_value: str, param_type: str) -> Any |
Convert value based on parameter type Args: param_value: Parameter value param_type: Parameter type Returns: Converted value | #L1317-L1371 |
vllm_mlx.tool_parsers.qwen3_xml_tool_parser.StreamingXMLToolCallParser._emit_delta |
method | vllm_mlx.tool_parsers.qwen3_xml_tool_parser.StreamingXMLToolCallParser._emit_delta(delta: DeltaMessage) -> not annotated |
Emit Delta response (streaming output) | #L750-L752 |
vllm_mlx.tool_parsers.qwen3_xml_tool_parser.StreamingXMLToolCallParser._end_element |
method | vllm_mlx.tool_parsers.qwen3_xml_tool_parser.StreamingXMLToolCallParser._end_element(name: str) -> not annotated |
Handle XML end element events | #L1027-L1206 |
vllm_mlx.tool_parsers.qwen3_xml_tool_parser.StreamingXMLToolCallParser._escape_xml_special_chars |
method | vllm_mlx.tool_parsers.qwen3_xml_tool_parser.StreamingXMLToolCallParser._escape_xml_special_chars(text: str) -> str |
Escape XML special characters Args: text: Original text Returns: Escaped text | #L332-L351 |
vllm_mlx.tool_parsers.qwen3_xml_tool_parser.StreamingXMLToolCallParser._extract_function_name |
method | vllm_mlx.tool_parsers.qwen3_xml_tool_parser.StreamingXMLToolCallParser._extract_function_name(name: str, attrs: dict[str, str]) -> Optional[str] |
Extract function name from various formats | #L1223-L1233 |
vllm_mlx.tool_parsers.qwen3_xml_tool_parser.StreamingXMLToolCallParser._extract_parameter_name |
method | vllm_mlx.tool_parsers.qwen3_xml_tool_parser.StreamingXMLToolCallParser._extract_parameter_name(name: str, attrs: dict[str, str]) -> Optional[str] |
Extract parameter name from various formats | #L1235-L1247 |
vllm_mlx.tool_parsers.qwen3_xml_tool_parser.StreamingXMLToolCallParser._find_next_complete_element |
method | vllm_mlx.tool_parsers.qwen3_xml_tool_parser.StreamingXMLToolCallParser._find_next_complete_element(start_pos: int) -> tuple[Optional[str], int] |
Find next complete XML element from specified position Args: start_pos: Position to start searching Returns: (Complete element string, element end position), returns (None, start_pos) if no complete element found | #L503-L569 |
vllm_mlx.tool_parsers.qwen3_xml_tool_parser.StreamingXMLToolCallParser._flush_pending_implicit_delta |
method | vllm_mlx.tool_parsers.qwen3_xml_tool_parser.StreamingXMLToolCallParser._flush_pending_implicit_delta() -> None |
Emit a deferred bare-<function=> delta now that the call is confirmed. | #L890-L900 |
vllm_mlx.tool_parsers.qwen3_xml_tool_parser.StreamingXMLToolCallParser._get_next_call_id |
method | vllm_mlx.tool_parsers.qwen3_xml_tool_parser.StreamingXMLToolCallParser._get_next_call_id() -> not annotated |
Generate unique call ID | #L1219-L1221 |
vllm_mlx.tool_parsers.qwen3_xml_tool_parser.StreamingXMLToolCallParser._get_param_type |
method | vllm_mlx.tool_parsers.qwen3_xml_tool_parser.StreamingXMLToolCallParser._get_param_type(param_name: str) -> str |
Get parameter type based on tool configuration, defaults to string Args: param_name: Parameter name Returns: Parameter type | #L1249-L1287 |
vllm_mlx.tool_parsers.qwen3_xml_tool_parser.StreamingXMLToolCallParser._looks_like_partial_tool_open |
method | vllm_mlx.tool_parsers.qwen3_xml_tool_parser.StreamingXMLToolCallParser._looks_like_partial_tool_open(fragment: str) -> bool |
True if `fragment` could complete into a tool-related XML tag. | #L488-L501 |
vllm_mlx.tool_parsers.qwen3_xml_tool_parser.StreamingXMLToolCallParser._merge_new_deltas_to_single_response |
method | vllm_mlx.tool_parsers.qwen3_xml_tool_parser.StreamingXMLToolCallParser._merge_new_deltas_to_single_response(initial_count: int) -> DeltaMessage |
Merge newly generated deltas from this processing into a single DeltaMessage Args: initial_count: Delta count before processing Returns: Merged DeltaMessage containing all newly generated delta information | #L571-L633 |
vllm_mlx.tool_parsers.qwen3_xml_tool_parser.StreamingXMLToolCallParser._preprocess_xml_chunk |
method | vllm_mlx.tool_parsers.qwen3_xml_tool_parser.StreamingXMLToolCallParser._preprocess_xml_chunk(chunk: str) -> str |
Preprocess XML chunk, handle non-standard formats, and escape special characters Args: chunk: Original XML chunk Returns: Processed XML chunk | #L635-L748 |
vllm_mlx.tool_parsers.qwen3_xml_tool_parser.StreamingXMLToolCallParser._process_complete_xml_elements |
method | vllm_mlx.tool_parsers.qwen3_xml_tool_parser.StreamingXMLToolCallParser._process_complete_xml_elements() -> bool |
Process complete XML elements in buffer Returns: bool: Whether complete elements were found and processed | #L353-L438 |
vllm_mlx.tool_parsers.qwen3_xml_tool_parser.StreamingXMLToolCallParser._reset_xml_parser_after_tool_call |
method | vllm_mlx.tool_parsers.qwen3_xml_tool_parser.StreamingXMLToolCallParser._reset_xml_parser_after_tool_call() -> not annotated |
Each tool_call is treated as a separate XML document, so we need to reset the parser after each tool_call. | #L1397-L1427 |
vllm_mlx.tool_parsers.qwen3_xml_tool_parser.StreamingXMLToolCallParser._should_skip_element |
method | vllm_mlx.tool_parsers.qwen3_xml_tool_parser.StreamingXMLToolCallParser._should_skip_element(element: str) -> bool |
Determine whether an element should be skipped Args: element: Element to evaluate Returns: bool: True means should skip, False means should process | #L440-L474 |
vllm_mlx.tool_parsers.qwen3_xml_tool_parser.StreamingXMLToolCallParser._start_element |
method | vllm_mlx.tool_parsers.qwen3_xml_tool_parser.StreamingXMLToolCallParser._start_element(name: str, attrs: dict[str, str]) -> not annotated |
Handle XML start element events | #L779-L888 |
vllm_mlx.tool_parsers.qwen3_xml_tool_parser.StreamingXMLToolCallParser.parse_single_streaming_chunks |
method | vllm_mlx.tool_parsers.qwen3_xml_tool_parser.StreamingXMLToolCallParser.parse_single_streaming_chunks(xml_chunk: str) -> DeltaMessage |
Parse single streaming XML chunk and return Delta response This is the actual streaming interface that receives chunks one by one and maintains internal state Args: xml_chunk: Single XML chunk string Returns: DeltaMessage: Contains delta information generated by this chunk, returns empty response if no complete elements | #L210-L330 |
vllm_mlx.tool_parsers.qwen3_xml_tool_parser.StreamingXMLToolCallParser.repair_param_type |
method | vllm_mlx.tool_parsers.qwen3_xml_tool_parser.StreamingXMLToolCallParser.repair_param_type(param_type: str) -> str |
Repair unknown parameter types by treating them as string Args: param_type: Parameter type Returns: Repaired parameter type | #L1289-L1315 |
vllm_mlx.tool_parsers.qwen3_xml_tool_parser.StreamingXMLToolCallParser.reset_streaming_state |
method | vllm_mlx.tool_parsers.qwen3_xml_tool_parser.StreamingXMLToolCallParser.reset_streaming_state() -> not annotated |
Reset streaming parsing state | #L158-L208 |
vllm_mlx.tool_parsers.qwen3_xml_tool_parser.StreamingXMLToolCallParser.set_tools |
method | vllm_mlx.tool_parsers.qwen3_xml_tool_parser.StreamingXMLToolCallParser.set_tools(tools: Union[list[ChatCompletionToolsParam], None]) -> not annotated |
Set tool configuration information | #L1215-L1217 |
vllm_mlx.tool_parsers.qwen3_xml_tool_parser.StreamingXMLToolCallParser.setup_parser |
method | vllm_mlx.tool_parsers.qwen3_xml_tool_parser.StreamingXMLToolCallParser.setup_parser() -> not annotated |
Set up XML parser event handlers | #L1208-L1213 |
vllm_mlx.tool_parsers.qwen_tool_parser._parse_param_value |
function | vllm_mlx.tool_parsers.qwen_tool_parser._parse_param_value(val: str) -> Any |
Parse a parameter value, handling JSON literals and plain strings. | #L25-L40 |
vllm_mlx.tool_parsers.qwen_tool_parser.generate_tool_id |
function | vllm_mlx.tool_parsers.qwen_tool_parser.generate_tool_id() -> str |
Generate a unique tool call ID. | #L43-L45 |
vllm_mlx.tool_parsers.qwen_tool_parser.QwenToolParser |
class | vllm_mlx.tool_parsers.qwen_tool_parser.QwenToolParser() |
Tool call parser for Qwen models. | #L49-L351 |
vllm_mlx.tool_parsers.qwen_tool_parser.QwenToolParser._get_partial_marker_len |
method | vllm_mlx.tool_parsers.qwen_tool_parser.QwenToolParser._get_partial_marker_len(text: str) -> int |
Return the length of a partial tool call marker suffix at end of text. | #L239-L248 |
vllm_mlx.tool_parsers.qwen_tool_parser.QwenToolParser._has_partial_marker |
method | vllm_mlx.tool_parsers.qwen_tool_parser.QwenToolParser._has_partial_marker(text: str) -> bool |
Check if text ends with an incomplete tool call marker prefix. | #L235-L237 |
vllm_mlx.tool_parsers.qwen_tool_parser.QwenToolParser._strip_unclosed_markup |
method | vllm_mlx.tool_parsers.qwen_tool_parser.QwenToolParser._strip_unclosed_markup(text: str) -> str |
Strip a trailing unclosed tool-call marker (truncated output). | #L200-L228 |
vllm_mlx.tool_parsers.qwen_tool_parser.QwenToolParser._was_buffering |
method | vllm_mlx.tool_parsers.qwen_tool_parser.QwenToolParser._was_buffering(previous_text: str) -> bool |
Check if the previous call was buffering a partial marker. | #L250-L252 |
vllm_mlx.tool_parsers.qwen_tool_parser.QwenToolParser.extract_tool_calls |
method | vllm_mlx.tool_parsers.qwen_tool_parser.QwenToolParser.extract_tool_calls(model_output: str, request: dict[str, Any] | None = None) -> ExtractedToolCallInformation |
Extract tool calls from a complete Qwen model response. | #L78-L197 |
vllm_mlx.tool_parsers.qwen_tool_parser.QwenToolParser.extract_tool_calls_streaming |
method | vllm_mlx.tool_parsers.qwen_tool_parser.QwenToolParser.extract_tool_calls_streaming(previous_text: str, current_text: str, delta_text: str, previous_token_ids: Sequence[int] | None = None, current_token_ids: Sequence[int] | None = None, delta_token_ids: Sequence[int] | None = None, request: dict[str, Any] | None = None) -> dict[str, Any] | None |
Extract tool calls from streaming Qwen model output. | #L254-L351 |
vllm_mlx.tool_parsers.xlam_tool_parser.generate_tool_id |
function | vllm_mlx.tool_parsers.xlam_tool_parser.generate_tool_id() -> str |
Generate a unique tool call ID. | #L24-L26 |
vllm_mlx.tool_parsers.xlam_tool_parser.xLAMToolParser |
class | vllm_mlx.tool_parsers.xlam_tool_parser.xLAMToolParser() |
Tool call parser for Salesforce xLAM models. | #L30-L177 |
vllm_mlx.tool_parsers.xlam_tool_parser.xLAMToolParser._try_extract_json |
method | vllm_mlx.tool_parsers.xlam_tool_parser.xLAMToolParser._try_extract_json(text: str) -> tuple[str | None, list | None] |
Try to extract JSON tool calls from text. | #L47-L91 |
vllm_mlx.tool_parsers.xlam_tool_parser.xLAMToolParser.extract_tool_calls |
method | vllm_mlx.tool_parsers.xlam_tool_parser.xLAMToolParser.extract_tool_calls(model_output: str, request: dict[str, Any] | None = None) -> ExtractedToolCallInformation |
Extract tool calls from xLAM model output. | #L93-L131 |
vllm_mlx.tool_parsers.xlam_tool_parser.xLAMToolParser.extract_tool_calls_streaming |
method | vllm_mlx.tool_parsers.xlam_tool_parser.xLAMToolParser.extract_tool_calls_streaming(previous_text: str, current_text: str, delta_text: str, previous_token_ids: Sequence[int] | None = None, current_token_ids: Sequence[int] | None = None, delta_token_ids: Sequence[int] | None = None, request: dict[str, Any] | None = None) -> dict[str, Any] | None |
Extract tool calls from streaming xLAM model output. | #L133-L177 |
vllm_mlx.utils.download.DownloadConfig |
class | vllm_mlx.utils.download.DownloadConfig(download_timeout: int = 300, max_retries: int = 3, retry_backoff_base: float = 2.0, offline: bool = False) |
Configuration for model download behavior. | #L45-L51 |
vllm_mlx.utils.download.ensure_model_downloaded |
function | vllm_mlx.utils.download.ensure_model_downloaded(model_name: str, config: DownloadConfig | None = None, is_mllm: bool = False) -> Path |
Ensure a model is available locally, downloading with retry if needed. | #L54-L144 |
vllm_mlx.utils.harmony_render._build_tools |
function | vllm_mlx.utils.harmony_render._build_tools(tools: list[dict] | None) -> list[Any] | None |
Function `_build_tools` calls `t.get`, `fn.get`, `tool_descs.append`, `_oh.ToolDescription.new`; has 2 explicit return paths. | #L64-L80 |
vllm_mlx.utils.harmony_render._content_to_text |
function | vllm_mlx.utils.harmony_render._content_to_text(content: Any) -> str |
Flatten OpenAI content (str | list[dict]) to plain text. | #L83-L97 |
vllm_mlx.utils.harmony_render._convert_message |
function | vllm_mlx.utils.harmony_render._convert_message(msg: dict) -> list[Any] |
Convert one OpenAI-format message to one or more ``openai_harmony.Message``. | #L100-L182 |
vllm_mlx.utils.harmony_render._harmony_encoding |
function | vllm_mlx.utils.harmony_render._harmony_encoding() -> Any |
Load the harmony encoding once and reuse it across requests. | #L45-L53 |
vllm_mlx.utils.harmony_render._resolve_tool_names |
function | vllm_mlx.utils.harmony_render._resolve_tool_names(messages: list[dict]) -> list[dict] |
Stamp ``name=functions.X`` on each ``role=tool`` message by tracing back the most recent assistant ``tool_call_id`` -> function name. | #L185-L214 |
vllm_mlx.utils.harmony_render.is_harmony_parser_name |
function | vllm_mlx.utils.harmony_render.is_harmony_parser_name(parser_name: str | None) -> bool |
Return True when the active --tool-call-parser is a harmony alias. | #L56-L61 |
vllm_mlx.utils.harmony_render.render_messages |
function | vllm_mlx.utils.harmony_render.render_messages(messages: list[dict], tools: list[dict] | None = None, reasoning_effort: str | None = None) -> str |
Render OpenAI-format messages as a harmony-format prompt string. | #L217-L303 |
vllm_mlx.utils.mamba_cache.BatchMambaCache |
class | vllm_mlx.utils.mamba_cache.BatchMambaCache(left_padding: Optional[List[int]] = None, size: int = 2) |
Batch-aware MambaCache for continuous batching. | #L24-L96 |
vllm_mlx.utils.mamba_cache.BatchMambaCache.__init__ |
method | vllm_mlx.utils.mamba_cache.BatchMambaCache.__init__(left_padding: Optional[List[int]] = None, size: int = 2) -> not annotated |
Initialize BatchMambaCache. | #L32-L43 |
vllm_mlx.utils.mamba_cache.BatchMambaCache.extract |
method | vllm_mlx.utils.mamba_cache.BatchMambaCache.extract(idx: int) -> MambaCache |
Extract a single cache from the batch. | #L45-L63 |
vllm_mlx.utils.mamba_cache.BatchMambaCache.merge |
method | vllm_mlx.utils.mamba_cache.BatchMambaCache.merge(caches: List[MambaCache]) -> 'BatchMambaCache' |
Merge multiple MambaCache objects into a BatchMambaCache. | #L66-L96 |
vllm_mlx.utils.mamba_cache.ensure_mamba_support |
function | vllm_mlx.utils.mamba_cache.ensure_mamba_support() -> not annotated |
Ensure MambaCache batching support is enabled. | #L201-L215 |
vllm_mlx.utils.mamba_cache.patch_mlx_lm_for_mamba |
function | vllm_mlx.utils.mamba_cache.patch_mlx_lm_for_mamba() -> not annotated |
Patch mlx-lm to support MambaCache in BatchGenerator. | #L99-L194 |
vllm_mlx.utils.mamba_cache.patch_mlx_lm_for_mamba._patched_make_cache |
nested function | vllm_mlx.utils.mamba_cache.patch_mlx_lm_for_mamba._patched_make_cache(model, left_padding, max_kv_size = None) -> not annotated |
Convert a list of regular caches into their corresponding batch-aware caches, with support for MambaCache. | #L126-L166 |
vllm_mlx.utils.mamba_cache.patch_mlx_lm_for_mamba._patched_make_cache.to_batch_cache |
nested function | vllm_mlx.utils.mamba_cache.patch_mlx_lm_for_mamba._patched_make_cache.to_batch_cache(c) -> not annotated |
Nested Function `patch_mlx_lm_for_mamba._patched_make_cache.to_batch_cache` calls `isinstance`, `BatchKVCache`, `BatchMambaCache`, `mx.array`; can raise `ValueError`; has 5 explicit return paths. | #L137-L155 |
vllm_mlx.utils.mamba_cache.patch_mlx_lm_for_mamba._patched_merge_caches |
nested function | vllm_mlx.utils.mamba_cache.patch_mlx_lm_for_mamba._patched_merge_caches(caches) -> not annotated |
Merge caches with MambaCache support. | #L174-L190 |
vllm_mlx.utils.tokenizer._load_strict_false |
function | vllm_mlx.utils.tokenizer._load_strict_false(model_name: str, tokenizer_config: dict = None) -> not annotated |
Load model with strict=False to discard extra weights. | #L114-L153 |
vllm_mlx.utils.tokenizer._load_with_tokenizer_fallback |
function | vllm_mlx.utils.tokenizer._load_with_tokenizer_fallback(model_name: str) -> not annotated |
Load model with fallback tokenizer for non-standard models like Nemotron. | #L218-L280 |
vllm_mlx.utils.tokenizer._needs_strict_false |
function | vllm_mlx.utils.tokenizer._needs_strict_false(model_name: str) -> bool |
Check if model needs strict=False loading (VLM models with extra weights). | #L31-L49 |
vllm_mlx.utils.tokenizer._needs_tokenizer_fallback |
function | vllm_mlx.utils.tokenizer._needs_tokenizer_fallback(model_name: str) -> bool |
Check if model needs tokenizer fallback. | #L25-L28 |
vllm_mlx.utils.tokenizer._try_inject_mtp |
function | vllm_mlx.utils.tokenizer._try_inject_mtp(model, model_path, config) -> not annotated |
Inject MTP support if model has MTP config + weights. | #L156-L176 |
vllm_mlx.utils.tokenizer._try_inject_mtp_post_load |
function | vllm_mlx.utils.tokenizer._try_inject_mtp_post_load(model, model_name) -> not annotated |
Check if MTP weights exist but were stripped by sanitize(), and inject. | #L179-L215 |
vllm_mlx.utils.tokenizer.load_model_with_fallback |
function | vllm_mlx.utils.tokenizer.load_model_with_fallback(model_name: str, tokenizer_config: dict = None) -> not annotated |
Load model and tokenizer with fallback for non-standard tokenizers. | #L52-L111 |
vllm_mlx.utils.truncation._config_get |
function | vllm_mlx.utils.truncation._config_get(config: Any, key: str) -> Any |
Read ``key`` from a model config that may be a dict or an object. | #L25-L31 |
vllm_mlx.utils.truncation._positive_int |
function | vllm_mlx.utils.truncation._positive_int(value: Any) -> int | None |
Function `_positive_int` calls `isinstance`; has 2 explicit return paths. | #L39-L42 |
vllm_mlx.utils.truncation.inner_tokenizer |
function | vllm_mlx.utils.truncation.inner_tokenizer(tokenizer: Any) -> Any |
Unwrap a wrapping tokenizer to its inner ``_tokenizer`` when present. | #L34-L36 |
vllm_mlx.utils.truncation.resolve_max_length |
function | vllm_mlx.utils.truncation.resolve_max_length(config: Any, tokenizer: Any, *, default: int = MAX_LENGTH_DEFAULT, sentinel_threshold: int = TOKENIZER_SENTINEL_THRESHOLD) -> int |
Resolve the tokenizer truncation length for a model. | #L45-L83 |
vllm_mlx.vision_embedding_cache.compute_image_hash |
function | vllm_mlx.vision_embedding_cache.compute_image_hash(image_path: str) -> str |
Compute hash of image content. | #L105-L124 |
vllm_mlx.vision_embedding_cache.compute_images_hash |
function | vllm_mlx.vision_embedding_cache.compute_images_hash(images: List[str]) -> str |
Compute combined hash for multiple images. | #L127-L132 |
vllm_mlx.vision_embedding_cache.EncodingCacheEntry |
class | vllm_mlx.vision_embedding_cache.EncodingCacheEntry(logits: mx.array, first_token: int, logprobs: mx.array, encoding_time: float = 0.0) |
Cached vision encoding output. | #L96-L102 |
vllm_mlx.vision_embedding_cache.PixelCacheEntry |
class | vllm_mlx.vision_embedding_cache.PixelCacheEntry(pixel_values: mx.array, input_ids: mx.array, attention_mask: Optional[mx.array], image_grid_thw: Optional[mx.array], extra_kwargs: Dict[str, Any], processing_time: float = 0.0) |
Cached pixel values from prepare_inputs. | #L70-L78 |
vllm_mlx.vision_embedding_cache.PixelOnlyCacheEntry |
class | vllm_mlx.vision_embedding_cache.PixelOnlyCacheEntry(pixel_values: mx.array, image_grid_thw: Optional[mx.array], processing_time: float = 0.0) |
Cached pixel values only (prompt-independent). | #L82-L92 |
vllm_mlx.vision_embedding_cache.VisionCacheStats |
class | vllm_mlx.vision_embedding_cache.VisionCacheStats(pixel_cache_hits: int = 0, pixel_cache_misses: int = 0, encoding_cache_hits: int = 0, encoding_cache_misses: int = 0, total_time_saved: float = 0.0, total_images_processed: int = 0) |
Statistics for vision cache performance. | #L30-L66 |
vllm_mlx.vision_embedding_cache.VisionCacheStats.encoding_hit_rate |
method | vllm_mlx.vision_embedding_cache.VisionCacheStats.encoding_hit_rate() -> float |
Return successful encoding lookups divided by all encoding lookups. | #L48-L52 |
vllm_mlx.vision_embedding_cache.VisionCacheStats.pixel_hit_rate |
method | vllm_mlx.vision_embedding_cache.VisionCacheStats.pixel_hit_rate() -> float |
Return successful pixel-cache lookups divided by all pixel lookups. | #L41-L45 |
vllm_mlx.vision_embedding_cache.VisionCacheStats.to_dict |
method | vllm_mlx.vision_embedding_cache.VisionCacheStats.to_dict() -> dict |
Return pixel, encoding, timing, and image counters. | #L54-L66 |
vllm_mlx.vision_embedding_cache.VisionEmbeddingCache |
class | vllm_mlx.vision_embedding_cache.VisionEmbeddingCache(max_pixel_entries: int = 100, max_encoding_entries: int = 50, enabled: bool = True) |
Two-level cache for vision processing in MLLM. | #L135-L413 |
vllm_mlx.vision_embedding_cache.VisionEmbeddingCache.__init__ |
method | vllm_mlx.vision_embedding_cache.VisionEmbeddingCache.__init__(max_pixel_entries: int = 100, max_encoding_entries: int = 50, enabled: bool = True) -> not annotated |
Initialize the vision embedding cache. | #L162-L185 |
vllm_mlx.vision_embedding_cache.VisionEmbeddingCache.__repr__ |
method | vllm_mlx.vision_embedding_cache.VisionEmbeddingCache.__repr__() -> str |
Method `VisionEmbeddingCache.__repr__` calls `len`; returns `f'<VisionEmbeddingCache pixel={len(self._pixel_cache)}/{self.max_pixel_entries} pixel_only={len(self._pixel_only_cache)…`. | #L407-L413 |
vllm_mlx.vision_embedding_cache.VisionEmbeddingCache._make_image_only_key |
method | vllm_mlx.vision_embedding_cache.VisionEmbeddingCache._make_image_only_key(images: List[str]) -> str |
Create cache key from images only (prompt-independent). | #L194-L196 |
vllm_mlx.vision_embedding_cache.VisionEmbeddingCache._make_key |
method | vllm_mlx.vision_embedding_cache.VisionEmbeddingCache._make_key(images: List[str], prompt: str) -> str |
Create cache key from images and prompt. | #L187-L192 |
vllm_mlx.vision_embedding_cache.VisionEmbeddingCache.clear |
method | vllm_mlx.vision_embedding_cache.VisionEmbeddingCache.clear() -> None |
Clear all caches and reset stats. | #L400-L405 |
vllm_mlx.vision_embedding_cache.VisionEmbeddingCache.get_encoding_cache |
method | vllm_mlx.vision_embedding_cache.VisionEmbeddingCache.get_encoding_cache(images: List[str], prompt: str) -> Optional[EncodingCacheEntry] |
Get cached vision encoding output. | #L330-L358 |
vllm_mlx.vision_embedding_cache.VisionEmbeddingCache.get_pixel_cache |
method | vllm_mlx.vision_embedding_cache.VisionEmbeddingCache.get_pixel_cache(images: List[str], prompt: str) -> Optional[PixelCacheEntry] |
Get cached pixel values for images+prompt. | #L200-L229 |
vllm_mlx.vision_embedding_cache.VisionEmbeddingCache.get_pixel_values |
method | vllm_mlx.vision_embedding_cache.VisionEmbeddingCache.get_pixel_values(images: List[str]) -> Optional[PixelOnlyCacheEntry] |
Get cached pixel values for images (prompt-independent). | #L268-L299 |
vllm_mlx.vision_embedding_cache.VisionEmbeddingCache.get_stats |
method | vllm_mlx.vision_embedding_cache.VisionEmbeddingCache.get_stats() -> dict |
Get cache statistics. | #L392-L398 |
vllm_mlx.vision_embedding_cache.VisionEmbeddingCache.set_encoding_cache |
method | vllm_mlx.vision_embedding_cache.VisionEmbeddingCache.set_encoding_cache(images: List[str], prompt: str, logits: mx.array, first_token: int, logprobs: mx.array, encoding_time: float = 0.0) -> None |
Store vision encoding output in cache. | #L360-L388 |
vllm_mlx.vision_embedding_cache.VisionEmbeddingCache.set_pixel_cache |
method | vllm_mlx.vision_embedding_cache.VisionEmbeddingCache.set_pixel_cache(images: List[str], prompt: str, pixel_values: mx.array, input_ids: mx.array, attention_mask: Optional[mx.array] = None, image_grid_thw: Optional[mx.array] = None, extra_kwargs: Optional[Dict[str, Any]] = None, processing_time: float = 0.0) -> None |
Store pixel values in cache. | #L231-L264 |
vllm_mlx.vision_embedding_cache.VisionEmbeddingCache.set_pixel_values |
method | vllm_mlx.vision_embedding_cache.VisionEmbeddingCache.set_pixel_values(images: List[str], pixel_values: mx.array, image_grid_thw: Optional[mx.array] = None, processing_time: float = 0.0) -> None |
Store pixel values in cache (prompt-independent). | #L301-L326 |
vllm_mlx.vllm_platform._get_apple_chip_name |
function | vllm_mlx.vllm_platform._get_apple_chip_name() -> str |
Get the name of the Apple Silicon chip. | #L24-L35 |
vllm_mlx.vllm_platform._get_unified_memory_size |
function | vllm_mlx.vllm_platform._get_unified_memory_size() -> int |
Get the total unified memory size in bytes. | #L38-L50 |
vllm_mlx.vllm_platform._is_apple_silicon |
function | vllm_mlx.vllm_platform._is_apple_silicon() -> bool |
Check if running on Apple Silicon. | #L66-L68 |
vllm_mlx.vllm_platform._is_mlx_available |
function | vllm_mlx.vllm_platform._is_mlx_available() -> bool |
Check if MLX is available and working. | #L53-L63 |
vllm_mlx.vllm_platform.MLXPlatform |
class | vllm_mlx.vllm_platform.MLXPlatform() |
Platform implementation for Apple Silicon using MLX. | #L71-L351 |
vllm_mlx.vllm_platform.MLXPlatform.__repr__ |
method | vllm_mlx.vllm_platform.MLXPlatform.__repr__() -> str |
Method `MLXPlatform.__repr__` returns `f'<MLXPlatform device={self.device_name}>'`. | #L350-L351 |
vllm_mlx.vllm_platform.MLXPlatform._enum |
method | vllm_mlx.vllm_platform.MLXPlatform._enum() -> not annotated |
Method `MLXPlatform._enum` returns `PlatformEnum.OOT`. | #L90-L93 |
vllm_mlx.vllm_platform.MLXPlatform.check_and_update_config |
method | vllm_mlx.vllm_platform.MLXPlatform.check_and_update_config(vllm_config: 'VllmConfig') -> None |
Check and update vLLM configuration for MLX. | #L251-L280 |
vllm_mlx.vllm_platform.MLXPlatform.get_attn_backend_cls |
method | vllm_mlx.vllm_platform.MLXPlatform.get_attn_backend_cls(selected_backend, head_size: int, dtype: torch.dtype, kv_cache_dtype, block_size: int, use_mla: bool, has_sink: bool, use_sparse: bool, attn_type: str | None = None) -> str |
Return MLX attention backend class path. | #L234-L248 |
vllm_mlx.vllm_platform.MLXPlatform.get_current_memory_usage |
method | vllm_mlx.vllm_platform.MLXPlatform.get_current_memory_usage(device = None) -> float |
Get current memory usage in bytes. | #L312-L323 |
vllm_mlx.vllm_platform.MLXPlatform.get_device_communicator_cls |
method | vllm_mlx.vllm_platform.MLXPlatform.get_device_communicator_cls() -> str |
Return the communicator class for distributed. | #L341-L343 |
vllm_mlx.vllm_platform.MLXPlatform.get_device_name |
method | vllm_mlx.vllm_platform.MLXPlatform.get_device_name(device_id: int = 0) -> str |
Get the Apple Silicon chip name. | #L181-L183 |
vllm_mlx.vllm_platform.MLXPlatform.get_device_total_memory |
method | vllm_mlx.vllm_platform.MLXPlatform.get_device_total_memory(device_id: int = 0) -> int |
Get total unified memory in bytes. | #L191-L193 |
vllm_mlx.vllm_platform.MLXPlatform.get_device_uuid |
method | vllm_mlx.vllm_platform.MLXPlatform.get_device_uuid(device_id: int = 0) -> str |
Get device UUID (not applicable for MLX). | #L186-L188 |
vllm_mlx.vllm_platform.MLXPlatform.get_punica_wrapper |
method | vllm_mlx.vllm_platform.MLXPlatform.get_punica_wrapper() -> str |
Return LoRA wrapper (not yet implemented for MLX). | #L346-L348 |
vllm_mlx.vllm_platform.MLXPlatform.import_kernels |
method | vllm_mlx.vllm_platform.MLXPlatform.import_kernels() -> None |
Import MLX kernels (no custom C kernels). | #L228-L231 |
vllm_mlx.vllm_platform.MLXPlatform.inference_mode |
method | vllm_mlx.vllm_platform.MLXPlatform.inference_mode() -> not annotated |
Return inference mode context manager. | #L196-L200 |
vllm_mlx.vllm_platform.MLXPlatform.is_cpu |
method | vllm_mlx.vllm_platform.MLXPlatform.is_cpu() -> bool |
Return ``False`` because MLX targets Apple GPU acceleration here. | #L155-L158 |
vllm_mlx.vllm_platform.MLXPlatform.is_cuda |
method | vllm_mlx.vllm_platform.MLXPlatform.is_cuda() -> bool |
Return ``False`` because this platform does not use CUDA. | #L135-L138 |
vllm_mlx.vllm_platform.MLXPlatform.is_cuda_alike |
method | vllm_mlx.vllm_platform.MLXPlatform.is_cuda_alike() -> bool |
Return ``False`` because MLX does not implement CUDA semantics. | #L170-L173 |
vllm_mlx.vllm_platform.MLXPlatform.is_mlx |
method | vllm_mlx.vllm_platform.MLXPlatform.is_mlx() -> bool |
Return ``True`` to identify the MLX platform plugin. | #L160-L163 |
vllm_mlx.vllm_platform.MLXPlatform.is_out_of_tree |
method | vllm_mlx.vllm_platform.MLXPlatform.is_out_of_tree() -> bool |
Return ``True`` because MLX is registered as a vLLM plugin. | #L165-L168 |
vllm_mlx.vllm_platform.MLXPlatform.is_pin_memory_available |
method | vllm_mlx.vllm_platform.MLXPlatform.is_pin_memory_available() -> bool |
Pin memory not needed with unified memory. | #L307-L309 |
vllm_mlx.vllm_platform.MLXPlatform.is_rocm |
method | vllm_mlx.vllm_platform.MLXPlatform.is_rocm() -> bool |
Return ``False`` because this platform does not use ROCm. | #L140-L143 |
vllm_mlx.vllm_platform.MLXPlatform.is_sleep_mode_available |
method | vllm_mlx.vllm_platform.MLXPlatform.is_sleep_mode_available() -> bool |
Return ``False`` because vLLM sleep mode is unavailable on MLX. | #L175-L178 |
vllm_mlx.vllm_platform.MLXPlatform.is_tpu |
method | vllm_mlx.vllm_platform.MLXPlatform.is_tpu() -> bool |
Return ``False`` because this platform is not a TPU backend. | #L145-L148 |
vllm_mlx.vllm_platform.MLXPlatform.is_xpu |
method | vllm_mlx.vllm_platform.MLXPlatform.is_xpu() -> bool |
Return ``False`` because this platform does not use Intel XPU. | #L150-L153 |
vllm_mlx.vllm_platform.MLXPlatform.seed_everything |
method | vllm_mlx.vllm_platform.MLXPlatform.seed_everything(seed: int | None = None) -> None |
Set random seeds for reproducibility. | #L209-L225 |
vllm_mlx.vllm_platform.MLXPlatform.set_device |
method | vllm_mlx.vllm_platform.MLXPlatform.set_device(device: torch.device) -> None |
Set the device (no-op for MLX, uses default device). | #L203-L206 |
vllm_mlx.vllm_platform.MLXPlatform.support_static_graph_mode |
method | vllm_mlx.vllm_platform.MLXPlatform.support_static_graph_mode() -> bool |
Static graph mode (CUDA graphs) not supported. | #L336-L338 |
vllm_mlx.vllm_platform.MLXPlatform.supported_dtypes |
method | vllm_mlx.vllm_platform.MLXPlatform.supported_dtypes() -> list[torch.dtype] |
Return supported dtypes for MLX. | #L122-L133 |
vllm_mlx.vllm_platform.MLXPlatform.supports_fp8 |
method | vllm_mlx.vllm_platform.MLXPlatform.supports_fp8() -> bool |
FP8 not supported on MLX. | #L326-L328 |
vllm_mlx.vllm_platform.MLXPlatform.use_custom_allreduce |
method | vllm_mlx.vllm_platform.MLXPlatform.use_custom_allreduce() -> bool |
Custom allreduce not available. | #L331-L333 |
vllm_mlx.vllm_platform.MLXPlatform.verify_model_arch |
method | vllm_mlx.vllm_platform.MLXPlatform.verify_model_arch(model_arch: str) -> None |
Verify model architecture is supported on MLX. | #L283-L294 |
vllm_mlx.vllm_platform.MLXPlatform.verify_quantization |
method | vllm_mlx.vllm_platform.MLXPlatform.verify_quantization(quant: str) -> None |
Verify quantization method is supported. | #L297-L304 |
vllm_mlx.worker.MLXWorker |
class | vllm_mlx.worker.MLXWorker(vllm_config: 'VllmConfig', local_rank: int, rank: int, distributed_init_method: str, is_driver_worker: bool = False) |
Worker implementation for MLX-based inference on Apple Silicon. | #L23-L278 |
vllm_mlx.worker.MLXWorker.__init__ |
method | vllm_mlx.worker.MLXWorker.__init__(vllm_config: 'VllmConfig', local_rank: int, rank: int, distributed_init_method: str, is_driver_worker: bool = False) -> None |
Initialize MLX worker. | #L36-L75 |
vllm_mlx.worker.MLXWorker.__repr__ |
method | vllm_mlx.worker.MLXWorker.__repr__() -> str |
Method `MLXWorker.__repr__` returns `f'<MLXWorker rank={self.rank} local_rank={self.local_rank}>'`. | #L277-L278 |
vllm_mlx.worker.MLXWorker.add_lora |
method | vllm_mlx.worker.MLXWorker.add_lora(lora_request) -> bool |
Report that dynamically adding a LoRA adapter is unsupported. | #L222-L226 |
vllm_mlx.worker.MLXWorker.check_health |
method | vllm_mlx.worker.MLXWorker.check_health() -> None |
Check worker health. | #L191-L200 |
vllm_mlx.worker.MLXWorker.compile_or_warm_up_model |
method | vllm_mlx.worker.MLXWorker.compile_or_warm_up_model() -> None |
Warm up model for inference. | #L161-L165 |
vllm_mlx.worker.MLXWorker.determine_available_memory |
method | vllm_mlx.worker.MLXWorker.determine_available_memory() -> int |
Determine available memory for KV cache. | #L113-L143 |
vllm_mlx.worker.MLXWorker.execute_model |
method | vllm_mlx.worker.MLXWorker.execute_model(scheduler_output: 'SchedulerOutput') -> 'ModelRunnerOutput | None' |
Execute model inference for the given scheduler output. | #L167-L183 |
vllm_mlx.worker.MLXWorker.get_cache_block_size_bytes |
method | vllm_mlx.worker.MLXWorker.get_cache_block_size_bytes() -> int |
Get size of a cache block in bytes. | #L259-L271 |
vllm_mlx.worker.MLXWorker.get_kv_cache_spec |
method | vllm_mlx.worker.MLXWorker.get_kv_cache_spec() -> dict |
Get KV cache specification. | #L155-L159 |
vllm_mlx.worker.MLXWorker.get_model |
method | vllm_mlx.worker.MLXWorker.get_model() -> not annotated |
Get the underlying model. | #L185-L189 |
vllm_mlx.worker.MLXWorker.init_device |
method | vllm_mlx.worker.MLXWorker.init_device() -> None |
Initialize MLX device and verify it's working. | #L77-L103 |
vllm_mlx.worker.MLXWorker.initialize_cache |
method | vllm_mlx.worker.MLXWorker.initialize_cache(num_gpu_blocks: int, num_cpu_blocks: int) -> None |
Initialize KV cache with the given size. | #L145-L153 |
vllm_mlx.worker.MLXWorker.list_loras |
method | vllm_mlx.worker.MLXWorker.list_loras() -> set[int] |
Return the empty set because runtime LoRA adapters are unsupported. | #L238-L241 |
vllm_mlx.worker.MLXWorker.load_model |
method | vllm_mlx.worker.MLXWorker.load_model() -> None |
Load model using mlx-lm. | #L105-L111 |
vllm_mlx.worker.MLXWorker.pin_lora |
method | vllm_mlx.worker.MLXWorker.pin_lora(lora_id: int) -> bool |
Report that pinning a LoRA adapter is unsupported. | #L233-L236 |
vllm_mlx.worker.MLXWorker.profile |
method | vllm_mlx.worker.MLXWorker.profile(is_start: bool = True) -> None |
Profiling (not yet implemented for MLX). | #L273-L275 |
vllm_mlx.worker.MLXWorker.remove_lora |
method | vllm_mlx.worker.MLXWorker.remove_lora(lora_id: int) -> bool |
Report that dynamically removing a LoRA adapter is unsupported. | #L228-L231 |
vllm_mlx.worker.MLXWorker.shutdown |
method | vllm_mlx.worker.MLXWorker.shutdown() -> None |
Clean up resources. | #L202-L219 |
vllm_mlx.worker.MLXWorker.sleep |
method | vllm_mlx.worker.MLXWorker.sleep(level: int = 1) -> None |
Leave the worker active because MLX unified memory has no sleep mode. | #L244-L247 |
vllm_mlx.worker.MLXWorker.vocab_size |
method | vllm_mlx.worker.MLXWorker.vocab_size() -> int |
Get vocabulary size. | #L255-L257 |
vllm_mlx.worker.MLXWorker.wake_up |
method | vllm_mlx.worker.MLXWorker.wake_up(tags: list[str] | None = None) -> None |
Perform no work because the MLX worker never enters sleep mode. | #L249-L252 |