vllm_mlx.tool_parsers.auto_tool_parser

Auto-detecting tool call parser for vllm-mlx.

View the complete module source at #L1-L414.

API details

Each callable below includes its exact signature, type annotations, inputs, defaults, return contract, documented exceptions, implementation source, and parsed docstring sections when the source provides them.

vllm_mlx.tool_parsers.auto_tool_parser

Auto-detecting tool call parser for vllm-mlx.

Automatically detects and parses tool calls from various model formats.

vllm_mlx.tool_parsers.auto_tool_parser.AutoToolParser

AutoToolParser(tokenizer: PreTrainedTokenizerBase | None = None)

Bases: ToolParser

Auto-detecting tool call parser.

Tries multiple formats in order: 1. Gemma 4: <|tool_call>call:name{...} 2. Mistral: [TOOL_CALLS] ... 3. Qwen bracket: [Calling tool: func_name({...})] 4. Qwen/Hermes XML: {"name": "...", "arguments": {...}} 5. Llama: {"arg": "value"} 6. Nemotron: ... 7. Raw JSON: {"name": "...", "arguments": {...}}

This is the default parser when no specific parser is selected.

Source code in vllm_mlx/tool_parsers/abstract_tool_parser.py

def __init__(self, tokenizer: PreTrainedTokenizerBase | None = None):
    """
    Initialize the tool parser.

    Args:
        tokenizer: The tokenizer for the model (optional, some parsers need it)
    """
    self.model_tokenizer = tokenizer
    # State for streaming parsing
    self.current_tool_id: int = -1
    self.prev_tool_call_arr: list[dict] = []

vllm_mlx.tool_parsers.auto_tool_parser.AutoToolParser.MISTRAL_TOKEN class-attribute instance-attribute

MISTRAL_TOKEN = '[TOOL_CALLS]'

vllm_mlx.tool_parsers.auto_tool_parser.AutoToolParser.QWEN_BRACKET_PATTERN class-attribute instance-attribute

QWEN_BRACKET_PATTERN = re.compile('\\[Calling tool:\\s*(\\w+)\\((\\{.*?\\})\\)\\]', re.DOTALL)

vllm_mlx.tool_parsers.auto_tool_parser.AutoToolParser.QWEN_XML_PATTERN class-attribute instance-attribute

QWEN_XML_PATTERN = re.compile('<tool_call>\\s*(\\{.*?\\})\\s*</tool_call>', re.DOTALL)

vllm_mlx.tool_parsers.auto_tool_parser.AutoToolParser.LLAMA_PATTERN class-attribute instance-attribute

LLAMA_PATTERN = re.compile('<function=([^>]+)>(\\{.*?\\})</function>', re.DOTALL)

vllm_mlx.tool_parsers.auto_tool_parser.AutoToolParser.NEMOTRON_PATTERN class-attribute instance-attribute

NEMOTRON_PATTERN = re.compile('<tool_call>\\s*<function=([^>]+)>(.*?)</function>\\s*</tool_call>', re.DOTALL)

vllm_mlx.tool_parsers.auto_tool_parser.AutoToolParser.NEMOTRON_PARAM_PATTERN class-attribute instance-attribute

NEMOTRON_PARAM_PATTERN = re.compile('<parameter=([^>]+)>\\s*(.*?)\\s*</parameter>', re.DOTALL)

vllm_mlx.tool_parsers.auto_tool_parser.AutoToolParser.BARE_BRACKET_PATTERN class-attribute instance-attribute

BARE_BRACKET_PATTERN = re.compile('\\[(\\w+)\\((\\{.*?\\})\\)\\]', re.DOTALL)

vllm_mlx.tool_parsers.auto_tool_parser.AutoToolParser.BARE_BRACKET_PARTIAL_PATTERN class-attribute instance-attribute

BARE_BRACKET_PARTIAL_PATTERN = re.compile('\\[\\w+\\($')

vllm_mlx.tool_parsers.auto_tool_parser.AutoToolParser.extract_tool_calls

extract_tool_calls(model_output: str, request: dict[str, Any] | None = None) -> ExtractedToolCallInformation

Extract tool calls by trying all known formats.

Source code in vllm_mlx/tool_parsers/auto_tool_parser.py

def extract_tool_calls(
    self, model_output: str, request: dict[str, Any] | None = None
) -> ExtractedToolCallInformation:
    """
    Extract tool calls by trying all known formats.
    """
    tool_calls: list[dict[str, Any]] = []
    cleaned_text = model_output

    # 1. Try Gemma 4 format (most distinctive marker)
    if "<|tool_call>" in model_output:
        gemma_parser = Gemma4ToolParser()
        result = gemma_parser.extract_tool_calls(model_output, request)
        if result.tools_called:
            return result

    # 2. Try Mistral format
    if self.MISTRAL_TOKEN in model_output:
        parts = model_output.split(self.MISTRAL_TOKEN)
        content = parts[0].strip()
        raw_tool_calls = parts[1:]

        for raw in raw_tool_calls:
            raw = raw.strip()
            if not raw:
                continue

            # New Mistral format: func_name{"args"}
            if not raw.startswith("[") and "{" in raw:
                end_name = raw.find("{")
                name = raw[:end_name].strip()
                args = raw[end_name:]
                if name:
                    tool_calls.append(
                        {"id": generate_tool_id(), "name": name, "arguments": args}
                    )
                continue

            # Old Mistral format: [{"name": "...", "arguments": {...}}]
            try:
                parsed = json.loads(raw)
                if isinstance(parsed, list):
                    for item in parsed:
                        if isinstance(item, dict) and "name" in item:
                            args = item.get("arguments", {})
                            tool_calls.append(
                                {
                                    "id": generate_tool_id(),
                                    "name": item["name"],
                                    "arguments": (
                                        json.dumps(args, ensure_ascii=False)
                                        if isinstance(args, dict)
                                        else str(args)
                                    ),
                                }
                            )
            except json.JSONDecodeError:
                pass

        if tool_calls:
            return ExtractedToolCallInformation(
                tools_called=True,
                tool_calls=tool_calls,
                content=content if content else None,
            )

    # 3. Try Qwen bracket pattern
    bracket_matches = self.QWEN_BRACKET_PATTERN.findall(model_output)
    for name, args_str in bracket_matches:
        try:
            arguments = json.loads(args_str)
            tool_calls.append(
                {
                    "id": generate_tool_id(),
                    "name": name.strip(),
                    "arguments": (
                        json.dumps(arguments, ensure_ascii=False)
                        if isinstance(arguments, dict)
                        else str(arguments)
                    ),
                }
            )
        except json.JSONDecodeError:
            tool_calls.append(
                {
                    "id": generate_tool_id(),
                    "name": name.strip(),
                    "arguments": args_str,
                }
            )

    if bracket_matches:
        cleaned_text = self.QWEN_BRACKET_PATTERN.sub("", cleaned_text).strip()

    # 4. Try bare bracket format: [func({...})]
    bare_matches = self.BARE_BRACKET_PATTERN.findall(cleaned_text)
    for name, args_str in bare_matches:
        try:
            arguments = json.loads(args_str)
            tool_calls.append(
                {
                    "id": generate_tool_id(),
                    "name": name.strip(),
                    "arguments": (
                        json.dumps(arguments, ensure_ascii=False)
                        if isinstance(arguments, dict)
                        else str(arguments)
                    ),
                }
            )
        except json.JSONDecodeError:
            tool_calls.append(
                {
                    "id": generate_tool_id(),
                    "name": name.strip(),
                    "arguments": args_str,
                }
            )

    if bare_matches:
        cleaned_text = self.BARE_BRACKET_PATTERN.sub("", cleaned_text).strip()

    # 5. Try Nemotron pattern (before Qwen XML as it's more specific)
    nemotron_matches = self.NEMOTRON_PATTERN.findall(cleaned_text)
    for name, params_block in nemotron_matches:
        params = self.NEMOTRON_PARAM_PATTERN.findall(params_block)
        arguments = {p_name.strip(): p_value.strip() for p_name, p_value in params}
        tool_calls.append(
            {
                "id": generate_tool_id(),
                "name": name.strip(),
                "arguments": json.dumps(arguments, ensure_ascii=False),
            }
        )

    if nemotron_matches:
        cleaned_text = self.NEMOTRON_PATTERN.sub("", cleaned_text).strip()

    # 6. Try Qwen/Hermes XML pattern
    xml_matches = self.QWEN_XML_PATTERN.findall(cleaned_text)
    for match in xml_matches:
        try:
            data = json.loads(match)
            name = data.get("name", "")
            arguments = data.get("arguments", {})
            if name:
                tool_calls.append(
                    {
                        "id": generate_tool_id(),
                        "name": name,
                        "arguments": (
                            json.dumps(arguments, ensure_ascii=False)
                            if isinstance(arguments, dict)
                            else str(arguments)
                        ),
                    }
                )
        except json.JSONDecodeError:
            continue

    if xml_matches:
        cleaned_text = self.QWEN_XML_PATTERN.sub("", cleaned_text).strip()

    # 7. Try Llama pattern
    llama_matches = self.LLAMA_PATTERN.findall(cleaned_text)
    for name, args_str in llama_matches:
        try:
            arguments = json.loads(args_str)
            tool_calls.append(
                {
                    "id": generate_tool_id(),
                    "name": name.strip(),
                    "arguments": (
                        json.dumps(arguments, ensure_ascii=False)
                        if isinstance(arguments, dict)
                        else str(arguments)
                    ),
                }
            )
        except json.JSONDecodeError:
            tool_calls.append(
                {
                    "id": generate_tool_id(),
                    "name": name.strip(),
                    "arguments": args_str,
                }
            )

    if llama_matches:
        cleaned_text = self.LLAMA_PATTERN.sub("", cleaned_text).strip()

    # 8. Fallback: Try raw JSON
    if not tool_calls:
        raw_calls = self._parse_raw_json_tool_calls(cleaned_text)
        if raw_calls:
            tool_calls.extend(raw_calls)
            cleaned_text = ""

    if tool_calls:
        return ExtractedToolCallInformation(
            tools_called=True,
            tool_calls=tool_calls,
            content=cleaned_text if cleaned_text else None,
        )
    else:
        return ExtractedToolCallInformation(
            tools_called=False, tool_calls=[], content=model_output
        )

vllm_mlx.tool_parsers.auto_tool_parser.AutoToolParser._parse_raw_json_tool_calls

_parse_raw_json_tool_calls(text: str) -> list[dict[str, Any]]

Parse raw JSON tool calls from text.

Handles: - Single JSON object: {"name": "func", "arguments": {...}} - JSON array: [{...}, {...}]

Source code in vllm_mlx/tool_parsers/auto_tool_parser.py

def _parse_raw_json_tool_calls(self, text: str) -> list[dict[str, Any]]:
    """
    Parse raw JSON tool calls from text.

    Handles:
    - Single JSON object: {"name": "func", "arguments": {...}}
    - JSON array: [{...}, {...}]
    """
    if not text:
        return []

    text = text.strip()
    tool_calls = []

    # Try JSON array first
    if text.startswith("["):
        try:
            parsed = json.loads(text)
            if isinstance(parsed, list):
                for item in parsed:
                    if isinstance(item, dict):
                        # Support "name" and "type" fields (Granite)
                        func_name = item.get("name") or item.get("type")
                        if func_name:
                            args = item.get("arguments", {})
                            tool_calls.append(
                                {
                                    "id": generate_tool_id(),
                                    "name": func_name,
                                    "arguments": (
                                        json.dumps(args, ensure_ascii=False)
                                        if isinstance(args, dict)
                                        else str(args)
                                    ),
                                }
                            )
                return tool_calls
        except json.JSONDecodeError:
            pass

    # Find JSON objects with balanced braces
    depth = 0
    start = None

    for i, char in enumerate(text):
        if char == "{":
            if depth == 0:
                start = i
            depth += 1
        elif char == "}":
            depth -= 1
            if depth < 0:
                # Reset on unbalanced braces
                depth = 0
                start = None
                continue
            if depth == 0 and start is not None:
                json_str = text[start : i + 1]
                try:
                    obj = json.loads(json_str)
                    if isinstance(obj, dict):
                        # Support both "name" and "type" fields
                        func_name = obj.get("name") or obj.get("type")
                        if func_name:
                            args = obj.get("arguments", {})
                            tool_calls.append(
                                {
                                    "id": generate_tool_id(),
                                    "name": func_name,
                                    "arguments": (
                                        json.dumps(args, ensure_ascii=False)
                                        if isinstance(args, dict)
                                        else str(args)
                                    ),
                                }
                            )
                except json.JSONDecodeError:
                    pass
                start = None

    return tool_calls

vllm_mlx.tool_parsers.auto_tool_parser.AutoToolParser.extract_tool_calls_streaming

extract_tool_calls_streaming(previous_text: str, current_text: str, delta_text: str, previous_token_ids: Sequence[int] | None = None, current_token_ids: Sequence[int] | None = None, delta_token_ids: Sequence[int] | None = None, request: dict[str, Any] | None = None) -> dict[str, Any] | None

Extract tool calls from streaming model output.

Uses simple heuristics to detect when a tool call might be complete.

Source code in vllm_mlx/tool_parsers/auto_tool_parser.py

def extract_tool_calls_streaming(
    self,
    previous_text: str,
    current_text: str,
    delta_text: str,
    previous_token_ids: Sequence[int] | None = None,
    current_token_ids: Sequence[int] | None = None,
    delta_token_ids: Sequence[int] | None = None,
    request: dict[str, Any] | None = None,
) -> dict[str, Any] | None:
    """
    Extract tool calls from streaming model output.

    Uses simple heuristics to detect when a tool call might be complete.
    """
    # Check for any tool call markers
    markers = [
        "<|tool_call>",
        self.MISTRAL_TOKEN,
        "[Calling tool:",
        "[",
        "<tool_call>",
        "<function=",
    ]

    has_marker = any(m in current_text for m in markers) and (
        self.BARE_BRACKET_PARTIAL_PATTERN.search(current_text) is not None
        or self.BARE_BRACKET_PATTERN.search(current_text) is not None
        or "[Calling tool:" in current_text
        or self.MISTRAL_TOKEN in current_text
        or "<" in current_text
    )

    if (
        self.BARE_BRACKET_PARTIAL_PATTERN.search(current_text) is not None
        and self.BARE_BRACKET_PATTERN.search(current_text) is None
    ):
        return None

    if not has_marker:
        return {"content": delta_text}

    # Check for completion markers
    end_markers = ["<tool_call|>", "</tool_call>", "</function>", ")]"]
    if any(m in delta_text for m in end_markers):
        result = self.extract_tool_calls(current_text)
        if result.tools_called:
            return {
                "tool_calls": [
                    {
                        "index": i,
                        "id": tc["id"],
                        "type": "function",
                        "function": {
                            "name": tc["name"],
                            "arguments": tc["arguments"],
                        },
                    }
                    for i, tc in enumerate(result.tool_calls)
                ]
            }

    return None

vllm_mlx.tool_parsers.auto_tool_parser.generate_tool_id

generate_tool_id() -> str

Generate a unique tool call ID.

Source code in vllm_mlx/tool_parsers/auto_tool_parser.py

def generate_tool_id() -> str:
    """Generate a unique tool call ID."""
    return f"call_{uuid.uuid4().hex[:8]}"

Complete contract reference

Expand any definition for its exact inputs, annotations, defaults, return contract, directly raised exceptions, source-grounded behavior, and immutable line link. This section includes private and nested definitions that ordinary API generators omit.

vllm_mlx.tool_parsers.auto_tool_parser.generate_tool_id · function

vllm_mlx.tool_parsers.auto_tool_parser.generate_tool_id() -> str

Generate a unique tool call ID.

Parameters

This callable has no explicit inputs.

Returns

• Type: str
• Direct return expressions: f'call_{uuid.uuid4().hex[:8]}'

Exceptions and behavior

Function generate_tool_id calls uuid.uuid4; returns f'call_{uuid.uuid4().hex[:8]}'. No direct raise statement appears in this definition.

View source #L22-L24.

vllm_mlx.tool_parsers.auto_tool_parser.AutoToolParser · class

vllm_mlx.tool_parsers.auto_tool_parser.AutoToolParser()

Auto-detecting tool call parser.

Parameters

This callable has no explicit inputs.

Returns

• Constructs: vllm_mlx.tool_parsers.auto_tool_parser.AutoToolParser

Exceptions and behavior

Class AutoToolParser derives from ToolParser and declares 3 direct member(s). No direct raise statement appears in this definition.

View source #L28-L414.

vllm_mlx.tool_parsers.auto_tool_parser.AutoToolParser.extract_tool_calls · method

vllm_mlx.tool_parsers.auto_tool_parser.AutoToolParser.extract_tool_calls(model_output: str, request: dict[str, Any] | None = None) -> ExtractedToolCallInformation

Extract tool calls by trying all known formats.

Parameters

Name	Type	Required	Default	Description
model_output	str	yes	none	Required positional or keyword input.
request	dict[str, Any] \| None	no	None	Optional positional or keyword input; defaults to None.

Returns

• Type: ExtractedToolCallInformation
• Direct return expressions: result; ExtractedToolCallInformation(tools_called=True, tool_calls=tool_calls, content=content if content else None); ExtractedToolCallInformation(tools_called=True, tool_calls=tool_calls, content=cleaned_text if cleaned_text else None); ExtractedToolCallInformation(tools_called=False, tool_calls=[], content=model_output)

Exceptions and behavior

Method AutoToolParser.extract_tool_calls calls Gemma4ToolParser, gemma_parser.extract_tool_calls, model_output.split, parts[0].strip; has 4 explicit return paths. No direct raise statement appears in this definition.

View source #L61-L268.

vllm_mlx.tool_parsers.auto_tool_parser.AutoToolParser._parse_raw_json_tool_calls · method

vllm_mlx.tool_parsers.auto_tool_parser.AutoToolParser._parse_raw_json_tool_calls(text: str) -> list[dict[str, Any]]

Parse raw JSON tool calls from text.

Parameters

Name	Type	Required	Default	Description
text	str	yes	none	Required positional or keyword input.

Returns

• Type: list[dict[str, Any]]
• Direct return expressions: []; tool_calls

Exceptions and behavior

Method AutoToolParser._parse_raw_json_tool_calls calls text.strip, text.startswith, json.loads, isinstance; has 2 explicit return paths. No direct raise statement appears in this definition.

View source #L270-L350.

vllm_mlx.tool_parsers.auto_tool_parser.AutoToolParser.extract_tool_calls_streaming · method

vllm_mlx.tool_parsers.auto_tool_parser.AutoToolParser.extract_tool_calls_streaming(previous_text: str, current_text: str, delta_text: str, previous_token_ids: Sequence[int] | None = None, current_token_ids: Sequence[int] | None = None, delta_token_ids: Sequence[int] | None = None, request: dict[str, Any] | None = None) -> dict[str, Any] | None

Extract tool calls from streaming model output.

Parameters

Name	Type	Required	Default	Description
previous_text	str	yes	none	Required positional or keyword input.
current_text	str	yes	none	Required positional or keyword input.
delta_text	str	yes	none	Required positional or keyword input.
previous_token_ids	Sequence[int] \| None	no	None	Optional positional or keyword input; defaults to None.
current_token_ids	Sequence[int] \| None	no	None	Optional positional or keyword input; defaults to None.
delta_token_ids	Sequence[int] \| None	no	None	Optional positional or keyword input; defaults to None.
request	dict[str, Any] \| None	no	None	Optional positional or keyword input; defaults to None.

Returns

• Type: dict[str, Any] | None
• Direct return expressions: None; {'content': delta_text}; {'tool_calls': [{'index': i, 'id': tc['id'], 'type': 'function', 'function': {'name': tc['name'], 'arguments': tc['argu…

Exceptions and behavior

Method AutoToolParser.extract_tool_calls_streaming calls any, self.BARE_BRACKET_PARTIAL_PATTERN.search, self.BARE_BRACKET_PATTERN.search, self.extract_tool_calls; has 3 explicit return paths. No direct raise statement appears in this definition.

View source #L352-L414.

Complete symbol map

This map also includes private definitions and nested helpers. The signature column exposes every explicit input even when an internal helper has no dedicated parameter prose.

Symbol	Kind	Signature and inputs	What it does	Source
generate_tool_id	function	generate_tool_id() -> str	Generate a unique tool call ID.	#L22-L24
AutoToolParser	class	AutoToolParser()	Auto-detecting tool call parser.	#L28-L414
AutoToolParser.extract_tool_calls	method	AutoToolParser.extract_tool_calls(model_output: str, request: dict[str, Any] \| None = None) -> ExtractedToolCallInformation	Extract tool calls by trying all known formats.	#L61-L268
AutoToolParser._parse_raw_json_tool_calls	method	AutoToolParser._parse_raw_json_tool_calls(text: str) -> list[dict[str, Any]]	Parse raw JSON tool calls from text.	#L270-L350
AutoToolParser.extract_tool_calls_streaming	method	AutoToolParser.extract_tool_calls_streaming(previous_text: str, current_text: str, delta_text: str, previous_token_ids: Sequence[int] \| None = None, current_token_ids: Sequence[int] \| None = None, delta_token_ids: Sequence[int] \| None = None, request: dict[str, Any] \| None = None) -> dict[str, Any] \| None	Extract tool calls from streaming model output.	#L352-L414