# `examples.mllm_benchmark`

MLLM Benchmark Script for vllm-mlx Tests Multimodal Language Models with real images of dogs from Wikimedia Commons at different resolutions and measures performance metrics.

[View the complete module source at #L1-L443](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/mllm_benchmark.py#L1-L443).

## API details

Each callable below includes its exact signature, type annotations, inputs, defaults, return contract, documented exceptions, implementation source, and parsed docstring sections when the source provides them.

::: examples.mllm_benchmark
    options:
      members:
        - TEST_IMAGES
        - PRIMARY_DOG_IMAGE
        - BenchmarkResult
        - download_image
        - resize_image
        - image_to_base64
        - run_mllm_request
        - benchmark_resolution
        - run_benchmark
        - print_results
        - save_results
        - main
      filters: []
      show_if_no_docstring: true

## Complete contract reference

Expand any definition for its exact inputs, annotations, defaults, return contract, directly raised exceptions, source-grounded behavior, and immutable line link. This section includes private and nested definitions that ordinary API generators omit.

<details class="api-contract" id="contract-examples.mllm_benchmark.BenchmarkResult" markdown="1">
<summary><code>examples.mllm_benchmark.BenchmarkResult</code> · class</summary>

```python
examples.mllm_benchmark.BenchmarkResult(resolution: str, width: int, height: int, pixels: int, time_seconds: float, tokens_generated: int, tokens_per_second: float, response_preview: str)
```

Result from a single benchmark run.

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `resolution` | `str` | `yes` | `none` | Required constructor field. |
| `width` | `int` | `yes` | `none` | Required constructor field. |
| `height` | `int` | `yes` | `none` | Required constructor field. |
| `pixels` | `int` | `yes` | `none` | Required constructor field. |
| `time_seconds` | `float` | `yes` | `none` | Required constructor field. |
| `tokens_generated` | `int` | `yes` | `none` | Required constructor field. |
| `tokens_per_second` | `float` | `yes` | `none` | Required constructor field. |
| `response_preview` | `str` | `yes` | `none` | Required constructor field. |

**Returns**

- Constructs: `examples.mllm_benchmark.BenchmarkResult`

**Exceptions and behavior**

Class `BenchmarkResult` declares 0 direct member(s).
No direct `raise` statement appears in this definition.

[View source #L65-L74](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/mllm_benchmark.py#L65-L74).

</details>

<details class="api-contract" id="contract-examples.mllm_benchmark.download_image" markdown="1">
<summary><code>examples.mllm_benchmark.download_image</code> · function</summary>

```python
examples.mllm_benchmark.download_image(url: str, timeout: int = 30) -> Image.Image
```

Download image from URL and return PIL Image.

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `url` | `str` | `yes` | `none` | Required positional or keyword input. |
| `timeout` | `int` | `no` | `30` | Optional positional or keyword input; defaults to `30`. |

**Returns**

- Type: `Image.Image`
- Direct return expressions: `Image.open(io.BytesIO(response.content))`

**Exceptions and behavior**

Function `download_image` calls `requests.get`, `response.raise_for_status`, `Image.open`, `io.BytesIO`; returns `Image.open(io.BytesIO(response.content))`.
No direct `raise` statement appears in this definition.

[View source #L77-L84](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/mllm_benchmark.py#L77-L84).

</details>

<details class="api-contract" id="contract-examples.mllm_benchmark.resize_image" markdown="1">
<summary><code>examples.mllm_benchmark.resize_image</code> · function</summary>

```python
examples.mllm_benchmark.resize_image(img: Image.Image, width: int, height: int) -> Image.Image
```

Resize image to specified dimensions.

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `img` | `Image.Image` | `yes` | `none` | Required positional or keyword input. |
| `width` | `int` | `yes` | `none` | Required positional or keyword input. |
| `height` | `int` | `yes` | `none` | Required positional or keyword input. |

**Returns**

- Type: `Image.Image`
- Direct return expressions: `img.resize((width, height), Image.Resampling.LANCZOS)`

**Exceptions and behavior**

Function `resize_image` calls `img.resize`; returns `img.resize((width, height), Image.Resampling.LANCZOS)`.
No direct `raise` statement appears in this definition.

[View source #L87-L89](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/mllm_benchmark.py#L87-L89).

</details>

<details class="api-contract" id="contract-examples.mllm_benchmark.image_to_base64" markdown="1">
<summary><code>examples.mllm_benchmark.image_to_base64</code> · function</summary>

```python
examples.mllm_benchmark.image_to_base64(img: Image.Image, format: str = 'JPEG') -> str
```

Convert PIL Image to base64 data URL.

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `img` | `Image.Image` | `yes` | `none` | Required positional or keyword input. |
| `format` | `str` | `no` | `'JPEG'` | Optional positional or keyword input; defaults to `'JPEG'`. |

**Returns**

- Type: `str`
- Direct return expressions: `f'data:{mime};base64,{b64}'`

**Exceptions and behavior**

Function `image_to_base64` calls `Image.new`, `background.paste`, `img.split`, `img.convert`; returns `f'data:{mime};base64,{b64}'`.
No direct `raise` statement appears in this definition.

[View source #L92-L106](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/mllm_benchmark.py#L92-L106).

</details>

<details class="api-contract" id="contract-examples.mllm_benchmark.run_mllm_request" markdown="1">
<summary><code>examples.mllm_benchmark.run_mllm_request</code> · function</summary>

```python
examples.mllm_benchmark.run_mllm_request(server_url: str, image_b64: str, prompt: str = 'Describe this image in detail. What do you see?', max_tokens: int = 256, model: str = 'default') -> tuple[str, float, int]
```

Send an MLLM request to the server.

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `server_url` | `str` | `yes` | `none` | Required positional or keyword input. |
| `image_b64` | `str` | `yes` | `none` | Required positional or keyword input. |
| `prompt` | `str` | `no` | `'Describe this image in detail. What do you see?'` | Optional positional or keyword input; defaults to `'Describe this image in detail. What do you see?'`. |
| `max_tokens` | `int` | `no` | `256` | Optional positional or keyword input; defaults to `256`. |
| `model` | `str` | `no` | `'default'` | Optional positional or keyword input; defaults to `'default'`. |

**Returns**

- Type: `tuple[str, float, int]`
- Direct return expressions: `(text, elapsed, tokens)`

**Exceptions and behavior**

Function `run_mllm_request` calls `time.perf_counter`, `requests.post`, `response.raise_for_status`, `response.json`; returns `(text, elapsed, tokens)`.
No direct `raise` statement appears in this definition.

[View source #L109-L153](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/mllm_benchmark.py#L109-L153).

</details>

<details class="api-contract" id="contract-examples.mllm_benchmark.benchmark_resolution" markdown="1">
<summary><code>examples.mllm_benchmark.benchmark_resolution</code> · function</summary>

```python
examples.mllm_benchmark.benchmark_resolution(server_url: str, base_image: Image.Image, width: int, height: int, model: str, warmup: bool = False) -> BenchmarkResult
```

Run benchmark for a specific resolution.

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `server_url` | `str` | `yes` | `none` | Required positional or keyword input. |
| `base_image` | `Image.Image` | `yes` | `none` | Required positional or keyword input. |
| `width` | `int` | `yes` | `none` | Required positional or keyword input. |
| `height` | `int` | `yes` | `none` | Required positional or keyword input. |
| `model` | `str` | `yes` | `none` | Required positional or keyword input. |
| `warmup` | `bool` | `no` | `False` | Optional positional or keyword input; defaults to `False`. |

**Returns**

- Type: `BenchmarkResult`
- Direct return expressions: `BenchmarkResult(resolution=resolution_name, width=width, height=height, pixels=pixels, time_seconds=elapsed, tokens_gen…`

**Exceptions and behavior**

Function `benchmark_resolution` calls `resize_image`, `image_to_base64`, `print`, `run_mllm_request`; returns `BenchmarkResult(resolution=resolution_name, width=width, height=height, pixels=pixels, time_seconds=elapsed, tokens_gen…`.
No direct `raise` statement appears in this definition.

[View source #L156-L198](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/mllm_benchmark.py#L156-L198).

</details>

<details class="api-contract" id="contract-examples.mllm_benchmark.run_benchmark" markdown="1">
<summary><code>examples.mllm_benchmark.run_benchmark</code> · function</summary>

```python
examples.mllm_benchmark.run_benchmark(server_url: str = 'http://localhost:8000', resolutions: list[tuple[int, int]] = None, warmup_runs: int = 1, image_url: str = None) -> list[BenchmarkResult]
```

Run full MLLM benchmark across multiple resolutions.

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `server_url` | `str` | `no` | `'http://localhost:8000'` | URL of the vllm-mlx server |
| `resolutions` | `list[tuple[int, int]]` | `no` | `None` | List of (width, height) tuples to test |
| `warmup_runs` | `int` | `no` | `1` | Number of warmup runs before measuring |
| `image_url` | `str` | `no` | `None` | URL of image to use (default: dog from Wikimedia) |

**Returns**

- Type: `list[BenchmarkResult]`
- Direct return expressions: `[]`; `results`

**Exceptions and behavior**

Function `run_benchmark` calls `print`, `requests.get`, `health.raise_for_status`, `health.json`; has 2 explicit return paths.
No direct `raise` statement appears in this definition.

[View source #L201-L293](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/mllm_benchmark.py#L201-L293).

</details>

<details class="api-contract" id="contract-examples.mllm_benchmark.print_results" markdown="1">
<summary><code>examples.mllm_benchmark.print_results</code> · function</summary>

```python
examples.mllm_benchmark.print_results(results: list[BenchmarkResult]) -> not annotated
```

Print benchmark results in a nice table.

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `results` | `list[BenchmarkResult]` | `yes` | `none` | Required positional or keyword input. |

**Returns**

- Type: `not annotated`
- Direct return expressions: `None`

**Exceptions and behavior**

Function `print_results` calls `print`, `table_data.append`, `tabulate`, `sum`; returns `None`.
No direct `raise` statement appears in this definition.

[View source #L296-L338](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/mllm_benchmark.py#L296-L338).

</details>

<details class="api-contract" id="contract-examples.mllm_benchmark.save_results" markdown="1">
<summary><code>examples.mllm_benchmark.save_results</code> · function</summary>

```python
examples.mllm_benchmark.save_results(results: list[BenchmarkResult], output_path: str) -> not annotated
```

Save benchmark results to JSON file.

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `results` | `list[BenchmarkResult]` | `yes` | `none` | Required positional or keyword input. |
| `output_path` | `str` | `yes` | `none` | Required positional or keyword input. |

**Returns**

- Type: `not annotated`

**Exceptions and behavior**

Function `save_results` calls `time.strftime`, `open`, `json.dump`, `print`.
No direct `raise` statement appears in this definition.

[View source #L341-L364](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/mllm_benchmark.py#L341-L364).

</details>

<details class="api-contract" id="contract-examples.mllm_benchmark.main" markdown="1">
<summary><code>examples.mllm_benchmark.main</code> · function</summary>

```python
examples.mllm_benchmark.main() -> not annotated
```

Function `main` calls `argparse.ArgumentParser`, `parser.add_argument`, `parser.parse_args`, `run_benchmark`.

**Parameters**

This callable has no explicit inputs.

**Returns**

- Type: `not annotated`

**Exceptions and behavior**

Function `main` calls `argparse.ArgumentParser`, `parser.add_argument`, `parser.parse_args`, `run_benchmark`.
No direct `raise` statement appears in this definition.

[View source #L367-L439](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/mllm_benchmark.py#L367-L439).

</details>

## Complete symbol map

This map also includes private definitions and nested helpers. The signature column exposes every explicit input even when an internal helper has no dedicated parameter prose.

| Symbol | Kind | Signature and inputs | What it does | Source |
| --- | --- | --- | --- | --- |
| [`BenchmarkResult`](#contract-examples.mllm_benchmark.BenchmarkResult) | class | `BenchmarkResult(resolution: str, width: int, height: int, pixels: int, time_seconds: float, tokens_generated: int, tokens_per_second: float, response_preview: str)` | Result from a single benchmark run. | [#L65-L74](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/mllm_benchmark.py#L65-L74) |
| [`download_image`](#contract-examples.mllm_benchmark.download_image) | function | `download_image(url: str, timeout: int = 30) -> Image.Image` | Download image from URL and return PIL Image. | [#L77-L84](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/mllm_benchmark.py#L77-L84) |
| [`resize_image`](#contract-examples.mllm_benchmark.resize_image) | function | `resize_image(img: Image.Image, width: int, height: int) -> Image.Image` | Resize image to specified dimensions. | [#L87-L89](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/mllm_benchmark.py#L87-L89) |
| [`image_to_base64`](#contract-examples.mllm_benchmark.image_to_base64) | function | `image_to_base64(img: Image.Image, format: str = 'JPEG') -> str` | Convert PIL Image to base64 data URL. | [#L92-L106](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/mllm_benchmark.py#L92-L106) |
| [`run_mllm_request`](#contract-examples.mllm_benchmark.run_mllm_request) | function | `run_mllm_request(server_url: str, image_b64: str, prompt: str = 'Describe this image in detail. What do you see?', max_tokens: int = 256, model: str = 'default') -> tuple[str, float, int]` | Send an MLLM request to the server. | [#L109-L153](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/mllm_benchmark.py#L109-L153) |
| [`benchmark_resolution`](#contract-examples.mllm_benchmark.benchmark_resolution) | function | `benchmark_resolution(server_url: str, base_image: Image.Image, width: int, height: int, model: str, warmup: bool = False) -> BenchmarkResult` | Run benchmark for a specific resolution. | [#L156-L198](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/mllm_benchmark.py#L156-L198) |
| [`run_benchmark`](#contract-examples.mllm_benchmark.run_benchmark) | function | `run_benchmark(server_url: str = 'http://localhost:8000', resolutions: list[tuple[int, int]] = None, warmup_runs: int = 1, image_url: str = None) -> list[BenchmarkResult]` | Run full MLLM benchmark across multiple resolutions. | [#L201-L293](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/mllm_benchmark.py#L201-L293) |
| [`print_results`](#contract-examples.mllm_benchmark.print_results) | function | `print_results(results: list[BenchmarkResult]) -> not annotated` | Print benchmark results in a nice table. | [#L296-L338](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/mllm_benchmark.py#L296-L338) |
| [`save_results`](#contract-examples.mllm_benchmark.save_results) | function | `save_results(results: list[BenchmarkResult], output_path: str) -> not annotated` | Save benchmark results to JSON file. | [#L341-L364](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/mllm_benchmark.py#L341-L364) |
| [`main`](#contract-examples.mllm_benchmark.main) | function | `main() -> not annotated` | Function `main` calls `argparse.ArgumentParser`, `parser.add_argument`, `parser.parse_args`, `run_benchmark`. | [#L367-L439](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/examples/mllm_benchmark.py#L367-L439) |