# `vllm_mlx.benchmark`

Performance Benchmark for vllm-mlx.

[View the complete module source at #L1-L1684](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/benchmark.py#L1-L1684).

## API details

Each callable below includes its exact signature, type annotations, inputs, defaults, return contract, documented exceptions, implementation source, and parsed docstring sections when the source provides them.

::: vllm_mlx.benchmark
    options:
      members:
        - ResourceMetrics
        - reset_mlx_peak_memory
        - get_mlx_memory_info
        - get_process_memory
        - get_system_memory
        - ResourceMonitor
        - VIDEO_SAMPLE_URLS
        - DEFAULT_VIDEO_URL
        - VLM_TEST_VIDEO_URLS
        - BenchmarkResult
        - BenchmarkSummary
        - calculate_percentile
        - benchmark_single_prompt
        - run_benchmark
        - MLLM_PATTERNS
        - MLLM_TEST_IMAGE_URL
        - MLLM_TEST_IMAGE_URLS
        - is_mllm_model
        - MLLMBenchmarkResult
        - download_test_image
        - resize_image
        - image_to_base64
        - benchmark_mllm_resolution
        - run_mllm_benchmark
        - print_mllm_summary
        - VideoBenchmarkResult
        - create_test_video
        - download_video
        - get_video_info
        - benchmark_video_config
        - run_video_benchmark
        - print_video_summary
        - print_summary
        - main
      filters: []
      show_if_no_docstring: true

## Complete contract reference

Expand any definition for its exact inputs, annotations, defaults, return contract, directly raised exceptions, source-grounded behavior, and immutable line link. This section includes private and nested definitions that ordinary API generators omit.

<details class="api-contract" id="contract-vllm_mlx.benchmark.ResourceMetrics" markdown="1">
<summary><code>vllm_mlx.benchmark.ResourceMetrics</code> · class</summary>

```python
vllm_mlx.benchmark.ResourceMetrics(process_memory_gb: float = 0.0, mlx_cache_gb: float = 0.0, mlx_peak_memory_gb: float = 0.0, system_memory_used_gb: float = 0.0, system_memory_total_gb: float = 0.0)
```

Resource usage metrics during benchmark.

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `process_memory_gb` | `float` | `no` | `0.0` | Optional constructor field; defaults to `0.0`. |
| `mlx_cache_gb` | `float` | `no` | `0.0` | Optional constructor field; defaults to `0.0`. |
| `mlx_peak_memory_gb` | `float` | `no` | `0.0` | Optional constructor field; defaults to `0.0`. |
| `system_memory_used_gb` | `float` | `no` | `0.0` | Optional constructor field; defaults to `0.0`. |
| `system_memory_total_gb` | `float` | `no` | `0.0` | Optional constructor field; defaults to `0.0`. |

**Returns**

- Constructs: `vllm_mlx.benchmark.ResourceMetrics`

**Exceptions and behavior**

Class `ResourceMetrics` declares 0 direct member(s).
No direct `raise` statement appears in this definition.

[View source #L72-L80](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/benchmark.py#L72-L80).

</details>

<details class="api-contract" id="contract-vllm_mlx.benchmark.reset_mlx_peak_memory" markdown="1">
<summary><code>vllm_mlx.benchmark.reset_mlx_peak_memory</code> · function</summary>

```python
vllm_mlx.benchmark.reset_mlx_peak_memory() -> not annotated
```

Reset MLX peak memory counter.

**Parameters**

This callable has no explicit inputs.

**Returns**

- Type: `not annotated`
- Direct return expressions: `None`

**Exceptions and behavior**

Function `reset_mlx_peak_memory` calls `hasattr`, `mx.reset_peak_memory`, `mx.metal.reset_peak_memory`; returns `None`.
No direct `raise` statement appears in this definition.

[View source #L83-L95](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/benchmark.py#L83-L95).

</details>

<details class="api-contract" id="contract-vllm_mlx.benchmark.get_mlx_memory_info" markdown="1">
<summary><code>vllm_mlx.benchmark.get_mlx_memory_info</code> · function</summary>

```python
vllm_mlx.benchmark.get_mlx_memory_info(reset_peak: bool = True) -> dict
```

Get MLX memory usage information.

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `reset_peak` | `bool` | `no` | `True` | If True, reset peak memory counter after reading. |

**Returns**

- Type: `dict`
- Direct return expressions: `{}`; `info`

**Exceptions and behavior**

Function `get_mlx_memory_info` calls `hasattr`, `mx.get_cache_memory`, `mx.get_peak_memory`, `mx.get_active_memory`; has 2 explicit return paths.
No direct `raise` statement appears in this definition.

[View source #L98-L135](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/benchmark.py#L98-L135).

</details>

<details class="api-contract" id="contract-vllm_mlx.benchmark.get_process_memory" markdown="1">
<summary><code>vllm_mlx.benchmark.get_process_memory</code> · function</summary>

```python
vllm_mlx.benchmark.get_process_memory() -> float
```

Get current process memory usage in GB.

**Parameters**

This callable has no explicit inputs.

**Returns**

- Type: `float`
- Direct return expressions: `0.0`; `process.memory_info().rss / 1024 ** 3`

**Exceptions and behavior**

Function `get_process_memory` calls `psutil.Process`, `process.memory_info`; has 2 explicit return paths.
No direct `raise` statement appears in this definition.

[View source #L138-L147](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/benchmark.py#L138-L147).

</details>

<details class="api-contract" id="contract-vllm_mlx.benchmark.get_system_memory" markdown="1">
<summary><code>vllm_mlx.benchmark.get_system_memory</code> · function</summary>

```python
vllm_mlx.benchmark.get_system_memory() -> tuple[float, float]
```

Get system memory (used, total) in GB.

**Parameters**

This callable has no explicit inputs.

**Returns**

- Type: `tuple[float, float]`
- Direct return expressions: `(0.0, 0.0)`; `(mem.used / 1024 ** 3, mem.total / 1024 ** 3)`

**Exceptions and behavior**

Function `get_system_memory` calls `psutil.virtual_memory`; has 2 explicit return paths.
No direct `raise` statement appears in this definition.

[View source #L150-L159](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/benchmark.py#L150-L159).

</details>

<details class="api-contract" id="contract-vllm_mlx.benchmark.ResourceMonitor" markdown="1">
<summary><code>vllm_mlx.benchmark.ResourceMonitor</code> · class</summary>

```python
vllm_mlx.benchmark.ResourceMonitor()
```

Monitor system resources during benchmark runs.

**Parameters**

This callable has no explicit inputs.

**Returns**

- Constructs: `vllm_mlx.benchmark.ResourceMonitor`

**Exceptions and behavior**

Class `ResourceMonitor` declares 4 direct member(s).
No direct `raise` statement appears in this definition.

[View source #L162-L213](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/benchmark.py#L162-L213).

</details>

<details class="api-contract" id="contract-vllm_mlx.benchmark.ResourceMonitor.__init__" markdown="1">
<summary><code>vllm_mlx.benchmark.ResourceMonitor.__init__</code> · method</summary>

```python
vllm_mlx.benchmark.ResourceMonitor.__init__() -> not annotated
```

Method `ResourceMonitor.__init__` updates `self.samples`, `self._start_time`, `self._start_memory`.

**Parameters**

This callable has no explicit inputs.

**Returns**

- Type: `not annotated`

**Exceptions and behavior**

Method `ResourceMonitor.__init__` updates `self.samples`, `self._start_time`, `self._start_memory`.
No direct `raise` statement appears in this definition.

[View source #L165-L168](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/benchmark.py#L165-L168).

</details>

<details class="api-contract" id="contract-vllm_mlx.benchmark.ResourceMonitor.start" markdown="1">
<summary><code>vllm_mlx.benchmark.ResourceMonitor.start</code> · method</summary>

```python
vllm_mlx.benchmark.ResourceMonitor.start() -> not annotated
```

Start monitoring.

**Parameters**

This callable has no explicit inputs.

**Returns**

- Type: `not annotated`

**Exceptions and behavior**

Method `ResourceMonitor.start` updates `self._start_time`, `self._start_memory`; calls `time.perf_counter`, `get_process_memory`, `reset_mlx_peak_memory`.
No direct `raise` statement appears in this definition.

[View source #L170-L176](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/benchmark.py#L170-L176).

</details>

<details class="api-contract" id="contract-vllm_mlx.benchmark.ResourceMonitor.sample" markdown="1">
<summary><code>vllm_mlx.benchmark.ResourceMonitor.sample</code> · method</summary>

```python
vllm_mlx.benchmark.ResourceMonitor.sample() -> ResourceMetrics
```

Take a resource sample.

**Parameters**

This callable has no explicit inputs.

**Returns**

- Type: `ResourceMetrics`
- Direct return expressions: `metrics`

**Exceptions and behavior**

Method `ResourceMonitor.sample` calls `get_mlx_memory_info`, `get_system_memory`, `ResourceMetrics`, `get_process_memory`; returns `metrics`.
No direct `raise` statement appears in this definition.

[View source #L178-L192](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/benchmark.py#L178-L192).

</details>

<details class="api-contract" id="contract-vllm_mlx.benchmark.ResourceMonitor.get_summary" markdown="1">
<summary><code>vllm_mlx.benchmark.ResourceMonitor.get_summary</code> · method</summary>

```python
vllm_mlx.benchmark.ResourceMonitor.get_summary() -> ResourceMetrics
```

Get summary of all samples.

**Parameters**

This callable has no explicit inputs.

**Returns**

- Type: `ResourceMetrics`
- Direct return expressions: `ResourceMetrics()`; `ResourceMetrics(process_memory_gb=peak_process, mlx_cache_gb=peak_mlx_cache, mlx_peak_memory_gb=peak_mlx, system_memory…`

**Exceptions and behavior**

Method `ResourceMonitor.get_summary` calls `ResourceMetrics`, `max`; has 2 explicit return paths.
No direct `raise` statement appears in this definition.

[View source #L194-L213](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/benchmark.py#L194-L213).

</details>

<details class="api-contract" id="contract-vllm_mlx.benchmark.BenchmarkResult" markdown="1">
<summary><code>vllm_mlx.benchmark.BenchmarkResult</code> · class</summary>

```python
vllm_mlx.benchmark.BenchmarkResult(prompt: str, prompt_tokens: int, generated_tokens: int, ttft: float, total_time: float, tpot: float = 0.0, generation_tps: float = 0.0, processing_tps: float = 0.0)
```

Results from a single benchmark run.

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `prompt` | `str` | `yes` | `none` | Required constructor field. |
| `prompt_tokens` | `int` | `yes` | `none` | Required constructor field. |
| `generated_tokens` | `int` | `yes` | `none` | Required constructor field. |
| `ttft` | `float` | `yes` | `none` | Required constructor field. |
| `total_time` | `float` | `yes` | `none` | Required constructor field. |
| `tpot` | `float` | `no` | `0.0` | Optional constructor field; defaults to `0.0`. |
| `generation_tps` | `float` | `no` | `0.0` | Optional constructor field; defaults to `0.0`. |
| `processing_tps` | `float` | `no` | `0.0` | Optional constructor field; defaults to `0.0`. |

**Returns**

- Constructs: `vllm_mlx.benchmark.BenchmarkResult`

**Exceptions and behavior**

Class `BenchmarkResult` declares 1 direct member(s).
No direct `raise` statement appears in this definition.

[View source #L235-L268](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/benchmark.py#L235-L268).

</details>

<details class="api-contract" id="contract-vllm_mlx.benchmark.BenchmarkResult.__post_init__" markdown="1">
<summary><code>vllm_mlx.benchmark.BenchmarkResult.__post_init__</code> · method</summary>

```python
vllm_mlx.benchmark.BenchmarkResult.__post_init__() -> not annotated
```

Method `BenchmarkResult.__post_init__` updates `self.tpot`, `self.generation_tps`, `self.processing_tps`.

**Parameters**

This callable has no explicit inputs.

**Returns**

- Type: `not annotated`

**Exceptions and behavior**

Method `BenchmarkResult.__post_init__` updates `self.tpot`, `self.generation_tps`, `self.processing_tps`.
No direct `raise` statement appears in this definition.

[View source #L251-L268](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/benchmark.py#L251-L268).

</details>

<details class="api-contract" id="contract-vllm_mlx.benchmark.BenchmarkSummary" markdown="1">
<summary><code>vllm_mlx.benchmark.BenchmarkSummary</code> · class</summary>

```python
vllm_mlx.benchmark.BenchmarkSummary(model_name: str, num_runs: int, total_prompt_tokens: int, total_generated_tokens: int, total_time: float, ttft_mean: float, ttft_min: float, ttft_max: float, ttft_p50: float, ttft_p95: float, tpot_mean: float, tpot_min: float, tpot_max: float, generation_tps_mean: float, generation_tps_max: float, processing_tps_mean: float, latency_mean: float, latency_min: float, latency_max: float, latency_p50: float, latency_p95: float, total_throughput_tps: float, requests_per_second: float, hardware_chip: str = '', hardware_memory_gb: float = 0.0, hardware_bandwidth_gbs: float = 0.0, resources: ResourceMetrics = field(default_factory=ResourceMetrics))
```

Summary statistics across all benchmark runs.

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `model_name` | `str` | `yes` | `none` | Required constructor field. |
| `num_runs` | `int` | `yes` | `none` | Required constructor field. |
| `total_prompt_tokens` | `int` | `yes` | `none` | Required constructor field. |
| `total_generated_tokens` | `int` | `yes` | `none` | Required constructor field. |
| `total_time` | `float` | `yes` | `none` | Required constructor field. |
| `ttft_mean` | `float` | `yes` | `none` | Required constructor field. |
| `ttft_min` | `float` | `yes` | `none` | Required constructor field. |
| `ttft_max` | `float` | `yes` | `none` | Required constructor field. |
| `ttft_p50` | `float` | `yes` | `none` | Required constructor field. |
| `ttft_p95` | `float` | `yes` | `none` | Required constructor field. |
| `tpot_mean` | `float` | `yes` | `none` | Required constructor field. |
| `tpot_min` | `float` | `yes` | `none` | Required constructor field. |
| `tpot_max` | `float` | `yes` | `none` | Required constructor field. |
| `generation_tps_mean` | `float` | `yes` | `none` | Required constructor field. |
| `generation_tps_max` | `float` | `yes` | `none` | Required constructor field. |
| `processing_tps_mean` | `float` | `yes` | `none` | Required constructor field. |
| `latency_mean` | `float` | `yes` | `none` | Required constructor field. |
| `latency_min` | `float` | `yes` | `none` | Required constructor field. |
| `latency_max` | `float` | `yes` | `none` | Required constructor field. |
| `latency_p50` | `float` | `yes` | `none` | Required constructor field. |
| `latency_p95` | `float` | `yes` | `none` | Required constructor field. |
| `total_throughput_tps` | `float` | `yes` | `none` | Required constructor field. |
| `requests_per_second` | `float` | `yes` | `none` | Required constructor field. |
| `hardware_chip` | `str` | `no` | `''` | Optional constructor field; defaults to `''`. |
| `hardware_memory_gb` | `float` | `no` | `0.0` | Optional constructor field; defaults to `0.0`. |
| `hardware_bandwidth_gbs` | `float` | `no` | `0.0` | Optional constructor field; defaults to `0.0`. |
| `resources` | `ResourceMetrics` | `no` | `field(default_factory=ResourceMetrics)` | Optional constructor field; defaults to `field(default_factory=ResourceMetrics)`. |

**Returns**

- Constructs: `vllm_mlx.benchmark.BenchmarkSummary`

**Exceptions and behavior**

Class `BenchmarkSummary` declares 0 direct member(s).
No direct `raise` statement appears in this definition.

[View source #L272-L315](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/benchmark.py#L272-L315).

</details>

<details class="api-contract" id="contract-vllm_mlx.benchmark.calculate_percentile" markdown="1">
<summary><code>vllm_mlx.benchmark.calculate_percentile</code> · function</summary>

```python
vllm_mlx.benchmark.calculate_percentile(data: list, percentile: float) -> float
```

Calculate percentile from a list.

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `data` | `list` | `yes` | `none` | Required positional or keyword input. |
| `percentile` | `float` | `yes` | `none` | Required positional or keyword input. |

**Returns**

- Type: `float`
- Direct return expressions: `0.0`; `sorted_data[index]`

**Exceptions and behavior**

Function `calculate_percentile` calls `sorted`, `int`, `len`, `min`; has 2 explicit return paths.
No direct `raise` statement appears in this definition.

[View source #L318-L325](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/benchmark.py#L318-L325).

</details>

<details class="api-contract" id="contract-vllm_mlx.benchmark.benchmark_single_prompt" markdown="1">
<summary><code>vllm_mlx.benchmark.benchmark_single_prompt</code> · function</summary>

```python
vllm_mlx.benchmark.benchmark_single_prompt(model, tokenizer, prompt: str, max_tokens: int = 256, temperature: float = 0.7) -> Optional[BenchmarkResult]
```

Benchmark a single prompt with detailed timing.

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `model` | `not annotated` | `yes` | `none` | The loaded MLX model |
| `tokenizer` | `not annotated` | `yes` | `none` | The tokenizer |
| `prompt` | `str` | `yes` | `none` | The prompt to benchmark |
| `max_tokens` | `int` | `no` | `256` | Maximum tokens to generate |
| `temperature` | `float` | `no` | `0.7` | Sampling temperature |

**Returns**

- Type: `Optional[BenchmarkResult]`
- Direct return expressions: `BenchmarkResult(prompt=prompt[:50] + '...' if len(prompt) > 50 else prompt, prompt_tokens=prompt_token_count, generated…`; `None`

**Exceptions and behavior**

Function `benchmark_single_prompt` calls `tokenizer.encode`, `len`, `make_sampler`, `time.perf_counter`; has 2 explicit return paths.
No direct `raise` statement appears in this definition.

[View source #L328-L394](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/benchmark.py#L328-L394).

</details>

<details class="api-contract" id="contract-vllm_mlx.benchmark.run_benchmark" markdown="1">
<summary><code>vllm_mlx.benchmark.run_benchmark</code> · function</summary>

```python
vllm_mlx.benchmark.run_benchmark(model_name: str, num_prompts: int = 5, max_tokens: int = 256, temperature: float = 0.7, warmup_runs: int = 1) -> Optional[BenchmarkSummary]
```

Run the full benchmark suite.

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `model_name` | `str` | `yes` | `none` | HuggingFace model name or local path |
| `num_prompts` | `int` | `no` | `5` | Number of prompts to test |
| `max_tokens` | `int` | `no` | `256` | Maximum tokens per generation |
| `temperature` | `float` | `no` | `0.7` | Sampling temperature |
| `warmup_runs` | `int` | `no` | `1` | Number of warmup runs before measuring |

**Returns**

- Type: `Optional[BenchmarkSummary]`
- Direct return expressions: `None`; `summary`

**Exceptions and behavior**

Function `run_benchmark` calls `detect_hardware`, `len`, `print`, `tabulate`; has 2 explicit return paths.
No direct `raise` statement appears in this definition.

[View source #L397-L610](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/benchmark.py#L397-L610).

</details>

<details class="api-contract" id="contract-vllm_mlx.benchmark.is_mllm_model" markdown="1">
<summary><code>vllm_mlx.benchmark.is_mllm_model</code> · function</summary>

```python
vllm_mlx.benchmark.is_mllm_model(model_name: str) -> bool
```

Check if model name indicates a multimodal language model.

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `model_name` | `str` | `yes` | `none` | Required positional or keyword input. |

**Returns**

- Type: `bool`
- Direct return expressions: `True`; `False`

**Exceptions and behavior**

Function `is_mllm_model` calls `model_name.lower`, `pattern.lower`; has 2 explicit return paths.
No direct `raise` statement appears in this definition.

[View source #L651-L657](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/benchmark.py#L651-L657).

</details>

<details class="api-contract" id="contract-vllm_mlx.benchmark.MLLMBenchmarkResult" markdown="1">
<summary><code>vllm_mlx.benchmark.MLLMBenchmarkResult</code> · class</summary>

```python
vllm_mlx.benchmark.MLLMBenchmarkResult(resolution: str, width: int, height: int, pixels: int, time_seconds: float, tokens_generated: int, tokens_per_second: float, response_preview: str, memory_gb: float = 0.0, mlx_memory_gb: float = 0.0)
```

Result from a single MLLM benchmark run.

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `resolution` | `str` | `yes` | `none` | Required constructor field. |
| `width` | `int` | `yes` | `none` | Required constructor field. |
| `height` | `int` | `yes` | `none` | Required constructor field. |
| `pixels` | `int` | `yes` | `none` | Required constructor field. |
| `time_seconds` | `float` | `yes` | `none` | Required constructor field. |
| `tokens_generated` | `int` | `yes` | `none` | Required constructor field. |
| `tokens_per_second` | `float` | `yes` | `none` | Required constructor field. |
| `response_preview` | `str` | `yes` | `none` | Required constructor field. |
| `memory_gb` | `float` | `no` | `0.0` | Optional constructor field; defaults to `0.0`. |
| `mlx_memory_gb` | `float` | `no` | `0.0` | Optional constructor field; defaults to `0.0`. |

**Returns**

- Constructs: `vllm_mlx.benchmark.MLLMBenchmarkResult`

**Exceptions and behavior**

Class `MLLMBenchmarkResult` declares 0 direct member(s).
No direct `raise` statement appears in this definition.

[View source #L661-L674](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/benchmark.py#L661-L674).

</details>

<details class="api-contract" id="contract-vllm_mlx.benchmark.download_test_image" markdown="1">
<summary><code>vllm_mlx.benchmark.download_test_image</code> · function</summary>

```python
vllm_mlx.benchmark.download_test_image(url: str, timeout: int = 30) -> Image.Image
```

Download image from URL and return PIL Image.

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `url` | `str` | `yes` | `none` | Required positional or keyword input. |
| `timeout` | `int` | `no` | `30` | Optional positional or keyword input; defaults to `30`. |

**Returns**

- Type: `Image.Image`
- Direct return expressions: `Image.open(io.BytesIO(response.content))`

**Exceptions and behavior**

Function `download_test_image` calls `requests.get`, `response.raise_for_status`, `Image.open`, `io.BytesIO`; returns `Image.open(io.BytesIO(response.content))`.
No direct `raise` statement appears in this definition.

[View source #L677-L684](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/benchmark.py#L677-L684).

</details>

<details class="api-contract" id="contract-vllm_mlx.benchmark.resize_image" markdown="1">
<summary><code>vllm_mlx.benchmark.resize_image</code> · function</summary>

```python
vllm_mlx.benchmark.resize_image(img: Image.Image, width: int, height: int) -> Image.Image
```

Resize image to specified dimensions.

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `img` | `Image.Image` | `yes` | `none` | Required positional or keyword input. |
| `width` | `int` | `yes` | `none` | Required positional or keyword input. |
| `height` | `int` | `yes` | `none` | Required positional or keyword input. |

**Returns**

- Type: `Image.Image`
- Direct return expressions: `img.resize((width, height), Image.Resampling.LANCZOS)`

**Exceptions and behavior**

Function `resize_image` calls `img.resize`; returns `img.resize((width, height), Image.Resampling.LANCZOS)`.
No direct `raise` statement appears in this definition.

[View source #L687-L689](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/benchmark.py#L687-L689).

</details>

<details class="api-contract" id="contract-vllm_mlx.benchmark.image_to_base64" markdown="1">
<summary><code>vllm_mlx.benchmark.image_to_base64</code> · function</summary>

```python
vllm_mlx.benchmark.image_to_base64(img: Image.Image, format: str = 'JPEG') -> str
```

Convert PIL Image to base64 data URL.

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `img` | `Image.Image` | `yes` | `none` | Required positional or keyword input. |
| `format` | `str` | `no` | `'JPEG'` | Optional positional or keyword input; defaults to `'JPEG'`. |

**Returns**

- Type: `str`
- Direct return expressions: `f'data:{mime};base64,{b64}'`

**Exceptions and behavior**

Function `image_to_base64` calls `Image.new`, `background.paste`, `img.split`, `img.convert`; returns `f'data:{mime};base64,{b64}'`.
No direct `raise` statement appears in this definition.

[View source #L692-L705](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/benchmark.py#L692-L705).

</details>

<details class="api-contract" id="contract-vllm_mlx.benchmark.benchmark_mllm_resolution" markdown="1">
<summary><code>vllm_mlx.benchmark.benchmark_mllm_resolution</code> · function</summary>

```python
vllm_mlx.benchmark.benchmark_mllm_resolution(model, processor, config, base_image: Image.Image, width: int, height: int, max_tokens: int = 256, warmup: bool = False) -> MLLMBenchmarkResult
```

Run MLLM benchmark for a specific resolution.

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `model` | `not annotated` | `yes` | `none` | Required positional or keyword input. |
| `processor` | `not annotated` | `yes` | `none` | Required positional or keyword input. |
| `config` | `not annotated` | `yes` | `none` | Required positional or keyword input. |
| `base_image` | `Image.Image` | `yes` | `none` | Required positional or keyword input. |
| `width` | `int` | `yes` | `none` | Required positional or keyword input. |
| `height` | `int` | `yes` | `none` | Required positional or keyword input. |
| `max_tokens` | `int` | `no` | `256` | Optional positional or keyword input; defaults to `256`. |
| `warmup` | `bool` | `no` | `False` | Optional positional or keyword input; defaults to `False`. |

**Returns**

- Type: `MLLMBenchmarkResult`
- Direct return expressions: `MLLMBenchmarkResult(resolution=resolution_name, width=width, height=height, pixels=pixels, time_seconds=elapsed, tokens…`

**Exceptions and behavior**

Function `benchmark_mllm_resolution` calls `reset_mlx_peak_memory`, `resize_image`, `tempfile.NamedTemporaryFile`, `img.save`; returns `MLLMBenchmarkResult(resolution=resolution_name, width=width, height=height, pixels=pixels, time_seconds=elapsed, tokens…`.
No direct `raise` statement appears in this definition.

[View source #L708-L800](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/benchmark.py#L708-L800).

</details>

<details class="api-contract" id="contract-vllm_mlx.benchmark.run_mllm_benchmark" markdown="1">
<summary><code>vllm_mlx.benchmark.run_mllm_benchmark</code> · function</summary>

```python
vllm_mlx.benchmark.run_mllm_benchmark(model_name: str, quick: bool = False, max_tokens: int = 256, warmup_runs: int = 1) -> list[MLLMBenchmarkResult]
```

Run MLLM benchmark across multiple image resolutions.

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `model_name` | `str` | `yes` | `none` | HuggingFace model name |
| `quick` | `bool` | `no` | `False` | If True, test only 4 resolutions |
| `max_tokens` | `int` | `no` | `256` | Max tokens to generate |
| `warmup_runs` | `int` | `no` | `1` | Number of warmup runs |

**Returns**

- Type: `list[MLLMBenchmarkResult]`
- Direct return expressions: `[]`; `results`

**Exceptions and behavior**

Function `run_mllm_benchmark` calls `detect_hardware`, `print`, `len`, `tabulate`; has 2 explicit return paths.
No direct `raise` statement appears in this definition.

[View source #L803-L913](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/benchmark.py#L803-L913).

</details>

<details class="api-contract" id="contract-vllm_mlx.benchmark.print_mllm_summary" markdown="1">
<summary><code>vllm_mlx.benchmark.print_mllm_summary</code> · function</summary>

```python
vllm_mlx.benchmark.print_mllm_summary(results: list[MLLMBenchmarkResult], model_name: str) -> not annotated
```

Print MLLM benchmark summary.

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `results` | `list[MLLMBenchmarkResult]` | `yes` | `none` | Required positional or keyword input. |
| `model_name` | `str` | `yes` | `none` | Required positional or keyword input. |

**Returns**

- Type: `not annotated`
- Direct return expressions: `None`

**Exceptions and behavior**

Function `print_mllm_summary` calls `print`, `table_data.append`, `tabulate`, `sum`; returns `None`.
No direct `raise` statement appears in this definition.

[View source #L916-L975](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/benchmark.py#L916-L975).

</details>

<details class="api-contract" id="contract-vllm_mlx.benchmark.VideoBenchmarkResult" markdown="1">
<summary><code>vllm_mlx.benchmark.VideoBenchmarkResult</code> · class</summary>

```python
vllm_mlx.benchmark.VideoBenchmarkResult(config_name: str, fps: float, max_frames: int, frames_extracted: int, video_duration: float, time_seconds: float, prompt_tokens: int, completion_tokens: int, tokens_per_second: float, response_preview: str, memory_gb: float = 0.0, mlx_memory_gb: float = 0.0)
```

Result from a single video benchmark run.

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `config_name` | `str` | `yes` | `none` | Required constructor field. |
| `fps` | `float` | `yes` | `none` | Required constructor field. |
| `max_frames` | `int` | `yes` | `none` | Required constructor field. |
| `frames_extracted` | `int` | `yes` | `none` | Required constructor field. |
| `video_duration` | `float` | `yes` | `none` | Required constructor field. |
| `time_seconds` | `float` | `yes` | `none` | Required constructor field. |
| `prompt_tokens` | `int` | `yes` | `none` | Required constructor field. |
| `completion_tokens` | `int` | `yes` | `none` | Required constructor field. |
| `tokens_per_second` | `float` | `yes` | `none` | Required constructor field. |
| `response_preview` | `str` | `yes` | `none` | Required constructor field. |
| `memory_gb` | `float` | `no` | `0.0` | Optional constructor field; defaults to `0.0`. |
| `mlx_memory_gb` | `float` | `no` | `0.0` | Optional constructor field; defaults to `0.0`. |

**Returns**

- Constructs: `vllm_mlx.benchmark.VideoBenchmarkResult`

**Exceptions and behavior**

Class `VideoBenchmarkResult` declares 0 direct member(s).
No direct `raise` statement appears in this definition.

[View source #L984-L999](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/benchmark.py#L984-L999).

</details>

<details class="api-contract" id="contract-vllm_mlx.benchmark.create_test_video" markdown="1">
<summary><code>vllm_mlx.benchmark.create_test_video</code> · function</summary>

```python
vllm_mlx.benchmark.create_test_video(duration: float = 10.0, fps: float = 30.0, width: int = 640, height: int = 480) -> str
```

Create a synthetic test video with colored frames and text.

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `duration` | `float` | `no` | `10.0` | Optional positional or keyword input; defaults to `10.0`. |
| `fps` | `float` | `no` | `30.0` | Optional positional or keyword input; defaults to `30.0`. |
| `width` | `int` | `no` | `640` | Optional positional or keyword input; defaults to `640`. |
| `height` | `int` | `no` | `480` | Optional positional or keyword input; defaults to `480`. |

**Returns**

- Type: `str`
- Direct return expressions: `temp_file.name`

**Exceptions and behavior**

Function `create_test_video` calls `tempfile.NamedTemporaryFile`, `temp_file.close`, `cv2.VideoWriter_fourcc`, `cv2.VideoWriter`; returns `temp_file.name`.
No direct `raise` statement appears in this definition.

[View source #L1002-L1056](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/benchmark.py#L1002-L1056).

</details>

<details class="api-contract" id="contract-vllm_mlx.benchmark.download_video" markdown="1">
<summary><code>vllm_mlx.benchmark.download_video</code> · function</summary>

```python
vllm_mlx.benchmark.download_video(url: str, timeout: int = 120) -> str
```

Download video from URL and return local path.

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `url` | `str` | `yes` | `none` | Required positional or keyword input. |
| `timeout` | `int` | `no` | `120` | Optional positional or keyword input; defaults to `120`. |

**Returns**

- Type: `str`
- Direct return expressions: `temp_file.name`

**Exceptions and behavior**

Function `download_video` calls `print`, `requests.get`, `response.raise_for_status`, `tempfile.NamedTemporaryFile`; returns `temp_file.name`.
No direct `raise` statement appears in this definition.

[View source #L1059-L1075](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/benchmark.py#L1059-L1075).

</details>

<details class="api-contract" id="contract-vllm_mlx.benchmark.get_video_info" markdown="1">
<summary><code>vllm_mlx.benchmark.get_video_info</code> · function</summary>

```python
vllm_mlx.benchmark.get_video_info(video_path: str) -> dict
```

Get information about a video file.

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `video_path` | `str` | `yes` | `none` | Required positional or keyword input. |

**Returns**

- Type: `dict`
- Direct return expressions: `{'error': 'Cannot open video'}`; `info`

**Exceptions and behavior**

Function `get_video_info` calls `cv2.VideoCapture`, `cap.isOpened`, `int`, `cap.get`; has 2 explicit return paths.
No direct `raise` statement appears in this definition.

[View source #L1078-L1094](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/benchmark.py#L1078-L1094).

</details>

<details class="api-contract" id="contract-vllm_mlx.benchmark.benchmark_video_config" markdown="1">
<summary><code>vllm_mlx.benchmark.benchmark_video_config</code> · function</summary>

```python
vllm_mlx.benchmark.benchmark_video_config(model, video_path: str, fps: float, max_frames: int, config_name: str, video_info: dict, max_tokens: int = 150, warmup: bool = False) -> VideoBenchmarkResult
```

Run a single video benchmark configuration.

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `model` | `not annotated` | `yes` | `none` | Required positional or keyword input. |
| `video_path` | `str` | `yes` | `none` | Required positional or keyword input. |
| `fps` | `float` | `yes` | `none` | Required positional or keyword input. |
| `max_frames` | `int` | `yes` | `none` | Required positional or keyword input. |
| `config_name` | `str` | `yes` | `none` | Required positional or keyword input. |
| `video_info` | `dict` | `yes` | `none` | Required positional or keyword input. |
| `max_tokens` | `int` | `no` | `150` | Optional positional or keyword input; defaults to `150`. |
| `warmup` | `bool` | `no` | `False` | Optional positional or keyword input; defaults to `False`. |

**Returns**

- Type: `VideoBenchmarkResult`
- Direct return expressions: `VideoBenchmarkResult(config_name=config_name, fps=fps, max_frames=max_frames, frames_extracted=frames_extracted, video_…`

**Exceptions and behavior**

Function `benchmark_video_config` calls `reset_mlx_peak_memory`, `print`, `time.perf_counter`, `model.generate`; returns `VideoBenchmarkResult(config_name=config_name, fps=fps, max_frames=max_frames, frames_extracted=frames_extracted, video_…`.
No direct `raise` statement appears in this definition.

[View source #L1097-L1162](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/benchmark.py#L1097-L1162).

</details>

<details class="api-contract" id="contract-vllm_mlx.benchmark.run_video_benchmark" markdown="1">
<summary><code>vllm_mlx.benchmark.run_video_benchmark</code> · function</summary>

```python
vllm_mlx.benchmark.run_video_benchmark(model_name: str, video_url: str = None, video_path: str = None, quick: bool = False, max_tokens: int = 150, warmup_runs: int = 1) -> list[VideoBenchmarkResult]
```

Run video benchmark across multiple frame configurations.

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `model_name` | `str` | `yes` | `none` | HuggingFace MLLM model name |
| `video_url` | `str` | `no` | `None` | URL to download video from |
| `video_path` | `str` | `no` | `None` | Local video file path |
| `quick` | `bool` | `no` | `False` | If True, test only 3 configurations |
| `max_tokens` | `int` | `no` | `150` | Max tokens to generate |
| `warmup_runs` | `int` | `no` | `1` | Number of warmup runs |

**Returns**

- Type: `list[VideoBenchmarkResult]`
- Direct return expressions: `results`

**Exceptions and behavior**

Function `run_video_benchmark` calls `detect_hardware`, `print`, `len`, `tabulate`; returns `results`.
No direct `raise` statement appears in this definition.

[View source #L1165-L1285](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/benchmark.py#L1165-L1285).

</details>

<details class="api-contract" id="contract-vllm_mlx.benchmark.print_video_summary" markdown="1">
<summary><code>vllm_mlx.benchmark.print_video_summary</code> · function</summary>

```python
vllm_mlx.benchmark.print_video_summary(results: list[VideoBenchmarkResult], model_name: str) -> not annotated
```

Print video benchmark summary.

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `results` | `list[VideoBenchmarkResult]` | `yes` | `none` | Required positional or keyword input. |
| `model_name` | `str` | `yes` | `none` | Required positional or keyword input. |

**Returns**

- Type: `not annotated`
- Direct return expressions: `None`

**Exceptions and behavior**

Function `print_video_summary` calls `print`, `sorted`, `table_data.append`, `tabulate`; returns `None`.
No direct `raise` statement appears in this definition.

[View source #L1288-L1341](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/benchmark.py#L1288-L1341).

</details>

<details class="api-contract" id="contract-vllm_mlx.benchmark.print_summary" markdown="1">
<summary><code>vllm_mlx.benchmark.print_summary</code> · function</summary>

```python
vllm_mlx.benchmark.print_summary(summary: BenchmarkSummary) -> not annotated
```

Print a formatted summary of benchmark results using tabulate.

**Parameters**

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `summary` | `BenchmarkSummary` | `yes` | `none` | Required positional or keyword input. |

**Returns**

- Type: `not annotated`

**Exceptions and behavior**

Function `print_summary` calls `print`, `tabulate`, `resource_data.append`.
No direct `raise` statement appears in this definition.

[View source #L1349-L1441](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/benchmark.py#L1349-L1441).

</details>

<details class="api-contract" id="contract-vllm_mlx.benchmark.main" markdown="1">
<summary><code>vllm_mlx.benchmark.main</code> · function</summary>

```python
vllm_mlx.benchmark.main() -> not annotated
```

Run the benchmark.

**Parameters**

This callable has no explicit inputs.

**Returns**

- Type: `not annotated`

**Exceptions and behavior**

Function `main` calls `argparse.ArgumentParser`, `parser.add_argument`, `parser.parse_args`, `is_mllm_model`.
No direct `raise` statement appears in this definition.

[View source #L1444-L1680](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/benchmark.py#L1444-L1680).

</details>

## Complete symbol map

This map also includes private definitions and nested helpers. The signature column exposes every explicit input even when an internal helper has no dedicated parameter prose.

| Symbol | Kind | Signature and inputs | What it does | Source |
| --- | --- | --- | --- | --- |
| [`ResourceMetrics`](#contract-vllm_mlx.benchmark.ResourceMetrics) | class | `ResourceMetrics(process_memory_gb: float = 0.0, mlx_cache_gb: float = 0.0, mlx_peak_memory_gb: float = 0.0, system_memory_used_gb: float = 0.0, system_memory_total_gb: float = 0.0)` | Resource usage metrics during benchmark. | [#L72-L80](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/benchmark.py#L72-L80) |
| [`reset_mlx_peak_memory`](#contract-vllm_mlx.benchmark.reset_mlx_peak_memory) | function | `reset_mlx_peak_memory() -> not annotated` | Reset MLX peak memory counter. | [#L83-L95](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/benchmark.py#L83-L95) |
| [`get_mlx_memory_info`](#contract-vllm_mlx.benchmark.get_mlx_memory_info) | function | `get_mlx_memory_info(reset_peak: bool = True) -> dict` | Get MLX memory usage information. | [#L98-L135](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/benchmark.py#L98-L135) |
| [`get_process_memory`](#contract-vllm_mlx.benchmark.get_process_memory) | function | `get_process_memory() -> float` | Get current process memory usage in GB. | [#L138-L147](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/benchmark.py#L138-L147) |
| [`get_system_memory`](#contract-vllm_mlx.benchmark.get_system_memory) | function | `get_system_memory() -> tuple[float, float]` | Get system memory (used, total) in GB. | [#L150-L159](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/benchmark.py#L150-L159) |
| [`ResourceMonitor`](#contract-vllm_mlx.benchmark.ResourceMonitor) | class | `ResourceMonitor()` | Monitor system resources during benchmark runs. | [#L162-L213](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/benchmark.py#L162-L213) |
| [`ResourceMonitor.__init__`](#contract-vllm_mlx.benchmark.ResourceMonitor.__init__) | method | `ResourceMonitor.__init__() -> not annotated` | Method `ResourceMonitor.__init__` updates `self.samples`, `self._start_time`, `self._start_memory`. | [#L165-L168](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/benchmark.py#L165-L168) |
| [`ResourceMonitor.start`](#contract-vllm_mlx.benchmark.ResourceMonitor.start) | method | `ResourceMonitor.start() -> not annotated` | Start monitoring. | [#L170-L176](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/benchmark.py#L170-L176) |
| [`ResourceMonitor.sample`](#contract-vllm_mlx.benchmark.ResourceMonitor.sample) | method | `ResourceMonitor.sample() -> ResourceMetrics` | Take a resource sample. | [#L178-L192](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/benchmark.py#L178-L192) |
| [`ResourceMonitor.get_summary`](#contract-vllm_mlx.benchmark.ResourceMonitor.get_summary) | method | `ResourceMonitor.get_summary() -> ResourceMetrics` | Get summary of all samples. | [#L194-L213](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/benchmark.py#L194-L213) |
| [`BenchmarkResult`](#contract-vllm_mlx.benchmark.BenchmarkResult) | class | `BenchmarkResult(prompt: str, prompt_tokens: int, generated_tokens: int, ttft: float, total_time: float, tpot: float = 0.0, generation_tps: float = 0.0, processing_tps: float = 0.0)` | Results from a single benchmark run. | [#L235-L268](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/benchmark.py#L235-L268) |
| [`BenchmarkResult.__post_init__`](#contract-vllm_mlx.benchmark.BenchmarkResult.__post_init__) | method | `BenchmarkResult.__post_init__() -> not annotated` | Method `BenchmarkResult.__post_init__` updates `self.tpot`, `self.generation_tps`, `self.processing_tps`. | [#L251-L268](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/benchmark.py#L251-L268) |
| [`BenchmarkSummary`](#contract-vllm_mlx.benchmark.BenchmarkSummary) | class | `BenchmarkSummary(model_name: str, num_runs: int, total_prompt_tokens: int, total_generated_tokens: int, total_time: float, ttft_mean: float, ttft_min: float, ttft_max: float, ttft_p50: float, ttft_p95: float, tpot_mean: float, tpot_min: float, tpot_max: float, generation_tps_mean: float, generation_tps_max: float, processing_tps_mean: float, latency_mean: float, latency_min: float, latency_max: float, latency_p50: float, latency_p95: float, total_throughput_tps: float, requests_per_second: float, hardware_chip: str = '', hardware_memory_gb: float = 0.0, hardware_bandwidth_gbs: float = 0.0, resources: ResourceMetrics = field(default_factory=ResourceMetrics))` | Summary statistics across all benchmark runs. | [#L272-L315](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/benchmark.py#L272-L315) |
| [`calculate_percentile`](#contract-vllm_mlx.benchmark.calculate_percentile) | function | `calculate_percentile(data: list, percentile: float) -> float` | Calculate percentile from a list. | [#L318-L325](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/benchmark.py#L318-L325) |
| [`benchmark_single_prompt`](#contract-vllm_mlx.benchmark.benchmark_single_prompt) | function | `benchmark_single_prompt(model, tokenizer, prompt: str, max_tokens: int = 256, temperature: float = 0.7) -> Optional[BenchmarkResult]` | Benchmark a single prompt with detailed timing. | [#L328-L394](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/benchmark.py#L328-L394) |
| [`run_benchmark`](#contract-vllm_mlx.benchmark.run_benchmark) | function | `run_benchmark(model_name: str, num_prompts: int = 5, max_tokens: int = 256, temperature: float = 0.7, warmup_runs: int = 1) -> Optional[BenchmarkSummary]` | Run the full benchmark suite. | [#L397-L610](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/benchmark.py#L397-L610) |
| [`is_mllm_model`](#contract-vllm_mlx.benchmark.is_mllm_model) | function | `is_mllm_model(model_name: str) -> bool` | Check if model name indicates a multimodal language model. | [#L651-L657](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/benchmark.py#L651-L657) |
| [`MLLMBenchmarkResult`](#contract-vllm_mlx.benchmark.MLLMBenchmarkResult) | class | `MLLMBenchmarkResult(resolution: str, width: int, height: int, pixels: int, time_seconds: float, tokens_generated: int, tokens_per_second: float, response_preview: str, memory_gb: float = 0.0, mlx_memory_gb: float = 0.0)` | Result from a single MLLM benchmark run. | [#L661-L674](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/benchmark.py#L661-L674) |
| [`download_test_image`](#contract-vllm_mlx.benchmark.download_test_image) | function | `download_test_image(url: str, timeout: int = 30) -> Image.Image` | Download image from URL and return PIL Image. | [#L677-L684](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/benchmark.py#L677-L684) |
| [`resize_image`](#contract-vllm_mlx.benchmark.resize_image) | function | `resize_image(img: Image.Image, width: int, height: int) -> Image.Image` | Resize image to specified dimensions. | [#L687-L689](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/benchmark.py#L687-L689) |
| [`image_to_base64`](#contract-vllm_mlx.benchmark.image_to_base64) | function | `image_to_base64(img: Image.Image, format: str = 'JPEG') -> str` | Convert PIL Image to base64 data URL. | [#L692-L705](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/benchmark.py#L692-L705) |
| [`benchmark_mllm_resolution`](#contract-vllm_mlx.benchmark.benchmark_mllm_resolution) | function | `benchmark_mllm_resolution(model, processor, config, base_image: Image.Image, width: int, height: int, max_tokens: int = 256, warmup: bool = False) -> MLLMBenchmarkResult` | Run MLLM benchmark for a specific resolution. | [#L708-L800](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/benchmark.py#L708-L800) |
| [`run_mllm_benchmark`](#contract-vllm_mlx.benchmark.run_mllm_benchmark) | function | `run_mllm_benchmark(model_name: str, quick: bool = False, max_tokens: int = 256, warmup_runs: int = 1) -> list[MLLMBenchmarkResult]` | Run MLLM benchmark across multiple image resolutions. | [#L803-L913](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/benchmark.py#L803-L913) |
| [`print_mllm_summary`](#contract-vllm_mlx.benchmark.print_mllm_summary) | function | `print_mllm_summary(results: list[MLLMBenchmarkResult], model_name: str) -> not annotated` | Print MLLM benchmark summary. | [#L916-L975](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/benchmark.py#L916-L975) |
| [`VideoBenchmarkResult`](#contract-vllm_mlx.benchmark.VideoBenchmarkResult) | class | `VideoBenchmarkResult(config_name: str, fps: float, max_frames: int, frames_extracted: int, video_duration: float, time_seconds: float, prompt_tokens: int, completion_tokens: int, tokens_per_second: float, response_preview: str, memory_gb: float = 0.0, mlx_memory_gb: float = 0.0)` | Result from a single video benchmark run. | [#L984-L999](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/benchmark.py#L984-L999) |
| [`create_test_video`](#contract-vllm_mlx.benchmark.create_test_video) | function | `create_test_video(duration: float = 10.0, fps: float = 30.0, width: int = 640, height: int = 480) -> str` | Create a synthetic test video with colored frames and text. | [#L1002-L1056](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/benchmark.py#L1002-L1056) |
| [`download_video`](#contract-vllm_mlx.benchmark.download_video) | function | `download_video(url: str, timeout: int = 120) -> str` | Download video from URL and return local path. | [#L1059-L1075](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/benchmark.py#L1059-L1075) |
| [`get_video_info`](#contract-vllm_mlx.benchmark.get_video_info) | function | `get_video_info(video_path: str) -> dict` | Get information about a video file. | [#L1078-L1094](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/benchmark.py#L1078-L1094) |
| [`benchmark_video_config`](#contract-vllm_mlx.benchmark.benchmark_video_config) | function | `benchmark_video_config(model, video_path: str, fps: float, max_frames: int, config_name: str, video_info: dict, max_tokens: int = 150, warmup: bool = False) -> VideoBenchmarkResult` | Run a single video benchmark configuration. | [#L1097-L1162](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/benchmark.py#L1097-L1162) |
| [`run_video_benchmark`](#contract-vllm_mlx.benchmark.run_video_benchmark) | function | `run_video_benchmark(model_name: str, video_url: str = None, video_path: str = None, quick: bool = False, max_tokens: int = 150, warmup_runs: int = 1) -> list[VideoBenchmarkResult]` | Run video benchmark across multiple frame configurations. | [#L1165-L1285](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/benchmark.py#L1165-L1285) |
| [`print_video_summary`](#contract-vllm_mlx.benchmark.print_video_summary) | function | `print_video_summary(results: list[VideoBenchmarkResult], model_name: str) -> not annotated` | Print video benchmark summary. | [#L1288-L1341](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/benchmark.py#L1288-L1341) |
| [`print_summary`](#contract-vllm_mlx.benchmark.print_summary) | function | `print_summary(summary: BenchmarkSummary) -> not annotated` | Print a formatted summary of benchmark results using tabulate. | [#L1349-L1441](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/benchmark.py#L1349-L1441) |
| [`main`](#contract-vllm_mlx.benchmark.main) | function | `main() -> not annotated` | Run the benchmark. | [#L1444-L1680](https://github.com/waybarrios/vllm-mlx/blob/a69d47912bcb21d8fe04d48f75fa896b620ffcfa/vllm_mlx/benchmark.py#L1444-L1680) |