Skip to content

examples.benchmark_detokenizer

Benchmark: Streaming Detokenizer vs Naive Decode Compares performance of: 1.

View the complete module source at #L1-L186.

API details

Each callable below includes its exact signature, type annotations, inputs, defaults, return contract, documented exceptions, implementation source, and parsed docstring sections when the source provides them.

examples.benchmark_detokenizer

Benchmark: Streaming Detokenizer vs Naive Decode

Compares performance of: 1. Old method: tokenizer.decode([token]) for each token 2. New method: StreamingDetokenizer.add_token() + last_segment

Run

python examples/benchmark_detokenizer.py

examples.benchmark_detokenizer.benchmark_naive_decode 

benchmark_naive_decode(tokenizer, tokens, iterations=10)

Benchmark naive decode approach (old method).

Source code in examples/benchmark_detokenizer.py 
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
def benchmark_naive_decode(tokenizer, tokens, iterations=10):
    """Benchmark naive decode approach (old method)."""
    times = []

    for _ in range(iterations):
        start = time.perf_counter()

        # Old method: decode each token individually
        texts = []
        for t in tokens:
            text = tokenizer.decode([t])
            texts.append(text)

        elapsed = time.perf_counter() - start
        times.append(elapsed)

    return {
        "method": "naive_decode",
        "mean_ms": statistics.mean(times) * 1000,
        "std_ms": statistics.stdev(times) * 1000 if len(times) > 1 else 0,
        "min_ms": min(times) * 1000,
        "max_ms": max(times) * 1000,
    }

examples.benchmark_detokenizer.benchmark_streaming_detokenizer 

benchmark_streaming_detokenizer(tokenizer, tokens, detokenizer_class, iterations=10)

Benchmark streaming detokenizer approach (new method).

Source code in examples/benchmark_detokenizer.py 
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
def benchmark_streaming_detokenizer(tokenizer, tokens, detokenizer_class, iterations=10):
    """Benchmark streaming detokenizer approach (new method)."""
    times = []

    for _ in range(iterations):
        detok = detokenizer_class(tokenizer)
        detok.reset()

        start = time.perf_counter()

        # New method: streaming detokenizer
        for t in tokens:
            detok.add_token(t)
            _ = detok.last_segment  # Get incremental text

        detok.finalize()
        _ = detok.text  # Get final text

        elapsed = time.perf_counter() - start
        times.append(elapsed)

    return {
        "method": detokenizer_class.__name__,
        "mean_ms": statistics.mean(times) * 1000,
        "std_ms": statistics.stdev(times) * 1000 if len(times) > 1 else 0,
        "min_ms": min(times) * 1000,
        "max_ms": max(times) * 1000,
    }

examples.benchmark_detokenizer.main 

main()
Source code in examples/benchmark_detokenizer.py 
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
def main():
    print("=" * 70)
    print(" Detokenizer Performance Benchmark")
    print("=" * 70)
    print()

    # Load tokenizer using mlx-lm's optimized loader
    print("Loading tokenizer with mlx-lm...")
    model_path = Path(snapshot_download('mlx-community/Qwen3-0.6B-8bit'))
    tokenizer_wrapper = load_tokenizer(model_path)

    # Also get raw tokenizer for naive decode
    raw_tokenizer = AutoTokenizer.from_pretrained('mlx-community/Qwen3-0.6B-8bit')

    print(f"Tokenizer type: {type(tokenizer_wrapper._detokenizer_class).__name__}")
    print()

    # Test with different sequence lengths (targeting realistic generation sizes)
    base_text = "The development of large language models has revolutionized natural language processing and artificial intelligence. These models can understand context, generate coherent text, and perform a wide variety of tasks. "
    test_texts = [
        ("Short", "Hello, how are you doing today?"),
        ("Medium", base_text * 3),  # ~100 tokens
        ("Long", base_text * 15),  # ~500 tokens
        ("1K tokens", base_text * 35),  # ~1000 tokens
        ("2K tokens", base_text * 70),  # ~2000 tokens
        ("4K tokens", base_text * 140),  # ~4000 tokens
    ]

    results = []

    for name, text in test_texts:
        tokens = raw_tokenizer.encode(text)
        actual_tokens = len(tokens)
        print(f"{name} ({actual_tokens} tokens)")
        print("-" * 50)

        # Benchmark all methods
        naive_result = benchmark_naive_decode(raw_tokenizer, tokens, iterations=20)

        # Use mlx-lm's auto-selected detokenizer (via wrapper)
        optimized_result = benchmark_streaming_detokenizer(
            tokenizer_wrapper,
            tokens,
            tokenizer_wrapper._detokenizer_class,
            iterations=20
        )

        # Calculate speedup
        speedup = naive_result["mean_ms"] / optimized_result["mean_ms"] if optimized_result["mean_ms"] > 0 else float('inf')

        print(f"  Naive decode():      {naive_result['mean_ms']:8.3f}ms")
        print(f"  {optimized_result['method']}: {optimized_result['mean_ms']:8.3f}ms")
        print(f"  Speedup:             {speedup:8.2f}x")
        print()

        results.append({
            "name": name,
            "tokens": actual_tokens,
            "naive_ms": naive_result["mean_ms"],
            "optimized_ms": optimized_result["mean_ms"],
            "optimized_name": optimized_result["method"],
            "speedup": speedup,
        })

    # Summary table
    print("=" * 70)
    print(" Summary")
    print("=" * 70)
    print(f"{'Sequence':<12} {'Tokens':>8} {'decode()':>12} {'Streaming':>12} {'Speedup':>10}")
    print("-" * 70)
    for r in results:
        print(f"{r['name']:<12} {r['tokens']:>8} {r['naive_ms']:>11.3f}ms {r['optimized_ms']:>11.3f}ms {r['speedup']:>9.2f}x")

    # Average speedup
    avg_speedup = statistics.mean([r["speedup"] for r in results])
    print("-" * 70)
    print(f"{'Average speedup:':<55} {avg_speedup:>9.2f}x")
    print()

    # Verify correctness
    print("Verifying correctness...")
    for name, text in test_texts[:1]:
        tokens = raw_tokenizer.encode(text)

        # Naive
        naive_texts = [raw_tokenizer.decode([t]) for t in tokens]

        # Streaming
        detok = tokenizer_wrapper._detokenizer_class(tokenizer_wrapper)
        detok.reset()
        for t in tokens:
            detok.add_token(t)
        detok.finalize()

        batch_result = raw_tokenizer.decode(tokens)

        if detok.text == batch_result:
            print(f"  ✓ {name}: Streaming matches batch decode")
        else:
            print(f"  ✗ {name}: MISMATCH!")
            print(f"    Streaming: {repr(detok.text)}")
            print(f"    Batch:     {repr(batch_result)}")

Complete contract reference

Expand any definition for its exact inputs, annotations, defaults, return contract, directly raised exceptions, source-grounded behavior, and immutable line link. This section includes private and nested definitions that ordinary API generators omit.

examples.benchmark_detokenizer.benchmark_naive_decode · function 
examples.benchmark_detokenizer.benchmark_naive_decode(tokenizer, tokens, iterations = 10) -> not annotated

Benchmark naive decode approach (old method).

Parameters

Name Type Required Default Description
tokenizer not annotated yes none Required positional or keyword input.
tokens not annotated yes none Required positional or keyword input.
iterations not annotated no 10 Optional positional or keyword input; defaults to 10.

Returns

  • Type: not annotated
  • Direct return expressions: {'method': 'naive_decode', 'mean_ms': statistics.mean(times) * 1000, 'std_ms': statistics.stdev(times) * 1000 if len(ti…

Exceptions and behavior

Function benchmark_naive_decode calls range, time.perf_counter, tokenizer.decode, texts.append; returns {'method': 'naive_decode', 'mean_ms': statistics.mean(times) * 1000, 'std_ms': statistics.stdev(times) * 1000 if len(ti…. No direct raise statement appears in this definition.

View source #L26-L48.

examples.benchmark_detokenizer.benchmark_streaming_detokenizer · function 
examples.benchmark_detokenizer.benchmark_streaming_detokenizer(tokenizer, tokens, detokenizer_class, iterations = 10) -> not annotated

Benchmark streaming detokenizer approach (new method).

Parameters

Name Type Required Default Description
tokenizer not annotated yes none Required positional or keyword input.
tokens not annotated yes none Required positional or keyword input.
detokenizer_class not annotated yes none Required positional or keyword input.
iterations not annotated no 10 Optional positional or keyword input; defaults to 10.

Returns

  • Type: not annotated
  • Direct return expressions: {'method': detokenizer_class.__name__, 'mean_ms': statistics.mean(times) * 1000, 'std_ms': statistics.stdev(times) * 10…

Exceptions and behavior

Function benchmark_streaming_detokenizer calls range, detokenizer_class, detok.reset, time.perf_counter; returns {'method': detokenizer_class.__name__, 'mean_ms': statistics.mean(times) * 1000, 'std_ms': statistics.stdev(times) * 10…. No direct raise statement appears in this definition.

View source #L51-L78.

examples.benchmark_detokenizer.main · function 
examples.benchmark_detokenizer.main() -> not annotated

Function main calls print, Path, snapshot_download, load_tokenizer.

Parameters

This callable has no explicit inputs.

Returns

  • Type: not annotated

Exceptions and behavior

Function main calls print, Path, snapshot_download, load_tokenizer. No direct raise statement appears in this definition.

View source #L81-L182.

Complete symbol map

This map also includes private definitions and nested helpers. The signature column exposes every explicit input even when an internal helper has no dedicated parameter prose.

Symbol Kind Signature and inputs What it does Source
benchmark_naive_decode function benchmark_naive_decode(tokenizer, tokens, iterations = 10) -> not annotated Benchmark naive decode approach (old method). #L26-L48
benchmark_streaming_detokenizer function benchmark_streaming_detokenizer(tokenizer, tokens, detokenizer_class, iterations = 10) -> not annotated Benchmark streaming detokenizer approach (new method). #L51-L78
main function main() -> not annotated Function main calls print, Path, snapshot_download, load_tokenizer. #L81-L182