examples.simple_generate

Simple text generation example using vllm-mlx.

View the complete module source at #L1-L71.

API details

Each callable below includes its exact signature, type annotations, inputs, defaults, return contract, documented exceptions, implementation source, and parsed docstring sections when the source provides them.

examples.simple_generate

Simple text generation example using vllm-mlx.

This example demonstrates basic LLM inference on Apple Silicon using the MLX backend.

examples.simple_generate.main

main()

Source code in examples/simple_generate.py

def main():
    # Use a small quantized model for quick testing
    model_name = "mlx-community/Llama-3.2-3B-Instruct-4bit"

    print(f"Loading model: {model_name}")
    model = MLXLanguageModel(model_name)
    model.load()

    print("\n" + "=" * 50)
    print("Model loaded! Starting generation...")
    print("=" * 50 + "\n")

    # Simple generation
    prompt = "What is the meaning of life?"
    print(f"Prompt: {prompt}\n")

    output = model.generate(
        prompt,
        max_tokens=200,
        temperature=0.7,
    )

    print(f"Response:\n{output.text}")
    print(f"\nFinish reason: {output.finish_reason}")

    # Streaming generation
    print("\n" + "=" * 50)
    print("Streaming generation:")
    print("=" * 50 + "\n")

    prompt = "Write a haiku about coding:"
    print(f"Prompt: {prompt}\n")
    print("Response: ", end="", flush=True)

    for chunk in model.stream_generate(
        prompt,
        max_tokens=100,
        temperature=0.8,
    ):
        print(chunk.text, end="", flush=True)

    print("\n")

    # Chat interface
    print("=" * 50)
    print("Chat interface:")
    print("=" * 50 + "\n")

    messages = [
        {"role": "user", "content": "Hello! Can you introduce yourself?"}
    ]

    response = model.chat(messages, max_tokens=150)
    print(f"User: {messages[0]['content']}")
    print(f"Assistant: {response.text}")

Complete contract reference

Expand any definition for its exact inputs, annotations, defaults, return contract, directly raised exceptions, source-grounded behavior, and immutable line link. This section includes private and nested definitions that ordinary API generators omit.

examples.simple_generate.main · function

examples.simple_generate.main() -> not annotated

Function main calls print, MLXLanguageModel, model.load, model.generate.

Parameters

This callable has no explicit inputs.

Returns

• Type: not annotated

Exceptions and behavior

Function main calls print, MLXLanguageModel, model.load, model.generate. No direct raise statement appears in this definition.

View source #L13-L67.

Complete symbol map

This map also includes private definitions and nested helpers. The signature column exposes every explicit input even when an internal helper has no dedicated parameter prose.

Symbol	Kind	Signature and inputs	What it does	Source
main	function	main() -> not annotated	Function main calls print, MLXLanguageModel, model.load, model.generate.	#L13-L67