Skip to content

examples.demo_openai_text

Demo: OpenAI API - Text Chat Shows how to use vllm-mlx with the OpenAI Python SDK for text-only chat.

View the complete module source at #L1-L123.

API details

Each callable below includes its exact signature, type annotations, inputs, defaults, return contract, documented exceptions, implementation source, and parsed docstring sections when the source provides them.

examples.demo_openai_text

Demo: OpenAI API - Text Chat

Shows how to use vllm-mlx with the OpenAI Python SDK for text-only chat.

Usage

  1. Start the server with any model (served model name is defaulted to "mlx-community/Llama-3.2-3B-Instruct-4bit"): vllm-mlx serve mlx-community/Llama-3.2-3B-Instruct-4bit --port 8000

  2. Run this script: python examples/demo_openai_text.py

examples.demo_openai_text.client module-attribute 

client = OpenAI(base_url='http://localhost:8000/v1', api_key='not-needed')

examples.demo_openai_text.response module-attribute 

response = client.chat.completions.create(model='mlx-community/Llama-3.2-3B-Instruct-4bit', messages=messages, max_tokens=50)

examples.demo_openai_text.stream module-attribute 

stream = client.chat.completions.create(model='mlx-community/Llama-3.2-3B-Instruct-4bit', messages=[{'role': 'user', 'content': 'Tell me a short joke'}], max_tokens=150, stream=True)

examples.demo_openai_text.messages module-attribute 

messages = [{'role': 'user', 'content': 'What is 2 + 2?'}]

examples.demo_openai_text.prompt module-attribute 

prompt = 'Complete this sentence: The robot walked into the'

examples.demo_openai_text.response_low module-attribute 

response_low = client.chat.completions.create(model='mlx-community/Llama-3.2-3B-Instruct-4bit', messages=[{'role': 'user', 'content': prompt}], max_tokens=30, temperature=0.1)

examples.demo_openai_text.response_high module-attribute 

response_high = client.chat.completions.create(model='mlx-community/Llama-3.2-3B-Instruct-4bit', messages=[{'role': 'user', 'content': prompt}], max_tokens=30, temperature=1.0)

Complete contract reference

Expand any definition for its exact inputs, annotations, defaults, return contract, directly raised exceptions, source-grounded behavior, and immutable line link. This section includes private and nested definitions that ordinary API generators omit.

This module does not declare classes or functions.

Complete symbol map

This map also includes private definitions and nested helpers. The signature column exposes every explicit input even when an internal helper has no dedicated parameter prose.

This module does not declare classes or functions.