ACS Infrapreview
Research preview. Apply for access. We review each application and email you when it's approved, with your API key ready on the API keys page.

stream

Set stream: true to receive the completion as server-sent events. Each data: frame holds the next part of the completion, and the data: [DONE] frame ends the stream.

Whenever 5 seconds pass with no data, the API sends a comment line. The line is : keepalive, or the startup stage while a cold model starts. A hand-written parser must skip both, along with any other line that starts with :.

A stream that ends without data: [DONE] is incomplete, even when no error frame arrived.

curl

curl -N -s "$ACS_API_BASE/completions" \
  -H "Authorization: Bearer $ACS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "llama-8b", "prompt": "Five reasons to learn Rust:\n1.", "max_tokens": 80,
       "stream": true, "stream_options": {"include_usage": true}}'

-N turns off curl's output buffering so that each frame prints when it arrives.

Python

stream = client.completions.create(
    model="llama-8b",
    prompt="Five reasons to learn Rust:\n1.",
    max_tokens=80,
    stream=True,
    stream_options={"include_usage": True},
)
for chunk in stream:
    if chunk.choices:                            # final usage chunk has choices=[]
        print(chunk.choices[0].text, end="", flush=True)
    elif chunk.usage:
        print(f"\n[tokens: {chunk.usage.total_tokens}]")

The if chunk.choices: guard skips the usage chunk, whose empty choices list would make chunk.choices[0] raise IndexError. Keep the guard when you drop stream_options, so that the loop works with either setting.

Example response

This excerpt leaves out created from every frame and system_fingerprint from the usage frame. It also leaves out logprobs, stop_reason, prompt_token_ids, and token_ids, which are null in every content frame here.

data: {"id":"cmpl-ae7c4833c82a33d6","object":"text_completion","model":"meta-llama/Llama-3.1-8B","choices":[{"index":0,"text":" It","finish_reason":null}]}

data: {"id":"cmpl-ae7c4833c82a33d6","object":"text_completion","model":"meta-llama/Llama-3.1-8B","choices":[{"index":0,"text":"’s","finish_reason":null}]}

data: {"id":"cmpl-ae7c4833c82a33d6","object":"text_completion","model":"meta-llama/Llama-3.1-8B","choices":[{"index":0,"text":" fast","finish_reason":null}]}

data: {"id":"cmpl-ae7c4833c82a33d6","object":"text_completion","model":"meta-llama/Llama-3.1-8B","choices":[{"index":0,"text":".\n","finish_reason":null}]}

... (one chunk per token until max_tokens or stop) ...

data: {"id":"cmpl-ae7c4833c82a33d6","object":"text_completion","model":"meta-llama/Llama-3.1-8B","choices":[{"index":0,"text":" again","finish_reason":"length"}]}

data: {"id":"cmpl-ae7c4833c82a33d6","object":"text_completion","model":"meta-llama/Llama-3.1-8B","choices":[],"usage":{"prompt_tokens":9,"completion_tokens":80,"total_tokens":89}}

data: [DONE]

Each choice has finish_reason: null until its own last content chunk. With stream_options.include_usage: true, a chunk with choices: [] and the token counts in usage arrives before [DONE]. Without that flag or continuous_usage_stats, the stream has no token counts, but the API meters usage either way.

A streaming request rejects any prompt_logprobs value other than 0 with 400. Prompt logprobs come back only in a non-streaming response.

Check every frame for an error

The API answers HTTP 200 and opens the stream once its checks pass, before it contacts the model. A validation, authentication, budget, or circuit-breaker failure happens before the stream opens, so it gets the status and headers in Errors.

Every later failure, including a cold boot, the request deadline, and a model-server error, arrives as a data: {"error": ...} frame inside the HTTP 200 response. Check each frame for an error key before you read choices. The OpenAI Python SDK raises APIError on such a frame.

Next steps