Cold-boot waiting

These snippets keep one request open until the model answers, even when its startup takes many minutes.

A request waits for a startup when the model has scaled to zero. It can also wait on a kept-warm model during a restart or a deployment. Models lists which models scale to zero.

During a startup, the model waits for GPU capacity and then loads, and neither step has a fixed length. While the request waits, the API sends keepalive bytes so that neither the public endpoint nor your client closes the connection as idle.

curl

curl --fail-with-body --no-buffer --max-time 900 \
  "$ACS_API_BASE/completions" \
  -H "Authorization: Bearer $ACS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "llama-405b", "prompt": "Hello", "max_tokens": 4}'

--max-time 900 limits the whole transfer to 15 minutes, which covers the API's 14-minute limit on a non-streaming request. On a stream, the 14-minute limit applies only to the wait for the model's first byte. The public endpoint ends every HTTP request, a stream included, after 15 minutes.

If your key already runs 16 requests, a new request waits for a free slot before its 14 minutes start. --max-time counts that wait too, so it can expire first.

--fail-with-body catches only a failure that arrives before the HTTP 200 response, so also check the body for an error key.

Python

import os
from openai import APIError, OpenAI

client = OpenAI(
    api_key=os.environ["ACS_API_KEY"],
    base_url=os.environ.get("ACS_API_BASE", "https://infra.acsresearch.org/v1"),
    timeout=900,     # maximum idle read gap, not an elapsed-time deadline
    max_retries=0,   # keep one request, do not duplicate a long startup
)

try:
    with client.completions.create(
        model="llama-405b", prompt="Hello", max_tokens=4, stream=True,
    ) as stream:
        for chunk in stream:
            for choice in chunk.choices:
                print(choice.text, end="", flush=True)
except APIError as exc:
    # A structured SSE error also raises after HTTP 200 / heartbeat comments.
    # Any output already printed belongs to an incomplete request.
    print(f"Request failed: {exc}")
finally:
    client.close()

Because this snippet streams, each kind of failure reaches your code in a different way:

• An error that the API reports inside the stream raises APIError.
• With openai-python 3.14.0 or later, a dropped connection raises APIConnectionError, and a read gap longer than 900 seconds raises APITimeoutError. Both are subclasses of APIError, so this snippet catches them.
• Earlier versions raise an exception from the SDK's HTTP library in those two cases, and this snippet does not catch it.
• If the stream closes early without an error, the loop ends with no exception.

Treat the text as complete only after a chunk reports a finish_reason.

Errors inside a 200 response

Without streaming

If a non-streaming request runs longer than about 150 seconds, the API sends HTTP 200 and then whitespace until the result arrives. A warm model gets the same early HTTP 200 as a cold one. JSON allows whitespace before a value, so openai-python and other JSON parsers skip it.

The HTTP status cannot change once the API has sent it, so any later failure, including the 14-minute deadline, arrives in the body as an {"error": {...}} object. For an activation capture or a prompt_logprobs: -1 request, a failure after the first byte cuts the body short instead.

When the body contains the error object, openai-python does not raise an exception. It returns a Completion with choices set to None and the error in an extra error field. Check for the error before you read choices:

from openai import OpenAI


def completion_or_raise(resp):
    """Surface an error that arrived inside an HTTP 200 body."""
    err = getattr(resp, "error", None)  # extra field, None on a normal completion
    if err:
        raise RuntimeError(f"{err.get('code')}: {err.get('message')}")
    return resp


client = OpenAI(api_key="...", base_url="https://infra.acsresearch.org/v1",
                timeout=900, max_retries=0)
resp = completion_or_raise(
    client.completions.create(model="llama-405b", prompt="Hello", max_tokens=4)
)
print(resp.choices[0].text)

With client.completions.with_raw_response.create(...), .http_response.json().get("error") returns the same error object.

A failure before the HTTP 200 response gets its real status code and the headers in this table:

An upstream error that comes from a bad request returns 400 without retries.

To watch a non-streaming startup, poll the model's status from a second terminal and read warm and boot.stage_label.

curl -s "$ACS_API_BASE/models/llama-405b/status" \
  -H "Authorization: Bearer $ACS_API_KEY"

With streaming

With stream: true, the API returns HTTP 200 before it contacts the model, so every failure after that arrives as a data: {"error": ...} frame.

While the stream waits, the API sends a : keepalive comment every 5 seconds. If the API judged the model cold, a startup progress comment replaces each keepalive until the first token arrives. The openai-python SDK ignores both kinds of comment.

The 14-minute deadline

The deadline ends the request with one of two codes. A non-streaming request nearly always gets the code inside the HTTP 200 response body, because the HTTP 200 arrives after about 150 seconds.

The code depends on whether the API judged the model cold when the request started. A model counts as cold when no completion on it finished within its idle time. The rule also covers kept-warm models, so an idle kept-warm model counts as cold while its container runs. A kept-warm model can still count as warm during a restart.

Do not resend after a dropped connection

A broken connection does not prove that inference stopped, so resending can run the same work twice. Do not repeat a synchronous completion automatically after a timeout, a truncated response, or a stream that ended without its final frame.

For a plain text completion, submit a completion job once with a stable Idempotency-Key, and poll the job after you reconnect. Completion jobs reject streaming, logprobs, prompt_logprobs, and activation requests.

Next steps

• Run the long-requests script
• Look up every error code