Stream through a cold boot

You stream a completion from a cold model and keep one request open through a startup that can take many minutes.

Run it: curl -O https://infra.acsresearch.org/examples/python_long_requests.py && ACS_API_KEY="acs-bm-..." uv run python_long_requests.py --model llama-405b

Prerequisites

• You need an API key. Account explains where to create one.
• You need Python 3.10 or later and uv. The script declares its dependencies, openai and httpx, so uv run installs them.

Run the script

1. Download python_long_requests.py.
2. Run it against a model that scales to zero:

ACS_API_KEY="acs-bm-..." uv run python_long_requests.py --model llama-405b

The script prints tokens as they arrive, and the first token can take minutes if the model is cold. If the stream fails, the script prints a line on stderr that says the output is incomplete.

To point the script at another deployment, set ACS_API_BASE to a URL that ends in /v1.

Script walkthrough

1. It builds an OpenAI client with a 900-second read timeout. The read timeout limits the gap between received bytes, not the total time.
2. It sets max_retries=0, so a slow start never becomes two runs.
3. It streams the completion with stream: true. While the model starts, the API sends a comment line every 5 seconds, so the read timeout does not fire.
4. It exits with status 1 if the stream ends in an APIError, which the OpenAI SDK raises when a failure arrives inside the HTTP 200 stream.

Next steps

• Wait for a cold boot in your own code adds a curl version and shows how to catch an error inside an HTTP 200 response.
• Models lists which models stay warm.
• Submit and poll a completion job is the other way to get a plain text completion through a long wait.