Cold-boot waiting

Large on-demand models sleep when idle. The first request after scale-to-zero takes a few minutes — usually under 10, occasionally longer for the largest models — while Modal allocates GPUs and loads the model. Keep that one request open: the wrapper sends harmless whitespace bytes for non-streaming JSON (or SSE comments for stream=true) so Railway and your client do not treat the connection as idle.

Allow a long wait and disable automatic retries for startup. In the Python SDK, the read timeout limits the gap between received bytes: regular heartbeats can keep one request alive beyond ten minutes. In curl, --max-time instead limits the entire transfer.

curl

curl --fail-with-body --no-buffer --max-time 900 \
  "$ACS_API_BASE/completions" \
  -H "Authorization: Bearer $ACS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "llama-405b", "prompt": "Hello", "max_tokens": 4}'

Python

import os
from openai import APIError, OpenAI

client = OpenAI(
    api_key=os.environ["ACS_API_KEY"],
    base_url=os.environ.get("ACS_API_BASE", "https://infra.acsresearch.org/v1"),
    timeout=900,     # maximum idle read gap, not an elapsed-time deadline
    max_retries=0,   # keep one request; do not duplicate a long startup
)

try:
    with client.completions.create(
        model="llama-405b", prompt="Hello", max_tokens=4, stream=True,
    ) as stream:
        for chunk in stream:
            for choice in chunk.choices:
                print(choice.text, end="", flush=True)
except APIError as exc:
    # A structured SSE error also raises after HTTP 200 / heartbeat comments.
    # Any output already printed belongs to an incomplete request.
    print(f"Request failed: {exc}")
finally:
    client.close()

The repository includes a runnable examples/python_long_requests.py with separate connect/read/write timeouts and a nonzero exit status on failure. These settings configure the SDK; they do not change the server's startup limit.

Non-streaming errors inside a 200

These are rare. The wrapper keeps the HTTP status line open until the model either answers or Modal confirms a cold boot is under way. Every failure that happens before that point — a retried upstream 5xx, out-of-memory, an unreachable model server, the 14-minute deadline — is returned with its real status code (502/503/504), a Retry-After header and an X-Acs-Upstream-Error-Kind header, so the SDK raises APIStatusError like it does for any other error.

The one exception is a failure after a cold boot was confirmed: by then the wrapper is already sending whitespace keepalives, which commits the 200 status, so the error can only be delivered in the body. The body is still the standard OpenAI-shaped {"error": {...}} object. openai-python does not raise on it — it returns a Completion with choices=None and the error attached as an extra field — so check for it before touching choices:

from openai import OpenAI


def completion_or_raise(resp):
    """Surface a late (post-cold-boot) error that arrived inside HTTP 200."""
    err = getattr(resp, "error", None)  # extra field; None on a normal completion
    if err:
        raise RuntimeError(f"{err.get('code')}: {err.get('message')}")
    return resp


client = OpenAI(api_key="...", base_url="https://infra.acsresearch.org/v1",
                timeout=900, max_retries=0)
resp = completion_or_raise(
    client.completions.create(model="llama-405b", prompt="Hello", max_tokens=4)
)
print(resp.choices[0].text)

The same object is reachable on the raw response if you prefer to inspect the JSON directly: client.completions.with_raw_response.create(...) returns the HTTP response, and .http_response.json().get("error") is the error object. Streaming requests never have this ambiguity — a failure is a data: {"error": ...} frame, which the SDK raises as APIError.

Wire-format note

For stream=false, you may see whitespace arrive before the JSON object if you inspect raw bytes. Leading whitespace is valid JSON and normal clients buffer it before parsing. For stream=true, the keepalives are SSE comment frames and are ignored by OpenAI SDKs and EventSource clients.

If startup exceeds the wrapper's 14-minute safety budget, the already-open response ends with a JSON/SSE error payload. That is an allocation failure; do not retry indefinitely.

Interrupted connections

A broken connection does not prove that inference stopped. Avoid automatically repeating a synchronous completion after a timeout, truncated response, or missing terminal SSE frame: the original invocation may still be running.

For ordinary bounded text completions, use completion jobs to submit once with a stable idempotency key, reconnect, and poll the result. An interrupted job is never automatically replayed. Its usage_pending flag means unknown work needs reconciliation before a new invocation is submitted. The existing synchronous endpoint remains available for streaming, logprobs, and activation requests that the initial jobs profile does not support.