Batch rollouts
Each API key runs up to 16 requests at once, so one client with 16 requests in flight reaches that limit without a second key. The examples send 32 requests tagged with X-Acs-Workload: batch.
curl
# xargs -P 16 runs 16 requests at a time, the per-key limit.
seq 1 32 | xargs -I{} -P 16 curl -s "$ACS_API_BASE/completions" \
-H "Authorization: Bearer $ACS_API_KEY" -H "Content-Type: application/json" \
-H "X-Acs-Workload: batch" \
-d '{"model": "llama-8b", "prompt": "Sample {}: once upon a time", "max_tokens": 32}'
Python
import asyncio
import os
from openai import AsyncOpenAI
client = AsyncOpenAI(
base_url=os.environ["ACS_API_BASE"],
api_key=os.environ["ACS_API_KEY"],
# Tag every request from this client as batch traffic.
default_headers={"X-Acs-Workload": "batch"},
)
prompts = [f"Sample {i}: once upon a time" for i in range(32)]
async def one(p):
r = await client.completions.create(model="llama-8b", prompt=p, max_tokens=32)
return r.choices[0].text
async def _main():
# asyncio.gather() needs a running event loop, so wrap it in a
# coroutine and pass that coroutine to asyncio.run().
return await asyncio.gather(*(one(p) for p in prompts))
results = asyncio.run(_main())
The Python example sends all 32 requests at once. The server runs 16 of them and queues the other 16.
A request with more than one prompt or with n greater than 1 must set max_tokens. The output-work cap limits prompt_count × n × max_tokens to 32,000.
Set X-Acs-Workload: batch on bulk, offline, and eval requests, and interactive on latency-sensitive calls. The header has no effect on scheduling or priority.
Cap your own fan-out at 16
Beyond the 16 active requests, up to 64 more wait in a server-side queue. When the queue is full, the next request gets 429 queue_full with Retry-After: 5. The rate limit is 600 request starts per minute on one key, and requests past it get 429 rate_limited with Retry-After: 60.
A cap of 16 requests in flight with asyncio.Semaphore(16) avoids queue_full but does not limit the start rate. With 16 requests in flight that each finish in under 1.6 seconds, a client starts more than 600 requests per minute. For short requests, also pace your starts to at most 10 per second, as this version does:
import asyncio
import os
from openai import AsyncOpenAI
client = AsyncOpenAI(
base_url=os.environ["ACS_API_BASE"],
api_key=os.environ["ACS_API_KEY"],
default_headers={"X-Acs-Workload": "batch"},
)
prompts = [f"Sample {i}: once upon a time" for i in range(32)]
async def _main():
in_flight = asyncio.Semaphore(16) # 16 running at most, so no queue_full
async def one(p):
async with in_flight:
r = await client.completions.create(model="llama-8b", prompt=p, max_tokens=32)
return r.choices[0].text
tasks = []
for p in prompts:
tasks.append(asyncio.create_task(one(p)))
await asyncio.sleep(0.1) # at most 10 starts per second, which is 600 per minute
return await asyncio.gather(*tasks)
results = asyncio.run(_main())