API reference

OpenAI-compatible POST /v1/completions for raw next-token generation against a base model. No chat endpoint — see What's not supported.

Endpoint

POST https://infra.acsresearch.org/v1/completions

Sends a prompt to a base model and returns one or more continuations. The request body follows the OpenAI Completions shape with vLLM-native extras (top_k, min_p, repetition_penalty, prompt_logprobs); responses pass through verbatim from vLLM.

curl -s "$ACS_API_BASE/completions" \
  -H "Authorization: Bearer $ACS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "llama-8b", "prompt": "The capital of France is", "max_tokens": 8}'

Using the OpenAI Python SDK

Five params are vLLM-native and aren't on the SDK's typed signature, so the SDK rejects them as keyword arguments before the request leaves your machine: top_k, min_p, repetition_penalty, prompt_logprobs, add_special_tokens. Pass them through extra_body={...}. Everything else — temperature, top_p, max_tokens, n, the penalties, seed, logprobs, echo, stream, stop, user — works as a normal keyword argument.

resp = client.completions.create(
    model="llama-8b",
    prompt="The capital of France is",
    max_tokens=1,
    temperature=0.7,            # standard kwarg
    logprobs=5,                 # standard kwarg
    extra_body={                # vLLM-native — must go here
        "top_k": 20,
        "min_p": 0.05,
        "repetition_penalty": 1.1,
        "prompt_logprobs": 5,
    },
)

extra_body is an SDK construct, not a wire field. The SDK merges everything under extra_body into the top-level request JSON before it goes over the wire. Over raw HTTP (curl, requests, etc.) there's no SDK to do that merge — put these parameters at the top level of the JSON body alongside model and prompt. Sending a literal {"extra_body": {...}} over raw HTTP gets a 400 Unknown field 'extra_body'.

# Raw HTTP: top_k / min_p / repetition_penalty / prompt_logprobs are just top-level fields.
curl -s "$ACS_API_BASE/completions" \
  -H "Authorization: Bearer $ACS_API_KEY" -H "Content-Type: application/json" \
  -d '{"model": "llama-8b", "prompt": "The capital of France is",
       "max_tokens": 1, "temperature": 0.7, "logprobs": 5,
       "top_k": 20, "min_p": 0.05, "repetition_penalty": 1.1, "prompt_logprobs": 5}'

Strict validation

Unknown fields, out-of-range values, and wrong types return 400. For example, temperature=200 returns 400 instead of resetting to the default. Invalid types include logprobs: true (a boolean), max_tokens: "16" (a string), and stream: "true". Every numeric param (max_tokens, n, seed, top_k, temperature, top_p, min_p, the penalties, logprobs, prompt_logprobs) rejects booleans and strings; a JSON integer is accepted where a float is expected (temperature: 1). echo, stream, and add_special_tokens are the only fields that take booleans, and they take only booleans. stream_options is a typed object — an unknown key inside it is a 400 as well.

These constraints are published in a machine-readable OpenAPI spec at https://infra.acsresearch.org/openapi.json (browsable UI at https://infra.acsresearch.org/swagger). It covers the public API surface — the /v1/* endpoints plus /health — and lists the exact bounds, including seed in 0–2^63-1, temperature 0–100, top_p in (0, 1], and stop at most 4 items. Use it to check values before sending a request. The spec also documents every error status each operation can return (with the error envelope) and the X-Acs-* / Retry-After response headers.

Request body

Routing and prompt

model

Type: string  ·  Default: the deployment's default model (the entry with "default": true in GET /v1/models)

Short id from GET /v1/models — e.g. llama-8b, llama-405b. Use the short id, not the HF repo name. Unknown ids return 400 model_not_found. Always set it explicitly — a request without model is routed to the default model silently, and the default can change between deployments; the response's model field tells you which checkpoint actually served.

prompt

Type: string, string[], int[] (token ids), or int[][] (batch of token-id prompts)  ·  Required

Raw text — no chat template is applied (these are base models, so the prompt passes through verbatim and the model continues it). A string[] is a batch: one completion per prompt, subject to the output-work cap below.

Pre-tokenized prompts. You can pass token ids directly instead of text — int[] for one prompt, int[][] for a batch. Ids are forwarded to the model verbatim, with no detokenize→retokenize round-trip (which on a non-bijective tokenizer can quietly change the token sequence). Use this when the prompt must be byte-for-byte exact — interpretability / eval work, or replaying an already-tokenized dataset. usage.prompt_tokens for a token-id prompt is just the number of ids. Mixed lists (e.g. [1, "two"]), empty lists ([], [[]]) and a batch containing an empty token-id prompt return 400. An empty string prompt ("") is accepted: with the default add_special_tokens: true it tokenizes to just the BOS token, which is how you sample from the model unconditionally. (With add_special_tokens: false there is nothing to tokenize and the engine rejects it.)

# OpenAI SDK — prompt takes token ids directly (no extra_body needed)
resp = client.completions.create(model="llama-8b", prompt=[1, 2, 3, 4, 5], max_tokens=8)

Length and stop

max_tokens

Type: int, 1 ≤ n ≤ 1_000_000 (schema bound)  ·  Default: auto-injected for a single prompt with n=1; required otherwise

Maximum tokens to generate per completion. A single prompt with n=1 may omit it — the wrapper injects min(32_000, remaining model context) rather than letting vLLM fill an unbounded context. Batched prompts or n > 1 must set it explicitly because the total worst-case output work is capped:

prompt_count × n × max_tokens ≤ 32_000  —  exceed this and the request returns 400 invalid_request.

prompt_count is the number of prompts the request carries — not a token count: 1 for a single string (or a single pre-tokenized list[int] prompt), the list length for a batch (list[str] or list[list[int]]).

max_tokens=0 is not supported here — a deliberate choice, not an engine limit (vLLM itself can score with echo + max_tokens=0). To score existing text rather than generate, use prompt_logprobs with max_tokens=1 and echo=true; you get the prompt-position logprobs and ignore the one generated token. See the prompt_logprobs and echo examples.

The wrapper may further reduce max_tokens to fit a per-key budget; when that happens the response carries an X-Acs-Max-Tokens-Clamped header (requested=N,applied=M,reason=budget,source=client|default — source=default means you omitted max_tokens and N is the wrapper's injected default, not a value you sent). (The Workbench and Loom refuse such a run instead of clamping — see Budgets.)

n

Type: int, 1 ≤ n ≤ 16  ·  Default: 1

Number of independently-sampled completions per prompt. Counts against the prompt_count × n × max_tokens cap above.

stop

Type: string or string[] (up to 4 entries)  ·  Default: none

Strings that, when produced, stop sampling. Matched stop content is not included in the response. Lists longer than 4 return 400 invalid_request.

Sampling

temperature

Type: float, 0.0 ≤ x ≤ 100  ·  Default: 1.0

Sampling temperature. 0.0 is greedy decoding (top-1 at every step) and is fully deterministic regardless of seed. Higher values flatten the distribution toward uniform; the effect scales as ~1/T, so it's already ~90% of the way to uniform by 10 and negligible beyond. (The Workbench slider stops at 5 — the common range — but you can pass higher here.) Pair a high temperature with top_p / top_k / min_p to keep the flattened sampling inside a plausible set.

top_p

Type: float, 0.0 < x ≤ 1.0  ·  Default: 1.0

Nucleus sampling threshold — keep the smallest set of tokens whose cumulative probability reaches top_p. 1.0 disables. Must be strictly positive (0.0 returns 400).

top_k

Type: int, -1 (disabled) or n ≥ 1  ·  Default: -1

Keep only the top-k tokens at each step. -1 disables.

vLLM-native: with the OpenAI Python SDK, pass via extra_body={"top_k": ...}.

min_p

Type: float, 0.0 ≤ x ≤ 1.0  ·  Default: 0.0

Minimum probability — relative to the top token — a token must have to be sampled. 0.0 disables.

vLLM-native: with the OpenAI Python SDK, pass via extra_body={"min_p": ...}.

Penalties

presence_penalty

Type: float, -2.0 ≤ x ≤ 2.0  ·  Default: 0.0

Penalty applied once a token has appeared anywhere in the generation so far. Positive values push the model away from repeating any token; negative encourage it.

frequency_penalty

Type: float, -2.0 ≤ x ≤ 2.0  ·  Default: 0.0

Like presence_penalty, but scaled by how often the token has already appeared.

repetition_penalty

Type: float, 0.0 < x ≤ 2.0  ·  Default: 1.0

Multiplicative repetition penalty (vLLM-native, distinct from the additive presence / frequency penalties). 1.0 disables; values above 1.0 discourage repeats; values below 1.0 encourage them. Must be strictly positive.

vLLM-native: with the OpenAI Python SDK, pass via extra_body={"repetition_penalty": ...}.

Determinism

seed

Type: int, 0 ≤ n ≤ 2^63-1  ·  Default: none (random)

PRNG seed for sampling (a signed 64-bit integer, the engine's own bound — larger values are a 400). A fixed seed with identical params on the same underlying checkpoint gives a reproducible draw. Greedy decoding (temperature=0) is deterministic without seed.

Token-level inspection

logprobs

Type: int, 0 ≤ n ≤ 100  ·  Default: none

Top-k logprobs per generated token. Unset disables. 0 does not disable: it returns each generated token's own logprob with no alternatives (top_logprobs[i] holds just the sampled token — vLLM returns k + 1 entries, the sampled token plus k alternatives). The per-model cap is exposed in GET /v1/models as capabilities.max_logprobs — read that rather than hard-coding 100. See the logprobs example.

prompt_logprobs

Type: int, n = -1 or 0 ≤ n ≤ 100  ·  Default: none

Top-k logprobs at each prompt position — the model's predictions over the prompt, not the actual prompt tokens unless they happened to be in the top k. As with logprobs, 0 is not "off": every position after the first carries exactly its own token's logprob (the Inspect page uses this to recover token ids). Useful for likelihood / surprisal / interpretability work. See the prompt_logprobs example. A top-k (> 0) or full-vocab (-1) request is not available with stream: true — that combination is a 400; prompt logprobs come back only in the non-streaming body. (0 is allowed with stream, matching the engine.)

Pass -1 for the full vocabulary — the model's entire next-token distribution at every prompt position. This is prompt-only (completion logprobs can't be -1) and, because the payload is ~vocab_size values per token (tens of MB), prompt length is capped — see capabilities.full_vocab_max_prompt_tokens in GET /v1/models, currently 1024 tokens for every model; a longer full-vocab prompt returns 400 full_vocab_prompt_too_long. For longer prompts, use a fixed top-k. Send Accept-Encoding: gzip (most HTTP clients do by default) — the wrapper compresses the response, which shrinks these logprobs bodies several-fold. Expect a large download even with compression. The response streams: the first bytes can take a while on long prompts while the model server serializes the body.

vLLM-native: with the OpenAI Python SDK, pass via extra_body={"prompt_logprobs": ...}.

echo

Type: bool  ·  Default: false

If true, the prompt is prepended to choices[i].text (no separator). See the echo example.

Streaming

stream

Type: bool  ·  Default: false

If true, the response is text/event-stream (SSE) with one chunk per token, terminated by data: [DONE]. See the stream example.

stream_options

Type: object  ·  Default: none

Optional streaming flags (only accepted together with stream: true; unknown keys are a 400). Currently honored:

• include_usage: bool — emit a terminal chunk with choices: [] and a populated usage object before [DONE]. Without this, streamed responses skip the usage frame — you get only content chunks, each with a non-empty choices. (The wrapper always meters the stream server-side; the flag controls only what you receive.) Guard with if chunk.choices: anyway if your code may run with either setting — see the stream example.
• continuous_usage_stats: bool — vLLM-native: attach a running usage block to every content chunk (cumulative for that choice) instead of only at the end. Useful for live token meters; the terminal frame still carries the batch totals when you also set include_usage.

Bookkeeping

user

Type: string  ·  Default: none

Free-form tag echoed back for your own bookkeeping (matches the OpenAI shape). Telemetry only — does not affect routing, scheduling, or pricing.

add_special_tokens

Type: boolean  ·  Default: true

Whether the tokenizer prepends the model's special tokens — for these base models, the BOS (<|begin_of_text|>). Leave it true (the default) for plain text prompts.

Set it to false when your prompt text already starts with a BOS — most commonly when you render a chat template client-side, which emits <|begin_of_text|> itself. With the default true, the engine adds a second BOS, and every token position shifts by one: logprobs, echo offsets, and any activation capture no longer line up with the tokens you think you sent. Passing add_special_tokens=false tokenizes your text verbatim so exactly one BOS is present.

This is a vLLM-native field (not on the OpenAI SDK's typed signature), so pass it via extra_body={...} from the SDK, or as a top-level field over raw HTTP.

What's not supported

No chat endpoint. POST /v1/chat/completions returns 400 chat_completions_unsupported — these are base models, no chat template. The openai SDK defaults to chat, so call client.completions.create(...) (not client.chat.completions.create(...)).

Returns

A text_completion object (JSON for non-streaming; SSE frames for stream: true):

{
  "id": "cmpl-91ea94c6ed0e5b75",
  "object": "text_completion",
  "model": "meta-llama/Llama-3.1-8B",
  "choices": [
    {
      "index": 0,
      "text": " a",
      "logprobs": {
        "text_offset": [0],
        "tokens": [" a"],
        "token_logprobs": [-1.7437],
        "top_logprobs": [
          {" a": -1.7437, " Paris": -1.9937, " one": -2.5562, " the": -2.5562, " also": -3.1187}
        ]
      },
      "finish_reason": "length"
    }
  ],
  "usage": {"prompt_tokens": 6, "completion_tokens": 1, "total_tokens": 7}
}

Top-level

• id — unique per response. Distinct from the X-Request-Id response header (which is the wrapper's trace id — quote that one in bug reports).
• object — always "text_completion".
• model — the checkpoint the backend served (e.g. "meta-llama/Llama-3.1-8B"), not the short id you passed. Use this for reproducibility; the short id is a routing alias and can re-point.
• choices — array of length prompt_count × n, ordered first by prompt then by fan-out index.
• usage — prompt_tokens, completion_tokens, total_tokens. Streaming responses include usage only if you set stream_options.include_usage: true (as a final choices: [] chunk).

Per choice

• text — the generated continuation (or prompt + continuation, if echo: true).
• index — the choice's position in the response.
• finish_reason — "stop" (a stop string matched), "length" (hit max_tokens or the model's context limit), or null mid-stream.
• stop_reason — the matched stop string when finish_reason == "stop", else null. vLLM-specific; omitted from the abbreviated example above but present on the wire.
• logprobs — null unless you set logprobs. When present:
  • tokens[] — generated token strings.
  • token_logprobs[] — logprob of each generated token.
  • top_logprobs[] — array of {token: logprob} dicts, one per position; each dict holds the sampled token plus up to logprobs alternatives (so logprobs: 0 gives a one-entry dict).
  • text_offset[] — byte offsets of each token in text.

When you set prompt_logprobs, vLLM surfaces it under choices[i].prompt_logprobs — see the prompt_logprobs example for the full shape.

Responses are passed through verbatim from vLLM. The wrapper validates request bodies strictly but does not re-shape responses, so logprobs, prompt_logprobs, and token ids surface exactly as the engine produced them.

Limits

• Each key has a monthly token budget (prompt + completion tokens, reset on the 1st, UTC). Exceed it and requests return 429 budget_exceeded. Email to raise it.
• Up to 16 active requests per key; another 64 may wait server-side. Beyond that, requests return retryable 429 queue_full rather than consuming unbounded memory.
• Each key may start up to 600 requests/minute. Sustained floods above that return 429 rate_limited.
• A single request may ask for at most 32,000 worst-case output tokens, calculated as prompt_count × n × max_tokens (prompt_count = prompts in the request: 1 for a single prompt, the list length for a batch). Batched prompts and n > 1 must set max_tokens explicitly.

Errors

Every error body follows the OpenAI shape: {"error": {"code": "...", "message": "...", "type": "..."}} with extra fields tagged where useful, plus a top-level request_id on most errors — the same value as the X-Request-Id header, which is on every response. Read error.code for programmatic handling — the openai SDK exposes it directly as e.code on the raised APIStatusError (see Budget-cap recovery). This applies to every /v1/* route, including a malformed query/path parameter or a missing required header (400 invalid_request); you will never see FastAPI's bare {"detail": [...]} on the public API.

Request headers

• Authorization: Bearer <ACS_API_KEY> — required.
• Content-Type: application/json — for the JSON body.
• X-Acs-Workload: interactive | batch — please set this on your requests: batch for bulk / offline / eval jobs, interactive for live, latency-sensitive calls. We use it as the main signal for understanding usage and planning capacity / keep-warm policy. It does not change scheduling or priority; unrecognised values are ignored. See Batch rollouts.

Response headers

• X-Request-Id — unique trace id for the request; quote it in bug reports so we can find it in our logs.
• X-Acs-Upstream-Model / X-Acs-Upstream-Gpu — which backend served (or failed) this request. Cite this in bug reports.
• X-Acs-Upstream-Error-Kind — set on upstream-error 4xx/5xx responses (mirrors error.code). Absent only on the rare in-body error after a confirmed cold boot (the 200* row above), where headers were already sent.
• X-Acs-Max-Tokens-Clamped — present when the wrapper reduced max_tokens; format requested=N,applied=M,reason=budget|deadline|deadline+budget,source=client|default. reason=deadline means you omitted max_tokens on a non-streaming request and the injected default was cut to what can finish inside the request window (~15,600 tokens); send stream=true to keep the full default.
• X-Should-Retry: false — on responses that a client retry cannot fix (504 upstream_deadline, 413, 400 deadline_infeasible). openai-python honours it and skips its automatic retries.
• X-Acs-Longpoll: declined — set on a GET /v1/harvest/<id>?wait= that was answered early (as a normal 200 with the current job state) because too many long-polls were already waiting. Not an error — just poll again; the accompanying Retry-After tells you when.
• Retry-After — RFC-7231 header on every retryable error response (429, 502, 503), and on an early-declined harvest long-poll (the X-Acs-Longpoll: declined 200 above). A 504 upstream_deadline carries none: the same request would time out again.

Privacy

API requests to /v1/completions are logged as metadata only — key prefix, email, IP, endpoint, model, token counts, status, latency — which we use to run the service and prevent abuse. We don't log your prompts or completions. Note this might change: we may start logging API requests to prevent abuse. We will not train on them. (Keep your own copies if you need them for reproducibility.)

Workbench and Loom store your saved prompts, completions and run settings server-side so you can revisit and export them. Temporary recovery checkpoints for results waiting to be saved expire after seven days; saved histories and nodes are separate from those checkpoints. Don't keep anything there you wouldn't want stored.

Opt-in completion jobs also store request and result content for polling and recovery. Requests remain stored while queued or running. Queued jobs expire after 24 hours; request/result content expires 24 hours after the job becomes terminal. Compact deduplication and accounting metadata remains for seven days after completion, or longer while accounting is unresolved. Keep your own exported results for long-term reproducibility.

Getting help

Email infra@acsresearch.org, or use the in-app Feedback button if you're signed in. To help us trace a specific request, include the rough time you made it, your key prefix or name (found on your API keys page — not the secret), and the error body.