API reference

Look up the fields, errors, headers, and limits of POST /v1/completions, the OpenAI-compatible completions endpoint for base models.

Endpoint

POST https://infra.acsresearch.org/v1/completions

The request body is the OpenAI Completions shape plus five vLLM-native fields: top_k, min_p, repetition_penalty, prompt_logprobs, and add_special_tokens.

curl -s "$ACS_API_BASE/completions" \
  -H "Authorization: Bearer $ACS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "llama-8b", "prompt": "The capital of France is", "max_tokens": 8}'

Send vLLM-native fields from the OpenAI SDK

The OpenAI Python SDK has no keyword arguments for these five fields. If you pass one as a keyword argument, the SDK raises TypeError before it sends anything. Put them in extra_body, which the SDK merges into the top level of the JSON body.

resp = client.completions.create(
    model="llama-8b",
    prompt="The capital of France is",
    max_tokens=1,
    temperature=0.7,
    logprobs=5,
    extra_body={                # vLLM-native fields go here
        "top_k": 20,
        "min_p": 0.05,
        "repetition_penalty": 1.1,
        "prompt_logprobs": 5,
    },
)

Over raw HTTP, put these fields at the top level of the JSON body, because a literal extra_body key returns 400 invalid_request with the message Unknown field 'extra_body'.

curl -s "$ACS_API_BASE/completions" \
  -H "Authorization: Bearer $ACS_API_KEY" -H "Content-Type: application/json" \
  -d '{"model": "llama-8b", "prompt": "The capital of France is",
       "max_tokens": 1, "temperature": 0.7, "logprobs": 5,
       "top_k": 20, "min_p": 0.05, "repetition_penalty": 1.1, "prompt_logprobs": 5}'

Strict validation

The API rejects an unknown field, a wrong type, or an out-of-range value with 400 invalid_request and names the field in the message. A bad value never falls back to a default.

• Every top-level numeric field except priority rejects booleans and strings.
• A float field accepts a JSON integer.
• echo, stream, add_special_tokens, and the two stream_options flags accept only JSON true or false. They reject values such as 1 and "true".
• output_residual_stream accepts true, false, or a list of layer indices.
• A mixed prompt list such as [1, "two"], an empty list such as [] or [[]], and a batch with an empty token-id prompt return 400. An empty string "" is valid.

https://infra.acsresearch.org/openapi.json publishes these bounds for the /v1/* endpoints and /health, and https://infra.acsresearch.org/swagger renders that spec. Neither URL needs an API key.

The spec omits the 413 and 504 responses of POST /v1/completions, the X-Should-Retry header, and every X-Acs-Max-Tokens-Clamped reason except budget.

Sampling defaults

If a request omits a sampling field, the API sends the field's neutral value, so the model samples from its whole, unchanged distribution:

The values are the same for every model, and the model's own generation_config.json does not change them.

A local run with transformers samples differently, because transformers takes temperature and top_p from generation_config.json and uses top_k 50 when the file omits it.

Request body

model

Type: string. Default: the model that GET /v1/models marks with "default": true.

Always set model, because a request without it runs on the default model, and we can change the default. Use the short id from GET /v1/models, such as llama-8b, not the Hugging Face repo name. The short id is an alias that we can later point at another checkpoint, so record the response's model field for reproducibility.

prompt

Type: string, string[], int[], or int[][]. Required.

A string or a list of token ids is one prompt, which the model continues as written. A batch is a list of strings or a list of id lists.

The API forwards token ids unchanged, but vLLM tokenizes a text prompt itself. On some tokenizers, ids that you decode to text and send as text come back as different ids. Send ids when the tokens must match a dataset or an earlier run.

To sample from a Llama model unconditionally, send an empty string "", which the tokenizer turns into the BOS token alone. With add_special_tokens: false, an empty string has no tokens, and vLLM rejects the request.

# OpenAI SDK: prompt takes token ids directly (no extra_body needed)
resp = client.completions.create(model="llama-8b", prompt=[1, 2, 3, 4, 5], max_tokens=8)

max_tokens

Type: int, 1 to 32,000, which is the output-work cap. Default: min(32000, max_model_len - prompt_tokens) for one prompt with n: 1. A request with more than one prompt, or with n greater than 1, must set max_tokens.

max_tokens caps the number of tokens generated for each completion and counts toward the output-work cap. Near a budget cap, the API lowers it to fit, as Requests near the cap describes.

A non-streaming request may set max_tokens to at most 15,600, the largest value that the API expects to finish within the time limit. A higher value returns 400 deadline_infeasible. If you omit max_tokens and the default is higher than 15,600, the API lowers it to 15,600 and sets X-Acs-Max-Tokens-Clamped with reason=deadline. Streams skip both checks.

n

Type: int, 1 to 16. Default: 1.

n sets the number of completions to sample for each prompt and counts toward the output-work cap.

stop

Type: string, or a list of up to 4 strings. Default: none.

Generation stops when the model produces one of these strings.

temperature

Type: float, 0 to 100. Default: 1.0.

The sampler divides the logits by this value. A value of 0 selects the likeliest token at every step. Pair a high temperature with top_p, top_k, or min_p so that sampling stays within a plausible set of tokens.

top_p

Type: float greater than 0 and at most 1. Default: 1.0, which turns the filter off.

Nucleus sampling keeps the smallest set of tokens whose probabilities add up to top_p. A value of 1 turns the filter off.

top_k

Type: int, -1 or 1 and up. Default: -1, which turns the filter off.

The sampler keeps only the top_k likeliest tokens. A value of -1 turns the filter off, and 0 returns 400.

min_p

Type: float, 0 to 1. Default: 0, which turns the filter off.

The sampler keeps only tokens whose probability is at least min_p times that of the likeliest token. A value of 0 turns the filter off.

presence_penalty

Type: float, -2 to 2. Default: 0, which turns the penalty off.

vLLM applies a flat penalty to every token already in the generated text. Positive values discourage repeats, and negative values encourage them.

frequency_penalty

Type: float, -2 to 2. Default: 0, which turns the penalty off.

This penalty works like presence_penalty, but vLLM scales it by the number of times the token has appeared in the generated text.

repetition_penalty

Type: float greater than 0 and at most 2. Default: 1.0, which turns the penalty off.

vLLM applies a multiplicative penalty to every token that appears in the prompt or in the generated text, unlike the additive presence_penalty and frequency_penalty, which count generated tokens only. A value of 1 turns the penalty off. Values greater than 1 discourage repeats, and values below 1 encourage them.

seed

Type: int, 0 to 9223372036854775807. Default: none. Without a seed, each request draws new random numbers.

A request sent on its own with the same model, prompt, settings, and seed repeats its output. When the server is busy, the logprobs can move a little, and a sampled token can change.

logprobs

Type: int, 0 to 100. Default: none, so choices[i].logprobs is null.

Each generated position's top_logprobs contains the sampled token plus up to k alternatives, where k is this value. That list has at most k + 1 entries, or k entries when the sampled token is one of the top k tokens. With a value of 0, top_logprobs contains only the sampled token. GET /v1/models reports the upper bound as capabilities.max_logprobs.

prompt_logprobs

Type: int, -1 or 0 to 100. Default: none.

choices[i].prompt_logprobs contains the model's prediction at each prompt position, given the tokens before it. Use it for likelihood, surprisal, and interpretability work. Each position after position 0 contains the actual token plus up to k alternatives, where k is this value. Position 0 is always null.

To get the token ids that vLLM used for a text prompt, set a value of 0, which returns only the actual token at each later position. On the Llama models, position 0 is the BOS token, so add its id yourself before you resend the ids.

With stream: true, the API accepts only prompt_logprobs: 0, and the stream then returns no prompt logprobs.

A value of -1 returns the full vocabulary distribution at every prompt position:

• The prompts of one request may total at most 256 tokens, which is less than a page of text. GET /v1/models reports this cap as capabilities.full_vocab_max_prompt_tokens. With a capture, the cap is 16 tokens. Both caps count prompt tokens × n, because each choice repeats the prompt's distribution. Both caps also count each text prompt's BOS token, even with add_special_tokens: false.
• Each position has about vocab_size entries. Send Accept-Encoding: gzip to get the body compressed.
• The response starts only after vLLM has built it, so expect a delay before any byte arrives on a long prompt.

echo

Type: bool. Default: false.

With true, choices[i].text holds the prompt followed by the generated text, with no separator, as the echo example shows.

stream

Type: bool. Default: false.

With true, the response is a text/event-stream of server-sent events. Each event is a data: line with one JSON chunk, and the stream ends with data: [DONE]. Errors after the stream opens arrive inside the 200 response, as the stream example shows.

stream_options

Type: object. Default: none. The API accepts stream_options only with stream: true, and an unknown key in stream_options returns 400.

• include_usage: true adds a final chunk with choices: [] and a usage object before data: [DONE]. The API meters usage either way. Guard your loop with if chunk.choices:, as the stream example does.
• continuous_usage_stats: true makes vLLM add a running usage object to every content chunk. The object is cumulative for each choice. With include_usage, the final frame still contains the batch totals.

user

Type: string. Default: none.

The API forwards user to vLLM unchanged and does not log it.

add_special_tokens

Type: bool. Default: true.

With true, the tokenizer prepends the model's BOS token to a text prompt. On the Llama models, the BOS token is <|begin_of_text|>. Set false when your text already starts with a BOS token, as it does when you render a chat template yourself. A second BOS token shifts logprobs, echo offsets, and activation captures one position later.

Activation fields

output_residual_stream captures residual-stream activations from all layers or from a list of layers. apply_steering_vectors adds steering vectors, each with its own layer_indices. Only a model that reports capabilities.activations: true in GET /v1/models accepts these fields. See Activation harvesting and Activation steering.

Response

A non-streaming request returns a text_completion object like the example below, while a stream spreads the same fields across SSE chunks.

{
  "id": "cmpl-91ea94c6ed0e5b75",
  "object": "text_completion",
  "model": "meta-llama/Llama-3.1-8B",
  "choices": [
    {
      "index": 0,
      "text": " a",
      "logprobs": {
        "text_offset": [0],
        "tokens": [" a"],
        "token_logprobs": [-1.7437],
        "top_logprobs": [
          {" a": -1.7437, " Paris": -1.9937, " one": -2.5562, " the": -2.5562, " also": -3.1187}
        ]
      },
      "finish_reason": "length"
    }
  ],
  "usage": {"prompt_tokens": 6, "completion_tokens": 1, "total_tokens": 7}
}

The example leaves out created, system_fingerprint, and the fields that vLLM sends as null.

The response has these top-level fields:

• id is unique per response and differs from the X-Request-Id trace id.
• object is always "text_completion".
• model is the checkpoint that served the request, such as "meta-llama/Llama-3.1-8B", not the short id that you sent.
• choices has one entry for each prompt and each of its n samples. vLLM orders choices by prompt, then by sample, so choices[i * n + j] is sample j of prompt i.
• usage holds prompt_tokens, completion_tokens, and total_tokens.

Each choice has these fields:

• index is the position of the choice in choices.
• text is the generated text.
• finish_reason is "length" when generation reaches max_tokens or the context limit, and "stop" when the model emits its end-of-sequence token or a stop string. In a stream, each choice has null in every chunk before its own last chunk.
• stop_reason is the matched stop string, or null. With finish_reason: "stop", null means the model emitted its end-of-sequence token.
• logprobs holds four arrays with one entry per generated token. With echo: true, each array starts with entries for the prompt tokens. The arrays are:
  • tokens holds the token strings.
  • token_logprobs holds the logprob of each token.
  • top_logprobs holds a {token: logprob} dict for each position. With echo: true, the first entry of token_logprobs and of top_logprobs is null.
  • text_offset holds where each token starts, counted in code points, not UTF-8 bytes. Each offset is the total length of the earlier strings in tokens. With echo: true on a Llama text prompt, tokens starts with <|begin_of_text|>, which text does not contain. Each later offset is then 17 higher than the token's position in text.
• prompt_logprobs is present when you set prompt_logprobs.

The API passes vLLM's response through with these changes:

• In a stream, it drops vLLM's usage chunk unless you set include_usage. It also adds SSE comment lines as keepalives.
• On a long non-streaming request, it sends the 200 status early and then whitespace keepalives before the JSON body.
• It replaces a vLLM 401 or 403 response with 502 upstream_auth.

Errors

Wrapper errors have the body {"error": {"message": ..., "type": ..., "code": ...}}. Some errors add upstream_status, retryable, or retry_after_seconds. A bad query or path parameter on a /v1/* route also returns 400 invalid_request in this envelope. An unknown path returns {"detail": "Not Found"} instead, and a wrong method returns {"detail": "Method Not Allowed"}.

Many errors add a top-level request_id. Authentication errors, model_not_found, rate_limited, circuit_open, and upstream errors sent with an error status do not, so read the X-Request-Id header for those.

Branch on error.code, which the SDK exposes as e.code, because different codes can have the same HTTP status.

A vLLM 4xx response passes through with vLLM's status and body, so its error.code is an integer.

Errors inside a 200 response

In the two cases below, the API sends a 200 status before the error, so these responses have no Retry-After, X-Should-Retry, or X-Acs-Upstream-Error-Kind header.

• With stream: true, the stream opens with 200 text/event-stream once the checks for validation, authentication, limits, budgets, and the circuit breaker pass. Every later failure arrives as a data: {"error": {...}} frame. The OpenAI Python SDK raises APIError on that frame.
• Without streaming, a request that runs past about 150 seconds gets its 200 status at that point, before the result exists. A later error, including upstream_deadline and modal_cold_boot, arrives as a body with a top-level error instead of choices. The OpenAI Python SDK does not raise an exception. It returns a Completion with choices set to None and the error in resp.error.

Cold-boot waiting shows how to handle both cases and poll the model's status.

Limits

• Requests use prompt and completion tokens from your key's and your account's budgets. Monthly budgets reset on the first day of each month, and a key's daily budget resets at midnight, both in UTC.
• Near a budget cap, the API lowers max_tokens to the number of tokens that fit and still runs the request, as Requests near the cap describes.
• Each key may have up to 16 requests running at once and up to 64 requests waiting in its queue.
• Each key may start up to 600 requests per minute. This limit includes requests to POST /v1/completion-jobs.
• The output-work cap limits prompt_count × n × max_tokens to 32,000 tokens. A string or one token-id list counts as one prompt. A batch counts as one prompt for each entry in its list.
• A non-streaming request must finish within 14 minutes. Completion jobs have the same limit.
• The request body may be at most 8 MiB.
• Full-vocabulary prompt_logprobs allows up to 256 prompt tokens per request, or up to 16 with a capture, counted as prompt tokens × n.
• Activation requests count against the key's monthly activation quota, if it has one.
• We can set a daily limit on GPU spend for the whole service. When spend passes that limit, we refuse new inference requests from the API and the browser tools. The refusal lasts until an administrator lifts it. Running requests continue.

Request headers

• Every request needs Authorization: Bearer <ACS_API_KEY>.
• Accept-Encoding: gzip makes the API compress large non-streaming responses.
• X-Acs-Workload takes batch for bulk, offline, and eval jobs, or interactive for latency-sensitive calls. The header has no effect on scheduling or priority.

We ask you to set X-Acs-Workload, because our request records use it to tell batch traffic from interactive traffic. The API matches the value without regard to case. It ignores any other value without rejecting the request. The Workbench and the Loom record their requests as interactive.

Response headers

• X-Request-Id is the API's trace id and equals request_id in an error body. Every response has it except 500 internal_error.
• X-Acs-Upstream-Model and X-Acs-Upstream-Gpu name the backend that served or failed the request, once the API has chosen a backend.
• X-Acs-Upstream-Error-Kind equals error.code on a non-streaming error from the model server or its connection. For an unclassified vLLM 5xx response, this header is upstream_5xx and error.code is upstream_server_error. Streaming errors, 503 modal_cold_boot, a passed-through vLLM 4xx response, and errors inside a 200 body do not have this header.
• X-Acs-Max-Tokens-Clamped appears when the API lowers max_tokens. Its value has the form requested=N,applied=M,reason=R, where R is budget, deadline, or deadline+budget. A budget clamp appends ,source=client. If you omitted max_tokens and N is the injected default, a budget clamp appends ,source=default instead. A deadline-only clamp has no source.
• Retry-After gives a delay in seconds. The Errors table says which errors have it.
• X-Should-Retry: false marks a response that resending cannot fix. The OpenAI Python SDK skips its automatic retries on such a response. 413 request_too_large, 400 deadline_infeasible, 504 upstream_deadline, 502 upstream_auth, 502 capture_rows_missing, 503 activation_engine_unconfigured, 503 modal_cold_boot, and 502 upstream_server_error after a vLLM 500 response have this header.
• X-Acs-Longpoll: declined marks a harvest long poll that the server answered early, as Bulk harvest describes.

429 budget_exceeded and 429 activation_quota_exceeded have neither Retry-After nor X-Should-Retry. With default settings, the OpenAI Python SDK retries them twice before it raises an exception. Set max_retries=0, as Budget-cap recovery shows.

Unsupported features

• The API does not support chat. We do not apply a chat template, so POST /v1/chat/completions returns 400 chat_completions_unsupported. In the OpenAI SDK, call client.completions.create(...).
• The API does not support max_tokens: 0. To score text without generating new tokens, send it with prompt_logprobs and max_tokens: 1, and ignore the generated token. See the prompt_logprobs example.
• The API does not support logprobs: -1. Full-vocabulary logprobs exist only for prompt positions, through prompt_logprobs: -1.
• The API does not support best_of. A request with best_of returns 400. The error message points to n.
• priority has no effect. The API forwards it to vLLM, which accepts any value and ignores it, because the production models use first-come-first-served scheduling.
• The API does not support other fields, such as vLLM's return_token_ids. A request with such a field returns 400 invalid_request.

Privacy

For each /v1/completions request, we log these fields. We do not log prompt text, completion text, logprob values, or steering vector contents.

• request id, key id, key prefix, and account email
• IP address, endpoint, and model
• prompt and completion token counts, status, and error kind
• latency, time to first token, and whether the request streamed or waited for a cold boot
• the X-Acs-Workload value
• the requested max_tokens, temperature, and top_p, where the log records 1.0 for an omitted temperature or top_p
• upstream latency, time spent in budget admission, and the source of the usage count
• whether the request set seed or echo, and the requested logprobs and prompt_logprobs counts
• whether the request captured activations or steered, with the layer and vector counts
• for a steering request, the selector mode, the highest targeted position, whether that position is a generated token, how many vectors had a nonzero strength, and how many used norm matching
• the engine that served the request

We use this metadata to run the service and to investigate misuse and incidents. Our policy allows us to start logging request content for the same reasons, but it does not allow us to train on API requests.

The Workbench and the Loom store saved prompts, completions, and run settings on the server. Their recovery checkpoints for unsaved results expire after seven days. Saved histories and nodes are separate from those checkpoints and do not expire with them.

For opt-in completion jobs, we store request and result content until 24 hours after the job ends.

Get help

Write to infra@acsresearch.org, or use the Feedback button after you sign in. Include the rough time of the request, the X-Request-Id, the error body, and the key prefix or key name from your API keys page. Never send the secret key.

Next steps

• Read logprobs at each generated position
• Wait out a cold boot
• Capture residual-stream activations