API cheatsheet

Each section gives the smallest request body for one task and links to the page that explains it.

Send a first completion

Send this body to POST /v1/completions with your key, as Quick start shows. The continuation comes back in choices[0].text.

{"model": "llama-8b", "prompt": "The capital of France is", "max_tokens": 8}

Set the sampler

A sampler field that you leave out gets the neutral value that Sampling defaults lists, such as 1.0 for temperature and top_p. The values below are example values rather than recommended settings.

A request sent on its own with the same seed repeats its output, but when the server is busy, the logprobs can move a little and a token can change. stop ends generation at any of up to 4 strings.

{"model": "llama-8b", "prompt": "The capital of France is", "max_tokens": 8,
 "temperature": 0.7, "top_p": 0.9, "seed": 7, "stop": ["\n"]}

Sample many completions

A prompt list and n return one choice for each prompt and each sample. vLLM orders choices by prompt, then by sample, so choices[i * n + j] is sample j of prompt i. Such a request must set max_tokens and stay within the output-work cap.

{"model": "llama-8b", "prompt": ["Once upon a time", "The capital of France is"], "max_tokens": 16, "n": 2}

Read logprobs

logprobs: 5 returns the sampled token and up to 5 alternatives at each generated position, as logprobs shows.

{"model": "llama-8b", "prompt": "The capital of France is", "max_tokens": 1, "logprobs": 5}

Read prompt logprobs

prompt_logprobs: 5 returns the actual token and up to 5 alternatives at each prompt position after the first, as prompt_logprobs shows.

{"model": "llama-8b", "prompt": "The capital of France is", "max_tokens": 1, "prompt_logprobs": 5}

A value of -1 returns the full distribution at each position, for at most 256 prompt tokens per request, counted as prompt tokens × n.

{"model": "llama-8b", "prompt": "The capital of France is", "max_tokens": 1, "prompt_logprobs": -1}

The OpenAI SDK takes this field only in extra_body.

Echo the prompt

echo: true puts the prompt in front of the generated text, with no separator, as echo shows.

{"model": "llama-8b", "prompt": "Once upon a time", "max_tokens": 8, "echo": true}

Stream a completion

stream: true sends the completion as server-sent events that end with data: [DONE]. include_usage adds a final chunk with the token counts, as stream shows. Check every frame for an error key, because errors after the stream opens arrive inside the 200 response.

{"model": "llama-8b", "prompt": "Five reasons to learn Rust:\n1.", "max_tokens": 80,
 "stream": true, "stream_options": {"include_usage": true}}

Capture activations

output_residual_stream returns residual-stream activations in activations.residual_stream, from every layer for true or from the layers in a list. Activation harvesting shows how to decode them.

{"model": "llama-8b", "prompt": "The capital of France is", "max_tokens": 1, "temperature": 0,
 "output_residual_stream": [15, 20]}

Steer a completion

apply_steering_vectors adds each vector to the residual stream at the layers in its layer_indices. The encode_vector function on Activation steering builds the activations object from an np.ndarray.

{"model": "llama-8b", "prompt": "I think that", "max_tokens": 32, "temperature": 0,
 "apply_steering_vectors": [{
   "activations": {"data": "...", "dtype": "int16", "original_dtype": "torch.bfloat16",
                   "shape": [1, 4096], "compression": "zstd"},
   "layer_indices": [16],
   "scale": 8.0
 }]}

Run a harvest job

A harvest job captures activations for many prompts and writes them to safetensors files that you download by URL. Submit this body to POST /v1/harvest, then poll GET /v1/harvest/<job_id> until status is done, failed, or cancelled.

{"model": "llama-8b",
 "prompts": ["The quick brown fox jumps over the lazy dog.",
             "Interpretability research reads the residual stream."],
 "layers": [8, 16, 24]}

Submit a completion job

A completion job runs on the server after you close the connection. Submit a completion body with an Idempotency-Key header, then poll until the result is ready.

POST /v1/completion-jobs                submit the body below
GET  /v1/completion-jobs/{id}           read the job's status
GET  /v1/completion-jobs/{id}/result    202 while the job waits or runs, then the stored body

{"model": "llama-8b", "prompt": "Hello", "max_tokens": 128}

We store the request and the result until 24 hours after the job ends.

List the models

GET /v1/models returns the short ids to send as model and each model's limits under capabilities, in the form below. Models describes every field.

{
  "object": "list",
  "data": [
    {
      "id": "llama-8b",
      "object": "model",
      "default": ...,
      "status": "live",
      "capabilities": {"max_model_len": ..., "max_logprobs": 100,
                       "full_vocab_max_prompt_tokens": 256, "activations": ..., ...}
    },
    ...
  ]
}

Headers

Request headers and Response headers describe every header.

Errors

Branch on error.code, because different codes can have the same HTTP status. After a stream opens, or after a long request gets its early 200 status, errors arrive inside the 200 response.

Limits

We apply these limits, and Limits explains each one:

• Each key may run 16 requests at once and queue 64 more.
• Each key may start 600 requests per minute, including requests to POST /v1/completion-jobs.
• prompt_count × n × max_tokens may total at most 32,000 tokens, which is the output-work cap.
• A non-streaming request may set max_tokens to at most 15,600 and must finish within 14 minutes.
• n may be at most 16, and stop may hold at most 4 strings.
• logprobs and prompt_logprobs may be at most 100.
• prompt_logprobs: -1 allows at most 256 prompt tokens per request, or 16 with a capture, counted as prompt tokens × n.
• The request body may be at most 8 MiB.
• Monthly budgets reset on the first day of each month, and a key's daily budget resets at midnight, both in UTC.

Next steps

• Read the API reference for every field, error, and limit.