prompt_logprobs

Set prompt_logprobs to a number k to get the model's top-k predictions at each position of your prompt. Use it to see what the model expected at each position, or to score a text.

curl

curl -s "$ACS_API_BASE/completions" \
  -H "Authorization: Bearer $ACS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "llama-8b",
       "prompt": "The capital of France is",
       "max_tokens": 1,
       "prompt_logprobs": 5}' | jq '.choices[0].prompt_logprobs[:2]'

Python

resp = client.completions.create(
    model="llama-8b",
    prompt="The capital of France is",
    max_tokens=1,
    extra_body={"prompt_logprobs": 5},
)

# resp.choices[0].prompt_logprobs[i] is a dict for prompt position i, keyed by
# token id as a string, with logprob, rank, and decoded_token. Position 0 is
# None, because it has no conditioning context.
for i, choices in enumerate(resp.choices[0].prompt_logprobs[:3]):
    if choices is None:
        continue
    print(f"position {i}:")
    for token_id, info in choices.items():
        print(f"  rank={info['rank']:>2}  logprob={info['logprob']:+.3f}  {info['decoded_token']!r}")

Example response

The jq filter in the curl command prints choices[0].prompt_logprobs[:2]:

[
  null,
  {
    "791":   {"logprob": -3.679518699645996, "rank": 4, "decoded_token": "The"},
    "14924": {"logprob": -1.179518699645996, "rank": 1, "decoded_token": "Question"},
    "755":   {"logprob": -2.179518699645996, "rank": 2, "decoded_token": "def"},
    "2":     {"logprob": -2.742018699645996, "rank": 3, "decoded_token": "#"},
    "16309": {"logprob": -4.179518699645996, "rank": 5, "decoded_token": "Tags"}
  }
]

Position 0 is null because the first token has no context. At position 1, the model's top prediction is "Question", while the actual prompt token, "The", is at rank 4. When the actual token is outside the top k, vLLM adds it as an extra entry with a rank greater than k, so you can still read its logprob.

Get the full distribution at each prompt position

Set prompt_logprobs to -1 to get one logprob per vocabulary entry at each prompt position, about 128k per position on Llama. Use -1 for entropy, for Kullback-Leibler divergence between models, or for interpretability work that needs the tail of the distribution. A fixed top-k value is enough to rank candidates or to score a sequence, and a top-k request has no prompt-length cap of its own.

curl -s --compressed "$ACS_API_BASE/completions" \
  -H "Authorization: Bearer $ACS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "llama-8b", "prompt": "The capital of France is", "max_tokens": 1, "prompt_logprobs": -1}'

The prompts of one request may total at most 256 tokens, which is less than a page of text, or 16 tokens when the request also captures activations. Both caps count prompt tokens × n, because each choice repeats the prompt's distribution. GET /v1/models reports the 256-token cap as capabilities.full_vocab_max_prompt_tokens, which is the same for every model. A longer request returns 400 full_vocab_prompt_too_long.

The body is about 10 MB of JSON per prompt token before compression. Its first bytes arrive only after vLLM has built the whole response. If you send Accept-Encoding: gzip, the API gzips the response. The OpenAI SDK, requests, httpx, browsers, and curl --compressed send that header and decompress for you.

Get the full distribution over generated tokens

Completion logprobs stays top-k, so a request with logprobs: -1 returns 400. To get the full distribution at every generation step, generate the completion first, then score the prompt plus the completion as one prompt.

prompt = "The capital of France is"
gen = client.completions.create(
    model="llama-8b", prompt=prompt, max_tokens=8, seed=7,
    temperature=1.0, top_p=1.0,   # set the sampler explicitly, see below
)
full_text = prompt + gen.choices[0].text

rescored = client.completions.create(
    model="llama-8b",
    prompt=full_text,
    max_tokens=1,
    extra_body={"prompt_logprobs": -1},
)
# rescored.choices[0].prompt_logprobs[i] is now the full ~128k-entry
# distribution at position i, including every generated position.

The second call costs one extra prefill, and the combined text must fit within the full-vocabulary prompt cap. The second response is about 10 MB of JSON per position before compression, which is about 150 MB for this example with 8 generated tokens, so send Accept-Encoding: gzip.

The rescored distributions are the model's own, before any sampler setting applies, but the generated tokens come from the first call's sampler. The snippet sets temperature and top_p to 1.0, which are also the values the API sends when you leave them out, so the first call samples from the same distribution that the second call returns. A lower temperature or top_p in the first call draws its tokens from a sharper, truncated version of that distribution.

vLLM tokenizes the second call's text again, so positions can shift without an error. A base model can sample a token split that the tokenizer would not produce, or the last prompt token can merge with the first completion token. The API returns no generated token ids to send back, so you cannot avoid the second tokenization.

When gen.choices[0].finish_reason is "length", check that rescored.usage.prompt_tokens equals gen.usage.prompt_tokens + gen.usage.completion_tokens. Different counts mean that the positions no longer line up, but equal counts do not rule out a shift.

Score your own text with a token-id prompt

To score your text, read the entry for each actual token, which is not always the rank 1 entry:

Tokenize the text with the model's tokenizer and send the token ids as prompt. Put the tokenizer's bos_token_id first, because vLLM adds nothing to a token-id prompt and scores the ids as you sent them.
Set prompt_logprobs: 20 and max_tokens: 1. Ignore the one generated token.
From position 1 on, read the entry in prompt_logprobs[i] whose key is your token's id as a string. The entry is always there, because vLLM adds the actual token as an extra entry when that token falls outside the top k.
Add up those logprobs to get the log-likelihood of every token after the first token.

For "The capital of France is" on llama-8b, the body is:

{"model": "llama-8b", "prompt": [128000, 791, 6864, 315, 9822, 374], "max_tokens": 1, "prompt_logprobs": 20}

For a text prompt, prompt_logprobs: 0 returns the actual token at each position after position 0, keyed by its id, with the token's text in decoded_token. Match tokens by id, because different token ids can decode to the same text.