logprobs

Set logprobs to a number k to get the logprob of the sampled token and of the k highest-probability tokens at each generated position. Use it for calibration, for classification by next-token probability, or to inspect alternative tokens.

The largest value that logprobs accepts is 100 on every model. GET /v1/models reports this limit as capabilities.max_logprobs.

curl

curl -s "$ACS_API_BASE/completions" \
  -H "Authorization: Bearer $ACS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "llama-8b", "prompt": "The capital of France is", "max_tokens": 1, "logprobs": 5}'

Python

resp = client.completions.create(
    model="llama-8b",
    prompt="The capital of France is",
    max_tokens=1,
    logprobs=5,
)
choice = resp.choices[0]
print(choice.text, choice.logprobs.top_logprobs[0])  # {' a': -1.74, ' Paris': -1.99, ...}

Example response

Here the sampled token " a" is one of the 5 highest-probability tokens, so the top_logprobs dict has 5 entries, not k + 1:

{
  "id": "cmpl-91ea94c6ed0e5b75",
  "object": "text_completion",
  "model": "meta-llama/Llama-3.1-8B",
  "choices": [
    {
      "index": 0,
      "text": " a",
      "logprobs": {
        "text_offset": [0],
        "tokens": [" a"],
        "token_logprobs": [-1.7437232732772827],
        "top_logprobs": [
          {
            " a": -1.7437232732772827,
            " Paris": -1.9937232732772827,
            " one": -2.5562233924865723,
            " the": -2.5562233924865723,
            " also": -3.1187233924865723
          }
        ]
      },
      "finish_reason": "length"
    }
  ],
  "usage": {"prompt_tokens": 6, "completion_tokens": 1, "total_tokens": 7}
}

Base llama-8b continues the sentence instead of answering it, so it ranks " a" higher than " Paris".

Each array under logprobs has one entry per generated position. text_offset counts code points in text, not UTF-8 bytes.

Logprob values for the same request can differ slightly when the server is busy, so compare them with a tolerance.

A count, not a boolean

logprobs takes an integer, so logprobs: true returns 400 invalid_request, as Strict validation describes. logprobs: 0 does not turn logprobs off, because vLLM still returns the sampled token's logprob at each position. To turn logprobs off, leave the field out.

logprobs

curl

Python

Example response

A count, not a boolean

Next steps