echo
When you set echo: true, the API returns your prompt in front of the generated text in choices[0].text. To score a text, add logprobs and max_tokens: 1, then ignore the generated token.
curl
curl -s "$ACS_API_BASE/completions" \
-H "Authorization: Bearer $ACS_API_KEY" \
-H "Content-Type: application/json" \
-d '{"model": "llama-8b", "prompt": "Once upon a time", "max_tokens": 8, "echo": true}'
Python
resp = client.completions.create(
model="llama-8b",
prompt="Once upon a time",
max_tokens=8,
echo=True,
)
print(resp.choices[0].text) # "Once upon a time there was a man who was very poor"
Example response
{
"id": "cmpl-b986142b3fd7d215",
"object": "text_completion",
"model": "meta-llama/Llama-3.1-8B",
"choices": [
{
"index": 0,
"text": "Once upon a time there was a man who was very poor",
"logprobs": null,
"finish_reason": "length"
}
],
"usage": {"prompt_tokens": 5, "completion_tokens": 8, "total_tokens": 13}
}
usage.completion_tokens counts only the 8 generated tokens. prompt_tokens is 5 for a 4-token prompt because the Llama tokenizer adds a <|begin_of_text|> BOS token.
echo alone returns no logprobs, but with logprobs set, each logprobs array starts with entries for the prompt tokens.
Strip the prompt from the text
echo does not put a separator between the prompt and the continuation, so drop the first len(prompt) characters of text to get only the continuation:
prompt = "Once upon a time"
resp = client.completions.create(model="llama-8b", prompt=prompt, max_tokens=8, echo=True)
continuation = resp.choices[0].text[len(prompt):] # " there was a man who was very poor"