Evaluate with Inspect

Inspect, from the UK AI Security Institute, is a framework for LLM evals. Its openai-api-completions provider uses an OpenAI-compatible /v1/completions endpoint with raw prompts and no chat template, so it can evaluate any base model we host. The provider name, variable names, and flags below match inspect-ai 0.3.258 and 0.3.277. If you use another version, check them against Inspect's documentation.

Inspect 0.3.245 to 0.3.257 fail every request with openai 3.0 or later, so install Inspect 0.3.258 or later and openai:

pip install "inspect-ai>=0.3.258" openai

Point Inspect at this API

Inspect's openai-api and openai-api-completions providers take a service name from the model string and read the key from <SERVICE>_API_KEY and the base URL from <SERVICE>_BASE_URL. <SERVICE> is the service name in upper case, with each hyphen changed to an underscore. This example uses the name acs:

export ACS_API_KEY="acs-bm-..."
export ACS_BASE_URL="https://infra.acsresearch.org/v1"

inspect eval your_task.py --model openai-api-completions/acs/llama-8b

Inspect reads ACS_BASE_URL, not the ACS_API_BASE that the other pages use. The part after the last / is a short model id from GET /v1/models, not the Hugging Face repository name.

Run a minimal task

Save this task as capitals.py:

from inspect_ai import Task, task
from inspect_ai.dataset import Sample
from inspect_ai.scorer import includes
from inspect_ai.solver import generate

@task
def capitals():
    return Task(
        dataset=[
            Sample(
                input="The capital of Spain is Madrid.\n"
                "The capital of Germany is Berlin.\n"
                "The capital of France is",
                target="Paris",
            )
        ],
        solver=generate(),
        scorer=includes(),
    )

inspect eval capitals.py --model openai-api-completions/acs/llama-8b --temperature 0

The provider generates one token unless you pass --max-tokens.

Write each prompt as text for the base model to continue, not as a chat turn. Few-shot examples such as the Spain and Germany lines work well, and instruction-style prompts often do not.

In a run on 2026-07-14, the eval of the bare prompt The capital of France is scored 0, because llama-8b's greedy completion of that prompt began with a city of many faces. It is. With the Spain and Germany lines, the first generated token was Paris, and that eval scored 1.

Set up a large eval

Set --max-connections to 16, the number of requests we run at once per key. With short requests, 16 connections can still exceed the limit of 600 request starts per minute and get 429 rate_limited.

The provider maps Inspect's generation settings to request fields:

It passes settings such as temperature, top_p, seed, and max_tokens, and it sends stop_seqs as stop.
With --logprobs, it sends top_logprobs as logprobs.
It does not send top_k.
It maps Inspect's num_choices to n, which the API caps at 16.

The API forwards token-id prompts unchanged, but a request with vLLM's return_token_ids field returns 400 invalid_request.

For a large eval, pick a kept-warm model, llama-8b or trinity-truebase, because the first calls on llama-405b wait for a cold boot.

Evaluate with Inspect

Point Inspect at this API

Run a minimal task

Set up a large eval

Next steps