Models

Choose a model from the Workbench picker or GET /v1/models. The picker shows configured startup policies and recent-use hints, not a live readiness check. A kept-warm model can still wait during a restart or deployment.

An existing Workbench session keeps its saved model. A new session currently prefers the largest model with a configured keep-warm policy (including an enabled warm-window schedule), falling back to the configured default if none qualify. This can differ from the model used in a docs example; check the picker before running.

Available models

Built with Llama. llama-8b and llama-405b are Meta Llama 3.1 models, made available under the Llama 3.1 Community License; use of them through this service is subject to that licence.

This catalog is illustrative and can change; GET /v1/models is the source of truth for models enabled on this deployment. Its live status means enabled in the registry, not currently warm or healthy. Pass the short model id (e.g. llama-8b), never the HF repo name.

What GET /v1/models returns

Query the fields in each entry rather than hard-coding values that may change:

• id — the short id you pass as model.
• default — true on exactly one entry: the model a /v1/completions request without model is routed to. Don't rely on it — pass model explicitly.
• served_model_name — the underlying checkpoint the backend serves.
• gpu_shape — the GPU shape backing this model (e.g. which accelerator / how many), useful for reasoning about cold-start cost and throughput.
• status — "live" for models you can call (only live models are listed).
• capabilities:
  • max_model_len — the context length: total prompt + completion tokens the model accepts. Read this instead of memorising a per-model window. (null means the registry hasn't declared it, so the pre-flight length check is skipped.)
  • max_logprobs — the cap on a positive (top-k) logprobs / prompt_logprobs count.
  • logprobs, prompt_logprobs — feature flags (both true today).
  • prompt_logprobs_full_vocab — whether prompt_logprobs=-1 (the model's whole next-token distribution at each prompt position) is supported. Prompt-only — completion logprobs stays top-k.
  • full_vocab_max_prompt_tokens — max prompt length for a full-vocab prompt_logprobs=-1 request. The full distribution is ~vocab_size values per position, so longer prompts must use a fixed top-k.
  • n_layers — the model's decoder-layer count (activation models only; the field is absent, not null, when a model doesn't declare it, so read it with a .get()). Two uses: layer indices run 0 … n_layers-1, and it's what lets you compute your own capture cap when you request a subset of layers — see Activation harvesting. Both paths reject an out-of-range index up front with 400 invalid_request whenever n_layers is published — a steering layer_index and a capture layer list alike. Only for a model that doesn't publish n_layers is the check left to the engine (a steering index then surfaces as 400 vllm_invalid_request); in that case verify shape[0] of what you received.
  • chat_template — always null: these are base models, prompts pass through verbatim with no templating.

Availability & cold starts

Always-on policies request a running container; scheduled warm windows apply only during the configured schedule. On-demand models can scale to zero after an idle period. Their next request may wait several minutes for startup, with longer waits possible when GPUs are unavailable. Picker startup ranges are estimates, not deadlines or measured queue positions.

While a request waits, Workbench shows elapsed time and the last available server-reported stage. If no stage is available, it says so. Opening the same session in another tab can resume that generation's status; opening a different session does not subscribe to a global model-startup feed.

The wrapper holds the first request open and sends keepalive bytes while the model starts, so callers should set a timeout of at least 15 minutes and wait for that request to finish. See Cold-boot waiting. For interactive work, send one short request first to warm the model before a session.

Scheduled warm windows

If you'd like a model kept warm for a scheduled stretch — a work session, a deadline — tell us via the Feedback button or email, and we'll schedule a warm window.