ACS Infrapreview
Research preview. Apply for access. We review each application and email you when it's approved, with your API key ready on the API keys page.

Models

Pass a model's short id, such as llama-8b, as model. A Hugging Face repository name returns 400 model_not_found with a suggestion of the short id. GET /v1/models lists the models we serve on this deployment, which can differ from the table below.

Available models

model Checkpoint Precision API requests Workbench, Compare, and Loom
llama-8b meta-llama/Llama-3.1-8B bf16 Kept warm Kept warm
llama-405b meta-llama/Llama-3.1-405B bf16 Scales to zero after at most 10 idle minutes Scales to zero after at most 5 idle minutes
trinity-truebase arcee-ai/Trinity-Large-TrueBase bf16 Scales to zero after at most 10 idle minutes Kept warm

An API request can wait for a startup even when the Workbench shows the model as warm.

Each deployment sets the context length of each model, so read it from capabilities.max_model_len in GET /v1/models.

This service is "Built with Llama." llama-8b and llama-405b are Meta Llama 3.1 models, available under the Llama 3.1 Community License. Your use of them through this service is subject to that license.

GET /v1/models

The response has one entry per live model, in this format:

{
  "object": "list",
  "data": [
    {
      "id": "...",
      "object": "model",
      "owned_by": "acs",
      "served_model_name": "...",
      "gpu_shape": "...",
      "status": "live",
      "default": ...,
      "capabilities": {"max_model_len": ..., "max_logprobs": ..., ...}
    }
  ]
}
Field Type Meaning
id string The short id you pass as model.
object string The value is always "model".
owned_by string The value is always "acs".
served_model_name string The checkpoint the engine serves.
gpu_shape string GPU count and type, such as 1xL40S or 8xH200.
status string The value is always "live". It does not mean that the model is warm or healthy.
default boolean The value is true on the model that a request without model goes to. At most one entry has the value true. Set model on every request, because we can change the default.
capabilities object The fields in the next table.
capabilities field Type Meaning
max_model_len integer or null The context length in prompt plus completion tokens. When the value is null, the API skips its pre-flight length checks.
max_logprobs integer The largest positive logprobs or prompt_logprobs count.
max_output_work_tokens integer The cap on prompts × n × max_tokens per request.
max_inflight_per_key integer Active requests allowed per key.
max_queued_per_key integer Requests per key allowed to wait for a free slot.
rate_limit_per_minute integer Requests per key per minute.
logprobs, prompt_logprobs boolean These fields are feature flags. Both are true today.
prompt_logprobs_full_vocab boolean Whether the model accepts prompt_logprobs: -1, the whole next-token distribution at each prompt position. Completion logprobs stays top-k.
full_vocab_max_prompt_tokens integer The cap on prompt tokens × n for a prompt_logprobs: -1 request, with the prompt tokens summed over every prompt. A capture lowers the cap to 16 prompt tokens, which this field does not show. The cap exists because each position returns about 128k logprobs on Llama. For a longer prompt, use a positive top-k count.
activations boolean The value is true when the model accepts capture and steering requests. The next three fields appear only when the value is true. When the value is false, every API request for the model returns 503 activation_engine_unconfigured.
max_steering_vectors integer The most steering vectors in one request.
max_activation_prompt_tokens integer The cap on captured positions for a capture of every layer. Fewer layers get a larger cap. See Activation harvesting.
n_layers integer The decoder layer count. Layer indices run from 0 to n_layers - 1. When the layer count is unknown, the response leaves out the key instead of setting it to null.
chat_template null The value is always null, because these are base models and the API does not apply a template.

When a model's entry includes n_layers, the API rejects an out-of-range index in layer_indices or output_residual_stream with 400 invalid_request. When the entry has no n_layers, the engine rejects an out-of-range steering index with 400 vllm_invalid_request. In that case, read the layer count from shape[0] of a capture.

Availability and cold starts

We keep a container running for each kept-warm engine, but a restart or a deployment can still make a request wait. A scale-to-zero engine stops after its idle time, and the next request waits while Modal allocates GPUs and then loads the model. Neither step has a fixed length, so a startup can take many minutes, and longer when GPUs are scarce.

Set a client timeout of at least 15 minutes and keep that first request open, as Cold-boot waiting shows. The startup counts toward the API's 14-minute limit on a request, and the API sends keepalive bytes so that the connection does not look idle. To watch a startup from a script, poll GET /v1/models/{id}/status.

We recommend one short request to start the model before a work session. To keep a model warm for a work session or a deadline, choose Feedback or email infra@acsresearch.org to ask for a scheduled warm window.

GET /v1/models/{id}/status

The status endpoint reports whether one model is warm and its most recent startup stage, in this format:

{
  "id": "...",
  "object": "model.status",
  "warm": ...,
  "boot": {"stage": "...", "stage_label": "...", "stage_age_s": ...},
  "activation_boot": ...
}

warm and boot are estimates, so only a real request shows whether the model answers.

Field Type Meaning
id string The model id.
object string The value is always "model.status".
warm boolean The value is true when Modal reports at least one container that serves API requests for the model, or when Modal does not answer and an API request finished in the last 10 minutes.
boot object or null The most recent startup stage of a container that serves API requests for the model, with stage, stage_label, and stage_age_s in seconds. The value is null when there is no recent stage. On a cold model, a null value means that Modal has not started the container yet.
activation_boot object or null The value is a copy of boot, which we keep for older clients.

Next steps