Activation harvesting

Activation harvesting returns a model's residual-stream activations in the same response as a completion. The residual stream is the vector that each decoder layer passes to the next. Use the values for interpretability and probing work, such as sparse auto-encoders, linear probes, and logit-lens analysis. To capture activations, add output_residual_stream to a POST https://infra.acsresearch.org/v1/completions request.

For each token, the tensor contains the residual stream that the model used to predict the token after it, so the last generated token has no row.

Prerequisites

• You need a model that lists "activations": true under capabilities in GET /v1/models.

Capture the residual stream

Send a completion request with "output_residual_stream": true to capture every decoder layer:

curl -s "$ACS_API_BASE/completions" \
  -H "Authorization: Bearer $ACS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama-8b",
    "prompt": "The capital of France is",
    "max_tokens": 1,
    "temperature": 0,
    "output_residual_stream": true
  }'

Set max_tokens on every capture request. If you leave it out, the API sets it to the rest of the context window. Most captures then exceed the capture size cap and fail with 400 activation_prompt_too_long. A one-layer capture can stay under the cap, but you then generate and pay for thousands of tokens. With max_tokens: 1, the tensor contains only the prompt positions.

Capture only some layers

To capture fewer layers, pass a list of layer indices instead of true:

"output_residual_stream": [15, 20]   // capture only layers 15 and 20

The engine captures only the layers you list, so the response is smaller and the capture size cap allows more positions. The API removes duplicates and sorts the list in ascending order.

Decode the tensor

The example capture returns this activations.residual_stream object, with data shortened:

{
  "data": "...",
  "dtype": "int16",
  "original_dtype": "torch.bfloat16",
  "shape": [32, 6, 4096],
  "compression": "zstd"
}

Reinterpret the int16 bits as bfloat16 with .view(), because a numeric .astype() converts the integer values.

import base64, numpy as np, ml_dtypes, zstandard as zstd

def decode_residual_stream(resp):
    d = resp["activations"]["residual_stream"]
    raw = base64.b64decode(d["data"])
    if d.get("compression") == "zstd":
        raw = zstd.ZstdDecompressor().decompress(raw)
    # reinterpret the 16 bits as bfloat16 with .view (an .astype would corrupt them)
    arr = np.frombuffer(raw, dtype=np.uint16).view(ml_dtypes.bfloat16)
    return arr.reshape(d["shape"])          # (n_layers, n_tokens, hidden)

x = decode_residual_stream(resp)            # for example shape (32, 6, 4096) for llama-8b
layer16 = x[16]                             # slice the layer(s) you want, client-side

Check the tensor before you use it

The API checks only the layer count, so assert the dimensions and values of every capture yourself:

requested = body["output_residual_stream"]   # the value you sent: True or a list
n_layers = 32                                # llama-8b, or read n_layers from GET /v1/models
usage = resp["usage"]

x = decode_residual_stream(resp)
want_layers = n_layers if requested is True else len(set(requested))
want_positions = usage["prompt_tokens"] + usage["completion_tokens"] - 1
assert x.ndim == 3, x.shape
assert x.shape[0] == want_layers, f"partial capture: {x.shape[0]} layers"
assert x.shape[1] == want_positions, f"unexpected position count: {x.shape[1]}"
assert np.isfinite(x.astype(np.float32)).all(), "non-finite values in the capture"

When the layer count differs from your request, the API refuses the capture in one of three forms:

• The response is a 200 whose JSON body ends early, so your JSON parser fails. The API sends this form when it finds the mismatch after it has sent part of the body.
• The response is a 502 with the code capture_incomplete. The API sends this form when it finds the mismatch before it sends the status line.
• The response is a 200 whose body is a complete JSON error object with the code capture_incomplete. This form follows a cold boot, when the API has already sent the status line and keepalive bytes.

A refused capture costs no tokens, but it counts against the key's activation quota.

On a capture with prompt_logprobs: -1, the API skips the layer-count check, and a partial capture arrives as a valid 200.

Get activations for text you already have

Send the text as the prompt, with max_tokens: 1. Text from a completion can tokenize differently when you send it back, so send its token ids if you have them. Start the ids with the model's BOS id, because the engine adds BOS only to a text prompt.

Reference

Request fields

Response format

{
  "choices": [ /* the normal completion */ ],
  "usage":   { /* the normal usage */ },
  "activations": {
    "residual_stream": {
      "data": "<base64 string>",       // zstd-compressed, int16-encoded bf16 bytes
      "dtype": "int16",                // storage dtype (bf16 reinterpreted as int16)
      "original_dtype": "torch.bfloat16",
      "shape": [n_layers, n_tokens, hidden],   // n_layers = full count for `true`, else how many you requested
      "compression": "zstd"
    }
  }
}

The tensor has one row for each prompt token and for each generated token except the last one. Row 0 of a text prompt is the BOS token the engine adds, so the 5-word example prompt gives 6 rows. A 6-token prompt that generates 5 tokens gives 10 rows, 6 for the prompt and 4 for the first four generated tokens.

Capture size cap

GET /v1/models publishes max_activation_prompt_tokens, n_layers, and max_model_len under each activation model's capabilities. max_activation_prompt_tokens is the cap we set for a capture of every layer. Do not hard-code the values, because they can change.

The cap counts positions, which are prompt tokens and generated tokens:

positions = prompt_tokens + max_tokens - 1

A list of fewer layers than n_layers gets a larger cap, because the cap bounds the response size:

cap = min(floor(max_activation_prompt_tokens * n_layers / len(layers)), max_model_len)

llama-8b has 32 layers, a published cap of 256 positions, and a max_model_len of 8192. There, a list of 4 layers allows 8 times the cap, and the list [14] allows 32 times the cap. Both caps are at most max_model_len. The caps and the uncompressed tensor sizes are:

The uncompressed tensor is layers * positions * hidden * 2 bytes. One llama-405b layer takes 32 KiB per position, so a one-layer capture of 4,000 positions is about 125 MiB before compression. For long prompts on llama-405b or trinity-truebase, use bulk harvest.

Errors

Cold boot and quota

We keep llama-8b warm. We scale llama-405b and trinity-truebase to zero after at most 10 idle minutes, and the next request waits while the model starts. On those two models, every startup loads the model from storage, and the load can take minutes. Cold-boot waiting explains the wait.

Prepare your client for a cold boot:

• Set a client timeout of at least 15 minutes.
• Turn off automatic retries in your client, because a retry sends a second request while the first one still waits for the startup.
• Check a 200 body for an error key.

We can give a key a monthly activation quota for capture and steering requests.

Known limits

• Captured values for the same request can differ slightly when the server is busy, so compare an effect you measure with repeated runs of the same request.
• Chunked prefill splits a long prompt into chunks, and the tensor still has one row per token. We have checked this on llama-8b only.
• If the server pauses a capture to free memory, or skips part of a capture, the API returns an error and no activations, because the rows would not match the tokens. After a pause, the code is 503 capture_preempted, and you can send the request again after the Retry-After delay. After a skipped part, the code is 502 capture_rows_missing, and you should not resend the request unchanged.

Next steps

• Steer a completion with your own vector
• Harvest activations for a list of prompts
• Build a steering vector in the pirate notebook