Bulk harvest

Harvest residual-stream activations over a whole prompt corpus as downloadable files — the batch counterpart to inline harvesting, for thousands to millions of prompts, such as SAE training or large probing datasets.

You submit a job, poll it, and download safetensors shards by URL. Same API key as everything else. The GPU work runs on a dedicated batch app per model; nothing you do here slows the completions engines.

Submit a job

POST https://infra.acsresearch.org/v1/harvest with your prompts:

curl -s "$ACS_API_BASE/harvest" \
  -H "Authorization: Bearer $ACS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama-8b",
    "prompts": ["The quick brown fox jumps over the lazy dog.",
                "Interpretability research reads the residual stream."],
    "layers": [8, 16, 24]
  }'

{ "job_id": "071fa547-…", "status": "running", "model": "llama-8b", "n_prompts": 2, "reserved_tokens": 11 }

(Responses below are abridged — a real one carries a few more bookkeeping keys like run_id and created_at.)

Fields beyond model and prompts (a misspelled field name returns 400 instead of silently using a default):

layers — which decoder-block outputs to keep, e.g. [8, 16, 24]. Omit it for the default quartile subset (blocks at ~25/50/75% depth — [8, 16, 24] on llama-8b); pass the string "all" (lowercase, exact) to keep every block. Unlike the inline endpoint, the harvester subsets server-side, so the 3-layer default on a 32-layer model writes ~10× less data than "all". Same convention as inline: layer k is the output of block k.
shard_size — prompts per output file (default 32).
batch_size — forward-pass batch inside the job. Leave it unset: the default vLLM-Lens backend batches continuously on its own; the value only shapes the HF reference path (HARVEST_USE_VLLM=0).

Poll it

GET https://infra.acsresearch.org/v1/harvest/<job_id> with the same key. status moves from running to done (or failed, with a short error message):

curl -s "$ACS_API_BASE/harvest/071fa547-…" -H "Authorization: Bearer $ACS_API_KEY"

{
  "job_id": "071fa547-…",
  "status": "done",
  "model": "llama-8b",
  "n_prompts": 2,
  "result": {
    "n_shards": 1,
    "layer_indices": [8, 16, 24],
    "manifest_url": "https://…/manifest.json?…",      // presigned — no key needed
    "shard_urls":   ["https://…/shard_00000.safetensors?…"],
    "stats_url":    "https://…/stats.safetensors?…",
    "timings": { "forward_s": 1.59, "upload_s": 4.05, "tokens": 21, /* … */ }
  },
  "urls_expire_at": "2026-07-24T20:31:37+00:00"
}

A small llama-8b job goes submit → done in about a minute, most of it engine start-up. Poll every 20–30 s; there's no webhook.

Cancel a job

DELETE https://infra.acsresearch.org/v1/harvest/<job_id> with the same key cancels a job that is still pending or running. A slow or stuck job occupies a concurrency slot (see the limits below); cancelling frees that slot immediately and asks the GPU container to stop.

curl -s -X DELETE "$ACS_API_BASE/harvest/071fa547-…" \
  -H "Authorization: Bearer $ACS_API_KEY"

{
  "job_id": "071fa547-…",
  "status": "cancelled",
  "model": "llama-8b",
  "n_prompts": 2,
  "created_at": "2026-08-07T20:14:00+00:00",
  "completed_at": "2026-08-07T20:14:12+00:00"
}

404 — no such job for your key (unknown id, or someone else's job).
409 harvest_not_cancellable — the job already finished (done/failed) or was already cancelled.

Cancellation frees the slot the moment the 200 returns. Stopping the container is best-effort: if the wrapper can't reach Modal, the job still shows cancelled and the slot is free, but that container may keep running until it finishes on its own or hits its timeout.

Download and read the shards

The URLs are presigned and work in a plain browser, curl, or wget — no API key. They expire at urls_expire_at (7 days after completion; the response adds "urls_expired": true once they have).

Each shard holds two tensors per prompt — the activations and the input token ids, so you can line activations up with tokens without re-tokenizing:

import os, requests
from safetensors.torch import load_file

API_BASE = os.environ["ACS_API_BASE"]           # e.g. https://…/v1
KEY = os.environ["ACS_API_KEY"]

job = requests.get(f"{API_BASE}/harvest/{job_id}",
                   headers={"Authorization": f"Bearer {KEY}"}).json()

open("shard0.safetensors", "wb").write(requests.get(job["result"]["shard_urls"][0]).content)
shard = load_file("shard0.safetensors")

shard["prompt_0"]   # bf16, [n_layers_kept, n_tokens, hidden] — e.g. [3, 11, 4096]
shard["tokens_0"]   # int32, [n_tokens] — same order as the tensor's token axis

tokens_i is a stable part of the format. It's what lets you verify alignment against your own tokenization rather than trusting ours, and it's how the double-BOS trap below gets caught. We won't remove it.

Watch out: the engine adds BOS to text prompts

Harvest tokenizes each prompt with add_special_tokens=True, so it prepends the model's BOS token (Llama: <|begin_of_text|>). If your text already starts with one — which it does whenever you render a chat template client-side — you get two, and every position in your analysis is off by one against your own tokenization. No error is returned, but the numbers refer to the wrong tokens.

Detect it. Compare shard["tokens_i"] against your client-side render before comparing activations. A length mismatch of exactly one, with a duplicated first id, is this bug. Make this the first assertion in a parity harness to catch alignment mistakes, including client-side ones.
Fix it. Set add_special_tokens: false in the request. Harvest then tokenizes your text verbatim and adds no BOS, so a prompt that already carries one ends up with exactly one. /v1/harvest takes text only (prompts is a list of strings); unlike /v1/completions, it cannot accept pre-tokenized prompts. (Default stays true: plain text prompts still get their BOS.)

manifest_url points to a JSON index of every shard (which prompts are in which file, shapes, the layer convention spelled out). stats_url is a small safetensors file with per-layer token mean and std over the whole run — the normalization statistics SAE training wants, with each prompt's BOS token left out of the statistics because its outlier norms would skew them.

Get projections instead of activations

If you already know which directions you care about — probe vectors, SAE features, a trait axis — send them with the job and the shards come back holding per-token projections onto those directions instead of the raw residual stream:

{
  "model": "llama-8b",
  "prompts": ["…"],
  "layers": [14],
  "project_onto": {                 // shape (n_directions, hidden)
    "data": "<base64>",             // raw little-endian buffer, no compression
    "dtype": "float32",             // or float16 / bfloat16
    "shape": [8, 4096],
    "compression": "none"
  }
}

prompt_i then has shape [n_layers_kept, n_tokens, n_directions] in float32, and tokens_i is unchanged. For 8 directions on llama-8b that's 512× less data than the raw stream.

Directions are L2-normalized server-side, so values are components along each unit direction. A non-unit vector would rescale the values without indicating it in the output. The manifest records projection.normalized: true.
The manifest's contents field reads "projections" (raw runs say "activations"), so a shard can't be mistaken for the other kind. Its dtype reads "float32" accordingly — projections are always float32, whatever dtype you sent the directions in.
stats.safetensors and mean_token_norm describe the projections, not the residual stream: mean/std are [n_layers_kept, n_directions] and mean_token_norm is the mean L2 norm of the projection vectors. The manifest's statistics string says which one you have.
Directions must have the model's hidden size — a mismatch is rejected before the model loads, within seconds.
Up to 64 directions per job. The buffer must be exactly n_directions × hidden × itemsize bytes — a truncated or mis-shaped payload is rejected at submit time with the byte counts in the message.
Projection happens on the GPU, before the activations are copied to host memory, reducing the cost of long-context and layers: "all" jobs as well as download size.

Omit project_onto for the default raw-activation harvest.

Limits & practicalities

A submit may answer "status": "pending". If Modal is slow to accept the spawn, the 202 arrives before it does; the job exists and is pollable, and GET /v1/harvest/<job_id> reports running (or failed) once the spawn resolves. Poll exactly as you would for running.
Counting is rate-limited too. Before a submit is admitted the corpus is tokenized on the server, one submit per account at a time (two cluster-wide). If another of your submits is still being counted, or a very large corpus takes longer than 10 s to count, you get 429 harvest_count_busy with a Retry-After header — retry after that many seconds, or split the corpus into smaller jobs.
Harvests draw down your token budget. The tokens in your corpus are reserved against the same monthly / daily / account budgets a completion uses when you submit (the 202 body reports them as reserved_tokens), and charged at what actually ran once the job finishes. An exhausted budget gets 429 budget_exceeded on a harvest just like on a completion. A job that never starts (spawn failed) or fails on Modal releases its hold; a job you cancel after it started is charged the reserved amount — the GPU time was spent.
Concurrent jobs per account, in two lanes. Small single-GPU models (llama-8b) allow 3 jobs at once; the large multi-GPU models (llama-405b, trinity-truebase) allow 1, because each one occupies a whole 8×H200 container. The lanes are counted separately, so a running 405B job never blocks an 8B job; interactive sessions can run alongside corpus jobs. The lanes are counted across all your keys, so a second key does not add a lane. Over the limit returns 429 harvest_concurrency_exceeded, and the message names the lane you filled. That 429 carries a Retry-After: 30 header, so wait ~30 s between retries or DELETE the in-flight job (see above) to free the slot now. Keys also carry a monthly job quota (200 jobs by default for keys created from now on; 429 harvest_quota_exceeded when it is used up).
Corpus caps. Up to 4,096 prompts and ~2M estimated tokens per job, and request bodies up to 50 MB (413 beyond that). For more, split into several jobs.
Prompts longer than the model's context window are truncated to fit, and the token axis you get back matches what actually ran.
Big models are slow to start. llama-405b streams ~810 GB of weights on a cold start — the first job can spend ~50 minutes loading before any prompt runs. The load is paid once per job, so batch your corpus into one large job instead of many small ones.
Waiting for a job: add ?wait=. GET /v1/harvest/<job_id>?wait=30 holds the request open for up to 30 seconds (60 max) and returns the moment the job reaches a terminal state, with the same payload as an immediate poll. Omit ?wait= for immediate polling. Jobs routinely outlive the window, so keep polling once per minute.
- It can also return early without a terminal state. To protect /v1/completions latency, only so many long-polls may wait at once (4 per key, 8 overall). Over that ceiling your ?wait= is answered immediately with the current (possibly still-running) state, an X-Acs-Longpoll: declined header and a Retry-After — a normal 200, not an error. Just poll again after Retry-After; keeping a plain one-request-per-minute loop handles this transparently.
503 harvest_unavailable means the harvest app for that model isn't reachable right now; the response records a job id you can show us.

Inline or bulk?

Inline (output_residual_stream) answers in the same HTTP response and is right for probing and steering workflows up to ~10k examples. Bulk trades latency for throughput: the capture itself runs ~13× faster than the inline path's transport ceiling, keeps only the layers you ask for, and hands you files you can re-download for a week.