# Run an orchestrator agent with a sandbox per task

> Keep the orchestrator agent on a box and give each builder or task its own sandbox, deleted when the task ends: a Python fan-out script, cleanup, costs, and limits.

URL: https://prized.dev/docs/sandboxes-orchestrator

One agent plans and many build. Keep the orchestrator on a box, and give each builder a sandbox of its own:

* **The box** holds the orchestrator agent, your repo, and its tools for days, and keeps working with your laptop closed ([Boxes](https://prized.dev/docs/boxes)).
* **A sandbox per task** is a clean microVM that takes commands seconds after the create, runs one task, and is deleted when it ends. Tasks run side by side and share nothing ([Sandboxes](https://prized.dev/docs/sandboxes)).

The steps run a script that fans three test runs out to three sandboxes, collects each result, and deletes every sandbox, even when a task fails.

## 1. Put a sandbox key on the box [#1-put-a-sandbox-key-on-the-box]

Make a key at [https://prized.dev/dashboard/workspace/api-keys](https://prized.dev/dashboard/workspace/api-keys) with **Custom** access, **Sandboxes** at **Write** and the rest at **None**, so the orchestrator can create and delete sandboxes but cannot touch your boxes. On the box (`prized ssh`), export it in the shell the orchestrator starts from:

```bash
export PRIZED_API_KEY=dcp_…
```

## 2. Save the script [#2-save-the-script]

Boxes have Python 3 with venv ([Machines](https://prized.dev/docs/machines#preinstalled-software)); the script needs `requests` too:

```bash
python3 -m venv ~/.fanout && ~/.fanout/bin/pip install -q requests
```

```python title="fanout.py"
import contextlib, os, signal, sys, time, uuid
from concurrent.futures import ThreadPoolExecutor

import requests

API = "https://api.prized.dev/api/v1"
VCPU = int(os.environ.get("VCPU", "1"))  # each task's sandbox, with 2 GB of memory per vCPU
PARALLEL = int(os.environ.get("PARALLEL", "8"))  # sandboxes at once
TIMEOUT = 600  # seconds per task, under the sandboxes' 30 minute run time below
RUN = uuid.uuid4().hex[:8]  # labels this run's sandboxes, for cleanup
TASKS = {
    "itsdangerous": "git clone -q --depth 1 https://github.com/pallets/itsdangerous repo && cd repo && python3 -m venv .venv && .venv/bin/pip install -q . pytest freezegun && .venv/bin/pytest -q",
    "markupsafe": "git clone -q --depth 1 https://github.com/pallets/markupsafe repo && cd repo && python3 -m venv .venv && .venv/bin/pip install -q . pytest && .venv/bin/pytest -q",
    "more-itertools": "git clone -q --depth 1 https://github.com/more-itertools/more-itertools repo && cd repo && python3 -m unittest discover -s tests",
}

http = requests.Session()
http.headers["Authorization"] = f"Bearer {os.environ['PRIZED_API_KEY']}"
http.mount("https://", requests.adapters.HTTPAdapter(pool_maxsize=PARALLEL))

def call(method, url, ok=(200, 201, 202), **kw):
    r = http.request(method, url, timeout=kw.pop("timeout", 60), **kw)
    if r.status_code not in ok:
        raise RuntimeError(f"{method} {url.split('?')[0]}: {r.status_code} {r.text[:300]}")
    return r.json()

def create(name):
    for _ in range(20):  # no room yet: 503 sandbox_capacity, or 429 while other sandboxes hold the quota
        r = http.post(f"{API}/sandboxes?wait=running", headers={"Idempotency-Key": f"{RUN}-{name}"}, timeout=60, json={
            "template": "prized-base-v1",
            "resources": {"vcpu": VCPU, "memoryMb": 2048 * VCPU},
            "labels": {"fanout": RUN, "task": name},
            "maxRuntimeMin": 30,  # deleted after 30 minutes of running, even if this script dies
        })
        if r.status_code not in (429, 503):
            break
        time.sleep(float(r.headers.get("Retry-After") or 5))
    if r.status_code not in (200, 201):
        raise RuntimeError(f"create: {r.status_code} {r.text[:300]}")
    return r.json()["sandbox"]

def ready(sbx):
    for _ in range(120):  # a create can outlast the API's 8 second wait
        if sbx["state"] != "starting":
            break
        time.sleep(1)
        sbx = call("GET", f"{API}/sandboxes/{sbx['id']}")["sandbox"]
    if sbx["state"] != "running":
        raise RuntimeError(f"{sbx['id']} is {sbx['state']}: {sbx['errorReason']}")
    for _ in range(10):  # for a moment after running, the edge can answer 409, 502 or 503
        r = http.post(f"{sbx['toolboxBase']}/process/execute", json={"command": "true"}, timeout=30)
        if r.ok:
            return sbx
        time.sleep(1)
    raise RuntimeError(f"{sbx['id']} is not taking commands: {r.status_code}")

def delete(sbx_id):
    with contextlib.suppress(requests.RequestException):
        http.delete(f"{API}/sandboxes/{sbx_id}", timeout=60)

def run(name):
    t0, sbx = time.monotonic(), None
    try:
        sbx = create(name)
        sbx = ready(sbx)
        t1 = time.monotonic()
        out = call("POST", f"{sbx['toolboxBase']}/process/execute", json={"command": TASKS[name], "timeout": TIMEOUT}, timeout=TIMEOUT + 60)
        last = (out["stdout"].strip() or out["stderr"].strip()).splitlines()[-1:] or [""]
        return out["exitCode"] == 0, f"{name}: exit {out['exitCode']}, ready {t1 - t0:.1f}s, task {time.monotonic() - t1:.1f}s, {last[0]}"
    except Exception as e:
        return False, f"{name}: failed, {e}"
    finally:  # toss it as soon as its task is done
        if sbx:
            delete(sbx["id"])

def cleanup():  # anything else labeled with this run, except a sandbox you paused to keep
    for sbx in call("GET", f"{API}/sandboxes")["sandboxes"]:
        if (sbx.get("labels") or {}).get("fanout") == RUN and sbx["state"] != "paused":
            delete(sbx["id"])

def stop(signum, _):  # Ctrl-C or kill: clean up, then exit without waiting on the tasks' commands
    try:
        cleanup()
    finally:
        os._exit(128 + signum)

signal.signal(signal.SIGINT, stop)
signal.signal(signal.SIGTERM, stop)
t = time.monotonic()
with ThreadPoolExecutor(min(PARALLEL, len(TASKS))) as pool:
    results = list(pool.map(run, TASKS))
print(*(line for _, line in results), f"all {len(TASKS)} in {time.monotonic() - t:.1f}s", sep="\n")
cleanup()
sys.exit(0 if all(ok for ok, _ in results) else 1)
```

For each task in `TASKS`, a name and one shell command, it:

* **Creates a sandbox** from `prized-base-v1` (Python 3, Node.js 22, git), labeled with the run, with a 30 minute run time as a backstop. A create refused for room (`503`) or quota (`429`) waits its `Retry-After` and tries again; the `Idempotency-Key` keeps a retry from making a second sandbox.
* **Waits until it takes commands.** A create can still be `starting` when the API's wait ends, and for a moment after `running` a command can answer `409`, `502`, or `503`, so it retries a harmless `true` first ([The edge](https://prized.dev/docs/sandboxes-api#the-edge)).
* **Runs the task** as the user `prized`, without sudo, with a `TIMEOUT` of 600 seconds, and reads `exitCode`: a failed command still answers `200`.
* **Deletes the sandbox** as soon as the task ends.

> Python and TypeScript SDK versions of this example land on November 5. Until then it calls the [Sandbox API](https://prized.dev/docs/sandboxes-api) over plain HTTPS.

## 3. Run it [#3-run-it]

```bash
~/.fanout/bin/python fanout.py
# itsdangerous: exit 0, ready 4.2s, task 6.1s, 297 passed in 0.53s
# markupsafe: exit 0, ready 4.1s, task 6.2s, 39 passed, 41 skipped in 0.11s
# more-itertools: exit 0, ready 4.3s, task 10.8s, OK
# all 3 in 15.1s
```

`ready` runs from the create until the sandbox takes commands, about 4 seconds at 1 vCPU; a burst of creates can wait longer while capacity is added. Each line ends with the task's last line of output, and the script exits `1` if any task failed. `VCPU=2` gives each sandbox 2 vCPU and 4 GB, a shape that usually starts in under a second ([Quickstart](https://prized.dev/docs/sandboxes#quickstart)), and `PARALLEL` caps how many run at once (8 unless set).

## 4. Give it your builders' tasks [#4-give-it-your-builders-tasks]

* **Your work in `TASKS`:** a test shard, a build, a branch to check out and test.
* **A builder that is an agent** runs on `prized-agent-v1`, which has Claude Code and Codex. Nothing is signed in there, so add its key to the create in `envVars`, and make the task the agent's headless run ([Create](https://prized.dev/docs/sandboxes-api#create)).
* **Private code** goes in as an upload (`git archive`, then `POST {toolboxBase}/files/upload`), or as a clone with a token you pass in `envVars`.
* **Files a task writes** go with its sandbox: download them before the delete with `GET {toolboxBase}/files/download?path=…` ([Files](https://prized.dev/docs/sandboxes-api#files)).
* **The orchestrator agent** can run the script, or make the same calls itself: the [Prized skill](https://prized.dev/docs/sandboxes-coding-agents) teaches Claude Code and Codex the API. An agent's shell tool can stop a long command before its cleanup runs, so have it start the script in the background (`nohup ~/.fanout/bin/python fanout.py > fanout.log 2>&1 &`) and read `fanout.log`.

## Clean up [#clean-up]

The script deletes each sandbox when its task ends, then anything else labeled with its run that is not paused. Ctrl-C or `kill` runs the same cleanup and exits at once.

* **`maxRuntimeMin: 30`** deletes each sandbox after 30 minutes of running, even if the box running the script goes down, or a create still in flight when you stopped the script lands after the cleanup. Keep it above `TIMEOUT`.
* **Leftovers carry the label `fanout`.** `prized sandbox ls` shows each sandbox's labels. With no run going, this deletes every sandbox with that label, paused ones included:

```bash
API=https://api.prized.dev/api/v1
AUTH="Authorization: Bearer $PRIZED_API_KEY"
for ID in $(curl -s "$API/sandboxes" -H @<(echo "$AUTH") | jq -r '.sandboxes[] | select(.labels.fanout) | .id'); do curl -s -o /dev/null -X DELETE "$API/sandboxes/$ID" -H @<(echo "$AUTH"); done
```

## Optional: pause one, or fork a set-up one [#optional-pause-one-or-fork-a-set-up-one]

* **Keep a failed sandbox to look into.** In `run()`, pause it with `call("POST", f"{API}/sandboxes/{sbx['id']}/pause?wait=paused")` instead of deleting it; `cleanup()` leaves paused sandboxes alone. It keeps its memory, processes, and disk for the price of storage, and the next command resumes it, usually in under a second ([Pause and resume](https://prized.dev/docs/sandboxes#pause-and-resume)). Delete it when you are done: a paid workspace keeps a paused sandbox until you do.
* **The same slow setup for every task?** Set it up in one sandbox, then fork it with `POST /api/v1/sandboxes/{id}/fork` and `{"count": 8}`: each copy starts running with the setup done, and keeps the parent's labels, so the cleanup above finds it too ([Fork a running VM](https://prized.dev/docs/sandboxes-fork-parallel)).

## What it costs [#what-it-costs]

A sandbox bills per second while it runs, with no minimum: $0.04 per vCPU-hour, $0.0125 per GB-hour of memory, and $0.0001 per GB-hour of disk. A create still starting bills no compute, and one that never comes up costs nothing ([Billing: Sandboxes](https://prized.dev/docs/billing#sandboxes)).

The example's 1 vCPU, 2 GB sandboxes are $0.066 an hour each, so its three tasks, about 25 seconds of running in all, cost under a tenth of a cent, and a thousand 10 second tasks cost $0.18. The box bills like any box ([Billing](https://prized.dev/docs/billing)); a small one is enough, since the builds run in the sandboxes.

## Limits [#limits]

* **50 at once.** Once the workspace has paid in, 50 sandboxes can be up at once, with 200 vCPU, 400 GB of memory, and 500 GB of disk across them; email the support address shown on https://prized.dev/docs/troubleshooting for more ([Limits](https://prized.dev/docs/limits#sandboxes)). A sandbox still `deleting` counts until it is gone, so keep `PARALLEL` a few under. On the free credit alone, 10 at once, each up to 2 vCPU, 4 GB, and 60 minutes ([Free tier](https://prized.dev/docs/sandboxes#free-tier)).
* **48 commands at once per key.** A command holds its request open until it ends, and the edge takes 48 at once per API key and 8 per sandbox; past that it answers `429 rate_limited` ([Edge limits](https://prized.dev/docs/sandboxes-api#edge-limits)). For more builders at once, or a task past the 1 hour `timeout` maximum, start the task in the background (`nohup ./task.sh > task.log 2>&1 &`), read `task.log` with short commands, and raise `maxRuntimeMin` above it.
* **Paused sandboxes still count.** Their disk counts toward the 500 GB, and while one keeps CPU and memory reserved for its resume, they count against the running totals; at 500 paused, a create or fork answers `429 sandbox_quota_exceeded` with `reason` `paused_limit`. Delete what you will not resume.