# Evals and RL harnesses: fork a warm environment for every rollout

> Set up an eval or RL environment once in a sandbox, with its server running and its models in memory, and fork a live copy of it for every rollout with the Python SDK: the script, its measured time and cost against a cold start, and the limits.

URL: https://prized.dev/docs/sandboxes-evals-rl-harnesses
Updated: 2026-11-10

An eval or RL harness runs the same environment many times. Set it up once in a sandbox: packages, data, and a server with its models loaded. Then fork it for each rollout: every copy is a microVM of its own that starts running with that disk, memory and server, so no rollout repeats the setup ([Fork](https://prized.dev/docs/sandboxes#fork)).

In our runs, each later batch of 8 rollouts forked this way had all 8 rewards in 6.9 s, against 34.2 s for 8 new sandboxes set up from scratch, at about a sixth of the cost ([What it measured](#what-it-measured)).

## 1. Install the SDK and export a key [#1-install-the-sdk-and-export-a-key]

```bash
python3 -m venv .venv && . .venv/bin/activate
pip install prized
```

It needs Python 3.10 or later ([Python SDK](https://prized.dev/docs/sandboxes-python-typescript)). Make a key at [https://prized.dev/dashboard/workspace/api-keys](https://prized.dev/dashboard/workspace/api-keys) with **Full access**, or **Custom** with **Sandboxes: Write**, or choose **Create API key** on [Sandboxes](https://prized.dev/onboarding/sandboxes) (new to Prized, it signs you up with the [free credit](https://prized.dev/docs/billing#free-credit), no card):

```bash
export PRIZED_API_KEY=dcp_…
```

## 2. Save the environment [#2-save-the-environment]

A stand-in for yours: it loads GPT-2 as the policy and a sentiment classifier as the reward model, and plays one episode per request, in which the policy writes the start of a movie review and the reward model scores how positive it is. In your harness the policy is usually a call to your inference server or model API, which a copy can reach over the internet.

```python title="env_server.py"
import json, random
from http.server import BaseHTTPRequestHandler, HTTPServer
from urllib.parse import parse_qs, urlparse

import torch
from transformers import pipeline

# The slow part, done once in the base: every copy starts with both models loaded
policy = pipeline("text-generation", model="openai-community/gpt2")
reward_model = pipeline("sentiment-analysis", model="distilbert/distilbert-base-uncased-finetuned-sst-2-english")
PROMPTS = ["This movie was", "I went to see this film and", "The acting was", "Honestly, the plot", "My friends said this show"]

def episode(seed):  # the policy writes a review, the reward model scores how positive it is
    torch.manual_seed(seed)  # copies share the base's random state, so seed every rollout
    prompt = random.Random(seed).choice(PROMPTS)
    text = policy(prompt, max_new_tokens=24, do_sample=True, pad_token_id=50256)[0]["generated_text"]
    label = reward_model(text)[0]
    reward = label["score"] if label["label"] == "POSITIVE" else 1 - label["score"]
    return {"seed": seed, "text": text, "reward": round(reward, 3)}

class Env(BaseHTTPRequestHandler):
    def do_GET(self):  # GET /rollout?seed=N plays one episode
        seed = int(parse_qs(urlparse(self.path).query).get("seed", ["0"])[0])
        body = json.dumps(episode(seed)).encode()
        self.send_response(200)
        self.end_headers()
        self.wfile.write(body)

    def log_message(self, *args):
        pass

episode(0)  # warm up in the base, so every copy inherits a server that has already played once
HTTPServer(("127.0.0.1", 8000), Env).serve_forever()
```

## 3. Save the rollout script [#3-save-the-rollout-script]

It sets up a base sandbox once, then for each batch forks it into 8 copies, plays one episode in each, collects the rewards and deletes the copies. One label on the base and its copies deletes them all at the end, even when something fails.

```python title="rollouts.py"
import json, time, uuid
from concurrent.futures import ThreadPoolExecutor

from prized import Sandbox

COPIES, BATCHES = 8, 3  # 8 rollouts at a time, 3 times
RUN = {"run": "rl-" + uuid.uuid4().hex[:8]}  # a label on the base and every copy, for cleanup
SETUP = (
    "python3 -m venv ~/.venv"
    " && ~/.venv/bin/pip install -q torch==2.15.0 --index-url https://download.pytorch.org/whl/cpu"
    " && ~/.venv/bin/pip install -q transformers==5.22.1"
)

def ok(result):  # a command that fails comes back as a result, not an exception
    if result.exit_code != 0:
        raise RuntimeError(f"exit {result.exit_code}: {result.stderr[-2000:]}")
    return result

def rollout(copy, seed):  # one episode in one copy, then delete the copy
    out = ok(copy.exec(f"curl -sS --fail 'localhost:8000/rollout?seed={seed}'", timeout=120))
    copy.delete(wait=True)  # until it is gone: a copy being deleted still counts toward your quotas
    return json.loads(out.stdout)

start = time.monotonic()
base = Sandbox.create(vcpu=2, memory_gb=4, max_runtime_min=60, labels=RUN)  # the free tier's most; raise it on a paid workspace
print("label:", RUN)
try:
    # 1. Set up once: packages, then the server with both models in memory
    ok(base.exec(SETUP, timeout=900))
    base.files.upload("env_server.py", "env_server.py")
    base.exec("~/.venv/bin/python env_server.py > server.log 2>&1", background=True)
    ok(base.exec("curl -sS --fail --retry 120 --retry-connrefused --retry-delay 1 -o /dev/null localhost:8000/rollout", timeout=180))
    print(f"setup: {time.monotonic() - start:.1f} s")

    # 2. Fork one copy per rollout, run them all at once, collect the results
    for batch in range(BATCHES):
        start = time.monotonic()
        copies = base.fork(COPIES)  # returns once every copy is running
        forked = time.monotonic() - start
        seeds = range(batch * COPIES + 1, (batch + 1) * COPIES + 1)
        with ThreadPoolExecutor(COPIES) as pool:
            results = list(pool.map(rollout, copies, seeds))
        mean = sum(r["reward"] for r in results) / len(results)
        print(f"batch {batch + 1}: {COPIES} copies in {forked:.1f} s, done in {time.monotonic() - start:.1f} s, mean reward {mean:.2f}")
finally:
    Sandbox.delete_by_labels(RUN)  # the base and any copy left
```

## 4. Run it [#4-run-it]

```bash
python3 rollouts.py
# label: {'run': 'rl-32f7ebf4'}
# setup: 30.9 s
# batch 1: 8 copies in 3.1 s, done in 10.0 s, mean reward 0.73
# batch 2: 8 copies in 4.6 s, done in 10.5 s, mean reward 0.80
# batch 3: 8 copies in 3.6 s, done in 10.4 s, mean reward 0.65
```

Each copy answered from the server the base started, with both models already in memory; nothing installed or loaded them again. The rewards came out the same on every one of our runs, since every episode is seeded. A batch is done once its copies are deleted, which the script waits for.

Every sandbox here is 2 vCPU and 4 GB, which bills $0.131 an hour while it runs, and the free tier runs 10 of them at once, so the base and 8 copies fit it ([Free tier](https://prized.dev/docs/sandboxes#free-tier)). If the script dies before its cleanup, `Sandbox.delete_by_labels({"run": "rl-…"})` with the label it printed deletes what it made.

## What it measured [#what-it-measured]

### This page's script [#this-pages-script]

Three runs of each way on November 8, 2026, from a client close to the API (a command that does nothing took 11 ms; from farther away, every call adds your round trip): 13 batches of 8 rollouts a run for each kind of fork, and 3 batches of 8 new sandboxes a run. A batch runs from the fork, or the creates, to the 8th reward; each cell is the median, with the range in parentheses. Costs are what the meter billed, scaled to 100 rollouts, the base's setup and running time included.

| Each rollout runs in | First batch of 8 | Each later batch | 100 rollouts cost |
| --- | --- | --- | --- |
| A fork of the running base, as in the script | 8.8 s (8.2-11.6) | 6.9 s (6.0-9.1) | $0.020 ($0.020-0.022) |
| A fork of the base, paused | 20.4 s (19.9-20.8) | 8.1 s (7.5-8.9) | $0.027 ($0.027-0.028) |
| A fork of a paused base with the packages and model files only, starting the server | 26.9 s (26.1-27.7) | 13.9 s (13.0-161.6) | $0.074 ($0.047-0.088) |
| A new sandbox, set up from scratch | 35.7 s (34.0-170.0) | 34.2 s (33.8-35.0) | $0.118 ($0.116-0.282) |

More from the same runs, as medians:

* **Setup runs once:** 30.5 s in the base, before the first fork. A new sandbox paid it again for every rollout, 32.6 s from the create to its reward, so the more your setup does, the more each fork saves.
* **A copy's first episode is slower than the base's:** 3.3 s in a fork of the running base and 6.5 s in a fork of the paused one, against 0.53 s in the base. Its next episodes took 0.68 s (a separate run of 3 episodes in each of 24 copies of the running base).
* **The first fork of a paused base is slow:** 13.9 s, against 1.6 s for each later one. `base.pause()` itself took 0.16 s.
* **A fork of the running base took 3.6 s.** The base pauses while it is copied, longer the more memory it has used ([Fork](https://prized.dev/docs/sandboxes#fork)).
* **The script prints more than a batch here.** Its `done` also waits until the copies are gone, 10.6 s in all. In 9 runs of the script itself the first batch was no slower than the rest; the table's running-base runs played 5 episodes in the base before its first fork.
* **Setting up or starting in each sandbox can stall.** One of the 9 batches of new sandboxes and two of the 39 that started the server in each copy took 112 to 170 s, all 8 rollouts at once; none of the 78 batches whose copies inherited the running server did. The stalls also raise the packages-only row's cost: $0.047 in the run without one.

### A heavier environment, with our OpenEnv harness [#a-heavier-environment-with-our-openenv-harness]

These numbers were measured with our OpenEnv harness, not this page's script, on November 9, 2026: OpenEnv's `openapp_env` (web apps driven through a Chromium browser) in sandboxes of 2 vCPU and 4 GB, with one base on one machine and the client in the same region. The harness does more for each environment than fork or create it: a round runs from the first call until all 8 environments have answered their health check and finished a first reset, with no model in the loop.

| How the harness started the 8 environments | All 8 ready |
| --- | --- |
| One fork of the paused base into 8 copies | 6.1 s (5.8-6.6), 8 rounds |
| 8 forks of one copy each, sent at once, of the same base | 14.0 s (13.9-14.8), 8 rounds |
| 8 new sandboxes, each installing and starting the environment | 96.6 s, 1 round |

* **One base, already forked.** Every fork round used the same base, which the harness built in 65 s (install, start, replace it with a fork of itself, one reset, pause) and forked, after that pause, in the round that built it. So no round here paid the slow first fork of a paused base.
* **Seconds to an hour after the pause.** The one-fork rounds ran 7 to 61 minutes after it (6.1 and 6.4 s at 61 minutes), and the 8-fork rounds 16 seconds to 61 minutes after it (14.0 and 14.3 s at 61 minutes).
* **Fork a batch in one call:** one fork into 8 had all 8 ready in less than half the time of 8 single forks sent at once.

## Keep the base between runs [#keep-the-base-between-runs]

To fork the same base again later, pause it at the end instead of deleting it. In `rollouts.py`, replace the `finally:` block with this one, which deletes only the copies a failed rollout left:

```python
finally:
    for sb in Sandbox.list(labels=RUN):
        if sb.id != base.id:
            sb.delete()
    base.pause()  # disk, memory and the running server are kept, and it bills storage only
    print("base:", base.id)
```

Later, from any script with the same key, fork it by the id it printed:

```python
from prized import Sandbox

copies = Sandbox.connect("sbx_…").fork(8, labels={"run": "rl-next"})  # the base stays paused
```

Labels on a fork replace the base's, so `Sandbox.delete_by_labels({"run": "rl-next"})` deletes these copies and leaves the base. The first fork after the pause is the slow one, about 15 s in our runs even though the base had been forked while it ran.

A paused sandbox of this shape holds 14 GB of the workspace's paused storage, and the first 20 GB are included ([Billing: Sandboxes](https://prized.dev/docs/billing#sandboxes)). It is kept until you delete it (on the free tier, until 30 days after its last use) and lives on one machine: it is lost if that machine fails, and a move to a different kind of machine keeps its disk but not its memory, so keep its setup in a script ([Paused retention](https://prized.dev/docs/sandboxes#paused-retention), [Pause and resume](https://prized.dev/docs/sandboxes#pause-and-resume)).

## What a copy shares with the base [#what-a-copy-shares-with-the-base]

* **Random state.** A process that was already running, like this server, keeps the base's random state: in our tests the copies of one fork all drew the same next value from Python's `random` and from numpy (196 of 196 copies, in 34 forks of 4 or 8), while `os.urandom` differed in every one. Seed each rollout, as `episode()` does, or reseed from `os.urandom` after the fork ([Do copies share randomness or identity?](https://prized.dev/docs/sandboxes-faq#do-copies-share-randomness-or-identity)).
* **Connections.** A copy's connections to other machines are closed and loopback ones keep working: reopen a connection to your trainer or inference server after the fork.
* **Run time.** On the free tier copies keep the base's deletion time; on a paid workspace each copy gets its own run time from the fork ([Fork](https://prized.dev/docs/sandboxes#fork)).

## Limits [#limits]

* **Copies start on the base's machine,** so one fork makes only as many as fit there. A fork that does not fit makes nothing and answers `409 fork_capacity` with `fits` (`err.body["fits"]` in the SDK): fork that many, or set up a second base and fork both ([Fork](https://prized.dev/docs/sandboxes#fork)).
* **Every copy counts toward your [quotas](https://prized.dev/docs/limits#sandboxes)**, and a paused base still counts toward disk, and toward vCPU and memory while it keeps them for its resume. A top-up or a plan lifts the free tier to the workspace quotas; email the support address shown on https://prized.dev/docs/troubleshooting for more.
* **A command with a timeout of 240 seconds or less, like each rollout's, holds one of your key's 48 concurrent requests until it ends.** The SDK runs one with a longer timeout detached and polls it instead ([Edge limits](https://prized.dev/docs/sandboxes-api#edge-limits)).
* **Oregon, and CPUs only.** Sandboxes run in `us-west-2` (Oregon) and have no GPUs, so keep the policy on your inference server. Every copy has outbound internet; egress allow lists are not there yet ([Not yet](https://prized.dev/docs/sandboxes#not-yet)).

## In your own harness [#in-your-own-harness]

The same calls fit any harness's environment hooks:

| Harness step | SDK call |
| --- | --- |
| Build the environment, once | `Sandbox.create()`, then `exec()` and `files.upload()` |
| Start a batch of trials | `base.fork(n)`: one running copy per trial, in one call |
| Act and score in a trial | `copy.exec()`, `copy.files.read()` |
| End a trial | `copy.delete(wait=True)` |
| End the run | `Sandbox.delete_by_labels(RUN)` |