Evals and RL harnesses: fork a warm environment for every rollout

An eval or RL harness runs the same environment many times. Set it up once in a sandbox: packages, data, and a server with its models loaded. Then fork it for each rollout: every copy is a microVM of its own that starts running with that disk, memory and server, so no rollout repeats the setup (Fork).

In our runs, each later batch of 8 rollouts forked this way had all 8 rewards in 6.9 s, against 34.2 s for 8 new sandboxes set up from scratch, at about a sixth of the cost (What it measured).

11. Install the SDK and export a key

python3 -m venv .venv && . .venv/bin/activate
pip install prized

It needs Python 3.10 or later (Python SDK). Make a key at https://prized.dev/dashboard/workspace/api-keys with Full access, or Custom with Sandboxes: Write, or choose Create API key on Sandboxes (new to Prized, it signs you up with the free credit, no card):

export PRIZED_API_KEY=dcp_…

22. Save the environment

A stand-in for yours: it loads GPT-2 as the policy and a sentiment classifier as the reward model, and plays one episode per request, in which the policy writes the start of a movie review and the reward model scores how positive it is. In your harness the policy is usually a call to your inference server or model API, which a copy can reach over the internet.

import json, random
from http.server import BaseHTTPRequestHandler, HTTPServer
from urllib.parse import parse_qs, urlparse

import torch
from transformers import pipeline

# The slow part, done once in the base: every copy starts with both models loaded
policy = pipeline("text-generation", model="openai-community/gpt2")
reward_model = pipeline("sentiment-analysis", model="distilbert/distilbert-base-uncased-finetuned-sst-2-english")
PROMPTS = ["This movie was", "I went to see this film and", "The acting was", "Honestly, the plot", "My friends said this show"]


def episode(seed):  # the policy writes a review, the reward model scores how positive it is
    torch.manual_seed(seed)  # copies share the base's random state, so seed every rollout
    prompt = random.Random(seed).choice(PROMPTS)
    text = policy(prompt, max_new_tokens=24, do_sample=True, pad_token_id=50256)[0]["generated_text"]
    label = reward_model(text)[0]
    reward = label["score"] if label["label"] == "POSITIVE" else 1 - label["score"]
    return {"seed": seed, "text": text, "reward": round(reward, 3)}


class Env(BaseHTTPRequestHandler):
    def do_GET(self):  # GET /rollout?seed=N plays one episode
        seed = int(parse_qs(urlparse(self.path).query).get("seed", ["0"])[0])
        body = json.dumps(episode(seed)).encode()
        self.send_response(200)
        self.end_headers()
        self.wfile.write(body)

    def log_message(self, *args):
        pass


episode(0)  # warm up in the base, so every copy inherits a server that has already played once
HTTPServer(("127.0.0.1", 8000), Env).serve_forever()

33. Save the rollout script

It sets up a base sandbox once, then for each batch forks it into 8 copies, plays one episode in each, collects the rewards and deletes the copies. One label on the base and its copies deletes them all at the end, even when something fails.

import json, time, uuid
from concurrent.futures import ThreadPoolExecutor

from prized import Sandbox

COPIES, BATCHES = 8, 3  # 8 rollouts at a time, 3 times
RUN = {"run": "rl-" + uuid.uuid4().hex[:8]}  # a label on the base and every copy, for cleanup
SETUP = (
    "python3 -m venv ~/.venv"
    " && ~/.venv/bin/pip install -q torch==2.15.0 --index-url https://download.pytorch.org/whl/cpu"
    " && ~/.venv/bin/pip install -q transformers==5.22.1"
)


def ok(result):  # a command that fails comes back as a result, not an exception
    if result.exit_code != 0:
        raise RuntimeError(f"exit {result.exit_code}: {result.stderr[-2000:]}")
    return result


def rollout(copy, seed):  # one episode in one copy, then delete the copy
    out = ok(copy.exec(f"curl -sS --fail 'localhost:8000/rollout?seed={seed}'", timeout=120))
    copy.delete(wait=True)  # until it is gone: a copy being deleted still counts toward your quotas
    return json.loads(out.stdout)


start = time.monotonic()
base = Sandbox.create(vcpu=2, memory_gb=4, max_runtime_min=60, labels=RUN)  # the free tier's most; raise it on a paid workspace
print("label:", RUN)
try:
    # 1. Set up once: packages, then the server with both models in memory
    ok(base.exec(SETUP, timeout=900))
    base.files.upload("env_server.py", "env_server.py")
    base.exec("~/.venv/bin/python env_server.py > server.log 2>&1", background=True)
    ok(base.exec("curl -sS --fail --retry 120 --retry-connrefused --retry-delay 1 -o /dev/null localhost:8000/rollout", timeout=180))
    print(f"setup: {time.monotonic() - start:.1f} s")

    # 2. Fork one copy per rollout, run them all at once, collect the results
    for batch in range(BATCHES):
        start = time.monotonic()
        copies = base.fork(COPIES)  # returns once every copy is running
        forked = time.monotonic() - start
        seeds = range(batch * COPIES + 1, (batch + 1) * COPIES + 1)
        with ThreadPoolExecutor(COPIES) as pool:
            results = list(pool.map(rollout, copies, seeds))
        mean = sum(r["reward"] for r in results) / len(results)
        print(f"batch {batch + 1}: {COPIES} copies in {forked:.1f} s, done in {time.monotonic() - start:.1f} s, mean reward {mean:.2f}")
finally:
    Sandbox.delete_by_labels(RUN)  # the base and any copy left

44. Run it

python3 rollouts.py
# label: {'run': 'rl-32f7ebf4'}
# setup: 30.9 s
# batch 1: 8 copies in 3.1 s, done in 10.0 s, mean reward 0.73
# batch 2: 8 copies in 4.6 s, done in 10.5 s, mean reward 0.80
# batch 3: 8 copies in 3.6 s, done in 10.4 s, mean reward 0.65

Each copy answered from the server the base started, with both models already in memory; nothing installed or loaded them again. The rewards came out the same on every one of our runs, since every episode is seeded. A batch is done once its copies are deleted, which the script waits for.

Every sandbox here is 2 vCPU and 4 GB, which bills $0.131 an hour while it runs, and the free tier runs 10 of them at once, so the base and 8 copies fit it (Free tier). If the script dies before its cleanup, Sandbox.delete_by_labels({"run": "rl-…"}) with the label it printed deletes what it made.

What it measured

This page's script

Three runs of each way on November 8, 2026, from a client close to the API (a command that does nothing took 11 ms; from farther away, every call adds your round trip): 13 batches of 8 rollouts a run for each kind of fork, and 3 batches of 8 new sandboxes a run. A batch runs from the fork, or the creates, to the 8th reward; each cell is the median, with the range in parentheses. Costs are what the meter billed, scaled to 100 rollouts, the base's setup and running time included.

More from the same runs, as medians:

• Setup runs once: 30.5 s in the base, before the first fork. A new sandbox paid it again for every rollout, 32.6 s from the create to its reward, so the more your setup does, the more each fork saves.
• A copy's first episode is slower than the base's: 3.3 s in a fork of the running base and 6.5 s in a fork of the paused one, against 0.53 s in the base. Its next episodes took 0.68 s (a separate run of 3 episodes in each of 24 copies of the running base).
• The first fork of a paused base is slow: 13.9 s, against 1.6 s for each later one. base.pause() itself took 0.16 s.
• A fork of the running base took 3.6 s. The base pauses while it is copied, longer the more memory it has used (Fork).
• The script prints more than a batch here. Its done also waits until the copies are gone, 10.6 s in all. In 9 runs of the script itself the first batch was no slower than the rest; the table's running-base runs played 5 episodes in the base before its first fork.
• Setting up or starting in each sandbox can stall. One of the 9 batches of new sandboxes and two of the 39 that started the server in each copy took 112 to 170 s, all 8 rollouts at once; none of the 78 batches whose copies inherited the running server did. The stalls also raise the packages-only row's cost: $0.047 in the run without one.

A heavier environment, with our OpenEnv harness

These numbers were measured with our OpenEnv harness, not this page's script, on November 9, 2026: OpenEnv's openapp_env (web apps driven through a Chromium browser) in sandboxes of 2 vCPU and 4 GB, with one base on one machine and the client in the same region. The harness does more for each environment than fork or create it: a round runs from the first call until all 8 environments have answered their health check and finished a first reset, with no model in the loop.

• One base, already forked. Every fork round used the same base, which the harness built in 65 s (install, start, replace it with a fork of itself, one reset, pause) and forked, after that pause, in the round that built it. So no round here paid the slow first fork of a paused base.
• Seconds to an hour after the pause. The one-fork rounds ran 7 to 61 minutes after it (6.1 and 6.4 s at 61 minutes), and the 8-fork rounds 16 seconds to 61 minutes after it (14.0 and 14.3 s at 61 minutes).
• Fork a batch in one call: one fork into 8 had all 8 ready in less than half the time of 8 single forks sent at once.

Keep the base between runs

To fork the same base again later, pause it at the end instead of deleting it. In rollouts.py, replace the finally: block with this one, which deletes only the copies a failed rollout left:

finally:
    for sb in Sandbox.list(labels=RUN):
        if sb.id != base.id:
            sb.delete()
    base.pause()  # disk, memory and the running server are kept, and it bills storage only
    print("base:", base.id)

Later, from any script with the same key, fork it by the id it printed:

from prized import Sandbox

copies = Sandbox.connect("sbx_…").fork(8, labels={"run": "rl-next"})  # the base stays paused

Labels on a fork replace the base's, so Sandbox.delete_by_labels({"run": "rl-next"}) deletes these copies and leaves the base. The first fork after the pause is the slow one, about 15 s in our runs even though the base had been forked while it ran.

A paused sandbox of this shape holds 14 GB of the workspace's paused storage, and the first 20 GB are included (Billing: Sandboxes). It is kept until you delete it (on the free tier, until 30 days after its last use) and lives on one machine: it is lost if that machine fails, and a move to a different kind of machine keeps its disk but not its memory, so keep its setup in a script (Paused retention, Pause and resume).

What a copy shares with the base

• Random state. A process that was already running, like this server, keeps the base's random state: in our tests the copies of one fork all drew the same next value from Python's random and from numpy (196 of 196 copies, in 34 forks of 4 or 8), while os.urandom differed in every one. Seed each rollout, as episode() does, or reseed from os.urandom after the fork (Do copies share randomness or identity?).
• Connections. A copy's connections to other machines are closed and loopback ones keep working: reopen a connection to your trainer or inference server after the fork.
• Run time. On the free tier copies keep the base's deletion time; on a paid workspace each copy gets its own run time from the fork (Fork).

Limits

• Copies start on the base's machine, so one fork makes only as many as fit there. A fork that does not fit makes nothing and answers 409 fork_capacity with fits (err.body["fits"] in the SDK): fork that many, or set up a second base and fork both (Fork).
• Every copy counts toward your quotas, and a paused base still counts toward disk, and toward vCPU and memory while it keeps them for its resume. A top-up or a plan lifts the free tier to the workspace quotas; email team@prized.dev for more.
• A command with a timeout of 240 seconds or less, like each rollout's, holds one of your key's 48 concurrent requests until it ends. The SDK runs one with a longer timeout detached and polls it instead (Edge limits).
• Oregon, and CPUs only. Sandboxes run in us-west-2 (Oregon) and have no GPUs, so keep the policy on your inference server. Every copy has outbound internet; egress allow lists are not there yet (Not yet).

In your own harness

The same calls fit any harness's environment hooks: