Benchmarks
A summary of Hotcell's time-to-interactive, burst, warm-pool and real-workload benchmarks, with the caveats needed to read them.
This page summarises docs/benchmarks.md, which has the full tables, methodology and reproduction steps. Raw output for the real-workload runs is in evidence/opencode-benchmark/.
Read this first
- The TTI runs used the ComputeSDK benchmarks harness unmodified, against Hotcell v0.1.10, on 2026-07-19 and 20.
- Hotcell's numbers have no network in them. The harness ran on the same machine as the daemon. Hosted providers on the public leaderboard are measured over the internet, so the numbers aren't directly comparable.
- Hotcell isn't listed on that leaderboard. Everything here is self-reported and reproducible.
- Hardware: Mac is an M-series machine with 8 cores and 16 GB, running Docker Desktop with an 8 GB VM. GCP is 8 vCPU and 32 GB with nested virtualization.
TTI (time to interactive) is the wall-clock time from create() to the first successful runCommand("node -v").
Sequential TTI
100 sandboxes, one at a time.
| Driver | Host | Median | P95 | Success |
|---|---|---|---|---|
| Container | Mac | 210 ms | 237 ms | 100/100 |
| Firecracker (cold) | GCP | 2,559 ms | 2,568 ms | 100/100 |
| Apple VZ (cold) | Mac | 2,618 ms | 2,712 ms | 100/100 |
Cold microVM boots are real kernel boots with no pre-warming. Their P95 is close to the median (9 ms on Firecracker, 94 ms on Apple VZ), so boot time is predictable.
Staggered and burst TTI
| Mode | Driver | Host | Concurrency | Median | Success |
|---|---|---|---|---|---|
| Staggered (200 ms apart) | Container | Mac | 100 | 300 ms | 100/100 |
| Staggered | Apple VZ | Mac | 25 | 2,603 ms | 25/25 |
| Staggered | Firecracker | GCP | 100 | 14,081 ms | 100/100 |
| Burst | Container | Mac | 25 | 1,796 ms | 25/25 |
| Burst | Container | Mac | 100 | 10,115 ms | 100/100 |
| Burst | Apple VZ | Mac | 25 | 3,176 ms | 25/25 |
| Burst | Firecracker | GCP | 100 | 32,027 ms | 97/100 |
A 100-wide burst on one machine is Hotcell's weakest result. Simultaneous kernel boots queue on the host's CPUs; hosted providers spread a burst across a fleet. The three Firecracker failures were typed 422 agent never came up errors (vsock handshake timeouts during the boot storm). Apple VZ was capped at 25 because 100 microVMs exceed a 16 GB laptop.
An earlier 100-wide container burst with the default 256 MiB per-sandbox reservation completed 30/100: admission control refused 70 creates with 503 host memory budget exhausted instead of over-subscribing the host. With a smaller reservation it completed 100/100.
Warm pools
- Container pool, 25-wide burst (2026-10-28, macOS, Docker Desktop, 64 MiB shape): median create plus probe 517 ms adopted from the pool, against 1,845 ms cold. This is a direct driver measurement, not a REST or SDK TTI number.
- Apple VZ pool adopt: 24 ms median over 10 iterations. This is not an official TTI measurement. It used a different image and probe, predates later warm-pool fixes, and shouldn't be quoted as a current TTI result.
A pool only helps while a matching spare is ready. Beyond pool depth, creates take the cold path, and a large microVM pool holds that many guests' worth of RAM.
Container guest agent
A same-host microbenchmark (2026-10-28, macOS arm64, Docker Desktop) compared the old Docker-exec path with the current guest agent:
| Operation | Docker exec median | Guest agent median |
|---|---|---|
exec |
53.63 ms | 6.44 ms |
writeFile |
54.51 ms | 3.47 ms |
readFile |
46.77 ms | 2.92 ms |
These figures are specific to that host and runtime.
Real workload: sandbox-dax
Inside one Firecracker sandbox: download Bun, clone the OpenCode repo, bun install, bun typecheck. Run with Hotcell's own harness on a bare-metal Ryzen 9 5950X, 5 cold runs at 32 vCPU and 32 GiB:
| Phase | Median |
|---|---|
| Clone | 4.34 s |
| Install | 13.44 s |
| Typecheck | 17.42 s |
| Workload total | 35.385 s |
| Full script (seven phases) | 58.45 s |
- The same script on the bare host with no VM totals 30.411 s, so the microVM costs about 16% on this workload.
- Peak memory varied from 11.0 to 14.2 GiB across identical runs. The source recommends 24 GiB or more for this workload.
- Typecheck gains flatten past 16 vCPU.
- Default-deny egress (no network device, all traffic over the vsock gateway) had no measurable cost. The workload needed three allowlist entries:
nodejs.org,api.github.comandpkg.pr.new.
Reproduce
The full page includes a ComputeSDK provider wrapper for @hotcell/sdk and the commands for each mode. Pick a concurrency your host can hold.