Docs / Hotcell / Observability

Observability

Per-sandbox metrics and cost, OpenTelemetry traces, the audit log, structured logs, the web dashboard and the terminal fleet monitor.

Metrics

A sampler reads every running sandbox's CPU, memory, network and PID counts every HOTCELL_METRICS_INTERVAL_MS (default 10000; 0 disables it). It integrates usage over time for the cost meter and keeps the last HOTCELL_METRICS_HISTORY samples (default 60) per sandbox for sparklines.

hotcell stats <id>

const { status, live, usage, cost } = await sandbox.metrics();
const samples = await sandbox.metricsHistory();
Object Fields
live (running only) cpuPercent, cpuTotalUsageNs, onlineCpus, memBytes, memLimitBytes, netRxBytes, netTxBytes, pids, sampledAt
usage (cumulative) cpuSeconds, memByteSeconds, egressBytes, providerCalls, providerBytes, providerTokensIn, providerTokensOut, providerCost
cost cpu, mem, egress, provider, total
history sample at, cpuPercent, memBytes, netRxBytes, netTxBytes, pids

Reading metrics doesn't count as activity, so polling won't keep an idle sandbox from auto-pausing.

Cost meter

Compute cost uses configurable rates:

Variable Default Unit
HOTCELL_COST_CPU_PER_HOUR 0.05 per vCPU-hour
HOTCELL_COST_MEM_GB_PER_HOUR 0.005 per GB-hour of memory
HOTCELL_COST_EGRESS_PER_GB 0.01 per GB of preview-proxy egress

LLM cost from the egress gateway is added as provider and included in total. See Egress for how it is computed.

Capacity

hotcell capacity

GET /capacity returns the memory and CPU budget, what is committed and available, the number of running sandboxes, how many more default-reservation sandboxes fit (fits), warm-pool spares (pool) and the microVM boot queue (boot).

Traces

The daemon records OpenTelemetry spans for API requests. Set HOTCELL_OTLP_ENDPOINT to an OTLP/HTTP collector base URL; spans are exported to /v1/traces.

HOTCELL_OTLP_ENDPOINT=http://127.0.0.1:4318 \
HOTCELL_OTLP_HEADERS='{"Authorization":"Bearer ..."}' hotcelld

The last HOTCELL_TRACE_RING finished spans (default 200) are also kept in memory and served at GET /traces?limit=N, newest first.

Audit log

The daemon writes audit events to ~/.hotcell/audit.jsonl by default. Events cover sandbox lifecycle, exec and processes, files, terminals, preview ports, backup and restore, auth failures, admission refusals, outbound egress, gateway policy, spend and tokens, and keys import.

  • With HOTCELL_OTLP_ENDPOINT set, audit events are also exported as OTLP/HTTP logs to /v1/logs, using the same headers. HOTCELL_OTEL_ENDPOINT and HOTCELL_OTEL_HEADERS remain as audit-only overrides.
  • HOTCELL_AUDIT_FILE changes the local path. An empty string disables local persistence.
  • Events include schema_version, event.action, severity, timestamp, sandbox.id, session.id, driver, host.name, service version and hashed token IDs.
  • Command text is redacted for common credential patterns and capped at 512 characters. File contents, keys and raw gateway tokens are never included. Key-import events contain variable names and dispositions only.
  • Export is best-effort and never blocks API work. Batches are capped at 128 records or 512 KiB, the queue at 2,048 events, and a failed request is retried at most twice.

On Linux with default-deny egress, the firewall logs new blocked flows before dropping them. The daemon maps the source container IP to its sandbox and records the destination IP, port, protocol and rule, rate-limited to 20 records per second.

Logs

The daemon writes structured logs. HOTCELL_LOG_LEVEL is debug, info (default), warn or error. HOTCELL_LOG_FORMAT=json emits one JSON object per line for ingestion; the default is pretty. With hotcell start, logs go to ~/.hotcell/daemon.log.

Web dashboard

Open http://127.0.0.1:4750/ after hotcell start. The dashboard is a single page embedded in the daemon, with no build step. It shows:

  • the sandbox list, with create, stop, start and destroy;
  • live CPU, memory and network with sparklines;
  • the cost meter;
  • preview links for exposed ports;
  • a live terminal (xterm.js) per sandbox.

It asks for the API key once, or you can open /dashboard#key=<key>.

Terminal fleet monitor

hotcell tui        # or: hotcell top

A full-screen view of every sandbox with live CPU, memory and cost. Use the arrow keys to move, Enter to attach a shell, p, r and d to pause, resume or destroy, and c to create.