Running unattended and on a server

#Overview

autopilot serve starts a headless HTTP server. Through its /runs API, an agent can work on a task much longer than one model turn: it starts a managed job, yields with wait_for, and is invoked again when the job finishes or a timer fires. No model request stays open while the job runs.

To send a task from your laptop to a server and get a pull request back, see Send tasks to your server.

#Start the server

KIMIFLARE_SERVER_PASSWORD='use-a-secret' autopilot serve

• The server refuses to listen on any address other than localhost unless KIMIFLARE_SERVER_PASSWORD is set. If you expose it beyond the machine, put TLS in front of it.
• Creating runs requires KIMIFLARE_SERVER_PASSWORD.
• Requests use HTTP Basic auth with the user name kimiflare and the server password.
• Requests carrying a browser Origin from another site get 403. On a localhost listener, the Host header must be a localhost name. To let a specific web app call the API, list its origins in KIMIFLARE_SERVER_CORS_ORIGINS (comma-separated).
• Run the server under a service manager so scheduled check-ins fire.

#Create a run

curl -u "kimiflare:$KIMIFLARE_SERVER_PASSWORD" \
  -H 'Content-Type: application/json' \
  -d '{
    "task": "Run the training job, inspect its checkpoints and health periodically, and report when it completes or needs intervention.",
    "cwd": "/workspace/model-training",
    "worktree": false,
    "allowedTools": ["job_start", "job_status", "job_logs", "job_cancel"]
  }' \
  http://127.0.0.1:4096/runs

#Run fields

New runs have no runtime, token, cost or tool-action budget by default. Usage is still accounted for. A single model turn is still bounded by the per-turn iteration limit and loop guardrails, and provider rate limits and machine resources still apply.

#Endpoints

#Keep a job running and check in

The agent uses job_start for a detached command, then yields instead of polling:

job_start({ command: "python train.py --checkpoint-every 10m", idempotency_key: "training-run-1" })
wait_for({ job_id: "<job-id>", poll_interval_ms: 300000 })

job_start is not an interactive shell. Stdin is closed, and Autopilot can't send later keystrokes to the process. Use it for commands that manage themselves, such as a training program with checkpoints. For interactive control, GPU allocation, cluster placement or infrastructure-level retries, use an external job scheduler.

#Budget pauses and resumption

If an explicit budget is reached, or cost can't be enforced because the provider didn't report authoritative cost, the run moves to needs_input. It is not marked completed or failed, and the background job is not killed. Resume with a revised limit, or null to remove it:

curl -u "kimiflare:$KIMIFLARE_SERVER_PASSWORD" \
  -H 'Content-Type: application/json' \
  -d '{"maxRuntimeMs": null, "maxTotalTokens": null}' \
  -X POST http://127.0.0.1:4096/runs/<run-id>/resume

The resume request must change at least one budget or the model. Omitted fields keep their current values.

#When a turn outgrows the context window

An unattended run's single turn keeps growing. Older turns are compacted, but the active turn is kept word for word. If it no longer fits the model's context window, the run moves to needs_input with a context_budget_exceeded stop reason, and connected clients receive a run.needs_input event. Resume on a model with a larger window:

curl -u "kimiflare:$KIMIFLARE_SERVER_PASSWORD" \
  -H 'Content-Type: application/json' \
  -d '{"model": "google/gemini-3.8-flash"}' \
  -X POST http://127.0.0.1:4096/runs/<run-id>/resume

#Email notifications

Autopilot can email you when a run completes, fails, needs input, is cancelled or is interrupted. It sends through Resend.

export RESEND_API_KEY=re_...
export AUTOPILOT_NOTIFY_EMAIL_TO=you@example.com    # comma-separated for several
export AUTOPILOT_NOTIFY_EMAIL_FROM="Autopilot <bot@your-verified-domain.com>"   # optional

Or in the config file:

{ "runNotifications": { "email": { "to": ["you@example.com"], "from": "Autopilot <bot@your-domain.com>", "events": ["needs_input", "failed", "completed"] } } }

The email gives the run id, the task (with credential-looking strings redacted), status and stop reason, branch, tokens and cost, and the command to inspect or resume it. It never includes transcript content.

• A notification is recorded in the same database transaction as the state change, so a provider outage never blocks a run.
• Temporary rejections (429, 5xx) are retried. Other rejections are marked failed.
• Unknown outcomes are retried with the same idempotency key, so Resend delivers at most once.
• Notifications older than 24 hours are not sent.

#Decide by email

With notifications on, you can answer a "needs input" email by replying. Autopilot polls a mailbox over IMAP, so the server needs no public endpoint. Use a mailbox dedicated to Autopilot.

Reply with one line:

A reply is acted on only if it comes from a notification recipient, passes DKIM or DMARC on your mail server, carries the email's [ref:...] token (single use, valid for 72 hours), and arrives while the run is still waiting. A reply can't change tools, permissions or other runs.

#Restarts

Waiting runs and their timers are stored in SQLite, and jobs are detached from the server process. If the server restarts during an agent turn, the run resumes automatically only when its latest checkpoint has no tool call with an unknown outcome. Otherwise it becomes interrupted_unknown, so an action such as starting a second training job is never replayed.

This is process-restart recovery, not machine-reboot durability. A detached local process does not survive a host reboot, power loss or container replacement. For multi-week work, checkpoint to durable storage and run the workload under a host or cluster service manager.

#Deploy on a Linux server

These steps follow docs/deploy-linux-server.md, written for a host such as Ubuntu 24.04.

#Host and user

Install Node.js (a version that meets the requirements), Git, SQLite and restic, then create a dedicated account:

sudo useradd --create-home --shell /bin/bash autopilot
sudo mkdir -p /srv/autopilot && sudo chown autopilot:autopilot /srv/autopilot
sudo npm install -g autopilot-ai

Clone the repositories runs will work on into /srv/autopilot as the autopilot user. Give that user only the credentials runs need.

#Secrets

sudo mkdir -p /etc/autopilot
sudo install -m 600 /dev/null /etc/autopilot/autopilot.env
sudoedit /etc/autopilot/autopilot.env

KIMIFLARE_SERVER_PASSWORD=<long random string>
OPENROUTER_API_KEY=sk-or-...
AUTOPILOT_MAX_CONCURRENT_RUNS=4
RESEND_API_KEY=re_...
AUTOPILOT_NOTIFY_EMAIL_TO=you@example.com
AUTOPILOT_NOTIFY_EMAIL_FROM=Autopilot <bot@your-verified-domain.com>

#Service

The repository ships a systemd unit, deploy/autopilot.service:

sudo cp deploy/autopilot.service /etc/systemd/system/autopilot.service
sudo systemctl daemon-reload
sudo systemctl enable --now autopilot
journalctl -u autopilot -f

The unit runs as autopilot, listens on 127.0.0.1:4096 only, restarts on failure, and keeps the filesystem read-only except the user's home and /srv/autopilot. It uses KillMode=process, so managed jobs keep running across restarts.

#Access

Don't expose the server publicly. Tunnel to it over SSH:

ssh -L 4096:127.0.0.1:4096 autopilot@your-host
curl -u "kimiflare:$KIMIFLARE_SERVER_PASSWORD" http://127.0.0.1:4096/runs

#Limits and the kill switch

• At most AUTOPILOT_MAX_CONCURRENT_RUNS runs (default 4) can be queued or running at once. Creating or resuming one past the limit returns 429. Waiting and paused runs don't count.
• The kill switch cancels every active run and refuses new ones. Add "jobs": true to also cancel running managed jobs. The halt is stored in the runs database, so it survives restarts.

curl -u "kimiflare:$KIMIFLARE_SERVER_PASSWORD" -H 'Content-Type: application/json' \
  -d '{"reason": "incident", "jobs": true}' -X POST http://127.0.0.1:4096/runs/stop-all

# Lift it
curl -u "kimiflare:$KIMIFLARE_SERVER_PASSWORD" -X DELETE http://127.0.0.1:4096/runs/stop-all

For a hard stop, run sudo systemctl stop autopilot. Managed jobs keep running until you cancel them.

#Upgrades

sudo npm install -g autopilot-ai@latest
sudo systemctl restart autopilot

Runs waiting on timers or jobs resume on their own. A run in the middle of a turn resumes from its last checkpoint, or becomes interrupted_unknown.

#State and backups

Copy the SQLite files with sqlite3 <db> ".backup '<dest>'", not cp, to get a consistent snapshot while the server writes. The source guide has a full nightly restic backup script and a restore test.

#Send tasks to your server

From your laptop, you can send a task to an Autopilot server you run. It works in its own sandbox and opens a pull request, and you can close your laptop while it runs.

autopilot remote add https://acme.example.com      # approve the code in your browser; no keys copied
autopilot --on acme "fix the flaky checkout test"  # runs on the server
autopilot runs                                     # what's running or done, with PR links
autopilot runs watch <run>                         # follow a run (Ctrl-C detaches)
autopilot pull <run>                               # check out the run's branch here

• The run starts from the last commit your branch shares with origin. Unpushed commits, staged and unstaged edits, and untracked files that aren't ignored are sent as a patch, up to 3 MiB.
• --detach returns right away. -m <model> picks one of the server's models. --repo <owner/name> overrides the repository.
• Each task gets its own sandbox and branch, so several can run at once.
• The server keeps the model key and GitHub credentials. Your laptop holds only a revocable 90-day pairing credential in ~/.config/autopilot/remotes.json.

Setting up the server side for paired tasks is covered in docs/aster-api.md.