Skip to content
Fuse
Esc
↑↓navigate↵open⌘Jpreview

Drive the environment's desktop

Relay one computer-use action to the environment’s guest agent: move the mouse, click, type, scroll, screenshot, or zoom into a region. The response carries a base64 PNG screenshot on every action that implies one, which is what a computer-use agent loop feeds back to the model as the tool_result.

The action vocabulary is the guest agent’s (it matches Anthropic’s computer_20251124 tool): screenshot, left_click, right_click, middle_click, double_click, triple_click, left_click_drag, left_mouse_down, left_mouse_up, mouse_move, type, key, hold_key, scroll, wait, zoom, cursor_position. Fields this server has not learned yet pass through to the guest untouched, so a newer image’s actions work through an older control plane.

Requires an environment booted from a desktop image; on any other image the guest answers with 503 and a reason. Unlike exec this accepts API keys: an action here is a mouse click on a display the image chose to run, not a root shell.

POST/v1/environments/{vmId}/computer
Authorization
AuthorizationBearer token · headerrequired
`Authorization: Bearer <token>` — either the static master token (`ORCH_AUTH_TOKEN`) or a revocable API key issued via /v1/api-keys (API keys require Postgres; without a database only the master token is accepted). Master-only surfaces (exec, attach, API key management) refuse API keys with 403. **Authentication can be disabled entirely.** When neither `ORCH_AUTH_TOKEN` nor an API key store (`DATABASE_URL`) is configured — the default local-development posture — the bearer middleware becomes a pass-through: every operation this document marks as secured is reachable with no credentials, requests carry no principal, and master-only surfaces therefore admit everyone. Never run that configuration on a reachable network; set `ORCH_REQUIRE_AUTH=true` to make the orchestrator refuse to start without a master token.
or
fuse_sessionAPI key · cookierequired
HttpOnly session cookie set by POST /login for browser callers. Consulted only when no Authorization header is present.
Path parameters
vmIdstringrequired
VM identifier assigned by the orchestrator.
Request body
requiredapplication/json
actionstringrequired
The action to perform, e.g. `screenshot`, `left_click`, `type`, `key`, `scroll`, `zoom`.
coordinateinteger[]
[x, y] target for pointer actions.
min items 2 · max items 2
start_coordinateinteger[]
[x, y] origin for left_click_drag.
min items 2 · max items 2
textstring
The payload for `type`, the keysym combo for `key` and `hold_key` (e.g. `ctrl+shift+t`), or modifiers held during a click action.
regioninteger[]
[x1, y1, x2, y2] crop for `zoom`.
min items 4 · max items 4
durationnumber
Seconds for `hold_key` and `wait`, capped guest-side at 100.
scroll_directionstring
Allowed:updownleftright
scroll_amountinteger
Wheel clicks for `scroll`, 1 to 100.
Responses
200The action's result, usually carrying a screenshot.
outputstring
Textual output, e.g. the pointer location for `cursor_position`.
screenshotstring
Base64 PNG of the display after the action settled. Present on every action that implies one; for `zoom` it is the cropped region at full resolution.
400Malformed request.
errorobjectrequired
Show properties
codestringrequired
Stable machine-readable code. `route_not_found` (404) means the URL matches no route this server exposes — usually a wrong host, port, or path prefix — as opposed to `not_found`, where the route exists but the resource does not. `forbidden` is emitted only by CIDR-allowlist rejections; master-only refusals (exec, attach, API key management) return 403 with code `unauthorized`.
Allowed:not_foundroute_not_foundconflictinvalid_argumentunauthorizedforbiddenunavailableinternalunimplemented
messagestringrequired
Human-readable description.
detailsobject
Optional stable, non-sensitive metadata (e.g. ids, counts). Omitted when empty.
404Resource does not exist. Note the distinct code `route_not_found`, returned when the URL matches no registered route at all (wrong host, port, or path prefix) rather than a known route with a missing resource.
errorobjectrequired
Show properties
codestringrequired
Stable machine-readable code. `route_not_found` (404) means the URL matches no route this server exposes — usually a wrong host, port, or path prefix — as opposed to `not_found`, where the route exists but the resource does not. `forbidden` is emitted only by CIDR-allowlist rejections; master-only refusals (exec, attach, API key management) return 403 with code `unauthorized`.
Allowed:not_foundroute_not_foundconflictinvalid_argumentunauthorizedforbiddenunavailableinternalunimplemented
messagestringrequired
Human-readable description.
detailsobject
Optional stable, non-sensitive metadata (e.g. ids, counts). Omitted when empty.
409The environment is not running.
errorobjectrequired
Show properties
codestringrequired
Stable machine-readable code. `route_not_found` (404) means the URL matches no route this server exposes — usually a wrong host, port, or path prefix — as opposed to `not_found`, where the route exists but the resource does not. `forbidden` is emitted only by CIDR-allowlist rejections; master-only refusals (exec, attach, API key management) return 403 with code `unauthorized`.
Allowed:not_foundroute_not_foundconflictinvalid_argumentunauthorizedforbiddenunavailableinternalunimplemented
messagestringrequired
Human-readable description.
detailsobject
Optional stable, non-sensitive metadata (e.g. ids, counts). Omitted when empty.
500Unexpected server error.
errorobjectrequired
Show properties
codestringrequired
Stable machine-readable code. `route_not_found` (404) means the URL matches no route this server exposes — usually a wrong host, port, or path prefix — as opposed to `not_found`, where the route exists but the resource does not. `forbidden` is emitted only by CIDR-allowlist rejections; master-only refusals (exec, attach, API key management) return 403 with code `unauthorized`.
Allowed:not_foundroute_not_foundconflictinvalid_argumentunauthorizedforbiddenunavailableinternalunimplemented
messagestringrequired
Human-readable description.
detailsobject
Optional stable, non-sensitive metadata (e.g. ids, counts). Omitted when empty.
501The operation is not supported by the backing provider (e.g. an exec or attach against an environment with no real guest).
errorobjectrequired
Show properties
codestringrequired
Stable machine-readable code. `route_not_found` (404) means the URL matches no route this server exposes — usually a wrong host, port, or path prefix — as opposed to `not_found`, where the route exists but the resource does not. `forbidden` is emitted only by CIDR-allowlist rejections; master-only refusals (exec, attach, API key management) return 403 with code `unauthorized`.
Allowed:not_foundroute_not_foundconflictinvalid_argumentunauthorizedforbiddenunavailableinternalunimplemented
messagestringrequired
Human-readable description.
detailsobject
Optional stable, non-sensitive metadata (e.g. ids, counts). Omitted when empty.
503The environment has no display (not a desktop image, or the display is still starting).
errorobjectrequired
Show properties
codestringrequired
Stable machine-readable code. `route_not_found` (404) means the URL matches no route this server exposes — usually a wrong host, port, or path prefix — as opposed to `not_found`, where the route exists but the resource does not. `forbidden` is emitted only by CIDR-allowlist rejections; master-only refusals (exec, attach, API key management) return 403 with code `unauthorized`.
Allowed:not_foundroute_not_foundconflictinvalid_argumentunauthorizedforbiddenunavailableinternalunimplemented
messagestringrequired
Human-readable description.
detailsobject
Optional stable, non-sensitive metadata (e.g. ids, counts). Omitted when empty.
Request
curl -X POST "http://localhost:8080/v1/environments/string/computer" \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
  "action": "left_click",
  "coordinate": [
    512,
    384
  ]
}'
Response
{
  "output": "string",
  "screenshot": "string"
}