Files
orca/skills/computer-use/SKILL.md
T
2026-06-03 16:45:34 -07:00

122 lines
6.2 KiB
Markdown

---
name: computer-use
description: >-
Use Orca's computer-use CLI to inspect and operate local desktop app windows
through accessibility trees, screenshots, and safe UI actions. Use for
desktop app interaction: list apps/windows, get app state, read visible UI,
click controls, type, press keys, scroll, drag, set values, or perform
accessibility actions. Also use for browser windows, webviews, Orca app UI,
or any desktop UI outside Orca's built-in browser. Triggers include "computer
use", "orca computer", "read Spotify", "read Slack", "control/click/read in
a desktop app", and "get app state".
---
# Computer Use
Use this skill for desktop UI through Orca's computer-use surface.
Routing:
- Use Orca built-in browser commands (`orca snapshot`, `orca click`, `orca fill`, etc.) only for the browser page embedded inside the Orca app.
- Use `orca computer` for native desktop apps, external browser windows, app/webview chrome, Orca settings/app UI, and any desktop UI outside Orca's embedded browser.
## Preconditions
- Prefer `orca computer ...`; on Linux, use `orca-ide computer ...` if `orca` is unavailable. In this Orca worktree, use `./config/scripts/orca-dev computer ...` only when testing the local dev runtime.
- Prefer `--json`. Screenshot bytes are omitted from JSON and written to `screenshot.path`.
- Do not push, submit forms, send messages, buy items, delete data, change account settings, or expose secrets unless the user explicitly asked for that action.
- If an app contains sensitive content, read only what the user requested.
```bash
orca status --json
orca computer capabilities --json
```
## Core Loop
```bash
orca computer list-apps --json
orca computer get-app-state --app com.spotify.client --json
orca computer click --app com.spotify.client --element-index 42 --json
```
Use the fresh state returned by each action for the next element index. Element indexes go stale after navigation, focus changes, scrolling, window changes, or app re-rendering.
## App Selectors
Prefer bundle IDs from `list-apps`; names are acceptable when unambiguous. Use `pid:<number>` only when bundle ID or name matching is ambiguous.
```bash
orca computer get-app-state --app com.microsoft.edgemac --json
orca computer get-app-state --app Spotify --json
orca computer get-app-state --app pid:12345 --json
```
For apps with multiple windows or ambiguous titles, run `list-windows` first. Once you choose a window, pass the same `--window-id <id>` or `--window-index <n>` to `get-app-state` and later actions until the target window changes.
## Commands
```bash
orca computer permissions --json
orca computer capabilities --json
orca computer list-apps --json
orca computer list-windows --app <app> --json
orca computer get-app-state --app <app> --json
orca computer get-app-state --app <app> --restore-window --json
orca computer click --app <app> --element-index <index> --json
orca computer click --app <app> --x 100 --y 100 --json
orca computer perform-secondary-action --app <app> --element-index <index> --action <name> --json
orca computer set-value --app <app> --element-index <index> --value "text" --json
orca computer type-text --app <app> --text "text" --json
orca computer press-key --app <app> --key Return --json
orca computer hotkey --app <app> --key CmdOrCtrl+A --json
orca computer paste-text --app <app> --text "text" --json
orca computer scroll --app <app> (--element-index <index> | --x <x> --y <y>) --direction down --json
orca computer drag --app <app> --from-element-index <index> --to-element-index <index> --json
orca computer drag --app <app> --from-x 100 --from-y 100 --to-x 300 --to-y 300 --json
```
Use `--no-screenshot` only when pixels are not needed. Use `--text-stdin` or `--value-stdin` for sensitive text so payloads do not land in shell history. On Linux and Windows, action payloads still pass through a short-lived local operation file, so avoid sending secrets unless the user explicitly asked for them:
```bash
printf '%s' "$TEXT" | orca computer set-value --app <app> --element-index <index> --value-stdin --json
```
## Action Rules
- Prefer semantic actions: `set-value` for editable fields, `click` for controls, `perform-secondary-action` only for listed action names.
- Use `type-text` only after focusing a field and confirming the app has a focused text receiver.
- Use `press-key` for single/navigation keys such as Return, Escape, Tab, and arrows. Use `hotkey` for shortcuts; prefer `CmdOrCtrl+...` for cross-platform combos.
- Some actions work in background apps, but this is app-dependent. If success does not change the UI, refresh state and choose a more semantic action or restore/focus the window.
- Coordinates are window-local; use coordinates from the latest screenshot/state for the same target window.
## Screenshots
`get-app-state` returns tree+screenshot. Use the tree for indexes/actions and the screenshot for visual confirmation; failed capture usually means hidden, minimized, off-screen, or permission-blocked.
## App Notes
Browsers: for Edge, Chrome, and similar browser windows, set the address/search field directly, then press Return. Do not assume raw typing went to the address bar.
```bash
orca computer get-app-state --app com.microsoft.edgemac --json
orca computer set-value --app com.microsoft.edgemac --element-index <addressBarIndex> --value "test123" --json
orca computer press-key --app com.microsoft.edgemac --key Return --json
```
Spotify: refresh after playback clicks; the UI often changes asynchronously.
Slack: the accessibility tree may be shallow while the screenshot contains useful information. Reading visible Slack UI is fine when requested; sending messages or triggering workflows still needs explicit permission.
## Errors
- `app_not_found`: run `list-apps` and retry with the bundle ID.
- `element_not_found`: index is stale; run `get-app-state` again.
- `action_failed`: inspect the element role/actions and try a more semantic action.
- Empty tree or no screenshot: app may have no visible window, be minimized, or need permissions.
- Permission errors: run `orca computer permissions --json`, use the setup UI, then retry.
## Next Action
Confirm Orca status unless already checked, run `orca computer capabilities --json`, then get the target app state with `orca computer get-app-state --app <app> --json`.