name: osaurus-macos-use description: Drive macOS apps in the background. Use when the user asks you to interact with native Mac apps, automate UI tasks, browse the web in Safari, fill forms, navigate menus, or perform any on-screen action — without taking the user's screen or cursor away from them. metadata: author: osaurus version: "3.0.0"

Osaurus macOS Use

A backgrounded computer-use driver. Open apps, observe the UI, click buttons, type into fields, navigate menus, browse the web — all while the user keeps working in the foreground. Cursor never moves, focus never changes, Spaces never follow.

Built on the cua-driver recipe: SkyLight SLEventPostToPid for cursor-warp-free routing, yabai-style focusWithoutRaise for AppKit-active flips, the (-1,-1) Chromium primer click for renderer-IPC user-activation gates, and a per-pid CGEvent.postToPid fallback for the rest.

The Contract

Five rules. Follow them in order, every time.

1. Discover with list_apps() if the target is already running, or skip to step 2 to launch fresh.
2. Open with open_application (defaults to background: true — the app is NOT raised). It returns the app pid AND a starting capture (default mode: "som" — AX tree + screenshot + numeric elementIndex per element). Read the capture before doing anything else.
3. Locate with find_elements({ pid, text, role? }) whenever you know what you're looking for. Faster, cheaper, more reliable than scanning a get_ui_elements result by hand.
4. Act with click_element, set_value, type_text, clear_field, press_key. Always pass element ids in the s{snapshot}-{n} format (e.g. "s7-12"). For raw-coordinate click/scroll/type_text/press_key, always pass pid — that's what keeps routing per-pid (no cursor warp).
5. Re-observe only when the result tells you to:
   • "stale": true → call get_ui_elements (or find_elements) again, then retry.
   • "removed": true → element is gone; observe and find a new one.
   • delta.focusedWindow changed in a way you didn't expect → observe again.
   • Otherwise → keep going.

If you'd rather not think about re-observing, use act_and_observe — runs an action and returns a fresh capture in one call.

Capture modes (cua-style)

open_application, get_ui_elements, and act_and_observe accept mode:

• som (default) — AX tree + annotated screenshot + per-element elementIndex. Best for vision-first agents that ground on pixels.
• ax — tree only. Fastest. No Screen Recording permission needed. Best for AppKit/SwiftUI apps with rich AX trees.
• vision — screenshot only. Smallest payload for VLMs that don't need the tree.

In som mode, every element is addressable two ways: by snapshot id ("s7-12") AND by elementIndex (1, 2, 3, …). Use whichever your model prefers — both resolve to the same element.

Routing chain

You don't usually need to think about this, but it's useful when debugging:

1. AXPress / AXShowMenu / AXValue — click_element etc. try this first. Fully backgrounded.
2. SLEventPostToPid — SkyLight private framework, loaded at runtime. Trusted by Chromium renderers, no cursor warp.
3. CGEvent.postToPid — public CoreGraphics. Works for almost everything except Chromium web content.
4. HID tap — last resort. This is the only path that warps the user's cursor. Auto-falls-back here for canvas/Blender/Unity.

drag is the one operation that ALWAYS uses the HID tap (drop receivers key on the global cursor position).

Canonical Recipe

1. list_apps()
   → { apps: [..., { pid: 1234, name: "Safari", bundleId: "com.apple.Safari", active: false }, ...] }

2. open_application({ identifier: "Safari", mode: "som" })
   → { pid: 1234, som: { snapshot: { snapshotId: 1, ... }, image: { mimeType: "image/jpeg", data: "..." }, elements: [{ elementIndex: 1, id: "s1-3", role: "textfield", label: "Address" }, ...] } }

3. press_key({ key: "l", modifiers: ["command"], pid: 1234 })
   → { success: true, delta: { focusedElement: { role: "textfield", label: "Address" } } }

4. type_text({ text: "https://example.com", pid: 1234 })
5. press_key({ key: "return", pid: 1234 })

6. find_elements({ pid: 1234, text: "More information", role: "link" })
   → { snapshotId: 2, elements: [{ id: "s2-3", role: "link", label: "More information..." }] }

7. click_element({ id: "s2-3" })
   → { success: true, delta: { focusedWindow: "IANA-managed Reserved Domains" } }

The user never sees Safari come forward. Their own app stays focused throughout.

Snapshot Ids and the Cache

• Element ids look like s7-12. The s7 is the snapshot they came from.
• The plugin keeps the last 2 snapshots in cache. Ids from older snapshots return "stale": true.
• Each call to get_ui_elements, find_elements, or open_application (with default observe: true) starts a new snapshot and bumps the counter.
• This means: if you call find_elements twice in a row, the ids from the first call become stale on the third call, not the second.

If you ever see a result with "stale": true, the fix is always the same: re-observe and retry with the new id.

Tool Reference

Discovery / observation

Element actions (snapshot id required)

Coordinate / keyboard actions

Combined / utility

Session telemetry

Tips That Actually Matter

1. Pass pid to every coordinate-based action (click, scroll, type_text, press_key). Without it, the call falls back to the HID tap and warps the cursor.
2. set_value first, type_text if it fails. set_value is instant and correct for most fields. type_text (focus + clear + type) is the fallback for search fields, password fields, anything that needs per-keystroke events.
3. Use windowId from list_windows when capturing a specific window. It works for windows that are occluded, hidden, or on a different Space.
4. Check truncated: true in any snapshot. If true, raise maxElements or use find_elements.
5. focusedWindowOnly: true is your friend. Cheap re-observation when the action only changed the focused window.
6. Keyboard shortcuts beat menu navigation. press_key("s", modifiers: ["command"], pid: ...) is one tool call; clicking File > Save is three.
7. Roles can be passed in any case. "button", "Button", and "AXButton" all work.

Troubleshooting

Reference

For per-app recipes (Safari URL bar, web forms, tab management) and the full keyboard shortcut catalog, see REFERENCE.md. Don't load it unless you need it — most automation only needs the contract above.

Limitations

• Canvas apps (Figma, games, Blender, Unity): per-pid event routes are filtered. Driver auto-falls back to the HID tap, which warps the cursor.
• Chromium right-click on web content: coerced to left-click. Use AX paths instead.
• Highly dynamic web apps: re-observe more often; prefer find_elements over caching ids across navigation.
• Iframes: AX coverage varies. Safari is most reliable.