chore: bump Peekaboo submodule
parent
ece8a3e701
commit
0d3aacd316
2
Peekaboo
2
Peekaboo
|
|
@ -1 +1 @@
|
||||||
Subproject commit 3c5576b271be0ccbd1e42947fa4c655a394db8ff
|
Subproject commit 4807e6fdf016f013dba8a32fa4b2528182539fad
|
||||||
|
|
@ -14,7 +14,7 @@ Author: steipete · Status: draft spec · Date: 2025-12-05
|
||||||
- Owns TCC prompts (Notifications, Accessibility, Screen Recording, Automation/AppleScript, Microphone, Speech Recognition).
|
- Owns TCC prompts (Notifications, Accessibility, Screen Recording, Automation/AppleScript, Microphone, Speech Recognition).
|
||||||
- Brokers privileged actions via local IPC:
|
- Brokers privileged actions via local IPC:
|
||||||
- Clawdis control socket (app-specific actions like notify/run)
|
- Clawdis control socket (app-specific actions like notify/run)
|
||||||
- PeekabooBridge socket (`bridge.sock`) for UI automation (see `docs/mac/peekaboo.md`)
|
- PeekabooBridge socket (`bridge.sock`) for UI automation brokering (consumed by `peekaboo`; see `docs/mac/peekaboo.md`)
|
||||||
- Provides a tiny CLI (`clawdis-mac`) that talks to the app; Node/TS shells out to it.
|
- Provides a tiny CLI (`clawdis-mac`) that talks to the app; Node/TS shells out to it.
|
||||||
- Replace the separate notifier helper pattern (Oracle) with a built-in notifier.
|
- Replace the separate notifier helper pattern (Oracle) with a built-in notifier.
|
||||||
- Offer a first-run experience similar to VibeTunnel’s onboarding (permissions + CLI install).
|
- Offer a first-run experience similar to VibeTunnel’s onboarding (permissions + CLI install).
|
||||||
|
|
@ -47,7 +47,7 @@ struct Response { ok: Bool; message?: String; payload?: Data }
|
||||||
- The control-socket server rejects oversize/unknown cases and validates the caller by code signature TeamID (with a `DEBUG`-only same-UID escape hatch controlled by `CLAWDIS_ALLOW_UNSIGNED_SOCKET_CLIENTS=1`).
|
- The control-socket server rejects oversize/unknown cases and validates the caller by code signature TeamID (with a `DEBUG`-only same-UID escape hatch controlled by `CLAWDIS_ALLOW_UNSIGNED_SOCKET_CLIENTS=1`).
|
||||||
|
|
||||||
UI automation is not part of `ClawdisIPC.Request`:
|
UI automation is not part of `ClawdisIPC.Request`:
|
||||||
- `clawdis-mac ui …` speaks **PeekabooBridge** (see `docs/mac/peekaboo.md`).
|
- UI automation is handled via the separate PeekabooBridge socket and is surfaced by the `peekaboo` CLI (see `docs/mac/peekaboo.md`).
|
||||||
|
|
||||||
## App UX (Clawdis)
|
## App UX (Clawdis)
|
||||||
- MenuBarExtra icon only (LSUIElement; no Dock).
|
- MenuBarExtra icon only (LSUIElement; no Dock).
|
||||||
|
|
@ -71,15 +71,7 @@ UI automation is not part of `ClawdisIPC.Request`:
|
||||||
- Subcommands (text by default; `--json` for machine output; non-zero exit on failure):
|
- Subcommands (text by default; `--json` for machine output; non-zero exit on failure):
|
||||||
- `notify --title --body [--sound] [--priority passive|active|timeSensitive] [--delivery system|overlay|auto]`
|
- `notify --title --body [--sound] [--priority passive|active|timeSensitive] [--delivery system|overlay|auto]`
|
||||||
- `ensure-permissions --cap accessibility --cap screenRecording [--interactive]`
|
- `ensure-permissions --cap accessibility --cap screenRecording [--interactive]`
|
||||||
- `ui permissions status`
|
- UI automation + capture: use `peekaboo …` (Clawdis hosts PeekabooBridge; see `docs/mac/peekaboo.md`)
|
||||||
- `ui frontmost`
|
|
||||||
- `ui apps`
|
|
||||||
- `ui windows [--bundle-id <id>]`
|
|
||||||
- `ui screenshot [--screen-index <n>] [--bundle-id <id>] [--window-index <n>] [--watch] [--scale native|1x]`
|
|
||||||
- `ui see [--bundle-id <id>] [--window-index <n>] [--snapshot-id <id>]`
|
|
||||||
- `ui click --on <elementId> [--bundle-id <id>] [--snapshot-id <id>] [--double|--right]`
|
|
||||||
- `ui type --text <value> [--into <elementId>] [--bundle-id <id>] [--snapshot-id <id>] [--clear] [--delay-ms <n>]`
|
|
||||||
- `ui wait --on <elementId> [--bundle-id <id>] [--snapshot-id <id>] [--timeout <sec>]`
|
|
||||||
- `run -- cmd args... [--cwd] [--env KEY=VAL] [--timeout 30] [--needs-screen-recording]`
|
- `run -- cmd args... [--cwd] [--env KEY=VAL] [--timeout 30] [--needs-screen-recording]`
|
||||||
- `status`
|
- `status`
|
||||||
- Sounds: supply any macOS alert name with `--sound` per notification; omit the flag to use the system default. There is no longer a persisted “default sound” in the app UI.
|
- Sounds: supply any macOS alert name with `--sound` per notification; omit the flag to use the system default. There is no longer a persisted “default sound” in the app UI.
|
||||||
|
|
@ -87,14 +79,14 @@ UI automation is not part of `ClawdisIPC.Request`:
|
||||||
- Delivery: `overlay` and `auto` show an in-app toast panel (bypasses Notification Center/Focus).
|
- Delivery: `overlay` and `auto` show an in-app toast panel (bypasses Notification Center/Focus).
|
||||||
- Internals:
|
- Internals:
|
||||||
- For app-specific commands (`notify`, `ensure-permissions`, `run`, `status`): build `ClawdisIPC.Request`, send over the control socket.
|
- For app-specific commands (`notify`, `ensure-permissions`, `run`, `status`): build `ClawdisIPC.Request`, send over the control socket.
|
||||||
- For UI automation (`ui …`): connect to PeekabooBridge hosts (Peekaboo.app → Clawdis.app) and send one JSON request per command (see `docs/mac/peekaboo.md`).
|
- UI automation is intentionally not exposed via `clawdis-mac`; it lives behind PeekabooBridge and is surfaced by the `peekaboo` CLI.
|
||||||
|
|
||||||
## Integration with clawdis/Clawdis (Node/TS)
|
## Integration with clawdis/Clawdis (Node/TS)
|
||||||
- Add helper module that shells to `clawdis-mac`:
|
- Add helper module that shells to `clawdis-mac`:
|
||||||
- Prefer `ensure-permissions` before actions that need TCC.
|
- Prefer `ensure-permissions` before actions that need TCC.
|
||||||
- Use `notify` for desktop toasts; fall back to JS notifier only if CLI missing or platform ≠ macOS.
|
- Use `notify` for desktop toasts; fall back to JS notifier only if CLI missing or platform ≠ macOS.
|
||||||
- Use `run` for tasks requiring privileged UI context (screen-recorded terminal runs, etc.).
|
- Use `run` for tasks requiring privileged UI context (screen-recorded terminal runs, etc.).
|
||||||
- For UI automation, `clawdis ui …` is a convenience passthrough to `clawdis-mac ui …` (text by default; add `--json` to the outer `clawdis` command for structured output).
|
- For UI automation, shell out to `peekaboo …` (text by default; add `--json` for structured output) and rely on PeekabooBridge host selection (Peekaboo.app → Clawdis.app → local).
|
||||||
|
|
||||||
## Deep links (URL scheme)
|
## Deep links (URL scheme)
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -1,15 +1,15 @@
|
||||||
---
|
---
|
||||||
summary: "Plan for integrating Peekaboo automation into Clawdis via PeekabooBridge (socket-based TCC broker)"
|
summary: "Plan for integrating Peekaboo automation into Clawdis via PeekabooBridge (socket-based TCC broker)"
|
||||||
read_when:
|
read_when:
|
||||||
- Adding UI automation commands
|
- Hosting PeekabooBridge in Clawdis.app
|
||||||
- Integrating Peekaboo as a submodule
|
- Integrating Peekaboo as a submodule
|
||||||
- Changing clawdis-mac IPC/output formats
|
- Changing PeekabooBridge protocol/paths
|
||||||
---
|
---
|
||||||
# Peekaboo Bridge in Clawdis (macOS UI automation broker)
|
# Peekaboo Bridge in Clawdis (macOS UI automation broker)
|
||||||
|
|
||||||
## TL;DR
|
## TL;DR
|
||||||
- **Peekaboo removed its XPC helper** and now exposes privileged automation via a **UNIX domain socket bridge** (`PeekabooBridge` / `PeekabooBridgeHost`, socket name `bridge.sock`).
|
- **Peekaboo removed its XPC helper** and now exposes privileged automation via a **UNIX domain socket bridge** (`PeekabooBridge` / `PeekabooBridgeHost`, socket name `bridge.sock`).
|
||||||
- Clawdis integrates by **hosting the same bridge** inside **Clawdis.app** (optional, user-toggleable), and by making `clawdis-mac ui …` act as a **bridge client**.
|
- Clawdis integrates by **optionally hosting the same bridge** inside **Clawdis.app** (user-toggleable). The primary client is the **`peekaboo` CLI** (installed via npm); Clawdis does not need its own `ui …` CLI surface.
|
||||||
- For **visualizations**, we keep them in **Peekaboo.app** (best UX); Clawdis stays a thin broker host. No visualizer toggle in Clawdis.
|
- For **visualizations**, we keep them in **Peekaboo.app** (best UX); Clawdis stays a thin broker host. No visualizer toggle in Clawdis.
|
||||||
|
|
||||||
Non-goals:
|
Non-goals:
|
||||||
|
|
@ -31,9 +31,8 @@ Reference (Peekaboo submodule): `docs/bridge-host.md`.
|
||||||
- **Peekaboo.app** (preferred; also provides visualizations + controls)
|
- **Peekaboo.app** (preferred; also provides visualizations + controls)
|
||||||
- **Clawdis.app** (secondary; “thin host” only)
|
- **Clawdis.app** (secondary; “thin host” only)
|
||||||
- **Bridge clients** (trigger single actions):
|
- **Bridge clients** (trigger single actions):
|
||||||
- `clawdis-mac ui …`
|
- `peekaboo …` (preferred; humans + agents)
|
||||||
- `clawdis ui …` (Node/TS convenience wrapper; shells out to `clawdis-mac ui …`)
|
- Optional: Clawdis/Node shells out to `peekaboo` when it needs UI automation/capture
|
||||||
- Node/Gateway shells out to `clawdis-mac`
|
|
||||||
|
|
||||||
### Host discovery (client-side)
|
### Host discovery (client-side)
|
||||||
Order is deliberate:
|
Order is deliberate:
|
||||||
|
|
@ -68,26 +67,22 @@ What Clawdis should *not* embed:
|
||||||
- **XPC**: don’t reintroduce helper targets; use the bridge.
|
- **XPC**: don’t reintroduce helper targets; use the bridge.
|
||||||
|
|
||||||
## IPC / CLI surface
|
## IPC / CLI surface
|
||||||
### Namespacing
|
### No `clawdis-mac ui …`
|
||||||
Add new automation commands behind a `ui` prefix:
|
We avoid a parallel “Clawdis UI automation CLI”. Instead:
|
||||||
- `clawdis-mac ui …` for UI automation + visualization-related actions.
|
- `peekaboo` is the user/agent-facing CLI surface for automation and capture.
|
||||||
- Keep existing top-level commands (`notify`, `run`, `canvas …`, etc.) for compatibility.
|
- Clawdis.app can host PeekabooBridge as a **thin TCC broker** so Peekaboo can piggyback on Clawdis permissions when Peekaboo.app isn’t running.
|
||||||
|
|
||||||
Screenshot cutover:
|
### Diagnostics
|
||||||
- Remove legacy screenshot endpoints/commands.
|
Use Peekaboo’s built-in diagnostics to see which host would be used:
|
||||||
- Ship only `clawdis-mac ui screenshot` (no aliases).
|
- `peekaboo bridge status`
|
||||||
|
- `peekaboo bridge status --verbose`
|
||||||
|
- `peekaboo bridge status --json`
|
||||||
|
|
||||||
### Output format
|
### Output format
|
||||||
Change `clawdis-mac` to default to human text output:
|
Peekaboo commands default to human text output. Add `--json` for a structured envelope.
|
||||||
- **Default**: plain text; errors are string messages to stderr; exit codes indicate success/failure.
|
|
||||||
- **`--json`**: structured output (for agents/scripts) with stable schemas.
|
|
||||||
|
|
||||||
This applies globally, not only `ui` commands.
|
|
||||||
|
|
||||||
Note (current state as of 2025-12-13): `clawdis-mac` prints text by default; use `--json` for structured output.
|
|
||||||
|
|
||||||
### Timeouts
|
### Timeouts
|
||||||
Default timeout for UI actions: **10 seconds** end-to-end.
|
Default timeout for UI actions: **10 seconds** end-to-end (client enforced; host should also enforce per-operation).
|
||||||
|
|
||||||
## Coordinate model (multi-display)
|
## Coordinate model (multi-display)
|
||||||
Requirement: coordinates are **per screen**, not global.
|
Requirement: coordinates are **per screen**, not global.
|
||||||
|
|
@ -111,18 +106,18 @@ Expose window/app targeting in the UI surface (align with Peekaboo targeting):
|
||||||
- by window title substring
|
- by window title substring
|
||||||
- by (app, index)
|
- by (app, index)
|
||||||
|
|
||||||
Current `clawdis-mac ui …` support:
|
Peekaboo CLI targeting (agent-friendly):
|
||||||
- `--bundle-id <id>` for app targeting
|
- `--bundle-id <id>` for app targeting
|
||||||
- `--window-index <n>` (0-based) for disambiguating within an app when capturing (see/screenshot)
|
- `--window-index <n>` (0-based) for disambiguating within an app when capturing
|
||||||
|
|
||||||
All “see/click/type/scroll/wait” requests should accept a target (default: frontmost).
|
All “see/click/type/scroll/wait” requests should accept a target (default: frontmost).
|
||||||
|
|
||||||
## “See” + click packs (Playwright-style)
|
## “See” + click packs (Playwright-style)
|
||||||
Behavior stays aligned with Peekaboo:
|
Behavior stays aligned with Peekaboo:
|
||||||
- `ui see` returns element IDs (e.g. `B1`, `T3`) with bounds/labels.
|
- `peekaboo see` returns element IDs (e.g. `B1`, `T3`) with bounds/labels.
|
||||||
- Follow-up actions reference those IDs without re-scanning.
|
- Follow-up actions reference those IDs without re-scanning.
|
||||||
|
|
||||||
`clawdis-mac ui see` should:
|
`peekaboo see` should:
|
||||||
- capture (optionally targeted) window/screen
|
- capture (optionally targeted) window/screen
|
||||||
- return a screenshot **file path** (default: temp directory)
|
- return a screenshot **file path** (default: temp directory)
|
||||||
- return a list of elements (text or JSON)
|
- return a list of elements (text or JSON)
|
||||||
|
|
@ -132,9 +127,9 @@ Snapshot lifecycle requirement:
|
||||||
- Snapshot scoping: “implicit snapshot” is **per target bundle id** (reuse last snapshot for that app when snapshot id is omitted).
|
- Snapshot scoping: “implicit snapshot” is **per target bundle id** (reuse last snapshot for that app when snapshot id is omitted).
|
||||||
|
|
||||||
Practical flow (agent-friendly):
|
Practical flow (agent-friendly):
|
||||||
- `clawdis-mac ui frontmost` returns the focused app (bundle id) + focused window (title/id) so follow-up calls can pass `--bundle-id …`.
|
- `peekaboo list apps` / `peekaboo list windows` provide bundle-id context for targeting.
|
||||||
- `clawdis-mac ui see --bundle-id X` updates the implicit snapshot for `X`.
|
- `peekaboo see --bundle-id X` updates the implicit snapshot for `X`.
|
||||||
- `clawdis-mac ui click --bundle-id X --on B1` reuses the most recent snapshot for `X` when `--snapshot-id` is omitted.
|
- `peekaboo click --bundle-id X --on B1` reuses the most recent snapshot for `X` when `--snapshot-id` is omitted.
|
||||||
|
|
||||||
## Visualizer integration
|
## Visualizer integration
|
||||||
Keep visualizations in **Peekaboo.app** for now.
|
Keep visualizations in **Peekaboo.app** for now.
|
||||||
|
|
@ -142,11 +137,11 @@ Keep visualizations in **Peekaboo.app** for now.
|
||||||
- Any “visualizer enabled/disabled” setting is controlled in Peekaboo.app.
|
- Any “visualizer enabled/disabled” setting is controlled in Peekaboo.app.
|
||||||
|
|
||||||
## Screenshots (legacy → Peekaboo takeover)
|
## Screenshots (legacy → Peekaboo takeover)
|
||||||
Clawdis uses `clawdis-mac ui screenshot` and returns a file path (default location: temp directory) instead of raw image bytes.
|
Clawdis should not grow a separate screenshot CLI surface.
|
||||||
|
|
||||||
Migration plan:
|
Migration plan:
|
||||||
- Bridge host performs capture and returns a temp file path.
|
- Use `peekaboo capture …` / `peekaboo see …` (returns a file path, default temp directory).
|
||||||
- No legacy aliases; make the old screenshot surface disappear cleanly.
|
- Once Clawdis’ legacy screenshot plumbing is replaced, remove it cleanly (no aliases).
|
||||||
|
|
||||||
## Permissions behavior
|
## Permissions behavior
|
||||||
If required permissions are missing:
|
If required permissions are missing:
|
||||||
|
|
@ -164,22 +159,9 @@ Debug-only escape hatch (development convenience):
|
||||||
- “allow same-UID callers” means: *skip codesign checks for clients running under the same Unix user*.
|
- “allow same-UID callers” means: *skip codesign checks for clients running under the same Unix user*.
|
||||||
- This must be **opt-in**, **DEBUG-only**, and guarded by an env var (Peekaboo uses `PEEKABOO_ALLOW_UNSIGNED_SOCKET_CLIENTS=1`).
|
- This must be **opt-in**, **DEBUG-only**, and guarded by an env var (Peekaboo uses `PEEKABOO_ALLOW_UNSIGNED_SOCKET_CLIENTS=1`).
|
||||||
|
|
||||||
## Current `clawdis-mac ui` commands (Dec 2025)
|
|
||||||
All commands default to text output. Add `--json` right after `clawdis-mac` for a structured envelope.
|
|
||||||
|
|
||||||
- `clawdis-mac ui permissions status`
|
|
||||||
- `clawdis-mac ui frontmost`
|
|
||||||
- `clawdis-mac ui apps`
|
|
||||||
- `clawdis-mac ui windows [--bundle-id <id>]`
|
|
||||||
- `clawdis-mac ui screenshot [--screen-index <n>] [--bundle-id <id>] [--window-index <n>] [--watch] [--scale native|1x]`
|
|
||||||
- `clawdis-mac ui see [--bundle-id <id>] [--window-index <n>] [--snapshot-id <id>]`
|
|
||||||
- `clawdis-mac ui click --on <elementId> [--bundle-id <id>] [--snapshot-id <id>] [--double|--right]`
|
|
||||||
- `clawdis-mac ui type --text <value> [--into <elementId>] [--bundle-id <id>] [--snapshot-id <id>] [--clear] [--delay-ms <n>]`
|
|
||||||
- `clawdis-mac ui wait --on <elementId> [--bundle-id <id>] [--snapshot-id <id>] [--timeout <sec>]`
|
|
||||||
|
|
||||||
## Next integration steps (after this doc)
|
## Next integration steps (after this doc)
|
||||||
1. Add Peekaboo as a git submodule (nested submodules OK).
|
1. Add Peekaboo as a git submodule (nested submodules OK).
|
||||||
2. Add a small `clawdis-mac ui …` surface that speaks PeekabooBridge (text by default, `--json` for structured).
|
2. Host `PeekabooBridgeHost` inside Clawdis.app behind a single setting (“Enable Peekaboo Bridge”, default on).
|
||||||
3. Host `PeekabooBridgeHost` inside Clawdis.app behind a single setting (“Enable Peekaboo Bridge”, default on).
|
3. Ensure Clawdis hosts the bridge at `~/Library/Application Support/clawdis/bridge.sock` and speaks the PeekabooBridge JSON protocol.
|
||||||
4. Implement the minimum operation set needed for agents (see/click/type/scroll/wait/screenshot, plus list apps/windows/screens).
|
4. Validate with `peekaboo bridge status --verbose` that Peekaboo can select Clawdis as the fallback host (no auto-launch).
|
||||||
5. Keep all protocol decisions aligned with Peekaboo (coordinate system, element IDs, snapshot scoping, error envelopes).
|
5. Keep all protocol decisions aligned with Peekaboo (coordinate system, element IDs, snapshot scoping, error envelopes).
|
||||||
|
|
|
||||||
Loading…
Reference in New Issue