chore: bump Peekaboo submodule

main
Peter Steinberger 2025-12-13 23:02:04 +00:00
parent ece8a3e701
commit 0d3aacd316
3 changed files with 35 additions and 61 deletions

@ -1 +1 @@
Subproject commit 3c5576b271be0ccbd1e42947fa4c655a394db8ff Subproject commit 4807e6fdf016f013dba8a32fa4b2528182539fad

View File

@ -14,7 +14,7 @@ Author: steipete · Status: draft spec · Date: 2025-12-05
- Owns TCC prompts (Notifications, Accessibility, Screen Recording, Automation/AppleScript, Microphone, Speech Recognition). - Owns TCC prompts (Notifications, Accessibility, Screen Recording, Automation/AppleScript, Microphone, Speech Recognition).
- Brokers privileged actions via local IPC: - Brokers privileged actions via local IPC:
- Clawdis control socket (app-specific actions like notify/run) - Clawdis control socket (app-specific actions like notify/run)
- PeekabooBridge socket (`bridge.sock`) for UI automation (see `docs/mac/peekaboo.md`) - PeekabooBridge socket (`bridge.sock`) for UI automation brokering (consumed by `peekaboo`; see `docs/mac/peekaboo.md`)
- Provides a tiny CLI (`clawdis-mac`) that talks to the app; Node/TS shells out to it. - Provides a tiny CLI (`clawdis-mac`) that talks to the app; Node/TS shells out to it.
- Replace the separate notifier helper pattern (Oracle) with a built-in notifier. - Replace the separate notifier helper pattern (Oracle) with a built-in notifier.
- Offer a first-run experience similar to VibeTunnels onboarding (permissions + CLI install). - Offer a first-run experience similar to VibeTunnels onboarding (permissions + CLI install).
@ -47,7 +47,7 @@ struct Response { ok: Bool; message?: String; payload?: Data }
- The control-socket server rejects oversize/unknown cases and validates the caller by code signature TeamID (with a `DEBUG`-only same-UID escape hatch controlled by `CLAWDIS_ALLOW_UNSIGNED_SOCKET_CLIENTS=1`). - The control-socket server rejects oversize/unknown cases and validates the caller by code signature TeamID (with a `DEBUG`-only same-UID escape hatch controlled by `CLAWDIS_ALLOW_UNSIGNED_SOCKET_CLIENTS=1`).
UI automation is not part of `ClawdisIPC.Request`: UI automation is not part of `ClawdisIPC.Request`:
- `clawdis-mac ui …` speaks **PeekabooBridge** (see `docs/mac/peekaboo.md`). - UI automation is handled via the separate PeekabooBridge socket and is surfaced by the `peekaboo` CLI (see `docs/mac/peekaboo.md`).
## App UX (Clawdis) ## App UX (Clawdis)
- MenuBarExtra icon only (LSUIElement; no Dock). - MenuBarExtra icon only (LSUIElement; no Dock).
@ -71,15 +71,7 @@ UI automation is not part of `ClawdisIPC.Request`:
- Subcommands (text by default; `--json` for machine output; non-zero exit on failure): - Subcommands (text by default; `--json` for machine output; non-zero exit on failure):
- `notify --title --body [--sound] [--priority passive|active|timeSensitive] [--delivery system|overlay|auto]` - `notify --title --body [--sound] [--priority passive|active|timeSensitive] [--delivery system|overlay|auto]`
- `ensure-permissions --cap accessibility --cap screenRecording [--interactive]` - `ensure-permissions --cap accessibility --cap screenRecording [--interactive]`
- `ui permissions status` - UI automation + capture: use `peekaboo …` (Clawdis hosts PeekabooBridge; see `docs/mac/peekaboo.md`)
- `ui frontmost`
- `ui apps`
- `ui windows [--bundle-id <id>]`
- `ui screenshot [--screen-index <n>] [--bundle-id <id>] [--window-index <n>] [--watch] [--scale native|1x]`
- `ui see [--bundle-id <id>] [--window-index <n>] [--snapshot-id <id>]`
- `ui click --on <elementId> [--bundle-id <id>] [--snapshot-id <id>] [--double|--right]`
- `ui type --text <value> [--into <elementId>] [--bundle-id <id>] [--snapshot-id <id>] [--clear] [--delay-ms <n>]`
- `ui wait --on <elementId> [--bundle-id <id>] [--snapshot-id <id>] [--timeout <sec>]`
- `run -- cmd args... [--cwd] [--env KEY=VAL] [--timeout 30] [--needs-screen-recording]` - `run -- cmd args... [--cwd] [--env KEY=VAL] [--timeout 30] [--needs-screen-recording]`
- `status` - `status`
- Sounds: supply any macOS alert name with `--sound` per notification; omit the flag to use the system default. There is no longer a persisted “default sound” in the app UI. - Sounds: supply any macOS alert name with `--sound` per notification; omit the flag to use the system default. There is no longer a persisted “default sound” in the app UI.
@ -87,14 +79,14 @@ UI automation is not part of `ClawdisIPC.Request`:
- Delivery: `overlay` and `auto` show an in-app toast panel (bypasses Notification Center/Focus). - Delivery: `overlay` and `auto` show an in-app toast panel (bypasses Notification Center/Focus).
- Internals: - Internals:
- For app-specific commands (`notify`, `ensure-permissions`, `run`, `status`): build `ClawdisIPC.Request`, send over the control socket. - For app-specific commands (`notify`, `ensure-permissions`, `run`, `status`): build `ClawdisIPC.Request`, send over the control socket.
- For UI automation (`ui …`): connect to PeekabooBridge hosts (Peekaboo.app → Clawdis.app) and send one JSON request per command (see `docs/mac/peekaboo.md`). - UI automation is intentionally not exposed via `clawdis-mac`; it lives behind PeekabooBridge and is surfaced by the `peekaboo` CLI.
## Integration with clawdis/Clawdis (Node/TS) ## Integration with clawdis/Clawdis (Node/TS)
- Add helper module that shells to `clawdis-mac`: - Add helper module that shells to `clawdis-mac`:
- Prefer `ensure-permissions` before actions that need TCC. - Prefer `ensure-permissions` before actions that need TCC.
- Use `notify` for desktop toasts; fall back to JS notifier only if CLI missing or platform ≠ macOS. - Use `notify` for desktop toasts; fall back to JS notifier only if CLI missing or platform ≠ macOS.
- Use `run` for tasks requiring privileged UI context (screen-recorded terminal runs, etc.). - Use `run` for tasks requiring privileged UI context (screen-recorded terminal runs, etc.).
- For UI automation, `clawdis ui …` is a convenience passthrough to `clawdis-mac ui …` (text by default; add `--json` to the outer `clawdis` command for structured output). - For UI automation, shell out to `peekaboo …` (text by default; add `--json` for structured output) and rely on PeekabooBridge host selection (Peekaboo.app → Clawdis.app → local).
## Deep links (URL scheme) ## Deep links (URL scheme)

View File

@ -1,15 +1,15 @@
--- ---
summary: "Plan for integrating Peekaboo automation into Clawdis via PeekabooBridge (socket-based TCC broker)" summary: "Plan for integrating Peekaboo automation into Clawdis via PeekabooBridge (socket-based TCC broker)"
read_when: read_when:
- Adding UI automation commands - Hosting PeekabooBridge in Clawdis.app
- Integrating Peekaboo as a submodule - Integrating Peekaboo as a submodule
- Changing clawdis-mac IPC/output formats - Changing PeekabooBridge protocol/paths
--- ---
# Peekaboo Bridge in Clawdis (macOS UI automation broker) # Peekaboo Bridge in Clawdis (macOS UI automation broker)
## TL;DR ## TL;DR
- **Peekaboo removed its XPC helper** and now exposes privileged automation via a **UNIX domain socket bridge** (`PeekabooBridge` / `PeekabooBridgeHost`, socket name `bridge.sock`). - **Peekaboo removed its XPC helper** and now exposes privileged automation via a **UNIX domain socket bridge** (`PeekabooBridge` / `PeekabooBridgeHost`, socket name `bridge.sock`).
- Clawdis integrates by **hosting the same bridge** inside **Clawdis.app** (optional, user-toggleable), and by making `clawdis-mac ui …` act as a **bridge client**. - Clawdis integrates by **optionally hosting the same bridge** inside **Clawdis.app** (user-toggleable). The primary client is the **`peekaboo` CLI** (installed via npm); Clawdis does not need its own `ui …` CLI surface.
- For **visualizations**, we keep them in **Peekaboo.app** (best UX); Clawdis stays a thin broker host. No visualizer toggle in Clawdis. - For **visualizations**, we keep them in **Peekaboo.app** (best UX); Clawdis stays a thin broker host. No visualizer toggle in Clawdis.
Non-goals: Non-goals:
@ -31,9 +31,8 @@ Reference (Peekaboo submodule): `docs/bridge-host.md`.
- **Peekaboo.app** (preferred; also provides visualizations + controls) - **Peekaboo.app** (preferred; also provides visualizations + controls)
- **Clawdis.app** (secondary; “thin host” only) - **Clawdis.app** (secondary; “thin host” only)
- **Bridge clients** (trigger single actions): - **Bridge clients** (trigger single actions):
- `clawdis-mac ui …` - `peekaboo …` (preferred; humans + agents)
- `clawdis ui …` (Node/TS convenience wrapper; shells out to `clawdis-mac ui …`) - Optional: Clawdis/Node shells out to `peekaboo` when it needs UI automation/capture
- Node/Gateway shells out to `clawdis-mac`
### Host discovery (client-side) ### Host discovery (client-side)
Order is deliberate: Order is deliberate:
@ -68,26 +67,22 @@ What Clawdis should *not* embed:
- **XPC**: dont reintroduce helper targets; use the bridge. - **XPC**: dont reintroduce helper targets; use the bridge.
## IPC / CLI surface ## IPC / CLI surface
### Namespacing ### No `clawdis-mac ui …`
Add new automation commands behind a `ui` prefix: We avoid a parallel “Clawdis UI automation CLI”. Instead:
- `clawdis-mac ui …` for UI automation + visualization-related actions. - `peekaboo` is the user/agent-facing CLI surface for automation and capture.
- Keep existing top-level commands (`notify`, `run`, `canvas …`, etc.) for compatibility. - Clawdis.app can host PeekabooBridge as a **thin TCC broker** so Peekaboo can piggyback on Clawdis permissions when Peekaboo.app isnt running.
Screenshot cutover: ### Diagnostics
- Remove legacy screenshot endpoints/commands. Use Peekaboos built-in diagnostics to see which host would be used:
- Ship only `clawdis-mac ui screenshot` (no aliases). - `peekaboo bridge status`
- `peekaboo bridge status --verbose`
- `peekaboo bridge status --json`
### Output format ### Output format
Change `clawdis-mac` to default to human text output: Peekaboo commands default to human text output. Add `--json` for a structured envelope.
- **Default**: plain text; errors are string messages to stderr; exit codes indicate success/failure.
- **`--json`**: structured output (for agents/scripts) with stable schemas.
This applies globally, not only `ui` commands.
Note (current state as of 2025-12-13): `clawdis-mac` prints text by default; use `--json` for structured output.
### Timeouts ### Timeouts
Default timeout for UI actions: **10 seconds** end-to-end. Default timeout for UI actions: **10 seconds** end-to-end (client enforced; host should also enforce per-operation).
## Coordinate model (multi-display) ## Coordinate model (multi-display)
Requirement: coordinates are **per screen**, not global. Requirement: coordinates are **per screen**, not global.
@ -111,18 +106,18 @@ Expose window/app targeting in the UI surface (align with Peekaboo targeting):
- by window title substring - by window title substring
- by (app, index) - by (app, index)
Current `clawdis-mac ui …` support: Peekaboo CLI targeting (agent-friendly):
- `--bundle-id <id>` for app targeting - `--bundle-id <id>` for app targeting
- `--window-index <n>` (0-based) for disambiguating within an app when capturing (see/screenshot) - `--window-index <n>` (0-based) for disambiguating within an app when capturing
All “see/click/type/scroll/wait” requests should accept a target (default: frontmost). All “see/click/type/scroll/wait” requests should accept a target (default: frontmost).
## “See” + click packs (Playwright-style) ## “See” + click packs (Playwright-style)
Behavior stays aligned with Peekaboo: Behavior stays aligned with Peekaboo:
- `ui see` returns element IDs (e.g. `B1`, `T3`) with bounds/labels. - `peekaboo see` returns element IDs (e.g. `B1`, `T3`) with bounds/labels.
- Follow-up actions reference those IDs without re-scanning. - Follow-up actions reference those IDs without re-scanning.
`clawdis-mac ui see` should: `peekaboo see` should:
- capture (optionally targeted) window/screen - capture (optionally targeted) window/screen
- return a screenshot **file path** (default: temp directory) - return a screenshot **file path** (default: temp directory)
- return a list of elements (text or JSON) - return a list of elements (text or JSON)
@ -132,9 +127,9 @@ Snapshot lifecycle requirement:
- Snapshot scoping: “implicit snapshot” is **per target bundle id** (reuse last snapshot for that app when snapshot id is omitted). - Snapshot scoping: “implicit snapshot” is **per target bundle id** (reuse last snapshot for that app when snapshot id is omitted).
Practical flow (agent-friendly): Practical flow (agent-friendly):
- `clawdis-mac ui frontmost` returns the focused app (bundle id) + focused window (title/id) so follow-up calls can pass `--bundle-id …`. - `peekaboo list apps` / `peekaboo list windows` provide bundle-id context for targeting.
- `clawdis-mac ui see --bundle-id X` updates the implicit snapshot for `X`. - `peekaboo see --bundle-id X` updates the implicit snapshot for `X`.
- `clawdis-mac ui click --bundle-id X --on B1` reuses the most recent snapshot for `X` when `--snapshot-id` is omitted. - `peekaboo click --bundle-id X --on B1` reuses the most recent snapshot for `X` when `--snapshot-id` is omitted.
## Visualizer integration ## Visualizer integration
Keep visualizations in **Peekaboo.app** for now. Keep visualizations in **Peekaboo.app** for now.
@ -142,11 +137,11 @@ Keep visualizations in **Peekaboo.app** for now.
- Any “visualizer enabled/disabled” setting is controlled in Peekaboo.app. - Any “visualizer enabled/disabled” setting is controlled in Peekaboo.app.
## Screenshots (legacy → Peekaboo takeover) ## Screenshots (legacy → Peekaboo takeover)
Clawdis uses `clawdis-mac ui screenshot` and returns a file path (default location: temp directory) instead of raw image bytes. Clawdis should not grow a separate screenshot CLI surface.
Migration plan: Migration plan:
- Bridge host performs capture and returns a temp file path. - Use `peekaboo capture …` / `peekaboo see …` (returns a file path, default temp directory).
- No legacy aliases; make the old screenshot surface disappear cleanly. - Once Clawdis legacy screenshot plumbing is replaced, remove it cleanly (no aliases).
## Permissions behavior ## Permissions behavior
If required permissions are missing: If required permissions are missing:
@ -164,22 +159,9 @@ Debug-only escape hatch (development convenience):
- “allow same-UID callers” means: *skip codesign checks for clients running under the same Unix user*. - “allow same-UID callers” means: *skip codesign checks for clients running under the same Unix user*.
- This must be **opt-in**, **DEBUG-only**, and guarded by an env var (Peekaboo uses `PEEKABOO_ALLOW_UNSIGNED_SOCKET_CLIENTS=1`). - This must be **opt-in**, **DEBUG-only**, and guarded by an env var (Peekaboo uses `PEEKABOO_ALLOW_UNSIGNED_SOCKET_CLIENTS=1`).
## Current `clawdis-mac ui` commands (Dec 2025)
All commands default to text output. Add `--json` right after `clawdis-mac` for a structured envelope.
- `clawdis-mac ui permissions status`
- `clawdis-mac ui frontmost`
- `clawdis-mac ui apps`
- `clawdis-mac ui windows [--bundle-id <id>]`
- `clawdis-mac ui screenshot [--screen-index <n>] [--bundle-id <id>] [--window-index <n>] [--watch] [--scale native|1x]`
- `clawdis-mac ui see [--bundle-id <id>] [--window-index <n>] [--snapshot-id <id>]`
- `clawdis-mac ui click --on <elementId> [--bundle-id <id>] [--snapshot-id <id>] [--double|--right]`
- `clawdis-mac ui type --text <value> [--into <elementId>] [--bundle-id <id>] [--snapshot-id <id>] [--clear] [--delay-ms <n>]`
- `clawdis-mac ui wait --on <elementId> [--bundle-id <id>] [--snapshot-id <id>] [--timeout <sec>]`
## Next integration steps (after this doc) ## Next integration steps (after this doc)
1. Add Peekaboo as a git submodule (nested submodules OK). 1. Add Peekaboo as a git submodule (nested submodules OK).
2. Add a small `clawdis-mac ui …` surface that speaks PeekabooBridge (text by default, `--json` for structured). 2. Host `PeekabooBridgeHost` inside Clawdis.app behind a single setting (“Enable Peekaboo Bridge”, default on).
3. Host `PeekabooBridgeHost` inside Clawdis.app behind a single setting (“Enable Peekaboo Bridge”, default on). 3. Ensure Clawdis hosts the bridge at `~/Library/Application Support/clawdis/bridge.sock` and speaks the PeekabooBridge JSON protocol.
4. Implement the minimum operation set needed for agents (see/click/type/scroll/wait/screenshot, plus list apps/windows/screens). 4. Validate with `peekaboo bridge status --verbose` that Peekaboo can select Clawdis as the fallback host (no auto-launch).
5. Keep all protocol decisions aligned with Peekaboo (coordinate system, element IDs, snapshot scoping, error envelopes). 5. Keep all protocol decisions aligned with Peekaboo (coordinate system, element IDs, snapshot scoping, error envelopes).