>> Break the Sandbox: OpenClaw + AppleScript Mac Desktop Control

OpenClaw AppleScript Mac automation gives a text-only agent gateway a physical body on the macOS desktop—shell tool, bridge script, and native app control in one logged-in session.

Introduction

Most teams run OpenClaw as a text-only operator: gateway channels, HTTP tools, and shell snippets that never touch the graphical session. That leaves a gap—native Mac apps (Keynote, Mail, Music, System Settings) live behind Apple’s GUI automation layer, not inside a headless SSH pipe.

OpenClaw AppleScript Mac automation closes that gap with a deliberate bridge pattern: OpenClaw’s native shell tool calls a small bash entrypoint, which dispatches AppleScript (osascript) or Automator workflows saved as .workflow / application bundles. The agent still plans in natural language; macOS still executes UI actions in the logged-in user session.

This is not “breaking” Apple’s sandbox in a security sense—you grant Automation and Accessibility permissions explicitly. You are giving OpenClaw a physical body on the Mac desktop: open Keynote, export a PDF, nudge volume, dim the display—while chat replies continue on Telegram or Slack.

Pair this guide with OpenClaw + Ollama on a cloud Mac mini when you want local models for planning and macOS automation for execution. Complete OpenClaw light deploy first so gateway health, launchd, and disk watermarks are stable before enabling GUI tools.

According to Apple’s macOS automation documentation, AppleScript and Automator are the supported bridges between scripts and scriptable applications.

Architecture: shell bridge → AppleScript / Automator

Data flow

Channel message → OpenClaw planner → shell tool "mac_desktop" → ~/bin/openclaw-mac-bridge.sh (JSON args) → osascript (inline .scpt) OR open MyWorkflow.app → Target app (Keynote, System Events, …) → stdout JSON { "ok": true, "artifact": "/path/file.pdf" }

Component table

Session rule: GUI automation requires a logged-in macOS user with a graphical session (local console, or screen sharing on a cloud Mac). SSH-only sessions without GUI cannot drive Keynote.

Why not pure shell?

Eight-step runbook

Step 1 — Baseline gateway and GUI session

ssh user@your-mac openclaw --version curl -s http://127.0.0.1:11430/health whoami ls /dev/console # should exist when GUI session active

Pass criteria: gateway HTTP 200, boot volume ≥25GB free (same as 72-hour guardrails).

Step 2 — Create bridge directories

mkdir -p ~/bin mkdir -p "$HOME/Library/Application Support/OpenClaw/scripts" mkdir -p "$HOME/Library/Application Support/OpenClaw/workflows" mkdir -p ~/.openclaw/logs chmod 700 ~/bin/openclaw-mac-bridge.sh 2>/dev/null || true

Step 3 — Install the bridge script

Save ~/bin/openclaw-mac-bridge.sh:

#!/usr/bin/env bash set -euo pipefail LOG="$HOME/.openclaw/logs/mac-desktop.log" ACTION="${1:-}" ARG_JSON="${2:-{}}" ts() { date -u +"%Y-%m-%dT%H:%M:%SZ"; } case "$ACTION" in keynote_export_pdf) DECK="${ARG_JSON:-}" OUT="${3:-$HOME/Desktop/openclaw-export.pdf}" osascript "$HOME/Library/Application Support/OpenClaw/scripts/keynote_export_pdf.applescript" "$DECK" "$OUT" echo "{\"ok\":true,\"artifact\":\"$OUT\"}" ;; set_volume) LEVEL="${ARG_JSON:-50}" osascript -e "set volume output volume $LEVEL" echo "{\"ok\":true,\"volume\":$LEVEL}" ;; set_brightness) LEVEL="${ARG_JSON:-0.5}" osascript "$HOME/Library/Application Support/OpenClaw/scripts/set_brightness.applescript" "$LEVEL" echo "{\"ok\":true,\"brightness\":$LEVEL}" ;; run_automator) WF="${ARG_JSON:-}" open -a "$WF" echo "{\"ok\":true,\"workflow\":\"$WF\"}" ;; *) echo "{\"ok\":false,\"error\":\"unknown_action\"}" >&2 exit 2 ;; esac >>"$LOG" 2>&1

chmod 750 ~/bin/openclaw-mac-bridge.sh

Step 4 — Add AppleScript helpers

keynote_export_pdf.applescript:

on run argv set deckPath to item 1 of argv set outPath to item 2 of argv tell application "Keynote" activate set docRef to open POSIX file deckPath export docRef to POSIX file outPath as PDF close docRef saving no end tell end run

set_brightness.applescript (uses System Events; requires Accessibility):

on run argv set level to item 1 of argv as number tell application "System Events" tell appearance preferences set brightness to level end tell end tell end run

Smoke test manually:

osascript "$HOME/Library/Application Support/OpenClaw/scripts/set_volume.applescript" 40 # or inline: osascript -e 'set volume output volume 40'

Step 5 — Build an Automator workflow (optional)

1. Open Automator → New Quick Action or Application.
2. Add Run AppleScript → paste Keynote export steps.
3. Save as ExportKeynotePDF.workflow under ~/Library/Application Support/OpenClaw/workflows/.
4. Test: open -a "$HOME/Library/Application Support/OpenClaw/workflows/ExportKeynotePDF.workflow"

Register in the bridge as run_automator when sequences exceed ~30 lines of AppleScript.

Step 6 — Register OpenClaw shell tool

Add under ~/.openclaw/config/tools/ (exact schema varies by build):

{ "name": "mac_desktop", "description": "Control native Mac apps via AppleScript/Automator. Actions: keynote_export_pdf, set_volume, set_brightness, run_automator. Args: action string, JSON payload, optional third path.", "type": "shell", "command": "/Users/OPERATOR/bin/openclaw-mac-bridge.sh", "timeout_seconds": 120, "allowed_actions": ["keynote_export_pdf", "set_volume", "set_brightness", "run_automator"] }

Replace OPERATOR with the macOS account running the gateway. Keep the tool disabled in lab until Step 7 passes.

Step 7 — Author SKILL.md policy

Create ~/.openclaw/skills/mac-desktop/SKILL.md:

# Mac Desktop Automation (AppleScript) ## When to use - User asks to export slides, adjust volume/brightness, or open a native app. - Channel message includes `/deck` or `export pdf` with a file path on the Mac. ## Never - Delete files outside ~/OpenClawExports without human confirmation. - Change System Settings security/privacy panes. - Run Automator workflows not listed in allowed_actions. ## Steps 1. Confirm deck path exists: `test -f path`. 2. Call mac_desktop with keynote_export_pdf. 3. Reply with artifact path and file size from `stat -f%z`.

Restart gateway after tool + skill changes.

Step 8 — Channel smoke with guardrails

1. Send a test message: “Set volume to 35%.”
2. Send: “Export /Users/you/Decks/Q2.key to PDF.”
3. Verify ~/.openclaw/logs/mac-desktop.log shows exit 0 and JSON stdout.

Enable one messaging channel only—see gateway channels. Cap concurrent desktop actions to 1 on 16GB hosts to avoid Keynote + heavy model contention.

Troubleshooting

Error: Not authorized to send Apple events to Keynote

Pattern: osascript exit 1, log contains Not authorized.

Fix: On the Mac → System Settings → Privacy & Security → Automation → allow osascript (or Terminal/iTerm) to control Keynote. Re-run Step 4 smoke test. For cloud Macs, perform once per snapshot image.

Error: GUI scripting is not allowed

Pattern: brightness script fails; System Events error -25211.

Fix: Grant Accessibility to Terminal/ssh session parent per Apple’s accessibility guide. Prefer lowering brightness via brightness only during interactive hours—document policy in SKILL.md.

Operational implications

Serialize desktop jobs with Ollama: do not run 14B local model + Keynote export concurrently on 16GB unified memory.

FAQ

Can OpenClaw control Mac apps without AppleScript?
Only partially. Shell tools handle CLI binaries; native GUI apps need AppleScript, Automator, or Shortcuts. The bridge pattern keeps OpenClaw’s planner unchanged while macOS executes UI work.

Does AppleScript work over SSH alone?
No. osascript targets the logged-in graphical session. Use console access, or screen sharing on a remote Mac, and run the gateway under that same user.

How do I trigger Automator from OpenClaw?
Register run_automator in the bridge script and call open -a /path/to/Workflow.app. Pass arguments via a plist or a sidecar JSON file the workflow reads in its first action.

Is this safe for production agents?
Treat desktop tools as high privilege. Restrict allowed_actions, log every invocation, and require human approval for destructive paths. Follow OpenClaw security for localhost gateway binds.

Will this work on Apple Silicon Mac mini M4?
Yes. AppleScript and Automator are architecture-neutral on macOS 14+. RAM pressure from Keynote + OpenClaw + Ollama is the practical limiter—keep one heavy job at a time on 16GB.

How does SKILL.md relate to AppleScript?
SKILL.md teaches when the planner invokes mac_desktop; AppleScript/Automator defines how macOS performs the action. Update SKILL.md whenever you add a new workflow file.

Conclusion

OpenClaw AppleScript Mac automation turns a text gateway into a desktop operator: shell tool → bridge script → osascript / Automator → native apps. The pattern is low competition because most tutorials stop at chat replies.

Start with volume + Keynote PDF smoke tests, harden permissions once per machine, then wire channel intents in SKILL.md. Keep promotion out of the hot path—this is an engineering bridge, not a rental pitch.

Related Articles

• GPT-5.6 Predictions: Codex Open Source for Agent Efficiency
• OpenClaw + Ollama on a cloud Mac mini
• OpenClaw light deploy
• OpenClaw security and networking
• Gateway channels and rate limits
• 72-hour guardrails