herds

the problemReal macOS is hard to get on demand

Anything that needs real macOS, like Xcode, simulators, codesigning, notarization, Safari or native apps, is hard to run on demand. Apple's licensing means nobody rents dense macOS capacity the way Modal or Fly rent Linux, while most developers already have a Mac sitting idle twenty hours a day.

SSH is the obvious answer, and it falls short for ordinary reasons. It gives you a shell without any of the things a runtime needs, and the machine is usually behind NAT anyway.

the ideaThe Mac connects out, and the work comes back over that connection

The Mac opens one outbound WebSocket to a control plane and never listens for anything. Self-hosted GitHub runners, Cloudflare Tunnel and Temporal workers do the same, for the same reason, and that one decision shapes the rest of the design.

The cost is that the socket is stateful, so the control plane can't scale by adding processes behind a load balancer. Every frame carries a request_id, and one connection carries exec, filesystem RPC, HTTP proxying and raw TCP.

control plane

FastAPI. holds the Mac sockets, dispatches exec and fans out logs

daemon

an asyncio process on the Mac with one outbound WebSocket, which runs the work

sdk + cli

synchronous Python. httpx to start a job, websockets to stream it

relay

gives a host a stable subdomain without opening any inbound port

how it worksFrom mac.run() to a process on the Mac

The control plane stores very little: machines, keys and job rows in SQLite. Volumes, sandboxes, images and all build output stay on the Mac. Its Hub buffers every frame per request and replays it to late subscribers, so a client that connects 100 ms in misses nothing. The fan-out is an in-memory map from request_id to asyncio queues with exactly two methods, which is where Redis would go once there is a second process.

macOS has no cgroups, so each sandbox is isolated in four layers instead. It gets its own directory tree with HOME and TMPDIR pointed into it, an environment built from an allowlist, its own caches, and a new process session per command so killpg stops everything it started. sandbox-exec limits writes to that tree plus any mounted volumes, and network access is a toggle. The caches have to move because two Xcode builds sharing DerivedData corrupt each other in a way that looks like a compiler bug. xcode-select --switch is avoided for a similar reason: it's global, and would repoint a job running at the same time.

the hard partsBugs I only found by running a real fleet

Every interesting bug came from something disappearing at the wrong moment. I found all of them by running two Macs against one account for days.

In each of the three cases below, the fix was turning a hang into an error, since an error at least says what went wrong. When the socket drops mid-job, for example, the control plane now fails every in-flight job with a synthetic exit -1, so the caller gets an error result and isn't left blocked on a dead read.

retry policy

retry forever after a dropped link, never after a rejected token, since that fails the same way every time

64KB reads

readline() fails on a 1.3MB base64 screenshot and takes the log pump down with it

admission

8 concurrent jobs, 32 queued behind them, exit 75 beyond that, plus a cpu high-water mark

reapers

idle sessions at 30 min, sandbox trees at 24h, and a machine with no metrics for 60s is marked offline

the surfaceWhat a caller gets

The SDK has 107 public methods and is synchronous on purpose. User code stays ordinary blocking Python, and all the asyncio lives in the control plane and the daemon.

Sending real mouse and keyboard events took a workaround. macOS no longer ships PyObjC with the system Python, so the input driver is a ctypes script against CoreGraphics, kept compatible with Python 3.9 because that's what every Mac has. AppleScript can type, but it can't move the mouse to a coordinate.

run, stream, map

a blocking Result, a generator of chunks, or one function run across the fleet

48 cli commands

connect, run, shell, machines, logs, volume, token, schedule, mcp

tokens

scoped read < run < admin, revocable, and checked on every route

SKILL.md

ships in the package, and a test checks every call in it against the live SDK

the stackWhat runs where, and why

FastAPI and websockets instead of gRPC. Bidirectional gRPC is the better long-term protocol and the documented upgrade path, but a browser dashboard and a pip install with no build step both want plain HTTP.

The daemon installs as a launchd LaunchAgent and runs as the user, so it has their toolchains, signing identities and keychain, which a root daemon wouldn't. inherit_home=True is an explicit opt-out of the sandbox for tools like git and gh, so leaving the sandbox is always a visible choice in the code.

where it standsWhat's shipped, and what's still wrong

It's on PyPI at 0.9.12 and works end to end. A remote agent with only a URL and a token can build an iOS app, boot a simulator, expose a port and stream the logs back. Many of the 369 tests are regressions written after a real two-Mac failure, with the observed symptom in the docstring. The code is 14,186 lines of Python and 9,916 lines of TypeScript for the dashboard.

Isolation here confines each job to its workspace, and it isn't built to contain a hostile one. Real multi-tenancy would need Tart VMs, and Apple's EULA allows only two macOS VMs per host, so VM slots would become a scarce resource the scheduler has to manage.