How I Built an Autonomous Desktop AI Companion Using AI Agents

The feature I wanted is the one that sounds like a malfunction: my mouse moving on its own, to the right button, while I sit there not touching it.

You cannot build that in a browser tab. A chatbot in a tab reads what you paste into it and writes back; it cannot see the web app you are actually working in, and it certainly cannot click anything. To get what I wanted — something that lived on my OS, could see what I was doing on web apps, and could literally take over my mouse to automate tasks for me — I needed three things a tab will never hand you: low-level OS hooks to control the mouse and capture audio, local AI inference so I wasn’t beaming my private screen data to the cloud, and a way to securely inject UI overlays into web apps. Each of those wants to live somewhere different, which is the whole reason the system has the shape it does.

Three layers, because nothing wanted to share

To make this work without freezing up the user’s computer, I had to separate the heavy lifting from the UI. I ended up splitting the system into three main layers:

  1. The Native Core (Rust): I needed something insanely fast, so I went with Rust. This layer talks directly to the OS. It handles hardware-accelerated local AI inference (running Speech-to-Text and a Small Language Model right on the GPU). It also manages a constant audio ring-buffer so it’s always listening, and it calculates the Bezier curves needed to make the synthetic mouse movements look like a real human is moving the cursor.
  2. The IPC Bridge: This is the nervous system. It’s a secure message bus that shuttles events (like microphone state or AI text tokens) between my Rust backend and the web frontend.
  3. The Web Overlay (Injected DOM): A JavaScript payload that gets injected right into whatever web app the user is looking at. It renders the floating command palette and the “ghost cursor”. More importantly, it scans the web page’s accessibility tree and slaps numerical markers on everything you can click. I call this the “Set-of-Marks” (SoM). It’s how the AI actually “sees” the screen to know exactly what X/Y coordinates to click.
Three-layer architecture of the desktop companion Voice is captured by an audio ring buffer in the Rust native core and passed to local GPU inference. Inference streams tokens down to an async event bus in the IPC layer, which sends render commands to DOM indexing in the injected web overlay. The marks flow through the floating palette to the ghost cursor, whose target coordinates travel back up to Bezier mouse actuation in the native core. NATIVE CORE · RUST IPC BRIDGE INJECTED WEB OVERLAY · JS voice tokens render cmds marks intent target x/y Audio Ring Buffer 750ms pre-roll · hotkeys Local GPU Inference whisper + slm · on device Mouse Actuation bezier curves Async Event Bus json payloads DOM Indexing set-of-marks · a11y tree Command Palette floating ui · hud Ghost Cursor autocomplete preview
Voice enters at the top left and never leaves the machine; only coordinates travel back up.

The two problems that earned the native core

Neither survives a round trip to a server, which is why there is a Rust layer at all.

The 750ms audio pre-roll around the hotkey A timeline in two parts. To the left of the hotkey, a rolling 750 millisecond buffer already holds the words spoken before the key was pressed. To the right, audio captured after the hotkey streams to local transcription. The hotkey sits on the boundary between them. AUDIO TIMELINE HOTKEY The 750ms you already said ring buffer · overwritten forever Everything after the hotkey streamed to whisper on the metal gpu
The buffer is always running, so the hotkey never marks the start of the sentence — only the moment you decided to send it.

Who wrote it

A multi-threaded Rust backend, an IPC layer, and an injected TypeScript frontend is months of work for a solo dev. I did not have months, so I built it agentically.

Instead of writing every line myself I acted as the system architect: I wrote the blueprints and spun up a team of AI coding agents to do the heavy lifting in parallel. A Systems Agent lived in the Rust codebase, wiring up the OS hooks. A Frontend Agent built out the isolated DOM injection scripts. And a QA Agent did nothing but enforce the IPC contracts between the two.

How the coding agents were directed A human architect writes the blueprint and hands it to a planner agent, which assigns Rust work to a native systems agent, TypeScript work to a frontend agent, and contract enforcement to an integration QA agent. The systems and frontend agents negotiate the IPC contract directly with each other, and all three commit into the same codebase. blueprint ipc contract Me human architect Planner Agent decomposes the blueprint Systems Agent rust · os hooks Frontend Agent typescript · dom QA Agent enforces the contract Codebase one repo
The QA agent exists because the other two negotiate a contract neither of them owns.

That third one is the agent I would not have thought to staff with humans. Whenever the Systems Agent changed how a Rust struct was serialized, the QA agent caught the breaking change over the IPC bridge and told the Frontend Agent to update the TypeScript interfaces. The seam between two specialists is where work rots, and neither of them owns it. Giving that seam its own agent was the closest I got to an engineering team rather than three fast strangers.

Building an agent out of other agents was a strange few weeks, and on the evidence so far it is how I intend to ship complex software from here on.