← Gautam Parab

How I Built an Autonomous Desktop AI Companion Using AI Agents

So, I had this idea: I wanted to build an autonomous desktop AI companion. I didn’t want just another chatbot stuck in a browser tab. I wanted something that actually lived on my OS, could see what I was doing on web apps, and could literally take over my mouse to automate tasks for me.

But I quickly realized that building this was going to be a cross-stack nightmare. I needed low-level OS hooks to control the mouse and capture audio, local AI inference so I wasn’t beaming my private screen data to the cloud, and a way to securely inject UI overlays into web apps.

Here is how I architected the solution, and how I actually managed to write the code for it by teaming up with autonomous AI coding agents.

The Architecture: Three Layers of Chaos

To make this work without freezing up the user’s computer, I had to separate the heavy lifting from the UI. I ended up splitting the system into three main layers:

  1. The Native Core (Rust): I needed something insanely fast, so I went with Rust. This layer talks directly to the OS. It handles hardware-accelerated local AI inference (running Speech-to-Text and a Small Language Model right on the GPU). It also manages a constant audio ring-buffer so it’s always listening, and it calculates the Bezier curves needed to make the synthetic mouse movements look like a real human is moving the cursor.
  2. The IPC Bridge: This is the nervous system. It’s a secure message bus that shuttles events (like microphone state or AI text tokens) between my Rust backend and the web frontend.
  3. The Web Overlay (Injected DOM): I wrote a JavaScript payload that gets injected right into whatever web app the user is looking at. This renders the floating command palette and the “ghost cursor”. More importantly, it scans the web page’s accessibility tree and slaps numerical markers on everything you can click. I call this the “Set-of-Marks” (SoM). It’s how the AI actually “sees” the screen to know exactly what X/Y coordinates to click.

Here is what the architecture looks like:

Three-layer architecture of the desktop companion Voice is captured by an audio ring buffer in the Rust native core and passed to local GPU inference. Inference streams tokens down to an async event bus in the IPC layer, which sends render commands to DOM indexing in the injected web overlay. The marks flow through the floating palette to the ghost cursor, whose target coordinates travel back up to Bezier mouse actuation in the native core. NATIVE CORE · RUST IPC BRIDGE INJECTED WEB OVERLAY · JS voice tokens render cmds marks intent target x/y Audio Ring Buffer 750ms pre-roll · hotkeys Local GPU Inference whisper + slm · on device Mouse Actuation bezier curves Async Event Bus json payloads DOM Indexing set-of-marks · a11y tree Command Palette floating ui · hud Ghost Cursor autocomplete preview
Voice enters at the top left and never leaves the machine; only coordinates travel back up.

Solving the Hard Problems

To make it actually feel autonomous, I had to solve a few really annoying problems:

The 750ms audio pre-roll around the hotkey A timeline in two parts. To the left of the hotkey, a rolling 750 millisecond buffer already holds the words spoken before the key was pressed. To the right, audio captured after the hotkey streams to local transcription. The hotkey sits on the boundary between them. AUDIO TIMELINE HOTKEY The 750ms you already said ring buffer · overwritten forever Everything after the hotkey streamed to whisper on the metal gpu
The buffer is always running, so the hotkey never marks the start of the sentence — only the moment you decided to send it.

Building it Agentically (Or: How I Didn’t Write All This Code Myself)

Let’s be real — writing a multi-threaded Rust backend, an IPC layer, and an injected TypeScript frontend as a solo dev would take months. So, I decided to build it agentically.

Instead of writing every line myself, I acted as the system architect. I wrote the blueprints and spun up a team of AI coding agents to do the heavy lifting in parallel.

How the coding agents were directed A human architect writes the blueprint and hands it to a planner agent, which assigns Rust work to a native systems agent, TypeScript work to a frontend agent, and contract enforcement to an integration QA agent. The systems and frontend agents negotiate the IPC contract directly with each other, and all three commit into the same codebase. blueprint ipc contract Me human architect Planner Agent decomposes the blueprint Systems Agent rust · os hooks Frontend Agent typescript · dom QA Agent enforces the contract Codebase one repo
The QA agent exists because the other two negotiate a contract neither of them owns.

Whenever the Systems Agent changed how a Rust struct was serialized, the QA agent would catch the breaking change over the IPC bridge and tell the Frontend Agent to update the TypeScript interfaces. It was like having a massive engineering team working at lightspeed.

Building an agent by using other agents was a wild experience, but honestly? It’s the only way I want to ship complex software from now on.