DSH Desktop — a browser and its annotations, beside your agent

Two pieces that meet on loopback:

| piece | what it is | | --- | --- | | this repo's root | the DSH plugin (dsh-browser-annotate): the agent's twelve browser tools, one dsh plugin command to install. It talks to the app over http://127.0.0.1:7391 — no service is shared, nothing is loaded into DSH beyond the plugin itself. | | desktop/ | an Electron app. Left pane: your DSH web UI, started by you, in its own tab. Right pane: real Chromium, any number of tabs. |

The DSH in the left pane is yours: you run it in your terminal (dsh web --port 3090), copy the tokenised URL it prints, and paste it into the app. The app never starts or owns DSH.

Run

./dev.sh 'http://127.0.0.1:3090/?token=…'   # first start, or to change the conversation URL
./dev.sh                                      # every later start; the URL persists in .dev-url

dev.sh kills only this app's Electron (other Electron apps and your dsh web are untouched), starts the app, waits for the annotation bridge, and prints the log path (/tmp/dsh-desktop.log).

To build a macOS disk image instead of running from source:

cd desktop && npm install && npm run dist    # unsigned DSH-<version>-arm64.dmg in desktop/dist/

The DMG is unsigned: the first launch needs right-click → Open (or System Settings → Privacy & Security → Open Anyway), after which it opens normally.

The window

• Left pane — one tab, the conversation. It is guarded: nothing can navigate it away from DSH, because a conversation moved off its URL is a conversation lost.
• Right pane — as many tabs as you like. The address bar takes any URL. ⌘T new tab, ⌘W close, ⌘L address, ⌘R reload, ⌘[/⌘] back/forward — in whichever pane the pointer is.
• The app always opens a Chrome DevTools Protocol port for itself; it is published as debugPort in GET /health, which is how the plugin's Playwright finds it.

Annotating

The gesture the app exists for:

• Quick — right-click anything → Add comment to this element. A comment box opens anchored to the element; type, press Enter. The element is recorded during the right-click itself, so pages that fight for their own events still annotate.
• Inspect — right-click → Inspect this element. Hovering walks the DOM with a DevTools-style panel (selector, size, position, computed styles); ↑/↓ walk to the parent/first child and move the selection with them; Enter annotates the selected node.

Every annotation carries the comment plus exact frontend facts: CSS selector, DOM path, tag, role, accessible name, bounding rect, computed styles, element HTML, page URL/title, and a screenshot cropped to the element.

Annotations queue rather than fire: the toolbar shows what is waiting, you can drop any of them, and Add to input delivers the whole batch into the conversation's composer — formatted text plus each screenshot as a real attachment, entering through the same events a human's typing and file-drop produce. Any number can accumulate first.

The agent reads the same queue through its annotations tool: list reports it with the screenshot attached to the result, inject additionally delivers it into the session as a context block, clear removes it once you have taken responsibility for the batch.

The agent's twelve tools

| tool | what it does | | --- | --- | | openBrowserPage | open a URL in a new tab (and refuse, with the real error, when the host cannot be reached) | | navigatePage | url / reload / back / forward | | readPage | accessibility-tree snapshot of a tab, optionally scoped to a selector | | screenshotPage | viewport or element crop, attached as an image | | clickElement / hoverElement / typeInPage | act on elements by @ref or selector; typing can submit | | dragElement | real drag via the engine, falling back to Playwright's dragAndDrop, reporting which ran | | handleDialog | report / accept / dismiss alert/confirm/prompt (the app no longer auto-answers) | | runPlaywrightCode | a real Playwright function over CDP against the app's Chromium — the honest route for anything bespoke | | cdp | any Chrome DevTools Protocol method against any tab | | annotations | the human's annotated feedback, as above |

Two engines answer this vocabulary: the desktop app (HTTP POST /rpc) and, if you attach the Chrome extension from extension/, that Chrome too. When both are attached the extension wins. Playwright exists only through the app — an extension exposes no CDP port — and runPlaywrightCode says so.

Chrome DevTools Protocol

cdp forwards any method to any tab (Input, DOM, Runtime, Network, Emulation, …). The app's own port is a normal Chromium one, so the usual client libraries work; Electron does not implement Target.createTarget or Target.createBrowserContext, and the tools say so rather than guessing — open tabs with openBrowserPage instead.

Installing the plugin

The plugin is a normal profile dependency of DSH, and this repository installs directly from GitHub — there is no npm package:

# install (or reinstall at the latest commit)
dsh plugin --profile web add github:iimaguest/dsh-browser-annotate

# update to the newest pushed commit
dsh plugin --profile web update dsh-browser-annotate

# remove — dependency and bundle layer together; the profile's own
# cordis.patch.yml is never touched
dsh plugin --profile web remove dsh-browser-annotate

Each command takes effect the next time DSH starts.

What that command installs: a package whose package.json declares itself a DSH plugin — dsh.bundle.patch points at cordis.patch.yml (one insert row carrying the host half: service, HTTP routes, tools) and dsh.client.inject mounts the browser sidebar into the web client. host/ is the DSH-side half, client/ the web-client half, extension/ an optional Chrome MV3 extension that can act as a second engine.

For developing on a checkout of this repo instead, the same wiring can be pointed at the working tree: symlink the profile's node_modules/dsh-browser-annotate at your checkout (or dsh plugin --profile web add <path>), and run npm run link-peers in the repo to re-link the @deepseek-ai/* peers if an install ever wipes them. Every edit is live the next time DSH starts.

Each half without the other

Neither piece assumes the other exists:

• Plugin without the app — DSH starts and works normally; the twelve tools register and answer with a plain "the desktop browser is not attached" (or the Chrome-extension equivalent) instead of hanging or crashing. Start the app and the next call works.
• App without the plugin — the app is a plain two-pane browser: tabs, address bar, annotations, the queue, delivery into the conversation's composer. It never calls DSH; it only answers HTTP on loopback, so nothing degrades when no plugin is installed anywhere.

What has been verified live

Against the real app and a real DSH, not fixtures:

• the twelve tools driving real pages, including a failed DNS lookup refused with Chromium's own error, scoped readPage, dialogs that genuinely block, and a drag verified by its drop effect;
• quick annotate from a trusted right-click: recorded element → composer → comment → queue with screenshot, selector, path, role, rect, styles, HTML;
• inspect annotate: hover walk, arrow keys moving the selection, Enter annotating the walked-to node (h1, not the hovered p);
• two annotations delivered into the conversation's composer as formatted text plus blob image attachments, then removed through their real chip buttons;
• the annotations tool against a populated queue from a real session: list with the screenshot attached to the result, inject delivering the context block, queue preserved.

Limits, stated plainly

• Closing a tab is a human affordance (⌘W, the tab strip); no tool verb closes one yet.
• Target.createTarget / Target.createBrowserContext are not implemented by Electron's CDP endpoint.
• The Chrome extension engine cannot serve runPlaywrightCode; only the app can.