dsh-uia-agent - Windows UI Automation agent for DeepSeek Harness

中文 | English

A dsh plugin that registers one uia tool, letting the model enumerate windows, snapshot UI trees, find controls, click, type text, scroll/drag/swipe, screenshot, and control window state (topmost / minimize / maximize / restore / close).

• Tool exe lives in its own repo: uia-agent (C++17 / MSVC single exe; the release ships uia_agent.exe).
• This plugin embeds the exe as a base64 asset (assets/uia_agent.exe.b64) and self-releases it to the fixed location %DSH_HOME%\bin\uia_agent.exe on startup. If release fails, it console.errors and the tool call reports how to add the exe.

One-click install (beginners)

Either of these two self-contained files installs everything. Download ONE of them from the repo root or from the v1.0 release page (they do not depend on each other):

• install.cmd — double-click it on Windows
• install.ps1 — powershell -ExecutionPolicy Bypass -File install.ps1

The script checks dsh, runs dsh plugin --profile web add github:cnyc6n/dsh-uia-agent, and tells you to restart. No compile, no exe download: the plugin embeds the exe as base64 and self-releases it (hash-verified) on first start.

Install from npm

npm i -g dsh-uia-agent
# then point dsh's profile at it (or follow the one-command install below for github:)

Published versions: 1.1.0 (latest), 1.0.0 (legacy tag).

Changelog

1.2.1

• desktop gains --limit (max icons returned; 0 = all) and --name (substring filter) — see every icon without truncation

1.2.0

• desktop command: auto-find the desktop icon host (Progman + WorkerW layouts) and list all icons with screen coords
• global timeout param: every command accepts timeout (ms) forwarded to the exe
• fixed: FindDesktopIconHost only checked WorkerW (missed Progman layout)

1.1.0 (latest)

• geometry / props commands (window geometry read/write; control properties)
• drag steps / hold_ms for precise drag control
• send_keys / clipboard / wait_for / foreach commands
• snapshot inlines the control tree into model text
• screenshot returns the saved file path (model chooses how to view)
• permission model: commands at or below the session permission run directly
• fixed: raw output always an object, get_text missing-field fallback

1.0.0

• Initial standalone release: list / snapshot / snapshot_all / find / click / set_text / get_text / scroll / drag / swipe / screenshot / window state (topmost / minimize / maximize / restore / close)
• permission tiers + approval escalation, embedded exe self-release (sha256-verified)

One-command install (manual)

dsh plugin --profile web add github:cnyc6n/dsh-uia-agent

Restart dsh afterwards; the uia tool then enters the model tool set. The exe is self-released by the plugin on startup - no manual build needed.

Permission tiers & escalation

Aligned with the dsh sandbox model (same sandbox_permissions + justification

• ctx.approval surface as the pwsh tool):

Tier	sandbox_permissions	commands
read-only	none (runs directly)	list / snapshot / snapshot_all / find / get_text / screenshot
workspace-write	workspace-write	minimize / maximize / restore / topmost / close
danger-full-access	danger-full-access	click / set_text / scroll / drag / swipe / send_keys / foreach

Higher-tier commands must pass sandbox_permissions + justification; they go through user approval (approveEscalation) before execution. Missing or mismatched permission returns an error guiding the model to retry with the correct tier.

uia tool parameters

command (enum of 20 actions), hwnd, depth, query (find JSON), text, keys, set, timeout_ms, interval_ms, cmd, args, mode, x/y/x1/y1/x2/y2, duration, amount, direction, distance, out, base64, off, plus escalation params sandbox_permissions / justification.

The model gets a short summary text; the structured raw result is persisted via presentationMeta.

Tested environment

• @deepseek-ai/dsh-tools 0.1.5-rc.3 (host deps resolve via the $DSH_HOME/profiles/node_modules fallback)
• Node.js 24 (dsh requires 22+)
• Windows 10 x64 + MSVC 2022 (exe build)

Development / self-check

node smoke.mjs                     # no-LLM smoke: module load + tool def + apply
powershell -ExecutionPolicy Bypass -File scripts/install.ps1   # manual exe release

Known limits

• Self-drawn UI / games / DirectX / stock Qt expose no UIA tree (snapshot returns unsupported; fall back to screenshot + image recognition).
• Elevated windows require the exe to run as administrator, else UIPI blocks.
• Chromium/Electron disable accessibility by default; the first access may be empty.
• UIA calls may hang: the exe has a built-in 5s timeout; tool spawn timeout is 15s.
• PrintWindow may return black for some DRM / self-drawn windows (capture_failed).