dsh-pve-vm

Proxmox VE VM control for DeepSeek Harness: see a VM's screen and click in its console over VNC, plus commands, files and typed text through the QEMU guest agent.

Almost every computer-use plugin drives the desktop the agent itself runs on — and therefore needs X11 or Wayland, a screenshot tool and an input injector on that host. This plugin does something different: it drives the console of a virtual machine on a Proxmox VE node, over one ssh connection. The host can be a headless server, a Wayland desktop or a laptop in another country; the guest can be Windows, Linux with any GUI, or a machine that has not booted yet.

agent ──► plugin ──► ssh ──► node: qm vncproxy ──► unix socket /var/run/qemu-server/<vmid>.vnc
                      └────► node: qm guest exec ──► qemu-guest-agent inside the VM

What you get

• Pixels — console_shot returns the VM screen as an inline image (with region and scale to keep the token cost sane), and mouse/keyboard tools operate in VM screen coordinates: move, click, double click, drag, wheel, type, key combinations, and a batch runner that performs a chain of actions and returns a single frame at the end.
• Commands — vm_exec, vm_file_read, vm_file_write, vm_status through the QEMU guest agent. Deterministic and cheap: use pixels only where a GUI really is in the way.
• Reliable input — uppercase goes through Caps Lock (qm monitor → HMP sendkey), shifted symbols through RFB keysyms with a settle delay. This is what makes typing a password on the Windows login screen work instead of producing xxXX → xxxX.
• Unicode text — a Windows guest gets a scheduled task in the logged-in user's session (SendKeys), a Linux guest gets xdotool/wtype through the guest agent. Layout and modifiers stop mattering.
• Clipboard — console_paste writes UTF-8 text into the guest clipboard over RFB and presses Ctrl+V; console_clipboard reads the guest clipboard back, so text can leave the VM without OCR.
• A settings card in the web UI (English and Russian labels), backed by the profile patch layer.
• A live screen panel — the “VM screen” tab in the right sidebar shows the frames the agent sees, taken from the very same VNC session, while the agent works. It follows the agent's VM automatically or stays on the VM you pick by hand. Watching only: the panel never clicks or types.
• Zero runtime dependencies — RFB, DES authentication, keysym tables and PNG encoding are implemented in this package on top of node:crypto and node:zlib. There is nothing to install on the host, and nothing to install on the node either.

Verified against pve-manager 8.4.21 with Windows 11 / Windows Server 2025 and Linux guests, on DeepSeek Harness 0.2.0-rc.2.

Requirements

What	Why
DeepSeek Harness ≥ 0.2.0-rc.2	The plugin uses the 0.2.x plugin contract (defineTool, ctx.tools, ctx.subprocess, ctx.attachments, the computerUse slot, client-side configForms). It does not load on 0.1.x.
Node.js ≥ 22	The harness runtime.
A Proxmox VE node with the qm CLI	/usr/sbin/qm vncproxy, qm guest exec, qm monitor. Containers (LXC) are not supported.
SSH access to that node as root (or an account that may run qm)	qm needs root on the node. Key-based access is expected (BatchMode, no password prompts).
AcceptEnv LC_* in the node's sshd_config	The VNC password is handed to qm as the LC_PVE_TICKET environment variable via SendEnv, so it never appears in ps. Without AcceptEnv, ssh silently drops it and the console cannot authenticate.
Host key already accepted	Connections run with BatchMode=yes; an unknown host key fails. Run ssh <node> once by hand, or add StrictHostKeyChecking=accept-new to sshExtraOptions.
qemu-guest-agent inside the guest (optional)	Only the vm_* tools need it. Console tools work without any agent at all.

Install

# 1. Add the bundle
dsh plugin --profile web add github:taxah92/dsh-pve-vm

# 2. Make sure the bundle is enabled in the profile: dsh.profile.bundles must list "dsh-pve-vm"
#    (the plugin manager / market toggle does the same thing)

# 3. Restart the harness:  the host half of a plugin is loaded once per process

The plugin adds one row to the profile patch layer:

- id: pve-vm
  name: dsh-pve-vm

The row id pve-vm is the settings namespace — do not rename it, or the settings card and the patch layer will stop finding each other.

Configure

Settings ▸ Plugins ▸ Proxmox VE in the web UI. Everything is written into the profile patch layer, and the same values can be edited there by hand:

- id: pve-vm
  name: dsh-pve-vm
  config:
    sshHost: proxmox            # alias from ~/.ssh/config, or an address
    sshUser: root
    sshPort: 22
    sshKeyPath: ''              # empty — default keys
    sshConfigFile: ''           # like `ssh -F`
    sshExtraOptions: []         # extra -o options, one per item
    allowVmids: [100, 101]      # VMs the plugin may touch; empty list forbids everything
    enableConsole: true
    enableGuest: true
    inputChannel: auto          # auto | hmp | vnc
    sendkeyDelayMs: 80
    consoleTimeoutMs: 20000
    sessionIdleMs: 300000
    guestTimeoutMs: 60000
    guestTypeCommand: ''        # script for typing non-ASCII text into a Linux guest
    viewAutoOpen: false         # open the "VM screen" tab by itself when the agent touches a console
    viewFollowAgent: true       # the panel follows the VM the agent works with
    viewFps: 2                  # frame pace in the panel, 1–4
    viewScale: 2                # frame resolution divisor: 1 native, 2 half, 4 quarter

First run checklist

1. allowVmids is empty by default — nothing is allowed until you list a VM id. This is the only gate the plugin has.
2. ssh <alias> 'qm status <vmid>' should work from the host without a password prompt.
3. ssh <alias> sshd -T | grep -i acceptenv must include LC_* (see Requirements).
4. Call vm_status — it reports whether the VM is allowed, whether the guest agent answers, which guest OS was detected, and whether a console session is open.

Tools

Console (VNC)

Tool	Parameters	What it does
console_shot	vmid, region?, scale?, drawCursor?, wake?	Frame as an inline image. scale 1 / 0.5 / 0.25, region in VM pixels. Nearly uniform frames are flagged as blank (a blanked display or a locked screen).
console_move	vmid, x, y, observe?, wake?, settleMs?, scale?, region?	Move the pointer.
console_click	vmid, x, y, button?, count?, observe?, …	Click (2 = double click); returns a frame by default.
console_drag	vmid, fromX, fromY, toX, toY, button?, …	Drag — move a window, select a region.
console_scroll	vmid, x, y, delta, …	Wheel: positive scrolls up, negative down.
console_type	vmid, text, …	Printable ASCII only, \n presses Enter. Capitals go through Caps Lock, so expect ≈0.1 s per character.
console_key	vmid, combo, …	enter, esc, tab, ctrl+alt+del, win+r, alt+f4, ctrl+shift+esc, f5, and modifier-only combinations such as alt+shift (layout switch).
console_paste	vmid, text, submit?, …	Put any Unicode text into the guest clipboard and press Ctrl+V. Needs the clipboard-enabled VGA device and a SPICE agent.
console_batch	vmid, actions[], observe?, …	A chain of actions (move, click, drag, scroll, type, key, wait) and one frame at the end.
console_clipboard	vmid, timeoutMs?	Read the guest text clipboard (extended VNC clipboard). Empty means "no text".
console_release	vmid	Close the console session (emergency brake: releases the ssh child and the QEMU connection).

Every action tool accepts a common tail: observe (return a frame after the action, default true), wake (Shift + pointer jitter to wake a blanked display), settleMs, scale, region.

Guest agent

Tool	Parameters	What it does
vm_exec	vmid, command | argv, shell?, timeoutMs?	Run a command in the guest (cmd/PowerShell on Windows, shell on Linux). Runs as root/SYSTEM and does not see the user's graphical session.
vm_file_read	vmid, path, maxBytes?, encoding?	Read a file (text, or base64 for binary) through the agent.
vm_file_write	vmid, path, data | dataBase64	Write a file (up to 1 MB per call); parent directories are created.
vm_type_text	vmid, text, settleMs?, observe?	Type text through the best available channel: ASCII → console, non-ASCII → the user session on Windows / xdotool–wtype on Linux.
vm_status	vmid?	Is the VM allowed, does the agent answer, which OS, are there open console sessions.

Watch the VM screen live

The VM screen tab in the right sidebar shows what the agent sees while it works. Add it from the sidebar guide, or let viewAutoOpen reveal it the moment the agent touches a console.

• The frames are taken from the agent's own RFB session — no second VNC session appears and the qm vncproxy password is never reset under the agent's feet. Captures and the agent's tools share one session queue, so they never tear the stream for each other.
• viewFollowAgent keeps the panel on the VM the agent is working with; clicking a VM chip pins the panel to that VM instead. viewFps (1–4) and viewScale (1 / 2 / 4) control the pace and the resolution, and both can be changed on the fly inside the panel.
• The panel is watch-only: no clicks, no typing, nothing is injected into the guest. A blank frame is called out explicitly — the guest blanked its display or is locked.
• Frames are served by the plugin's own routes (/pve-vm/console/frame, /state, /release) on the harness web server, behind a loopback-only fence: a request is answered only when the Host is loopback (localhost, 127.0.0.0/8, [::1]), a present Origin matches Host, and sec-fetch-site: cross-site is refused. If you bind the harness to your LAN, the panel stays dark by design — the same fence the harness applies to /api.
• The fence is loopback, not the harness token. These routes live outside /api, so the harness URL token does not cover them (the browser-skill overview works the same way). Any process on the same machine can therefore pull frames or call /release; that is the accepted price of loading frames without a token on a single-user box. Serve VMs you would not mind a local user seeing, and harden with a harness-issued token if that is not acceptable.
• The console session stays open while the panel streams (the idle reaper does not take it); the panel's “disconnect console” button closes it immediately. Streaming pauses by itself when the tab is hidden.

How it works

• Transport. A VM has no TCP VNC port: QEMU listens on the unix socket /var/run/qemu-server/<vmid>.vnc, and qm vncproxy bridges it to its own stdin/stdout. The plugin therefore speaks RFB through an ssh child process — no ports, no TLS, no websocket, and the VNC password never lands in argv. qm vncproxy treats EOF on its stdin as "quit", so the plugin keeps that pipe open (the bridge also drops idle sessions after 15 minutes; the plugin closes its own session after sessionIdleMs, 5 minutes by default).
• Input channels. inputChannel: auto (default) mixes them: Caps Lock through the QEMU monitor for uppercase, RFB keysyms for the rest. hmp sends everything through qm monitor sendkey (faster, but QEMU occasionally loses modifiers below ~80 ms), vnc uses RFB only (uppercase is unreliable — QEMU emulates Shift itself and our own Shift events desynchronize it).
• Unicode. Windows: the plugin drops a payload file and a PowerShell typer into the guest and registers a scheduled task with -LogonType Interactive, so the text is typed inside the user's session via SendKeys — layout and modifiers stop mattering, and the VM is not restarted. Linux: xdotool type (X11) or wtype (Wayland) inside the active session; both paths run through the guest agent in a single call.
• Clipboard. ClientCutText for writing, the extended clipboard pseudo-encoding (VNC_ENCODING_CLIPBOARD_EXT) with CAPS/REQUEST/PROVIDE for reading. QEMU passes the bytes to the guest through qemu-vdagent, so the VM needs vga: <type>,clipboard=vnc and a full stop/start (device arguments are fixed at start-up), plus a SPICE agent in the guest.
• Frames. RFB Raw/CopyRect rectangles are composited into RGBA (QEMU leaves alpha at zero), optionally cropped and downscaled, then encoded to PNG with node:zlib.

Troubleshooting

Symptom	Cause and fix
CONSOLE_FAILED right after connecting, node reports nothing useful	The node's sshd does not accept LC_*, so qm never received the VNC password. Add AcceptEnv LC_* to sshd_config and reload sshd.
Host key verification failed	The node's key is not in known_hosts and the plugin runs in batch mode. Run ssh <node> once, or set sshExtraOptions: ["StrictHostKeyChecking=accept-new"].
A frame comes back almost uniform (blank: true)	The guest blanked its display or is locked. The tools already send Shift + pointer jitter to wake it; a locked session needs a password (console_type + console_key).
unable to connect to VM qmp socket — timeout after 51 retries	Something else holds the VM's single QMP socket (a hung qm monitor/QMP client). Find and kill that client on the node; do not restart the VM.
Uppercase letters turn into garbage	You are on inputChannel: vnc, where QEMU's own Shift emulation fights our Shift events. Use auto.
Latin characters are not typed at all on a login screen	The guest layout is not Latin: console_key with alt+shift, then type.
vm_* tools report that the agent is not running	qemu-guest-agent is missing or wedged (Restart-Service qemu-ga on Windows, systemctl restart qemu-guest-agent on Linux). Console tools keep working.
Console dies after ~15 minutes of idling	That is the qm vncproxy bridge timeout. Any tool call reopens the session automatically.

Security

This plugin gives an agent root-level hands inside your VMs. Be deliberate about it:

• Every action is executed without an approval prompt. The only gate is allowVmids, which is empty by default; list only the VMs you are willing to let an agent touch, and keep the list short.
• vm_exec runs as root/SYSTEM in the guest, vm_file_write can write anywhere the agent can, and the console tools can drive a logged-in desktop. Treat an allowed VM as a compromised host: do not keep long-lived secrets in it while the plugin is enabled.
• Root SSH to the node is required (qm needs it). Use a dedicated key, restrict it to the nodes you need, and consider from=/command= restrictions in authorized_keys where possible. The plugin passes the whole host environment to the ssh child, as any ssh call does.
• sshExtraOptions accepts arbitrary -o options, including ProxyCommand — it is a privileged escape hatch for the person who configures the plugin, not for the model.
• VNC passwords are generated per session (8 printable ASCII characters) and handed to qm through the environment via SendEnv, so they do not show up in ps. The QEMU password is scoped to the VM and expires with the session.
• The plugin talks only to your node. No telemetry, no outbound connections of its own.

Limitations

• Proxmox VE only, QEMU VMs only — no LXC containers, no other hypervisors.
• Clipboard tools need clipboard=vnc on the VM's VGA device and a SPICE agent in the guest; they move text only (no files, no images) and only inside a logged-in session.
• Unicode typing in Windows needs an interactive session; on the login screen only ASCII through console_type works.
• One console session per VM id, serialized with a mutex; the session is closed on idle.
• The console tools do not work with a serial console (vga: serialN) — there is no framebuffer.
• Not yet verified: Proxmox VE 7 and 9, multi-node clusters, Linux guests with Wayland-only desktops.

Development

npm run check                       # node --check on every runtime file (no harness needed)
node tools/check-client.mjs         # browser bundle format (classic script, exports, inject)
node tools/check-plugin.mjs         # settings schema, tool set, JSON schemas, layout contract

check-plugin.mjs imports the plugin, and the plugin imports the harness packages (@deepseek-ai/dsh-tools, @deepseek-ai/schemastery), so run it where those resolve — the easiest way is to install the checkout into a profile with link: and run the script from there:

dsh plugin --profile web add link:/path/to/dsh-pve-vm
cd /path/to/dsh-pve-vm && node tools/check-plugin.mjs

For a bare checkout, point DSH_PACKAGES at the harness package directory (…/@deepseek-ai/dsh/node_modules/@deepseek-ai) — the script uses it as a fallback for the compiler it needs.

Live probes need node access only (no harness):

node tools/probe-rfb.mjs --vmid 100 --ssh-config ~/.ssh/config --out /tmp/shot.png
node tools/probe-rfb.mjs --vmid 100 --click 640,400 --after /tmp/after.png
node tools/live-test.mjs --vmid 100 --ssh-config ~/.ssh/config --out probe-out
node tools/clipboard-test.mjs --vmid 100 --text "hello" --out /tmp/clipboard.png

All of them talk to the node exactly like the plugin does (qm vncproxy over ssh), so they are the fastest way to tell a plugin bug from a node-side problem.

License

MIT — see LICENSE.

Proxmox® is a registered trademark of Proxmox Server Solutions GmbH. This project is an independent plugin and is not affiliated with or endorsed by Proxmox Server Solutions GmbH or by the DeepSeek Harness project.