dsh-maestro-core

Supervisor for DSH Web resilience — Phase 1 Guard & Report + Phase 3 Auto-Resume & Auto-Reload — together with the settings store, the Maestro settings card, the tool guard and the harness-to-harness sync engine that used to be four separate packages.

Runs outside the pnpm → sh → node tree (systemd daemon) to survive tree crashes, plus inside dsh web as a host+client Cordis plugin to auto-resume interrupted sessions and auto-reload the browser after restart.

• Daemon: Polls :3080 every 3s, keeps last-known-good (LKG) snapshots (~/.dsh/.supervisor/lkg/, rotate 3, sha256 verify, df >500MB guard), auto-rollbacks on crash (debounce 60s, flock lock), writes report-<ts>.md (health + git diff + log tail), and notifies via Telegram (loose, never blocks).
• Host plugin: runAutoResume() 8s after boot — findInterrupted (tail 100) + findDanglingOpenTurns (full scan for recent sessions) within autoResumeWithin (default 5m) → agents.resume({resumeSessionId, agentOptions: {provider,model}}) recovered from request/context → followup('continue'). Loopback RPC POST /dsh-maestro-supervisor-resume/{scan,resume} (reachable from loopback; connection.isLoopback reports it) for the daemon (resumeViaRpc).
• Client plugin: Hybrid auto-reload — fetch HEAD / polling 1s on offline/WebSocket close/visibilitychange → 200 → location.reload(). Served as window.__ModuleLoader__.load bundle at /plugins/@ddtcorex/dsh-maestro-core/client.js via dsh.client.

Modules

One package, four host rows and one client bundle.

Module	Source	Row id	Channel	What it does
Supervisor	src/host/*.ts	maestro-supervisor	/dsh-maestro-supervisor-resume (loopback-reachable)	Crash polling, LKG snapshots, auto-resume, auto-reload
Store	src/host/store/	—	—	The namespaced settings.json every other module reads and writes
Config	src/host/config/	maestro-config	/dsh-maestro-config	maestroConfig service over the store, backing the Maestro settings card
Guard	src/host/guard/	dsh-maestro-guard	/dsh-maestro-guard	Rule classification, the approval gate and its journal
Sync	src/host/sync/	dsh-maestro-sync	/dsh-maestro-sync	Backup, restore, retention GC and two-machine sync

Row names for the absorbed modules are subpaths of this package (@ddtcorex/dsh-maestro-core/lib/<module>/index.js), which is why exports["./lib/*"] exists: a row name the exports map cannot resolve is skipped silently at boot.

The store is also published on its own (@ddtcorex/dsh-maestro-core/store) and can be vendored into a consumer with scripts/vendor-store.mjs, which writes one self-verifying file with a body hash.

Registered tools

Host tools registered with ctx.tools.register:

Tool	Module	What it does
dsh_web_restart	supervisor	Schedule a supervised dsh web restart (consent-gated)
dsh_web_restart_status	supervisor	Report the state of the last restart request
dsh_web_dryboot	supervisor	Dry-boot on an ephemeral port with an isolated DSH_HOME
dsh_web_gc	supervisor	Preview, then reap, verified dry-boot orphans
maestro_session_health	supervisor	Session-log health scan (re-encode or quarantine)
maestro_resume_tool_health	supervisor	Tool-view health of a resumed session
maestro_repair_session_preset	supervisor	Re-link an agent that joined no preset
maestro_guard_status, maestro_guard_stats, maestro_full_scan	guard	Guard state, counters and a full rule scan
maestro_sync_status, maestro_sync_check_machines, maestro_sync_preview, maestro_sync_apply, maestro_sync_bidirectional_preview, maestro_sync_bidirectional_apply, maestro_sync_tunnel_restore	sync	Two-machine sync, preview first
maestro_backup_preview, maestro_backup_apply, maestro_backup_gc_preview, maestro_backup_gc_apply, maestro_restore_preview, maestro_restore_apply	sync	Backup, retention GC and restore, preview first

Requirements

• Node.js ^22.19.0 || >=24.0.0 and pnpm 11+. A shell that defaults to Node 20 fails with No such built-in module: node:sqlite; put a Node 22 bin first on PATH before running dsh or pnpm.
• DSH 0.1.x or 0.2.x (peer range <0.3.0-0).

Install

One package, one command:

dsh plugin --profile web add @ddtcorex/dsh-maestro-core

The package ships its own cordis.patch.yml, applied automatically. It includes a connection entry that declares webServer, which DSH 0.2.x needs before rpc.handle can register a channel. Do not copy it into the profile patch, and do not add the rows by hand (duplicate ids crash the loader). Restart dsh web after install.

To get an exact release instead of whatever the package manager resolves, pin it: dsh plugin --profile web add @ddtcorex/dsh-maestro-core@<version>.

Installed with link: from a checkout? After every git pull, run pnpm install && pnpm build (lib/ is gitignored) and restart dsh web.

In a workspace checkout, build and link instead:

pnpm --dir packages/dsh-maestro-core install
pnpm --dir packages/dsh-maestro-core build   # tsc host + tsc client + esbuild bundle -> lib/ + lib/client.js
pnpm --dir packages/dsh-maestro-core verify  # tsc --noEmit host + client
pnpm --dir packages/dsh-maestro-core test    # vitest run
test -f packages/dsh-maestro-core/lib/index.js
test -f packages/dsh-maestro-core/lib/client.js
# link:() the checkout into ~/.dsh/profiles/web/package.json, then
pnpm --dir ~/.dsh/profiles/web install

pnpm build is required after any src/ change; lib/ is gitignored build output. The client needs both tsc steps and the build-client.mjs bundle — plain tsc alone leaves lib/client.js as a bare ES module and dsh web will fail with exports no "./client" bundle.

The package declares dsh.client (platform: web, inject: ["@deepseek-ai/dsh-client-connection", "@deepseek-ai/dsh-client-ui-slots"]) so the browser half is auto-loaded — no extra dsh.client flag needed.

Pre-flight (required): before adding to a live profile's bundles, dry-boot must pass:

DSH_HOME=$(mktemp -d) pnpm --dir deepseek-harness dsh web --port 0 &
# wait for "dsh web: http://127.0.0.1:<port>" and curl 200, then kill
# This catches load-time failures (missing lib/index.js, stale build, bad cordis.patch.yml)
# that no in-code try/catch can catch. See dsh-safe-restart skill.

This exact failure class caused dsh web outages on 2026-08-27 (missing lib/index.js). See AGENTS.md Conventions.

Systemd daemon (optional, for crash detection outside the tree)

bash packages/dsh-maestro-core/scripts/install-systemd.sh
systemctl --user daemon-reload
systemctl --user enable --now dsh-web-supervisor
systemctl --user status dsh-web-supervisor
journalctl --user -u dsh-web-supervisor -f

The template leaves Environment=TELEGRAM_BOT_TOKEN/TELEGRAM_CHAT_ID commented — uncomment via systemctl --user edit dsh-web-supervisor if you want Telegram, otherwise it logs only.

To run without systemd (foreground, for debugging):

node packages/dsh-maestro-core/lib/index.js daemon   # poll every 3s
node packages/dsh-maestro-core/lib/index.js status
node packages/dsh-maestro-core/lib/index.js resume --within 5m   # list interrupted sessions

Configuration

All autoResumeWithin values are minutes when given as number (e.g. 5 → 5 minutes). Strings support 30s/5m/1h. Precedence (highest first):

1. Cordis config (cordis.patch.yml config: or apply(ctx, config)) — explicit per-install.
2. Env DSH_SUPERVISOR_AUTO_RESUME / DSH_SUPERVISOR_RESUME_WITHIN (bare 5 in env → 5m for ergonomics).
3. Supervisor config ~/.dsh/.supervisor/config.json (autoResumeEnabled, autoResumeWithin).
4. Maestro settings ~/.dsh/dsh-maestro-config/settings.json (domains.supervisor.*).
5. Default: true / 5.

Key	Type	Default	Env	File	Notes
autoResumeEnabled	boolean	true	DSH_SUPERVISOR_AUTO_RESUME (1/true/yes/on/enabled vs 0/false/no/…)	config.json: autoResumeEnabled, settings.json: domains.supervisor.autoResumeEnabled	false → notify only
autoResumeWithin	number (minutes) or string (5m)	5	DSH_SUPERVISOR_RESUME_WITHIN	config.json: autoResumeWithin, settings.json: domains.supervisor.autoResumeWithin	Window for findInterrupted/findDangling (mtime + event.time)

Example ~/.dsh/.supervisor/config.json:

{
  "autoResumeWithin": 5,
  "autoResumeEnabled": true
}

CLI

node packages/dsh-maestro-core/lib/index.js --help
node packages/dsh-maestro-core/lib/index.js status
node packages/dsh-maestro-core/lib/index.js daemon   # poll 3s, debounce 60s
node packages/dsh-maestro-core/lib/index.js resume [--within <dur>]   # list interrupted sessions, e.g. 5m, 30s, 1h
node packages/dsh-maestro-core/lib/index.js boot-guard acquire|release --pid <pid>   # boot.lock + boot-boundary, used by the safe-restart script

The CLI has no logs or rollback command: rollback runs inside the daemon, and reports are files under ~/.dsh/.supervisor/reports/.

RPC (reachable from loopback, connection.isLoopback)

# Scan (findInterrupted only, tail 100)
curl -s http://127.0.0.1:3080/dsh-maestro-supervisor-resume/scan -X POST \
  -H 'content-type: application/json' \
  -d '{"type":"client-request","rpcId":"r1","method":"scan","payload":{"withinMs":300000}}'
# → {"type":"server-response","rpcId":"r1","result":{"ok":true,"value":{"scanned":425,"interrupted":[]}}}

# Resume (re-attaches agent + followup continue, recovers provider/model)
curl -s http://127.0.0.1:3080/dsh-maestro-supervisor-resume/resume -X POST \
  -H 'content-type: application/json' \
  -d '{"type":"client-request","rpcId":"r2","method":"resume","payload":{"ids":["--example-project--/session-abc"]}}'
# → {"type":"server-response","rpcId":"r2","result":{"ok":true,"value":{"resumed":["--example-project--/session-abc"]}}}
# or {"ok":false,"error":{"code":"bad-request","message":"resume requires at least one session id"}}

The daemon uses resumeViaRpc() (supervisor.ts) which POSTs the same envelope to http://127.0.0.1:3080/dsh-maestro-supervisor-resume/resume with fetch and validates server-response + rpcId + result.ok.

Auto-Resume Details

• When: apply() sets setTimeout 8000 after boot, then runAutoResume() — only safe right after fresh boot when dsh web is sole owner (an open turn found then cannot belong to a still-running generation). resumeInterrupted additionally checks agents.get(sessionId) and skips if already live.
• What: findInterrupted (tail 100, looks for turn/end with reason.kind === 'interrupted' at time within window) + findDanglingOpenTurns (full scan for recent sessions, looks for turn/start without matching turn/end at time within window) → merged = Set([...interrupted, ...dangling]) → resumeInterrupted for each id.
• How: agents.get(sessionId) if live → followup('continue'); else sessionPersistence.load(sessionId) → find request/context with provider/model → agents.resume({resumeSessionId, agentOptions}) → followup('continue'). If load fails, still resumes without agentOptions (degrades). Returns string[] resumed and logs sent continue trigger.
• Subagents: findDangling does full log scan for recent sessions (mtime within window, 1-2 files) — not tail — because subagent b6487e33 had its only turn/start at seq 6 at the very beginning of a 1906-line log, missed by tail -100. findInterrupted stays tail 100 (interrupted closer is always at tail). The mtime pre-filter keeps full scans cheap (previously 5.5s for 425 sessions without it).

Auto-Reload Details (Hybrid)

• Client (src/client/auto-reload.ts, lib/client.js via window.__ModuleLoader__.load): ctx.effect hooks WebSocket (patches window.WebSocket to catch close for same-origin DSH ws), offline/online, visibilitychange → setInterval(fetch HEAD / 1s) when down → 200 → location.reload() (once, reloading guard). Also checks HEAD / on load in case the page was opened while down.
• Host (supervisor.ts pollHealth 3s + notify, plugin.ts runAutoResume): health check + restart + notify is the host half; together with client polling they cover manual, supervisor, and systemd restarts without F5. No extra host push channel needed — client polling is primary, host health is secondary; the window.__ModuleLoader__ bundle is served at /plugins/@ddtcorex/dsh-maestro-core/client.js via ClientModuleRegistry (dsh.client + exports["./client"]).

Verification

# Build & unit
pnpm --dir packages/dsh-maestro-core verify   # host + client
pnpm --dir packages/dsh-maestro-core test     # vitest run
test -f packages/dsh-maestro-core/lib/index.js
test -f packages/dsh-maestro-core/lib/client.js
curl -s http://127.0.0.1:3080/plugins/@ddtcorex/dsh-maestro-core/client.js | grep -c "window.location.reload"  # 2

# Live
curl -s -o /dev/null -w "%{http_code}\n" http://127.0.0.1:3080/  # 200
curl -s http://127.0.0.1:3080/dsh-maestro-supervisor-resume/scan -X POST -H 'content-type: application/json' -d '{"type":"client-request","rpcId":"t","method":"scan","payload":{"withinMs":300000}}' | head -c 200
node --input-type=module -e "import {findDanglingOpenTurns} from './packages/dsh-maestro-core/lib/resume.js'; console.log(await findDanglingOpenTurns(undefined,{withinMs:5*60*1000}))"
# Create a real dangling: pnpm --dir deepseek-harness dsh --profile headless "Run bash synchronously sleep 60" & sleep 4; kill $!; node -e "...findDangling..."  # should be 1
# After restart, it should have turn/end interrupted → continue → turn2

Troubleshooting

Symptom	Cause	Fix
Cannot find package '.../dsh-maestro-core/index.js'	pnpm build not run or lib/ stale	pnpm --dir packages/dsh-maestro-core build && pnpm --dir ~/.dsh/profiles/web install
exports no "./client" bundle / client bundle not found	Missing lib/client.js or exports["./client"]	pnpm build (runs tsc + tsc -p tsconfig.client.json + node scripts/build-client.mjs), check package.json exports and dsh.client, test -f lib/client.js, curl .../client.js
EADDRINUSE on a fixed port of dsh web	A previous dsh web still holds the listener tree: :3080 (LAN proxy), :3081 (public proxy), :3082 (raw webserver); :3000 is unbound	`ss -tlnp
uses .jsonl but backend is zstd	Hand-written session.jsonl while backend is zstd	Use zstd -c plain.jsonl > session.jsonl.zstd or JsonlSessionPersistence API. Never hand-write opposite encoding — listArtifacts checks every project dir on boot and one stray file blocks all of dsh web.
first frame is not exactly one header line	zstd without type: session header	Use toHeaderLine + compressZstdFrame(header) + compressZstdFrame(body) as in encodeMaterialization.
findDangling 0 but subagent still open	Tail window too small (before 63b7719)	Fixed: findDangling now full-scans recent sessions (mtime within window). findInterrupted stays tail 100.
resumed: [] or RESUME FAILED	agents.resume failed (no persistence, no provider/model)	Check session.jsonl.zstd exists and zstd -d -c ... | head -n 1 is valid header. request/context with provider/model is recovered — if missing, still resumes without agentOptions.
RESUME SKIPPED	No session within autoResumeWithin window	Increase autoResumeWithin to 10/"10m", check config.json and DSH_SUPERVISOR_RESUME_WITHIN, verify with withinMs: 60*60*1000.
Page does not reload after restart	lib/client.js not served or browser cache	`curl .../client.js
dangling found but agents.get says live	Session still live in current process (not a crash)	findDangling is only safe right after fresh boot when dsh web is sole owner. resumeInterrupted skips if agents.get is live — correct, wait for next boot.

Known Issues

• Manual session.jsonl vs zstd: One stray opposite-encoding file under ~/.dsh/sessions/ blocks the entire workspace listArtifacts on every boot (encodingMismatch). See Troubleshooting.
• Tail vs full scan: Before 63b7719, b6487e33 subagent missed because its only turn/start was at seq 6 at the very beginning of a 1906-line log. Fixed, but if you add a new scan variant, reuse readSessionAllLines with mtime pre-filter.
• Port map: :3080 is the LAN proxy (PIN login), :3081 the public proxy, :3082 the raw dsh web webserver; :3000 is unbound. Inspect with ss -tlnp; never pkill -f "dsh web".
• Client bundling: lib/client.js must be window.__ModuleLoader__.load wrapper via scripts/build-client.mjs, not bare export. Add new client files under src/client/ and ensure tsconfig.client.json includes them, then pnpm build.
• Config precedence: cordis.patch.yml config: > env (DSH_SUPERVISOR_RESUME_WITHIN bare 5 → 5m) > config.json > settings.json > default. See getAutoResumeEnabled() and getResumeWithinMs() in plugin.ts and supervisor.ts.

Development

pnpm --dir packages/dsh-maestro-core verify
pnpm --dir packages/dsh-maestro-core test
pnpm --dir packages/dsh-maestro-core build

For daemon changes: DSH_HOME=$(mktemp -d) pnpm --dir deepseek-harness dsh web --port 0 + corrupt settings.json → assert report + rollback.

Telegram

Loose by default: notifier.ts tries import('@ddtcorex/dsh-maestro-notifier'), then TELEGRAM_BOT_TOKEN/TELEGRAM_CHAT_ID env, then console.log. Enable via systemctl --user edit dsh-web-supervisor → uncomment Environment=TELEGRAM_* → daemon-reload + restart.

Hard mode (optional): package.json add "@ddtcorex/dsh-maestro-notifier": "workspace:^0.1.0" + pnpm-workspace.yaml packages: ["../dsh-maestro-notifier"] → pnpm install links it.

See Also

• Spec: <workspace-root>/docs/specs/2026-09-13-supervisor-restart-resilience-design.md
• Skill: skills/dsh-safe-restart/ (restart-dsh-web.sh with dry_boot_and_verify() and --auto)
• Client bundling: dsh-maestro-mobile (scripts/build-client.mjs pattern)