ego-browser — The Visible Agent Browser

Repository: github.com/Fisfzy/ego-browser | Version history in CHANGELOG.md | Detail page: dshfind

Compatibility Matrix

Dependency	Minimum Version	Recommended Version	Notes
DSH (DeepSeek Harness)	0.1.2-rc.1	≥ 0.1.2-rc.1 (verified up to v0.1.5-rc.2)	engines.dsh declares the floor; peer dependency is pinned to >=0.1.2-rc.1 as well. For 0.1.0-rc.x / 0.1.1-rc.x, use v0.8.0 or earlier
dsh-better-sidebar	0.12.2 (optional)	≥ 0.17.1	Without it, automatically falls back to the floating observation orb; < 0.12.2 works but external link interception (urlTarget) silently degrades
Node.js	22	—	Bundled with the harness environment

DSH full-version compatibility note: This version v0.8.3 has been confirmed via source-code audit to be compatible with all released DSH versions from 0.1.2-rc.1 through 0.1.5-rc.2 (core APIs such as defineTool, ctx.tools.register, ctx.subprocess.spawn, ctx.webServer.register, ctx.inject, ModuleLoader CJS factory, and cordis.patch.yml have no breaking changes from v0.1.0-rc.7 to v0.1.5-rc.2). The 0.1.2-alpha.x series is installable per the declared constraints but has not been tested.

dsh-better-sidebar compatibility note: ego-browser registers a sidebar Tab and listens for external links via the ctx.betterSidebar service (defensive acquisition with try-catch). Key API introduction versions:

API	Usage in ego-browser	Introduced in better-sidebar
registerTab() / openTab() / ctx.betterSidebar	Tab registration + open	v0.9.0+
TabDescriptor.single	Single-instance Tab	v0.9.0+
TabDescriptor.urlTarget	External link interception	v0.12.2+ (below this version, link interception silently fails)

---

DSH version support details: Major changes from v0.8.2 to v0.8.3: merged 6 community PRs (root/xvfb/macOS headless support, rc.1 compatibility, Windows stability), fixed an unauthenticated /api/ego/* route security vulnerability, client startup failure without dsh-better-sidebar host (#29), Windows cold-start regression (Xvfb misjudgment introduced by #22), and missing egoCliArgs/chromeArgs in the gateway settings allowlist. Adaptation points: client runtime rename (@deepseek-ai/dsh-client-store), client module registration id and load line name follow the declared package name, dsh.client.inject only declares real module graph lines, webServer is delivered as nested injection (optional service), and sidebar Tab (dsh-better-sidebar) pattern synchronization.

Sidebar support (dsh-better-sidebar): When the host has dsh-better-sidebar installed (recommended ≥ v0.12.2), the live observation window registers as a sidebar native Tab — "Agent Browser" appears in the sidebar "+" menu; clicking it opens the Tab and it stays pinned with the sidebar drawer; the Tab auto-opens when the agent first invokes an ego_* tool. Without dsh-better-sidebar, it automatically falls back to the bottom-right floating observation orb (#dsh-ego-fab) mode. Both forms share the same SSE real-time streaming / click / input / download capture capabilities.

Integrates CitroLabs/ego-lite (Chromium for AI Agents) into DeepSeek Harness: drives the browser with 32 structured ego_* tools, plus a real-time observation frontend — while the agent operates web pages in the background, you can watch every page it's browsing as if it were a live stream, and directly interact with it.

A unique private gem (self-observation): The agent uses this single Chromium — even when it operates DSH itself (managing sessions, task kanban, adjusting settings), the observation window shows it in real time, and you can take over at any moment. It's not just "seeing the agent work on web pages" — the agent operating the DSH interface itself is fully visible and controllable.

Works out of the box: The plugin package bundles the ego runtime (runtime/, MIT, see THIRD_PARTY_NOTICES.md) — no cloning of the official repo, no manual build; the --no-sandbox wrapper is included, so root / Docker / headless setups work with one click.

---

Our Real Advantages (Not Slogans — Verifiable Capabilities Against Code and Competitors)

Another plugin on the market that integrates ego-lite into DSH only built 3 tools with it — a run script, a help guide, and a status health check — leaving the browser as a background black box. ego-browser takes a different path: opens the black box and delivers both "see" and "control" capabilities from the start.

Capability	ego-browser (this repo)	Similar plugin (Da1dr1em/dsh-ego-browser)
Structured tool count	32, single-responsibility, deterministically callable	3 (run/help/status)
Real-time observation window (CDP JPEG / FFmpeg H.264 dual backends + tab bar + history drawer)	✅ Yes	❌ No
Mouse direct operation of the real browser via the monitor window (click/drag/scroll sent back through CDP)	✅ Yes	❌ No
Worker single-instance guard + crash/duplicate self-healing	✅ Yes	❌ No
Download capture ego_download / captcha detection ego_captcha/ego_page_info	✅ Yes	❌ No
Platform adaptation (Linux/macOS/Windows auto-detection + root/headless/--no-sandbox fallback)	✅ All platforms	Windows preview host only, manual setup required
Login persistence to disk ego_auth_flush	✅ Yes	⚠️ Documentation-level description only

Two key differences:

• Visible: Others give you a "run and tell you the result" black box; we stream in real time so you watch the agent operate and immediately spot it stuck on a captcha or going wrong.
• Controllable: Others are read-only; our monitor window drives directly the same agent browser, so when needed you take over by hand (zoom/drag/click) without restarting the agent.

The comparisons above are based on publicly available, verifiable facts: this repository's code (bin/ego-cast-worker.mjs real-time streaming + CDP input relay, lib/index.js 32 registered tools, lib/cast-server.js host bridge) and the similar plugin's source code/README. This document contains no disparagement of others — we simply state which capabilities we additionally implemented and verified.

Beyond the ego-lite itself, we added these (all verifiable against this repo's code):

Capability	Description (corresponding code)
Observation frontend	ego-lite itself is a headless CLI (heredoc scripts + text output only); we added on top of it SSE real-time streaming + tab bar + history drawer + monitor window mouse direct-operation (bin/ego-cast-worker.mjs, lib/cast-server.js, lib/client.js), making "see" and "control" first-class capabilities
Out of the box + cross-platform self-sufficiency	resolveEgoEnv auto-detects Chrome/Edge/Brave, includes a --no-sandbox wrapper, root / Docker / display-less setups need no configuration (lib/index.js); no need to install a GUI host first as the official version requires
Robustness layer	Cold-start auto-retry (only retries transient CDP errors, doesn't swallow real errors), worker single-instance guard + auto-restart on crash, plugin uninstall fire-and-forget that doesn't block host exit, frontend frame cache cap (withWarmupRetry / makeEnsureWorker / frameCache)
Ops tools	ego_doctor (environment health check), ego_captcha (captcha detection), ego_auth_flush (login persistence), ego_http (request in browser context), etc. — a layer the native CLI helper doesn't have
Self-observation	The agent operating DSH's own interface is likewise visible in real time and can be taken over

We don't claim parity with the official macOS App's kernel-level snapshots or native multi-window experience; this repo solves the problem of "bringing the same browser capability into DSH + Linux/WSL + being able to see it."

---

What Problem It Solves

Generic browsers aren't designed for agents, and a large portion of web interactions (login states, captchas, dynamic rendering, forms, sites requiring a real human session) can only be handled by a real browser — this is the origin of the ego suite's "let the agent use your already-logged-in browser, without disturbing you" (official site).

ego-browser integrates it into DSH and addresses the most painful point — you can't see what the agent is doing or get involved — with a set of observation windows:

🌐 Click the orb for a live stream; 🟦 tab bar to switch/close; 🕘 history drawer for review; 🔍 zoom & drag; 🖱️ monitor window takes over the real browser directly. In one line: let the agent work in the browser while you watch and can take over at any time.

Common use cases

• Literature / data scraping: have the agent log into CNKI / Google Scholar, flip through pages to collect; watch in the observation window as it scrolls, clicks next page, and downloads PDFs — spot any stuck state immediately.
• Forms and logins: the agent fills a form halfway, a captcha pops up in the observation window — take over directly to click through the captcha, then hand back to the agent to continue.
• QA / smoke testing: have the agent click through your product; the observation window acts as a "talking screen recorder" that you can also use to review historical trajectories.
• Watching the agent operate DSH itself (self-observation): when the agent is managing sessions / adjusting settings, the observation window keeps everything visible and takeover-ready throughout.

---

✨ Recent Highlights

• v0.8.0: Sidebar Tab integration — when dsh-better-sidebar is available, the live view window registers as a native sidebar Tab (not a floating widget); the ego_browser tool auto-expands it on first call; ships a built-in EgoBrowserTab React component plus the LivePreviewController real-time frame pipeline. dsh-better-sidebar is not a peer dependency (consumed opportunistically via ctx.get()); if not installed it falls back to the floating widget — clean in both deployments.
• v0.7.0: Observation window status light stays green while working, breathes when idle; ego_script per-run timeoutMs is now actually enforced; frontend frameCache/pageMeta is cleaned up per tab with a cap as fallback to prevent unbounded memory growth in long sessions; the state directory fallback for home paths was switched to os.homedir() for cross-platform consistency; a .gitattributes was added to enforce LF line endings.
• v0.6.1: Uninstall no longer blocks host exit (self-healing path stabilized); observation window worker now has a single-instance guard plus stale-state cleanup; the login / human-captcha banner is dismissable and mutually exclusive with the other; the observation window actively follows the page the agent is operating on (no longer preempted by background re-rendered tabs).
• v0.6.0: Engineering convergence — lib/ designated as the single source of truth, build downgraded to a syntax check, so "one build, all regressions" is eliminated. (After the TS refactor, source lives under src/ and lib/ is the build output; see the "Development" section.)
• v0.5.0: Real-time SSE streaming plus monitor-window direct operation of the agent browser.
• v0.4.0: Windows support.
• Full history in CHANGELOG.md.

---

Prerequisites

Requirement	Notes
Node ≥ 22	Bundled with the harness environment
Any Chrome / Chromium / Brave / Edge	Auto-discovered, or specified via EGO_LINUX_CHROME; a bundled wrapper is used when running as root
DSH + dshx	Plugin loading mechanism
DSH Web with GUI (watch window)	ego_* tools still work in headless sessions; only the watch window is unavailable

Installation

Package name migration (DSH Desktop 2.0.5+): This plugin's package name is dsh-ego-browser (not @dsh-external/ego-browser). Starting with DSH Desktop 2.0.5, a consistency check has been added to verify that the "profile dependency name == actual package name". If the profile still references the old name @dsh-external/ego-browser, startup will fail and fall back into recovery mode (profile package identity is invalid for @dsh-external/ego-browser). After upgrading to 2.0.5, update the profile's package.json dependency key and the dsh.profile.bundles entry — both places — to dsh-ego-browser:

- "@dsh-external/ego-browser": "git+https://github.com/Fisfzy/ego-browser.git",
+ "dsh-ego-browser": "git+https://github.com/Fisfzy/ego-browser.git",

- "@dsh-external/ego-browser",
+ "dsh-ego-browser",

dshx install ego-browser <ego-browser.tgz>                             # tarball 或 git URL 均可
dshx list                                                # 应显示：[on] ego-browser

In watch window settings you can select captureBackend=auto|cdp|ffmpeg (default auto, which currently resolves to CDP), a quality tier, CDP FPS/JPEG quality/max width, and FFmpeg FPS/max width/bitrate/encoder/custom path. The plugin first checks the custom path, then the system PATH, then the managed cache; FFmpeg cannot be selected in the settings page until a compatible FFmpeg build is detected, and a one-click download of a pinned version is provided. GitHub downloads can replace https://github.com via githubMirror, e.g. https://gh-proxy.com/github.com. FFmpeg bitrate range is 500–20000 kbps, with low/balanced/high defaults of 2000/4000/8000 kbps.

No host-side configuration is required: resolveEgoEnv auto-detects root / no-displayer and falls back accordingly. Watch-window host routes (/api/ego/spaces, etc.) are registered only when an HTTP server is present; in headless mode they are a safe no-op.

Tool List (32 tools, prefix ego_, full index in ego_help)

Category	Tools
Task spaces	ego_space_open ego_space_close ego_status
Page reading	ego_snapshot (semantic tree) ego_page_info ego_read_element
Navigation/waiting	ego_navigate (reuses tab) ego_wait ego_wait_for_selector ego_wait_for_url ego_wait_for_response
Interaction	ego_click ego_fill ego_hover ego_drag ego_select ego_check ego_key ego_scroll
Execution/debugging	ego_js (page eval) ego_cdp (raw CDP) ego_cli (arbitrary heredoc) ego_script (multi-step script)
Output	ego_screenshot ego_download ego_upload
Sessions/security	ego_auth_flush (persist login) ego_captcha ego_dialog
Meta tools	ego_help ego_doctor ego_http

Watch Window

Bottom-right corner 🌐 persistent ball → click to expand:

• Main view: live view of the agent's current page; click/drag/scroll to interact directly with the page. Ctrl+scroll zooms the view, Ctrl+drag pans, double-click resets. After clicking the view you can type directly with the keyboard, including Chinese IME, paste, Tab/Enter/arrow keys, and Ctrl/Cmd shortcuts.
• Tab strip: top horizontal row; click to switch, × to close.
• History drawer (🕘): review visit trails chronologically.
• During interactions, the URL bar at the bottom shows contextual hints in place, restoring after 2 seconds.
• When the panel is closed, the sidebar tab is hidden, or the component is unmounted, frame production stops 1.5 seconds after the grace period. Merely moving the DSH window to the background does not stop streaming, avoiding repeated WGC/FFmpeg teardown when returning to the foreground; abnormal termination is covered by a 120-second worker lease timeout.

Frame Backend

• cdp: Page.startScreencast JPEG, default 20 FPS. Each source frame is ACKed immediately with the frame ID provided by Chrome; only the latest pending frame is retained. Only the currently viewed tab is captured; static pages revert to screenshot mode by default every 3 seconds.
• ffmpeg: On Windows, gfxcapture(hwnd) captures the target Chrome window's D3D11 surface directly; other platforms use display-source crop. Then H.264 fragmented MP4 → HTTP binary chunks → MediaSource <video>, with no Base64/SSE in the path.
• auto: defaults to CDP and will not auto-download FFmpeg even on detection failure. FFmpeg becomes selectable only after installation and capability check pass; if a previously saved FFmpeg backend later becomes unavailable, this observation session falls back to CDP and displays the reason.
• Windows FFmpeg must include gfxcapture. The plugin matches the HWND by browser PID, target title, and CDP window bounds; it still captures the target page when the window is moved or occluded, and never falls back to desktop recording. If the target is a background tab within the same Chrome window, an explicit error is returned instead of showing the currently visible tab or stealing user focus. First use on macOS requires "Screen Recording" permission; X11 requires Chromium and FFmpeg to share DISPLAY; on Wayland, when Portal/PipeWire input is missing, you will be prompted to switch back to CDP.

Managed FFmpeg is installed to ~/.dsh/cache/ego-browser/ffmpeg/, never into the plugin directory. Windows/Linux use a pinned BtbN release tag; macOS uses a pinned ffmpeg-static GitHub release asset (Intel/Apple Silicon binaries sourced from Evermeet/OSXExperts). All downloads are pinned by resource SHA-256; only the FFmpeg main binary is extracted, not ffprobe or ffplay. Windows/Linux extraction uses system tar; if missing, a clear error is raised before download.

Note: Cookies across task spaces are isolated; log in within the corresponding space. After DSH restart, runtime login is cleared (Chrome Cookies persist only on graceful shutdown), requiring re-login—QR scan is quick.

How It Works

• Tool layer: Each tool assembles its parameters into a JS script, fed via stdin to ego-browser nodejs through ctx.subprocess; the host drives a shared Chromium over CDP. Results are parsed from the @@DSH_RESULT@@ sentinel line. All ego_* calls are serialized through an in-process mutex; errors are normalized uniformly.
• Watch window: lib/client.js manages the watcher lease, JPEG <img> and MSE <video>; lib/cast-server.js proxies metadata SSE, the watch API, and backpressured binary video; in the worker, CaptureManager guarantees at most one active backend and one current target at a time. The CDP control plane (tabs, viewport, input, captcha) is independent of the frame backend.

Development

Source lives in src/ (TypeScript); build artifacts are in lib/ (host + client bundle) and bin/ego-cast-worker.mjs (worker bundle).

pnpm typecheck   # tsc 类型门禁（tsconfig.json 主 + tsconfig.client.json 客户端）
pnpm test        # vitest 单元测试
pnpm run build   # tsdown 三 bundle：lib/index.js + lib/client.js + bin/ego-cast-worker.mjs

Modify src/ directly (src/index.ts tool layer, src/client/index.ts frontend, src/worker/ego-cast-worker.ts worker). New tools are added in registerActionTools using t({...}), and a corresponding entry must be added to the ego_help index (src/help.ts), then run pnpm typecheck && pnpm test && pnpm run build. lib/ and bin/ego-cast-worker.mjs are build artifacts (pre-built and committed); do not hand-edit them.

node_modules/ contains only symlinks pointing to the DSH checkout (compile-time type resolution); at runtime, @deepseek-ai/dsh-tools is resolved by the harness.

Known Limitations

• Windows: The plugin layer has been adapted for v0.4.0; the underlying ego-lite host is still a community port not officially supported on Windows, and stability of complex multi-step flows may be weaker than on macOS.
• FFmpeg platform capture: Windows already uses gfxcapture(HWND) and requires a build that includes this filter; older FFmpeg on PATH is skipped with a prompt to download a compatible version. Linux uses x11grab, macOS uses avfoundation display crop; macOS ScreenCaptureKit and Wayland Portal helpers are planned future enhancements.
• Installation environment: Not all of this repo's DSH peer packages are available on the public npm registry. A plain pnpm install may fail when resolving @deepseek-ai/* peers; DSH profile installation should provide these peers. CDP does not depend on FFmpeg, and no binary is downloaded during plugin installation.
• Snapshot quality: Linux rebuilds the semantic tree using CDP DOMSnapshot, which is not macOS kernel-level; complex iframe/canvas scenarios may degrade.
• Host reliability (Linux): Unmerged community PR; tab/space state may be lost between CLI invocations. The plugin has built-in defenses; simple flows are stable, but complex flows may require retries.
• Login session persistence: Chrome runtime Cookies are persisted only on graceful shutdown; force-killing and restarting requires re-login.
• The output schema is lenient with additionalProperties: true; clients should rely on the actual return values.

License & Attribution

The plugin itself is MIT. The embedded runtime contains ego-lite's MIT-licensed code; the optionally downloaded FFmpeg build carries GPL-3.0-or-later obligations. Please review the license and source-obtention information of the build provenance before use or redistribution; see THIRD_PARTY_NOTICES.md for details.

Supply Chain & Permissions

Facts for review:

• Runtime files: lib/ (build artifacts, deterministically generated from src/ TypeScript via tsdown through npm run build), bin/ (executable entry scripts for the worker and ffmpeg-probe), cordis.patch.yml (assembly layer), dsh-plugin.json (manifest). *.map files are debug-only sourcemaps, not used at runtime, and are declared excluded.
• Native/executable artifacts: runtime/ embeds the ego-lite runtime (MIT; provenance and per-file inventory in THIRD_PARTY_NOTICES.md)—it is the core functionality of this plugin (a bundled managed Chrome/CDP host) and is an intentionally carried executable artifact, not a build byproduct. runtime/PATCHES.md documents all local patches applied upstream.
• Dependencies: The only runtime dependency is @deepseek-ai/schemastery (provided by the DSH host as a peer implementation); all peer dependencies are @deepseek-ai/dsh-* host services. External modules in the client bundle are resolved via the host module table; no npm runtime dependencies are carried.
• External services: No telemetry, no external API calls. The only network behavior is the optional FFmpeg installer downloading builds from GitHub (or a user-configured mirror) on user instruction; source verification and license obligations are in THIRD_PARTY_NOTICES.md.
• Failure boundaries: When the host has no webServer (TUI/headless), watch routes are safely skipped; if worker startup fails, watch routes return an ok:false JSON instead of hanging; the browser process is terminated together with host teardown (--stop is fire-and-forget and does not block host exit).
• Permissions: The manifest permissions field is empty—file read/write by the tool set is confined to ego's self-managed space directory and the user's workspace; network access goes through the managed agent browser rather than the host process.

---

Links

Also part of the DeepSeek Harness plugin ecosystem—check each other out: