ego-browser — The Visible Agent Browser

Repository: github.com/Fisfzy/ego-browser｜ See CHANGELOG.md for version history｜ Details: dshfind

Version Compatibility

Dependency	Minimum Version	Recommended Version	Notes
DSH (DeepSeek Harness)	0.1.2-rc.1	≥ 0.1.2-rc.1 (verified up to v0.1.5-rc.2)	engines.dsh declares the floor; peer dependency also locks >=0.1.2-rc.1. Use v0.8.0 or earlier for 0.1.0-rc.x / 0.1.1-rc.x
dsh-better-sidebar	0.12.2 (Optional)	≥ 0.17.1	Falls back to floating observation ball if not installed; < 0.12.2 runs but external link interception (urlTarget) degrades silently
Node.js	22	—	Bundled with harness environment

DSH Full-Version Compatibility Note: This version v0.8.3 has been confirmed via source code audit to be compatible with all released DSH versions from 0.1.2-rc.1 to 0.1.5-rc.2 (Core APIs such as defineTool, ctx.tools.register, ctx.subprocess.spawn, ctx.webServer.register, ctx.inject, ModuleLoader CJS factory, cordis.patch.yml have no breaking changes from v0.1.0-rc.7 → v0.1.5-rc.2). 0.1.2-alpha.x series is installable per declaration but not tested.

dsh-better-sidebar Compatibility Note: ego-browser registers sidebar Tabs and listens for external links via the ctx.betterSidebar service (defensive acquisition via try-catch). Key API introduction versions:

API	ego-browser Usage	better-sidebar Introduction Version
registerTab() / openTab() / ctx.betterSidebar	Tab registration + opening	v0.9.0+
TabDescriptor.single	Single-instance Tab	v0.9.0+
TabDescriptor.urlTarget	External link interception	v0.12.2+ (Link interception silently fails for versions below this)

---

DSH Version Support Details: v0.8.2 → v0.8.3 main changes: merged 6 community PRs (root/xvfb/macOS headless support, rc.1 compatibility, Windows stability), fixed a security vulnerability in the unauthenticated /api/ego/* routes, fixed host client launch failure without dsh-better-sidebar (#29), fixed the Windows cold-start regression (the Xvfb misjudgment introduced by #22), and fixed the gateway settings whitelist missing egoCliArgs/chromeArgs. Adaptation points: client runtime rename (@deepseek-ai/dsh-client-store), client module registration id and loading row name follow the declared package name, dsh.client.inject only declares actual module graph rows, webServer delivered as nested injection (optional service), and synced the sidebar Tab (dsh-better-sidebar) mode.

Sidebar support (dsh-better-sidebar): When the host has dsh-better-sidebar installed (≥ v0.12.2 recommended), the live observation window is registered as a native sidebar Tab — “Agent Browser” appears in the sidebar “+” menu; clicking it opens and keeps the view pinned within the sidebar drawer. The agent automatically opens this Tab upon its first ego_* tool call (since v0.8.5, it opens within the calling session scope, preventing multi-session popup errors). If dsh-better-sidebar is not installed, it automatically falls back to the bottom-right floating observation ball (#dsh-ego-fab) mode. Both forms share the same SSE real-time streaming / click / input / download capture capabilities. The observation window also provides an “Popup Window” button: headless-running agent browsers can be switched with one click to a headed window using the same Profile (tabs are preserved) for manual takeover.

Login state import (new in v0.8.5): Use “Import login state from system browser” on the settings page or the ego_login_import tool to copy login cookies by domain from your daily Chrome/Edge/Brave into the agent browser (actual binary headless startup + CDP pass-through reading, compatible with Chrome 127+ App-Bound Encryption, no offline decryption; if the source browser is running, you can choose to close it gracefully before importing, and the window will auto-restore on next launch). Cookie values are not exposed in any logs or outputs; the source cookie database is automatically backed up before import, and any accidental clearing is automatically restored. Combined with the default disk-persistent Profile, imported login states are permanently retained across restarts.

Integrate CitroLabs/ego-lite (Chromium for AI Agents) into DeepSeek Harness: drive the browser with 33 structured ego_* tools, paired with a real-time observation frontend — while the agent operates web pages in the background, you can watch each page it is browsing like a live stream, and even interact with it directly.

A unique hidden feature (self-observation): The agent uses this same Chromium instance — even when it operates DSH itself (managing sessions, task boards, adjusting settings), the observation window displays it in real-time, allowing you to take over at any moment. It’s not just about “seeing the agent working on web pages”; even when the agent interacts with the DSH interface itself, the entire process is visible and controllable.

Out of the box: The plugin package includes the ego runtime (runtime/, MIT, see THIRD_PARTY_NOTICES.md) — no cloning the official repo or manual building required; the --no-sandbox wrapper is bundled, enabling one-click operation on root / Docker / display-less environments.

---

Our Real Advantages

Similarly, when integrating ego-lite into DSH, existing similar plugins have only implemented 3 tools using it—a run script, a help guide, and a status check—while the browser remains a background black box. ego-browser takes a different approach: it opens the black box and establishes robust 'viewing' and 'control' capabilities right from the start.

Feature	ego-browser (This Repo)	Similar Plugin (Da1dr1em/dsh-ego-browser)
Structured tool count	32 tools, with single responsibilities and deterministic invocation	3 tools (run/help/status)
Real-time observation window (CDP JPEG / FFmpeg H.264 dual backends + tab bar + history drawer)	✅ Available	❌ Not available
Mouse direct manipulation of the real browser in the monitor window (click/drag/scroll back to CDP)	✅ Available	❌ Not available
Worker single-instance guard + crash/duplicate self-healing	✅ Available	❌ Not available
Download capture ego_download / CAPTCHA detection ego_captcha/ego_page_info	✅ Available	❌ Not available
Platform adaptability (Auto-detection for Linux/macOS/Windows + root/headless/--no-sandbox fallbacks)	✅ All platforms	Windows preview host only, manual configuration required
Login state persistence to disk ego_auth_flush	✅ Available	⚠️ Documentation-level description only

Two key differences:

• Visibility: Others are a black box that "tells you the result after running"; we stream in real time, so you watch the agent operate and immediately spot CAPTCHAs or errors.
• Controllability: Others are read-only; our monitor window directly drives the same agent browser, allowing you to manually take over (zoom/drag/click) when needed without interrupting the agent to restart.

The above comparison is based on publicly visible, verifiable facts: this repository's code (bin/ego-cast-worker.mjs real-time streaming + CDP input back-propagation, lib/index.js 32 registered tools, lib/cast-server.js host bridge) and the source code/README of similar plugins. This document does not demean anyone—it only states the capabilities we have implemented and verified.

Compared to the core ego-lite entity, we have added the following (all verifiable against this repository's code):

Feature	Description (Corresponding Code)
Observation window frontend	The ego-lite core is a headless CLI (only heredoc scripts + text output); we added SSE real-time streaming + tab bar + history drawer + mouse direct operation in the monitor window (bin/ego-cast-worker.mjs, lib/cast-server.js, lib/client.js) to make "viewing" and "control" first-class capabilities
Out-of-the-box + Cross-platform self-sufficiency	resolveEgoEnv auto-detects Chrome/Edge/Brave, built-in --no-sandbox wrapper, requires no configuration for root / Docker / headless environments (lib/index.js); no need to install a GUI host beforehand like the official version
Robustness layer	Cold start auto-retry (retries only transient CDP errors, does not swallow real errors), worker single-instance guard + automatic crash restart, plugin uninstallation fire-and-forget does not block host exit, frontend frame cache limit (withWarmupRetry / makeEnsureWorker / frameCache)
Operational tools	ego_doctor (environment health check), ego_captcha (CAPTCHA detection), ego_auth_flush (login persistence), ego_login_import (system browser login state import), ego_http (browser context requests), etc., a layer not present in native CLI helpers
Self-observation	When the agent operates the DSH interface itself, it is also visible in real time and can be taken over

We do not claim parity with the official macOS App's kernel-level snapshots or native multi-window experience; this repository solves the problem of "bringing the same set of browser capabilities into DSH + Linux/WSL + visibility".

---

What Problem It Solves

General-purpose browsers are not designed for agents. However, many web interactions (login states, captchas, dynamic rendering, forms, sites requiring human sessions) can only be handled by a real browser. This is exactly why ego aims to "let agents use your logged-in browser without disturbing you" (official site).

ego-browser integrates this into DSH and solves the most painful point—you can’t see what the agent is doing and can’t intervene—with an observation window:

🌐 Click the ball to watch live; 🟦 Tab bar to switch/close; 🕘 History drawer to review; 🔍 Zoom and drag; 🖱️ Take over the real browser directly in the monitoring window. In short: let the agent work in the browser while you can watch and take over at any time.

Common Use Cases

• Literature / data crawling: Have the agent log into CNKI / Google Scholar to page through and collect; in the observation window you watch it scroll, click next page, and download PDFs — if it gets stuck midway you'll notice immediately.
• Forms and login: The agent fills a form halfway through and the observation window pops up a CAPTCHA — you take over to complete the CAPTCHA, then hand it back to the agent to continue.
• QA / smoke testing: Have the agent click through your own product; the observation window is like a "talking screen recording," and you can also review past traces on the fly.
• Watch the agent operate DSH itself (self-observation): when the agent manages sessions / adjusts settings, the observation window is fully visible and takes over throughout.

---

✨ Recent Highlights

• v0.8.0: Sidebar Tab Integration — when dsh-better-sidebar is available, the Live View registers as a native sidebar Tab (instead of a floating overlay), and the ego_browser tool auto-expands on first invocation; built-in EgoBrowserTab React component + LivePreviewController real-time frame pipeline. dsh-better-sidebar is not a peer dependency (opportunistic consumption via ctx.get()); if not installed, it falls back to the floating overlay, ensuring clean deployment in either mode.
• v0.7.0: Observation window status light stays green while active, breathes when idle; timeoutMs for ego_script runs now actually takes effect; frontend frameCache/pageMeta cleaned up per tab with an upper limit fallback to prevent memory growth in long sessions; state path home directory fallback switched to os.homedir() for cross-platform support; added .gitattributes to enforce LF line endings.
• v0.6.1: Uninstallation no longer blocks host exit (self-healing chain stable); observation window worker single-instance guard + stale state cleanup; login/CAPTCHA guidance bars are now dismissible and mutually exclusive; observation window actively follows the page the agent is operating on (no longer preempted by background repainted pages).
• v0.6.0: Engineering convergence — lib/ designated as the single source of truth, build changed to syntax validation only, eliminating "one build, total regression". (After TS refactoring, source code moved to src/, lib/ is build output, see "Development" section.)
• v0.5.0: Real-time SSE streaming + monitoring window directly operates agent browser.
• v0.4.0: Windows support.
• For complete history, see CHANGELOG.md.

---

Prerequisites

Requirement	Description
Node ≥ 22	Provided by the harness environment
Any Chrome / Chromium / Brave / Edge	Auto-discovered, or specified via EGO_LINUX_CHROME; uses built-in wrapper under root
DSH + dshx	Plugin loading mechanism
DSH Web with GUI (Observation Window)	Headless sessions can still use ego_* tools, just without the Observation Window

Installation

Package Name Migration (DSH Desktop 2.0.5+): This plugin's package name is dsh-ego-browser (not @dsh-external/ego-browser). Starting from DSH Desktop 2.0.5, a consistency check for "profile dependency name == actual package name" was added. If the profile still references the old name @dsh-external/ego-browser, startup will enter recovery mode (profile package identity is invalid for @dsh-external/ego-browser). After upgrading to 2.0.5, please change the dependency key in the profile's package.json and the dsh.profile.bundles entry in both places to dsh-ego-browser:

- "@dsh-external/ego-browser": "git+https://github.com/Fisfzy/ego-browser.git",
+ "dsh-ego-browser": "git+https://github.com/Fisfzy/ego-browser.git",

- "@dsh-external/ego-browser",
+ "dsh-ego-browser",

dshx install ego-browser <ego-browser.tgz>                             # tarball 或 git URL 均可
dshx list                                                # 应显示：[on] ego-browser

In the Observation Window settings, you can optionally configure captureBackend=auto|cdp|ffmpeg (default auto, currently resolves to CDP), quality levels, CDP FPS/JPEG quality/max width, and FFmpeg FPS/max width/bitrate/encoder/custom path. The plugin first checks the custom path, system PATH, and managed cache; before a compatible FFmpeg is detected, selecting FFmpeg in the settings page is disabled, and a one-click download of a fixed version is provided. For GitHub downloads, githubMirror can replace https://github.com, e.g., https://gh-proxy.com/github.com. The FFmpeg bitrate range is 500-20000 kbps, with default values of 2000/4000/8000 kbps for low/balanced/high levels.

Other switches: isolateSpaces (task space sandbox isolation, default off), idleTimeoutMin (automatically reclaims background browsers after N minutes of inactivity, default 0=off), disableFrameRelay (Disables frame relay, default off).

Disabling Frame Relay (disableFrameRelay): When enabled, the plugin stops relaying any frames—it does not start the ego-cast worker (thus, it will not perform CDP/WGC screen capture, nor will it use ffmpeg for streaming and fMP4 encapsulation). The /api/ego/stream, /api/ego/video, and worker-related routes will instead return {"ok":false,"reason":"frame relay disabled"}. The observation window will no longer establish any pull stream connections and will display a disabled state prompt. The ego_* tools themselves remain completely unaffected (the browser opens as usual, and snapshot/click/input/screenshot functions work as normal). This switch is a configuration item and takes effect immediately upon saving: running workers will be stopped, and they will automatically resume once re-enabled.

No host-side configuration is required: resolveEgoEnv automatically detects root / no display and provides a fallback. Observation window host routes (/api/ego/spaces, etc.) are only registered when an HTTP server is available; they are a safe no-op in headless mode.

Tool List (32 tools, prefix ego_, see ego_help for the complete index)

Category	Tools
Task Space	ego_space_open ego_space_close ego_status
Page Reading	ego_snapshot (semantic tree) ego_page_info ego_read_element
Navigation/Waiting	ego_navigate (reuse tab) ego_wait ego_wait_for_selector ego_wait_for_url ego_wait_for_response
Interaction	ego_click ego_fill ego_hover ego_drag ego_select ego_check ego_key ego_scroll
Execution/Debugging	ego_js (page evaluation) ego_cdp (raw CDP) ego_cli (any heredoc) ego_script (multi-step script)
Output	ego_screenshot ego_download ego_upload
Session/Security	ego_auth_flush (persist login) ego_captcha ego_dialog
Meta Tools	ego_help ego_doctor ego_http

Observation Window

Bottom-right corner 🌐 persistent ball → click to open:

• Main View: Live view of the agent's current page. Click/drag/scroll to operate the page directly. Use Ctrl+scroll to zoom the view, Ctrl+drag to pan, and double-click to reset. You can input keyboard commands directly after clicking the view; it supports Chinese IME, pasting, Tab/Enter/Arrow keys, and Ctrl/Cmd shortcuts.
• Tab Bar: Located at the top, click to switch tabs, use × to close.
• History Drawer (🕘): Review browsing history chronologically.
• During operations, hints are displayed in the URL line at the bottom and disappear after 2 seconds.
• After the panel is closed, the sidebar Tab is hidden, or the component is unmounted, frame production stops once a 1.5-second grace period ends. Merely moving the DSH window to the background does not stop streaming, avoiding repeated reconstruction of WGC/FFmpeg when returning to the foreground; abnormal closure is handled by a 120-second worker lease timeout.

Screen Backend

• cdp: Page.startScreencast JPEG, default 20 FPS. Each source frame is immediately ACKed with the frame ID provided by Chrome; only the latest pending frame is retained. It captures only the currently viewed tab, and resumes screenshots at a default interval of 3 seconds for static pages.
• ffmpeg: On Windows, gfxcapture(hwnd) directly captures the D3D11 surface of the target Chrome window; other platforms use display source cropping. Subsequent encoding goes H.264 fragmented MP4 → HTTP binary chunk → MediaSource <video>, without passing through Base64/SSE.
• auto: CDP is the default selection; successful detection does not automatically trigger an FFmpeg download. FFmpeg can be selected only after installation and capability check. If a saved FFmpeg backend later becomes invalid, the current observation will fall back to CDP and display the reason.
• Windows FFmpeg must include gfxcapture. The plugin matches the HWND by browser PID, target title, and CDP window bounds; it continues to capture the target page even when the window is moved or occluded, and does not allow fallback to desktop recording. If the target is a background tab within the same Chrome window, it explicitly reports an error rather than displaying the currently visible tab or stealing user focus. macOS requires "Screen Recording" permission on first use; X11 requires Chromium and FFmpeg to share DISPLAY; Wayland will prompt to switch back to CDP if Portal/PipeWire input is missing.

Hosted FFmpeg is installed to ~/.dsh/cache/ego-browser/ffmpeg/ and does not write to the plugin directory. Windows/Linux use a fixed BtbN release tag; macOS uses a fixed ffmpeg-static GitHub release asset (its Intel/Apple Silicon binaries are sourced from Evermeet/OSXExperts respectively). All downloads are pinned by resource SHA-256, extracting only the FFmpeg main program; ffprobe or ffplay are not installed. Windows/Linux unpacking uses the system tar; if missing, an explicit error is raised before download.

Note: Cookies are isolated per task space; log in within the corresponding space. After a DSH restart, runtime login state is cleared (Chrome runtime Cookies are only persisted on graceful shutdown), so you'll need to log in again — scanning the QR code is quick.

How It Works

• Tool layer: Each tool assembles parameters into a JS script, which is fed via stdin to ego-browser nodejs through ctx.subprocess to execute; the host drives the shared Chromium via CDP. Results are parsed using the @@DSH_RESULT@@ sentinel line. All ego_* calls are serialized through an in-process mutex, and errors are uniformly normalized.
• Observation window: lib/client.js manages the watcher lease, JPEG <img>, and MSE <video>; lib/cast-server.js proxies metadata SSE, the watch API, and backpressured binary video; CaptureManager in the worker ensures only one active backend and one current target at a time. The CDP control plane (tabs, viewport, input, captchas) is independent of the screen backend.

Development

Source code is in src/ (TypeScript); build artifacts are in lib/ (host + client bundle) and bin/ego-cast-worker.mjs (worker bundle).

pnpm typecheck   # tsc 类型门禁（tsconfig.json 主 + tsconfig.client.json 客户端）
pnpm test        # vitest 单元测试
pnpm run build   # tsdown 三 bundle：lib/index.js + lib/client.js + bin/ego-cast-worker.mjs

Edit src/ directly (src/index.ts tool layer, src/client/index.ts frontend, src/worker/ego-cast-worker.ts worker). Add new tools in registerActionTools using t({...}), and add an entry to the ego_help index (src/help.ts), then run pnpm typecheck && pnpm test && pnpm run build. lib/ and bin/ego-cast-worker.mjs are build artifacts (pre-built and checked in); do not edit them by hand.

node_modules/ contains only symlinks pointing to the DSH checkout (compile-time type resolution); at runtime the harness resolves @deepseek-ai/dsh-tools.

Known Limitations

• Windows: The plugin layer has v0.4.0 adaptation; the underlying ego-lite host remains a community port without official Windows support, and stability of complex multi-step flows may be weaker than on macOS.
• FFmpeg platform capture: Windows uses gfxcapture(HWND), which requires a newer build including that filter; old FFmpeg in PATH is skipped with a prompt to download a compatible version. Linux uses x11grab, macOS uses avfoundation display crop; macOS ScreenCaptureKit and Wayland Portal helper are future enhancements.
• Installation environment: The DSH peer packages of this repo are not all available in the public npm registry. A regular pnpm install may fail when resolving @deepseek-ai/* peers; DSH profile installs should provide these peers. CDP does not depend on FFmpeg and will not download binaries during the plugin installation stage.
• Snapshot quality: Linux rebuilds the semantic tree via CDP DOMSnapshot, not the macOS kernel-level approach; complex iframe/canvas scenarios may degrade.
• Host reliability (Linux): Unmerged community PRs may lose tab/space state across CLI invocations; the plugin has built-in defenses, simple flows are stable, but complex flows may need retries.
• Login state persistence: Chrome runtime Cookies are only persisted to disk on graceful shutdown; force-killing and restarting requires re-login.
• Output schema uses loose additionalProperties: true; the client follows actual return values.

License & Attribution

The plugin itself is MIT. The embedded runtime includes MIT code from ego-lite; the optionally downloadable FFmpeg build involves GPL-3.0-or-later obligations. Before use or redistribution, please read the license and source acquisition information of the build source; see THIRD_PARTY_NOTICES.md for details.

Supply Chain & Permissions

Facts for catalog inclusion and review:

• Runtime Files: lib/ (build artifacts, deterministically generated from src/ TypeScript via npm run build using tsdown), bin/ (executable entry scripts for workers and ffmpeg-probe), cordis.patch.yml (assembly layer), dsh-plugin.json (manifest). *.map files are sourcemaps for debugging only, not part of the runtime, and are declared as excluded.
• Native/Executable Artifacts: runtime/ includes the built-in ego-lite runtime (MIT; source and per-file manifest available in THIRD_PARTY_NOTICES.md). It is a core feature of this plugin (self-managed Chrome/CDP host) and constitutes an intentionally bundled executable artifact, not a build byproduct. runtime/PATCHES.md documents all local patches applied to upstream.
• Dependencies: Runtime dependencies include only @deepseek-ai/schemastery (provided by the DSH host as a peer implementation). All peer dependencies are @deepseek-ai/dsh-* host services. External modules in the client bundle are resolved by the host module table and do not include npm runtime dependencies.
• External Services: No telemetry or external API calls. The only network behavior is the optional FFmpeg installer, which downloads build artifacts from GitHub (or a user-configured mirror) upon user instruction. Source verification and licensing obligations are documented in THIRD_PARTY_NOTICES.md.
• Failure Boundaries: If the host lacks a webServer (TUI/headless), the watch route safely skips. If worker startup fails, the watch route returns a JSON with ok:false instead of hanging. Browser processes terminate along with the host teardown (--stop is fire-and-forget and does not block host exit).
• Permissions: The permissions field in the manifest is empty. The file read/write capabilities of the toolset are restricted to ego's self-managed space directory and the user's workspace. Network access is handled via the managed agent browser rather than the host process.

---

Friend Links

Other works in the DeepSeek Harness plugin ecosystem, recommending each other: