dsh-client-ui-voice-input

Composer voice control for DeepSeek Harness: a minimal linear mic button in the composer tool row that turns your speech into text — with a tap-to-monitor mode (continuous, live 逐字 streaming, send-anytime) and a hold-to-talk voice-chat mode (release to send, reply read aloud). Zero API key: recognition runs in the browser via the Web Speech API; reply reading uses the host's Edge TTS (/api/tts) with a browser speechSynthesis fallback.

dsh-plugin · TypeScript · React

Features

• Tap to monitor: click the mic, speak — text streams into the draft live (逐字输入), the mic keeps listening even in silence, and you can send or keep adding speech anytime. Tap again to stop.
• Auto-send on silence (optional): with autoSendOnSilenceMs configured, tap-to-monitor becomes hands-free — once you stop talking for the configured window, the message sends itself ("tap, speak, walk away"). Continued speech cancels the pending send.
• Hold to talk: press-and-hold to record a voice-chat message, release to send it; the assistant's reply is read aloud — host Edge neural TTS (/api/tts) first, browser speechSynthesis as fallback.
• Continuous across silences: each recognition segment auto-restarts so monitoring never drops.
• Respects the composer: speech appends to the draft (base preserved); a send clears the draft cleanly without re-filling old text; monitoring continues after a send on a fresh recognizer.
• DeepSeek-blue listening state: the icon pulses in DeepSeek brand blue while listening; borderless linear icon, no clutter.
• Configurable: recognition language (default zh-CN) and interim results.

Install

The package is a dsh.bundle installable, published on npm as @zhangbo-cn/dsh-client-ui-voice-input. One command:

dsh plugin add @zhangbo-cn/dsh-client-ui-voice-input

0.1.1+ required. 0.1.0 registered the browser bundle under the wrong ModuleLoader id (@deepseek-ai/...), so Harness failed with loaded without registering "@zhangbo-cn/dsh-client-ui-voice-input". Upgrade / reinstall, then hard-refresh the Web UI.

(It also installs from the GitHub repo via dsh plugin add github:Zhangbo-cn/dsh-voice-input-plugin.)

If you develop from a DeepSeek Harness checkout, you can mount it directly in the web-app browser roster (packages/bundle/web-app/cordis.patch.yml):

- id: ui-voice-input
  name: '@zhangbo-cn/dsh-client-ui-voice-input'

For reliable reply reading, also mount the host Edge TTS capability (@deepseek-ai/dsh-tts-edge), which registers /api/tts:

- id: tts-edge
  name: '@deepseek-ai/dsh-tts-edge'

Without it, reply reading still works but falls back to the browser's speechSynthesis (less natural, occasionally silent on Chrome after an idle gap).

Then build the client bundle with the repo's tsdown preset:

pnpm --filter @zhangbo-cn/dsh-client-ui-voice-input run bundle

Usage

After refreshing the Web UI, the composer tool row shows a linear mic button.

Voice input (tap)

1. Click the mic → the icon turns DeepSeek blue and pulses (listening).
2. Speak → text appears in the input box live, word by word.
3. Send anytime with the composer's send button; keep talking to add more.
4. Click the mic again to stop monitoring.
5. With autoSendOnSilenceMs set, stopping speech for that long auto-sends the message — the mic stops listening, so you don't need the extra tap.

Voice chat (hold)

1. Press-and-hold the mic (longer than ~250 ms) and speak.
2. Release → your message is sent.
3. The assistant's reply is read aloud automatically.

Reply reading after any send

A send that follows mic use (within 5 minutes) — hold or tap-monitoring + the composer send button — arms reply reading for the next assistant reply. Typed sends without recent mic use do not trigger it.

Configuration

- id: ui-voice-input
  name: '@zhangbo-cn/dsh-client-ui-voice-input'
  config:
    language: 'zh-CN'      # Web Speech recognition language tag
    interimResults: true   # stream live interim transcript into the draft
    autoSendOnSilenceMs: 0 # auto-submit after this many ms of silence following committed speech (0 = off)

How it works

MicButton (conversation.input.left)
  ├─ tap → beginMonitoring()
  │     → SpeechRecognition (continuous:false, interimResults)  // reliable results
  │     → onresult → TranscriptAccumulator → inputActions.setDraft(base + transcript)
  │     → onend (silence) → auto-restart (keep monitoring)      // continuous
  │     → onend + committed speech + autoSendOnSilenceMs>0 → silence window → auto-submit
  │     → tap again → stop
  └─ hold → submitChat()
        → on release: stop + inputActions.setDraft(text) + inputActions.submit()
        → reply streams → complete sentences read aloud WHILE the model
          generates (sentence-chunked queue)
        → tail (last incomplete sentence) read on finalize
        → each segment → fetch /api/tts (host Edge neural MP3)
              → play via gesture-unlocked AudioContext (else <audio> element)
              → fallback: browser speechSynthesis

• Recognition starts on pointer-down (a user gesture — required by the Web Speech API); tap vs hold is decided on release.
• The same pointer-down gesture unlocks reply audio (a shared AudioContext is resumed), so the assistant's reply — which arrives seconds later — is exempt from the browser autoplay policy that would otherwise block a plain HTMLMediaElement.play().
• Reply reading streams: complete sentences are read aloud while the model is still generating (a sentence-chunked queue, flushed at ~30 chars for delimiter-less runs); the last incomplete sentence is read on finalize. The mic icon pulses deep blue while reading, and tapping the mic stops the reading.
• No speaker-echo: while the reply is being read, recognition is paused (the mic physically picks up the speaker), then resumes when reading finishes if monitoring was on.
• continuous: false per segment is intentional: Chrome's continuous: true fails to deliver onresult, so monitoring is achieved by auto-restarting segments.
• The append base resets when the draft changes externally, so a send never lets stale voice text re-fill the box.
• The console logs [dsh-voice] diagnostics for each read segment and any fallback.

Compatibility

Browser	Mic (input, SpeechRecognition)	Reply playback (host /api/tts, fallback speechSynthesis)
Chrome / Edge (Windows)	✅ Web Speech	✅ host Edge neural MP3; browser speechSynthesis fallback
Safari	✅ webkitSpeechRecognition (re-trigger on each gesture)	✅ host Edge neural MP3 (playable); browser fallback works
Firefox	⚠️ not supported — browser limitation (Mozilla has not shipped SpeechRecognition; local on-device recognition is still early-stage)	✅ host Edge neural MP3 (playable); speechSynthesis fallback supported but less natural

Notes:

• Firefox mic input: this is a genuine browser limitation, not a plugin issue. The plugin feature-detects and disables the mic with a "not supported in this browser" hint. A cross-browser fallback would need MediaRecorder + an external transcription service (out of scope for a zero-backend plugin).
• Reply playback: the preferred path is the host's /api/tts (Microsoft Edge neural voices, synthesized server-side) — reliable and natural on every browser that can play MP3. Without the tts-edge host plugin, the client falls back to speechSynthesis (Chrome may silently drop speak() after an idle gap; voices are OS-default).
• Mic input requires a browser with Web Speech; reply playback requires either the tts-edge host plugin or a browser with speechSynthesis.

Tests

The plugin ships with two standalone-runnable suites plus a registration suite that needs the Harness monorepo.

npm install --legacy-peer-deps   # peers reference monorepo-only @deepseek-ai/* packages
npm test                         # 36 tests: tap monitoring, auto-send on silence, hold submit,
                                 # streaming reply reading, tap-send arming, stop-reading, markdown/emoji stripping

• tests/mic-button.client.spec.tsx — component behavior (tap-to-monitor, auto-send on silence, hold-to-talk, streaming reply reading) and the splitStreamSegments / commonPrefixLength helpers.
• tests/speech.client.spec.ts — stripMarkdownForSpeech and applyResults.
• tests/apply.client.spec.ts — plugin registration; it binds real Harness services (@deepseek-ai/cordis, -runtime, -locale) that are built from the DeepSeek Harness monorepo, so it runs there (a plain vitest run from this package) rather than standalone. CI runs the standalone suites on Linux.

License

MIT