dsh-unicode-guard

English | 中文

A DSH host-half plugin that keeps a small class of rare glyphs out of every model-visible surface.

The failure it prevents

Some inference endpoints reject a request outright when its body contains a particular glyph, answering with an HTTP 400 and the opaque reason Content Exists Risk. The usual way such a glyph enters a request is accidental: an agent fetches a web page whose language switcher or footer uses flag-style emoji, the whole page lands in a tool result, and from then on every request in that session replays it. Each retry is byte-identical, so each retry fails the same way. The session cannot recover on its own.

The glyph that matters here is a regional indicator pair: two adjacent code points from the U+1F1E6..U+1F1FF block, which is how flag emoji are encoded. Nothing about the page is objectionable - a page footer is enough.

What this plugin does

It rewrites the payload before the payload can hurt, at three layers:

Hook	Layer	Effect
tools/post-execute	tool results	rewrites result content before it is appended to the session
agent/pre-step	an already-affected session	replaces the offending surface node with a scrubbed copy, so the session's next turn is already clean
llm/stream	the outgoing request	scrubs the serialized body itself - the last hook before adapter dispatch

The three layers matter because they cover different producers. Only the first sees a tool result; only the third sees text that never travels as one (compaction summaries, instruction bodies, MCP server instructions, scheduled prompts).

A regional indicator pair is rewritten as a readable ASCII tag: the two code points become [FLAG:AB]. A variation selector (U+FE0F) directly after such a rewrite is dropped so no bare selector is left behind. Everything else is untouched - ordinary emoji, CJK text, and every other astral-plane character pass through unchanged.

Install

As a bundle (recommended)

dsh plugin add dsh-unicode-guard

The package ships a cordis.patch.yml that inserts its own row, so adding it to the profile bundle list is enough.

Manually, into a profile

Link the directory into the profile's node_modules and append one row to the profile's cordis.patch.yml:

- insert:
    - id: unicode-guard
      name: dsh-unicode-guard
      config:
        mode: regional
        verbose: true

A change to cordis.patch.yml is hot-reloaded; adding the link itself needs a restart.

Configuration

Key	Default	Meaning
mode	regional	regional masks regional indicator pairs only. astral additionally replaces every non-BMP character with replacement, which destroys legitimate output - opt in deliberately.
replacement	U+FFFD	Replacement used in astral mode only.
toolResults	true	Scrub tool results before they are logged.
userMessages	true	Scrub user-role messages during heal.
assistantMessages	false	Scrub assistant messages during heal.
request	true	Enable the llm/stream backstop.
verbose	true	Log one line per rewrite through ctx.logger.

Unknown keys or wrong value types fail at mount time rather than silently doing nothing.

Verifying it works

A mounted plugin logs exactly one line:

[I] [unicode-guard] unicode-guard: mounted (mode=regional, toolResults=true, ...)

Every rewrite logs a warning, so you can watch it work:

[W] [unicode-guard] unicode-guard: scrubbed the web_fetch tool result
[W] [unicode-guard] unicode-guard: healed 1 surface node(s), removed 2 glyph(s)
[W] [unicode-guard] unicode-guard: scrubbed the outgoing request

An end-to-end check that does not depend on any log: read a file that contains a regional indicator pair. The tool result must show [FLAG:AB] where the file has the raw pair.

How a session recovers

A session that is already rejected heals on its next turn, with no retry: agent/pre-step runs before that turn's request is assembled, so the node is replaced first and the very first request of the turn is already clean.

Repairs are surface replacements. The session log is append-only, so the original record stays on disk and is shadowed on the current surface. That is the same mechanism the kernel uses for its own tool-result pruning, and it is what keeps the log's integrity guarantees intact.

Design notes

• Projection, not mutation. A message in llm/stream arrives deep-frozen; a changed message is replaced by a clone. Rewriting in place would either throw or corrupt a message that other listeners still hold.
• assistant/message cannot carry sourceEventSeqs. The kernel rejects it because that event embeds its own source stream, so the replacement intent differs per event type.
• The whole payload, not just the message. A tool result can carry a second copy of its payload in data.meta (the read tool stores its numbered window in meta.lines). That copy is never sent to the provider, but leaving it tainted keeps the session file dirty for forks, exports, and UI replay, so it is scrubbed too.
• A repair must never break the turn it repairs. A node that throws during healing is logged and skipped; the turn proceeds.
• UTF-16 care. Regional indicators are surrogate pairs. Pairing walks code points, never bare code units, and a lone surrogate is passed through rather than mangled.

Tests

node --test test/

The suite has no dependencies and no fixtures to download. It covers the masker's edge cases (pairing, leftovers, variation selectors, idempotence, surrogate integrity), the hook contracts, and the request-backstop rewrite.

test/selfcheck.test.mjs fails the run if any file in this repository ever contains a literal astral-plane character. A plugin that shipped one would be a trigger source in its own right, so the property is enforced rather than promised.

Scope and limits

• It masks the glyph classes it is configured for. mode: astral widens that to all non-BMP characters.
• It changes what the model sees: a flag-style emoji becomes [FLAG:AB]. That is the point - an ASCII tag keeps the information readable while removing the bytes that make the request fail.
• Repairs do not shrink the log: shadowed records stay on disk.
• It is a client-side robustness measure, not a replacement for whatever the provider may do server-side. If the endpoint stops rejecting this glyph class, the plugin degrades into a harmless normalizer.

License

MIT