dsh-semantic-search

Local semantic code search for DeepSeek Harness (dsh) — and any Node script.

中文文档：README.zh.md

sema builds a fragment-level index of a workspace: source is tokenized by a language-aware tokenizer (camel/snake/kebab splitting, CJK n-grams), chunked with symbol-aware boundaries (functions/classes stay intact), and embedded into fixed dimension vectors — locally and dependency-free by default (feature-hashed lexical TF-IDF), or via any OpenAI-compatible embedding endpoint. Queries run a hybrid retrieval (vector cosine + BM25, fused with reciprocal-rank fusion) over the index, so meaning-based search works even when exact terms don't match.

Ships as a dsh plugin bundle — the sema_search, sema_reindex and sema_stats tools on the harness tool registry — plus a standalone sema CLI.

---

Highlights

• Works offline by default — the built-in lexical provider needs no network, no model download, no API key. Use it as a fast BM25-plus code search.
• Symbol-aware chunking — boundaries from a conservative per-language table (16 languages) keep functions/classes intact; a missed boundary degrades to a plain chunk rather than breaking.
• CJK-aware tokenizer — n-gram tokenization (default bigrams) aligns Chinese queries and documents without a segmentation library; full-width punctuation is folded, not treated as a hard break.
• Hybrid retrieval with RRF — vector cosine + BM25 channels are fused by reciprocal-rank fusion, so a document found by one channel still ranks.
• Graceful degradation — if a configured remote provider is unreachable, the index falls back to the local lexical provider (configurable via allowFallback).
• Incremental refresh + file watching — sema_reindex diffs by size+mtime, and an optional watcher keeps the index live.
• Persistence — the index is saved to <root>/.sema atomically (JSON metadata
  • binary vectors), with staleness detection when the provider/dimension changes.
• Deterministic — the same workspace and options produce the same index and the same ranked answers.

Supported languages

TypeScript, JavaScript, Python, Go, Rust, Java, Kotlin, Scala, C, C++, C#, Objective-C, Ruby, PHP, Swift, Bash, plus common data/markup formats (JSON, YAML, TOML, Markdown, HTML, XML...).

---

Installation

As a dsh bundle

The package declares "dsh": { "bundle": { "patch": "./cordis.patch.yml" } }. The patch inserts one plugin row that mounts the bundle and registers sema_search / sema_reindex / sema_stats on ctx.tools.

# from npm (name reserved; publish pending access setup)
npm install -g dsh-semantic-search

# straight from this repository
dsh plugin --profile demo add github:JohnXu22786/semantic-search

# or from a local checkout
dsh plugin --profile demo add /path/to/semantic-search

As a standalone CLI

npm install -g dsh-semantic-search   # or: npm run build && node bin/sema.mjs
sema --help

---

CLI usage

sema index                build the full index from the workspace
sema reindex [--full]     incremental refresh (or full rebuild with --full)
sema search <query...>    hybrid vector + BM25 search, prints top hits
sema stats [--json]       index health, provider, and sizing numbers

Global options:

--root <dir>          workspace root (default: current directory)
--data-dir <dir>      index storage directory (default: <root>/.sema)
--provider <kind>     embedding provider: lexical | openai (default: lexical)
--dim <n>             embedding dimension (lexical default: 4096; openai 0 = auto)
--base-url <url>      OpenAI-compatible embeddings endpoint base URL
--model <name>        embeddings model name (openai only)
--api-key <key>       API key (openai only; env: SEMA_EMBEDDING_API_KEY)
--top <n>             hits to print for search (default: 20)
--json                machine-readable output where supported
--help                show this help

CLI exit codes

• 0 — success (including a search with zero hits and a --version/--help call).
• 1 — a runtime failure (config error, build/index/search error).
• 2 — a usage error: unknown command, unknown flag, or a missing query.

Configuration

The plugin is configurable through the bundle row's config (see cordis.patch.yml for an example), the CLI flags above, or defaults in code:

| Option | Default | Meaning | | --- | --- | --- | | root | cwd | workspace root to index | | dataDir | .sema | index storage directory | | provider.kind | lexical | lexical (offline) or openai | | provider.dimension | 0 (lexical: 4096) | embedding dimension; 0 = auto-infer from the endpoint | | provider.baseUrl | https://api.openai.com/v1 | OpenAI-compatible endpoint root | | provider.model | text-embedding-3-small | embedding model name | | provider.apiKeyEnv | SEMA_EMBEDDING_API_KEY | env var holding the API key | | allowFallback | true | fall back to the lexical provider when a remote one fails | | include / ignore | defaults | glob sets of files to index / skip | | maxLinesPerChunk | 80 | hard chunk size upper bound | | nGram | 2 | CJK n-gram size (1 disables n-gramming) | | topK | 20 | hits returned by default | | rrfK | 60 | RRF fusion constant | | vectorK | 300 | candidates per channel before fusion | | autosave | true | persist the index after builds | | autoIndex | true | build lazily on first search | | watch | true | watch the workspace for changes |

Development

npm ci
npm test          # build + run the node:test suite (76 tests)
npm run typecheck
npm run build     # tsc -> lib/

---

License

MIT — see LICENSE. © 2026 dsh-semantic-search contributors.