Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@
node_modules/
vendor/
wp-agentic-admin.zip
agentic-admin.zip
*.pdf
tests/e2e/screenshots/
tests/e2e/RESULTS.md
Expand Down
19 changes: 13 additions & 6 deletions docs/RAG-CODEBASE-SEARCH.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ The RAG (Retrieval-Augmented Generation) system lets the LLM answer questions ab

1. **Extracting** code from your active theme and plugins (PHP backend)
2. **Embedding** code chunks into vectors using Transformers.js (CPU/WASM)
3. **Storing** vectors in a Voy search index, persisted in IndexedDB
3. **Storing** vectors in IndexedDB, as plain `Float32Array`s
4. **Searching** with semantic similarity when users ask about code

All processing happens locally — no code leaves the browser.
Expand All @@ -22,7 +22,7 @@ User: "find the login function"
↓
[Transformers.js] → embed query on CPU/WASM
↓
[Voy index] → nearest neighbor search across 1000+ code chunks
[vector index] → cosine similarity scan across 1000+ code chunks
↓
[Results] → top 3 matching code snippets with file paths + line numbers
↓
Expand All @@ -43,9 +43,16 @@ User: "find the login function"
| Dependency | How loaded | Size | Purpose |
|------------|-----------|------|---------|
| [Transformers.js v3](https://huggingface.co/docs/transformers.js) | CDN (lazy, on first use) | ~100MB + 23MB model | Text embeddings |
| [voy-search](https://github.com/tantaraio/voy) | Bundled via npm | ~168KB WASM | Vector nearest-neighbor search |
| IndexedDB | Browser native | — | Persist index across sessions |

**Why no vector-search library?** Search is an exhaustive cosine scan written in
plain JavaScript in `vector-store.js`. The embedding model emits L2-normalised
vectors, so cosine similarity is a dot product; at 384 dimensions a few thousand
chunks score in single-digit milliseconds, far below the cost of embedding the
query itself. An ANN index buys nothing at this scale, and dropping it removed
the only WebAssembly binary the plugin distributed — every shipped file now has
readable source in this repository.

**Why CDN for Transformers.js?** At ~100MB it would 17x the current 5.8MB bundle. Lazy-loading from CDN means zero cost until RAG is actually used, and the model is cached by the browser after first download.

**Why CPU for embeddings?** The LLM already uses ~1.5GB VRAM via WebGPU. Running embeddings on GPU too would risk OOM. WASM/CPU is slower but avoids contention entirely.
Expand All @@ -68,14 +75,14 @@ User: "index codebase"

```
User: "search code for authentication"
→ Embeds query, searches Voy index
→ Embeds query, scores it against every indexed vector
→ Returns top 3 matching code snippets
→ LLM summarizes the results
```

### The index persists

After indexing once, the Voy index is restored from IndexedDB on page reload. No need to re-index unless your code changes.
After indexing once, the index is restored from IndexedDB on page reload. No need to re-index unless your code changes.

To rebuild: say **"reindex the codebase"**.

Expand Down Expand Up @@ -120,7 +127,7 @@ Returns 50 files per page. The JS ability paginates automatically until `has_mor

### Persistence

- Serialized Voy index stored in IndexedDB (`wp-agentic-rag-db`)
- Vectors stored as `Float32Array`s in IndexedDB (`wp-agentic-rag-db`, store `embedding-index`)
- Chunk metadata (path, lines, content, type) stored alongside
- Restored automatically on `vectorStore.init()`

Expand Down
9 changes: 1 addition & 8 deletions package-lock.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

7 changes: 3 additions & 4 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,8 @@
"test:abilities": "node tests/abilities/runner.js",
"test:e2e": "node tests/abilities/e2e-runner.js",
"watch": "wp-scripts start src/extensions/index.js --output-path=build-extensions",
"clean": "del-cli build-extensions wp-agentic-admin.zip",
"dist": "npm run clean && npm run build && rm -f wp-agentic-admin.zip && zip -r wp-agentic-admin.zip -@ < .distpackage",
"clean": "del-cli build-extensions agentic-admin.zip wp-agentic-admin.zip",
"dist": "npm run clean && npm run build && zip -r agentic-admin.zip -@ < .distpackage",
"test": "wp-scripts test-unit-js --testMatch='**/__tests__/**/*.test.js'",
"test:watch": "npm run test -- --watch",
"lint:js": "wp-scripts lint-js src/",
Expand All @@ -33,8 +33,7 @@
"@huggingface/transformers": "^3.8.1",
"@mlc-ai/web-llm": "^0.2.82",
"@wordpress/icons": "^12.0.0",
"fastest-levenshtein": "^1.0.16",
"voy-search": "^0.6.3"
"fastest-levenshtein": "^1.0.16"
},
"keywords": [
"wordpress",
Expand Down
5 changes: 4 additions & 1 deletion readme.txt
Original file line number Diff line number Diff line change
Expand Up @@ -83,7 +83,7 @@ Agentic Admin is fully open source under GPL-2.0-or-later. The complete, human-r

https://github.com/pluginslab/wp-agentic-admin

The files under `build-extensions/` are generated from the sources in `src/` with @wordpress/scripts (webpack). They include the JavaScript bundles (`index.js`, `sw.js`, `indexing-worker.js`, and code-split chunks), the CSS, and the voy-search vector-index WebAssembly module (`*.module.wasm`, built from its npm package; source: https://github.com/tantaraio/voy). To regenerate them from a checkout:
The files under `build-extensions/` are generated from the sources in `src/` with @wordpress/scripts (webpack). They contain only JavaScript and CSS: the bundles (`index.js`, `sw.js`, `indexing-worker.js`, and code-split chunks) and the stylesheets. No WebAssembly, binaries, or other compiled artifacts are distributed with the plugin. To regenerate them from a checkout:

1. `npm install`
2. `npm run build`
Expand All @@ -107,6 +107,9 @@ There is no build step for the PHP. The WebLLM engine is bundled into the plugin
* Improved: ChatInput keyboard handling simplified (Space inserts a space, no push-to-talk hijacking).
* Improved: KB embedding moved to a Web Worker with persistent progress across tab switches.
* Pinned: Transformers.js CDN URL to @3.8.1 (was floating @3 range), privacy-first plugin shouldn't depend on a CDN range that can ship new code without a deliberate bump.
* Removed: the voy-search dependency and the WebAssembly module it distributed. Vector search is now plain JavaScript (exhaustive cosine over L2-normalised embeddings), so the plugin ships no compiled binaries at all and every distributed file has readable source in the repository.
* Fixed: re-running the knowledge base index no longer leaves vectors and chunk metadata misaligned, which could return the wrong code chunk for a query.
* Fixed: `.well-known` scanning resolves via get_home_path() instead of ABSPATH, so subdirectory installs scan the real site root.
* Removed: 7 stale tab references and 6+ stale docs files (FEEDBACK-DEV.md).
* Tests: 96 unit tests passing, plus the new manifest test suite (7 cases), index test suite (6 cases), knowledge-base test suite (16 cases), and react-agent regression tests (3 cases for the per-call state cleanup fix).

Expand Down
2 changes: 1 addition & 1 deletion src/extensions/abilities/code-search.js
Original file line number Diff line number Diff line change
Expand Up @@ -96,7 +96,7 @@ export function registerCodeSearch() {

execute: async ( params ) => {
try {
// Initialize vector store (loads Voy, restores from IndexedDB).
// Initialize vector store (restores index from IndexedDB).
await vectorStore.init();

if ( ! vectorStore.isReady() ) {
Expand Down
2 changes: 1 addition & 1 deletion src/extensions/services/indexing-worker.js
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
*
* Runs Transformers.js embedding inside a Web Worker to avoid blocking
* the main thread during knowledge base builds. Only handles the slow
* part (neural network inference); the main thread builds the Voy index
* part (neural network inference); the main thread builds the index
* and persists to IndexedDB.
*
* Message protocol:
Expand Down
4 changes: 2 additions & 2 deletions src/extensions/services/knowledge-base.js
Original file line number Diff line number Diff line change
Expand Up @@ -106,7 +106,7 @@ function clearKBStatus() {
/* ── Worker helper ───────────────────────────────────────────────────── */

/**
* Embed chunks in a Web Worker, then build Voy index on main thread.
* Embed chunks in a Web Worker, then build the index on the main thread.
*
* @param {Object[]} chunks Chunks to index.
* @param {Function} onProgress Callback: (done, total, message) => void.
Expand All @@ -133,7 +133,7 @@ function indexInWorker( chunks, onProgress ) {
} else if ( msg.type === 'complete' ) {
worker.terminate();

// Build Voy index on main thread (fast, < 1s).
// Build the index on the main thread (fast, < 1s).
try {
onProgress(
msg.chunkMetadata.length,
Expand Down
Loading
Loading