diff --git a/.github/workflows/pr-quality.yml b/.github/workflows/pr-quality.yml index 9e20bcb563..5f4896f4d5 100644 --- a/.github/workflows/pr-quality.yml +++ b/.github/workflows/pr-quality.yml @@ -73,6 +73,7 @@ jobs: --exclude '^https://github\.com/tinyhumansai/openhuman/stargazers' --exclude '^https://api\.star-history\.com/' --exclude '^https://x\.com/karpathy/status/2039805659525644595$' + --exclude '^https://www\.producthunt\.com/' 'docs/**/*.md' 'src/**/README.md' '.github/PULL_REQUEST_TEMPLATE.md' diff --git a/AGENTS.md b/AGENTS.md index b25206b984..838cfbe184 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -226,7 +226,9 @@ Consequences worth knowing before touching either seam: the second belongs accepts an address that only fails later, at signing time. - **Each crate's gates ride OpenHuman's existing ones**: `tinydocs` is exclusive to `documents`, `tinywallet` to `web3`. Both are default-ON and - already forwarded to the desktop shell. + already forwarded to the desktop shell. Note `tinydocs` is now taken with + `default-features = false` — the wire contract, not the writers, which run in + the TinyBus module instead (see the module host section). ### Backend API access — `src/api/` over `tinyhumans-sdk` @@ -384,7 +386,7 @@ Two independent runtime axes on `CoreBuilder` (`src/core/runtime/builder.rs`): - **`ServiceSet`** selects which *background services / transports* run (`rpc_http`, `socketio`, `cron`, `channels`, `heartbeat`, …). Presets: `desktop()` / `headless_api()` / `none()`. - **`DomainSet`** selects which *domain families* exist at runtime, one flag per `DomainGroup` (`src/core/all.rs`). Presets: `full()` (default — byte-identical to before #4796), `harness()` (agent + memory + threads + config + security only), `none()`. Every controller is tagged with its `DomainGroup` at the single registration site in `src/core/all.rs`; the live surface (controllers/`/schema`/dispatch, agent tools, stores, subscribers) is filtered by the ambient `CoreContext::domains()`. A gated domain's controllers become unknown-method, its agent tools absent, its stores/subscribers uninitialized. `examples/embed_headless.rs` uses `DomainSet::harness()`; `examples/embed_kernel.rs` uses `DomainSet::kernel()` — the floor (threads + config + security, with `agent`/`memory` OFF) that a host opts subsystems back into by field assignment. Per-gate Cargo `[features]` (children #4797–#4804) narrow the compile-time surface further; `DomainSet` is the runtime axis they compose with. -**`DomainGroup` tracks family directories 1:1.** After the domain reorg (#5328) each variant names a `src/openhuman/` family, so the runtime axis stopped sweeping half the surface into the `Platform` catch-all. Groups: the harness families (`Agent`, `Memory`, `Threads`, `Config`, `Security`), the compile-gate families (`Flows`, `Skills`, `Mcp`, `Meet`, `Channels`, `Web3`, `Voice`, `Media`, `Medulla`), the families carved out of `Platform` (`Inference`, `Integrations`, `Automation` = cron + subconscious, `Runtimes` = runtime + sandbox, `Desktop`, `Hosted`, `Relay` = tinyplace), and `Platform` itself — now only the kernel surfaces with no family of their own (`platform/`, `tools/`, `http_host/`, `test_support/`). +**`DomainGroup` tracks family directories 1:1.** After the domain reorg (#5328) each variant names a `src/openhuman/` family, so the runtime axis stopped sweeping half the surface into the `Platform` catch-all. Groups: the harness families (`Agent`, `Memory`, `Threads`, `Config`, `Security`), the compile-gate families (`Flows`, `Skills`, `Mcp`, `Meet`, `Channels`, `Web3`, `Voice`, `Media`, `Medulla`), the families carved out of `Platform` (`Inference`, `Integrations`, `Automation` = cron + subconscious, `Runtimes` = runtime + sandbox, `Desktop`, `Hosted`, `Relay` = tinyplace, `Modules` = the native module host), and `Platform` itself — now only the kernel surfaces with no family of their own (`platform/`, `tools/`, `http_host/`, `test_support/`). That realignment fixed two real defects, both pinned by tests in `src/core/all_tests.rs`: @@ -411,7 +413,7 @@ Per-domain Cargo features drop whole domains **at compile time** (smaller binary | **Contributor** | `[features] default` in `Cargo.toml` | What a bare `cargo check`, `cargo test` and rust-analyzer compile. 9 cheap gates. **353 packages / 3 native builds** (`libsqlite3-sys`, `lzma-sys`, `ring`). | | **Product** | `scripts/ci/product-features.txt` | What the shipped desktop app has. 16 gates. **540 packages / 7 native builds** (adds `bzip2-sys`, `libgit2-sys`, `libz-sys`, `zstd-sys`). | -`default` used to be the product set, which made the inner loop pay for the whole product on every edit — web3's ethers/secp256k1 cohort, `documents`' zstd/bzip2 native builds, the cpal/hound/arboard/enigo/rdev stack behind `voice`+`inference`, `contacts`' macOS objc2 cohort, `crash-reporting`'s sentry tree, `tui`'s ratatui. Those are default-OFF now. **This did not change what ships**: the shell has set `default-features = false` since #1061 and never inherited `default` anyway. +`default` used to be the product set, which made the inner loop pay for the whole product on every edit — web3's ethers/secp256k1 cohort, `documents`' zstd/bzip2 native builds (since removed from the graph entirely — the codecs run in a module now), the cpal/hound/arboard/enigo/rdev stack behind `voice`+`inference`, `contacts`' macOS objc2 cohort, `crash-reporting`'s sentry tree, `tui`'s ratatui. Those are default-OFF now. **This did not change what ships**: the shell has set `default-features = false` since #1061 and never inherited `default` anyway. What it *did* change: **a lane that relies on default features no longer covers the product.** Every CI lane that builds or tests the product passes `--features "$(bash scripts/ci/product-features.sh)"` — clippy, the unit lane, the coverage lane, `scripts/test-rust-with-mock.sh`. If you add a lane, decide which of the two sets it is testing and say so in a comment. Four `tests/*.rs` targets carry `required-features` for the same reason (`json_rpc_e2e`, `raw_coverage_all`, `observability_smoke`, `x402_twit_sh_live`); without those gates cargo **silently skips** them and the run still exits 0 — the same trap `--bins` without `bin-tools` already had. @@ -484,6 +486,8 @@ Two columns because there are two sets (see above): **Contrib** is `[features] d | `inference` | OFF | ON | the `cpal` audio-device stack: microphone capture for voice, plus `desktop::accessibility::permissions`' mic-permission probe. Implied by `voice`. Off ⇒ the probe reports `Unknown`. **The name is historical** — it used to gate the bundled whisper.cpp STT engine, which no longer exists (see the scope note below); do not rename it, it is forwarded by name from the shell manifest and asserted by `INFERENCE_COMPILED_IN` | `cpal` | | `web3` | OFF | ON | the `openhuman::web3` family (`web3`, `web3::wallet`, `web3::x402`) — crypto wallet (multi-chain sign/broadcast), swaps/bridges/dapp calls, x402 machine payments | `bitcoin`, `curve25519-dalek` | | `media` | ON | ON | `openhuman::media::generation` (the `media_generate_*` agent tools) + `openhuman::media::image` scaffold | none (surface-only) | +| `documents` | OFF | ON | the `generate_document` / `generate_presentation` agent tools and PDF text extraction during multimodal ingest. **The synthesis is not in this build** — all three run in the `tinydocs` TinyBus module (see below), so this gate turns on the tools and the host policy around them: the artifact pipeline, the deadlines, image resolution under the security policy. `tinydocs` is consumed with `default-features = false`, for the wire contract only. Implies `modules`. Off ⇒ both tools absent from the tool list rather than degraded, and PDF ingest degrades a file to a reference instead of extracted text | **39 crates**, and they leave `Cargo.lock` entirely: `docx-rs`, `ppt-rs`, `pdf-extract` plus `lopdf`, `syntect`, `pulldown-cmark`, `xml-rs`, `quick-xml`, `zip 0.6`, `zstd`, `bzip2`, `encoding_rs`, `euclid`, `ttf-parser`, the CFF/Type1/CMap parsers, … Product profile 505 → 448 names | +| `modules` | ON | ON | `openhuman::modules` — the dynamic module host: the loader that admits a compiled `cdylib` through tinybus's ABI descriptor, manifest, dependency and SHA-256 gates, the compiled-in registry of modules this build trusts, and the `modules` RPC namespace. Implied by `documents`. Off ⇒ `modules.*` is unknown-method and nothing can load a native module | none in the product profile (`ureq`, `flate2`, `tar`, `zip 2`, `tempfile`, `toml` are already there) — **but see the kernel-floor note**: this feature exists so `tinybus/modules` is not enabled on the dependency itself, which would put a `dlopen` loader into the kernel profile where `tinybus` is always-on | | `meet` | OFF | ON | `openhuman::meet` (join-URL validation) + `openhuman::meet::agent` (live STT/LLM/TTS loop) + `openhuman::meet::backend_bot` (backend-delegated Meet bot over Socket.IO) | none — see note | | `skills` | ON | ON | `openhuman::skills` + `openhuman::skills::runtime` + `openhuman::skills::catalog` domains — SKILL.md discovery/parse/install, workflow execution + run logs, remote catalogs, the `skill_setup` / `skill_executor` builtin agents, and the 16 skill agent tools | none (see below) | | `flows` | ON | ON | `openhuman::flows` (saved automation graphs — create/run/schedule, the `workflow_builder` + `flow_discovery` agents), `openhuman::flows::tinyflows` (engine seam), `openhuman::flows::rhai` (`.ragsh` language-workflow tool) | `tinyflows`, `jaq-core`, `jaq-std`, `jaq-json`, `rhai` | @@ -571,6 +575,88 @@ Follows the voice facade+stub pattern for `mcp::server` / `mcp::registry` / `mcp `src/core/all.rs` needs **no** `#[cfg]` for this gate: the stub aggregators return empty vecs, so the registration sites keep compiling unchanged. +### Loadable native modules — `src/openhuman/modules/` + +A capability can live outside this binary. A module is a compiled `cdylib` +speaking the tinybus module ABI: downloaded from a pinned release, verified +against a digest compiled into `modules::registry`, admitted through tinybus's +ABI and manifest gates, and attached to a private in-process broker as an +ordinary bus peer. The core then calls it over that bus like any other service. +`documents` is the first consumer — `.docx` / `.pptx` synthesis and PDF +extraction all happen in the `tinydocs` module. + +**What it buys is a dependency boundary that survives compilation.** A codec is +not kernel work, and each one drags a tree of parsers into a binary that mostly +does something else. Moving one out removes its dependencies from the build +rather than merely gating them: `documents` went from 39 crates to none. + +**What it costs is process isolation, and that is not small.** A loaded module +shares this address space, these privileges and this crash domain; tinybus's +deadlines, bounded queues and caught panics contain ordinary misbehaviour, not a +segfault. `dlopen` runs code before any symbol can be inspected, so the ABI, +manifest and digest gates decide what is **admitted**, never what is **safe**. +Modules are first-party code that ships separately. Anything untrusted belongs in +a process. + +**tinybus never unloads a library.** A module that is refused or faulted is +failed until the process restarts, which is why `modules::ops` caches failures +instead of retrying — the alternative is paying a download and a `dlopen` per +tool call to reach the same error. + +Five decisions worth knowing before touching this: + +- **The registry is a compiled-in `const` table.** Which modules exist, which + interfaces they claim, and which bytes are legitimate are build-time decisions. + Neither config nor RPC can name an artifact: a registry a server could add + entries to would be remote code execution with a download step. `[modules]` + config controls only whether modules load, whether this host may fetch them, + and where a developer's own build lives. +- **Digests are pinned in source as the host's half of a two-sided check.** + tinybus fetches the release's own `checksum.toml`, compares it with ours, + hashes the download, and extracts only after. Pinning here makes the check + auditable offline and makes a release re-cut under the same tag stop matching + rather than silently replacing what runs in-process. Take the values verbatim + from the release; never recompute them from a local build. +- **Artifact selection returns an ordered list, not one answer.** A target triple + is not enough — a `.so` built against glibc 2.39 fails to `dlopen` on a 2.35 + host with a symbol-version error the ABI gate cannot phrase helpfully. So + releases publish per-distro artifacts, `modules::platform` probes glibc, prefers + the newest build that could work, and falls through on admission failure. A musl + or BSD host gets an empty list: "unsupported" beats a download that cannot load. +- **Admission is permissive, deliberately.** Strict mode additionally refuses a + module whose rustc version differs from the host's, and the real published + artifact **is** refused that way — released artifacts are built on whatever + toolchain CI had and this crate pins its own, so mismatch is the normal case. + Strict mode would have meant the feature never worked in the field while every + local build looked fine. Everything protecting the address space is still + enforced; only the toolchain string is relaxed. +- **Modules run on their own broker**, because `OnceBus::init_in_process` builds + its `Broker` privately and `ModuleHost::new` needs one. The consequence: a + module cannot publish a `DomainEvent`. Fine for a codec; revisit if a module + ever needs to emit events. + +**The bus belongs to whichever runtime creates it.** In the core that is the one +runtime the process has. In tests it is not: two `#[tokio::test]` functions each +build their own, and the second to call a loaded module finds a broker whose tasks +died with the first — the call **hangs** until some deadline above it fires. Any +test driving a real module must be the only one in its process, which is why the +module-backed tool tests are `#[ignore]`d rather than merely gated on an artifact. +Run them one at a time with `OPENHUMAN_MODULE_PATH` pointing at a directory +holding the built library. + +**Payloads in and out are not symmetric.** Inbound bytes ride a tinybus stream +opened alongside the call, so flow control and the size cap are the bus's. Replies +cannot: `Interface::call` receives no caller identity and no connection, so a +served object cannot open a stream back to its caller. A produced document is held +by the module and pulled in chunks. A reply-stream seam upstream would remove that +half. + +**`modules` must not be enabled on the tinybus dependency directly.** `tinybus` is +always-on kernel surface, so `features = ["modules"]` there puts a loader plus +`ureq` and an archive stack into the kernel profile for a host that can never use +one — 305 → 308 packages, which the kernel-floor ratchet caught. It is forwarded +from this crate's own `modules` feature instead. + #### The `tui` gate The tabbed terminal UI (`openhuman`, or explicitly `openhuman tui` / alias `chat`) lives in `src/openhuman/tui/` and follows the **`mcp`/`voice` facade+stub** pattern: `pub mod tui;` is always compiled; the behavioural submodules (`app`, `render`, `state`, `terminal`, `runner`) are `#[cfg(feature = "tui")]`; and `#[cfg(not(feature = "tui"))] mod stub;` re-exposes the one symbol an always-compiled caller reaches — `run_from_cli` — with a build-fact error body (`"tui feature disabled at compile time … --features tui"`). Bare-command auto-launch requires terminal stdin/stdout and `HostKind::Cli`; Docker, CI, pipes, and `--no-tui` retain the non-TUI CLI path. diff --git a/Cargo.lock b/Cargo.lock index aa331b13f3..8b8c37ed9d 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -17,15 +17,6 @@ version = "2.0.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "320119579fcad9c21884f5c4861d16174d0e06250625266f50fe6898340abefa" -[[package]] -name = "adobe-cmap-parser" -version = "0.4.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ae8abfa9a4688de8fc9f42b3f013b6fffec18ed8a554f5f113577e0b9b3212a3" -dependencies = [ - "pom", -] - [[package]] name = "aead" version = "0.5.2" @@ -224,7 +215,7 @@ dependencies = [ "base64ct", "blake2", "cpufeatures 0.2.17", - "password-hash 0.5.0", + "password-hash", ] [[package]] @@ -511,15 +502,6 @@ version = "0.11.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "32637268377fc7b10a8c6d51de3e7fba1ce5dd371a96e342b34e6078db558e7f" -[[package]] -name = "bincode" -version = "1.3.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b1f45e9417d87227c7a56d22e471c6206462cba514c7590c09aff4cf6d1ddcad" -dependencies = [ - "serde", -] - [[package]] name = "bindgen" version = "0.72.1" @@ -739,12 +721,6 @@ version = "1.2.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7575182f7272186991736b70173b0ea045398f984bf5ebbb3804736ce1330c9d" -[[package]] -name = "bytecount" -version = "0.6.9" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "175812e0be2bccb6abe50bb8d566126198344f707e304f45c648fd8f2cc0365e" - [[package]] name = "bytemuck" version = "1.25.0" @@ -772,26 +748,6 @@ dependencies = [ "serde", ] -[[package]] -name = "bzip2" -version = "0.4.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "bdb116a6ef3f6c3698828873ad02c3014b3c85cadb88496095628e3ef1e347f8" -dependencies = [ - "bzip2-sys", - "libc", -] - -[[package]] -name = "bzip2-sys" -version = "0.1.13+1.0.8" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "225bff33b2141874fe80d71e07d6eec4f85c5c216453dd96388240f96e1acc14" -dependencies = [ - "cc", - "pkg-config", -] - [[package]] name = "castaway" version = "0.2.4" @@ -837,12 +793,6 @@ dependencies = [ "nom 7.1.3", ] -[[package]] -name = "cff-parser" -version = "0.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "31f5b6e9141c036f3ff4ce7b2f7e432b0f00dee416ddcd4f17741d189ddc2e9d" - [[package]] name = "cfg-if" version = "1.0.4" @@ -1067,12 +1017,6 @@ dependencies = [ "thiserror 1.0.69", ] -[[package]] -name = "color_quant" -version = "1.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3d7b894f5411737b7867f4827955924d7c254fc9f4d91a6aad6b097804b1018b" - [[package]] name = "colorchoice" version = "1.0.5" @@ -1193,12 +1137,6 @@ dependencies = [ "unicode-xid", ] -[[package]] -name = "constant_time_eq" -version = "0.1.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "245097e9a4535ee1e3e3931fcfcd55a796a44c643e8596ff6566d68f09b87bbc" - [[package]] name = "convert_case" version = "0.10.0" @@ -1846,21 +1784,6 @@ dependencies = [ "litrs", ] -[[package]] -name = "docx-rs" -version = "0.4.20" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ed73cbf5e1c37baa23f4132569ac1187829f03922c206bd68fe109e3001a343d" -dependencies = [ - "base64 0.22.1", - "image", - "quick-xml 0.36.2", - "serde", - "serde_json", - "thiserror 2.0.18", - "zip 0.6.6", -] - [[package]] name = "dotenvy" version = "0.15.7" @@ -1879,15 +1802,6 @@ version = "1.0.20" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "d0881ea181b1df73ff77ffaaf9c7544ecc11e82fba9b5f27b262a3c73a332555" -[[package]] -name = "ecb" -version = "0.1.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1a8bfa975b1aec2145850fcaa1c6fe269a16578c44705a532ae3edc92b8881c7" -dependencies = [ - "cipher", -] - [[package]] name = "ecdsa" version = "0.16.9" @@ -1968,15 +1882,6 @@ version = "0.2.9" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "e079f19b08ca6239f47f8ba8509c11cf3ea30095831f7fed61441475edd8c449" -[[package]] -name = "encoding_rs" -version = "0.8.35" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "75030f3c4f45dafd7586dd6780965a8c7e8e285a5ecb86713e63a79c5b2766f3" -dependencies = [ - "cfg-if", -] - [[package]] name = "engineioxide" version = "0.15.2" @@ -2204,15 +2109,6 @@ dependencies = [ "tracing", ] -[[package]] -name = "euclid" -version = "0.20.14" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2bb7ef65b3777a325d1eeefefab5b6d4959da54747e33bd6258e789640f307ad" -dependencies = [ - "num-traits", -] - [[package]] name = "euclid" version = "0.22.14" @@ -2271,17 +2167,6 @@ dependencies = [ "regex", ] -[[package]] -name = "fancy-regex" -version = "0.16.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "998b056554fbe42e03ae0e152895cd1a7e1002aec800fdc6635d20270260c46f" -dependencies = [ - "bit-set 0.8.0", - "regex-automata", - "regex-syntax", -] - [[package]] name = "fantoccini" version = "0.22.1" @@ -2633,15 +2518,6 @@ dependencies = [ "windows-link", ] -[[package]] -name = "getopts" -version = "0.2.24" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "cfe4fbac503b8d1f88e6676011885f34b7174f46e59956bba534ba83abded4df" -dependencies = [ - "unicode-width", -] - [[package]] name = "getrandom" version = "0.2.17" @@ -2702,16 +2578,6 @@ dependencies = [ "polyval 0.7.1", ] -[[package]] -name = "gif" -version = "0.14.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ee8cfcc411d9adbbaba82fb72661cc1bcca13e8bba98b364e62b2dba8f960159" -dependencies = [ - "color_quant", - "weezl", -] - [[package]] name = "gimli" version = "0.32.3" @@ -3268,14 +3134,10 @@ checksum = "85ab80394333c02fe689eaf900ab500fbd0c2213da414687ebf995a65d5a6104" dependencies = [ "bytemuck", "byteorder-lite", - "color_quant", - "gif", "moxcms", "num-traits", "png", "tiff", - "zune-core", - "zune-jpeg", ] [[package]] @@ -3806,34 +3668,6 @@ version = "0.4.29" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "5e5032e24019045c762d3c0f28f5b6b8bbf38563a65908389bf7978758920897" -[[package]] -name = "lopdf" -version = "0.38.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c7184fdea2bc3cd272a1acec4030c321a8f9875e877b3f92a53f2f6033fdc289" -dependencies = [ - "aes", - "bitflags 2.13.1", - "cbc", - "ecb", - "encoding_rs", - "flate2", - "getrandom 0.3.4", - "indexmap", - "itoa", - "log", - "md-5", - "nom 8.0.0", - "nom_locate", - "rand 0.9.4", - "rangemap", - "sha2 0.10.9", - "stringprep", - "thiserror 2.0.18", - "ttf-parser", - "weezl", -] - [[package]] name = "lru" version = "0.18.1" @@ -3921,16 +3755,6 @@ version = "0.8.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "47e1ffaa40ddd1f3ed91f717a33c8c0ee23fff369e3aa8772b9605cc1d22f4c3" -[[package]] -name = "md-5" -version = "0.10.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d89e7ee0cfbedfc4da3340218492196241d89eefb6dab27de5df917a6d2e78cf" -dependencies = [ - "cfg-if", - "digest 0.10.7", -] - [[package]] name = "md5" version = "0.8.0" @@ -4146,17 +3970,6 @@ dependencies = [ "memchr", ] -[[package]] -name = "nom_locate" -version = "5.0.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0b577e2d69827c4740cba2b52efaad1c4cc7c73042860b199710b3575c68438d" -dependencies = [ - "bytecount", - "memchr", - "nom 8.0.0", -] - [[package]] name = "ntapi" version = "0.4.3" @@ -4693,8 +4506,6 @@ dependencies = [ "objc2-foundation 0.3.2", "once_cell", "parking_lot", - "pdf-extract", - "ppt-rs", "proptest", "rand 0.10.1", "ratatui", @@ -4753,7 +4564,7 @@ dependencies = [ "x25519-dalek", "xz2", "zeroize", - "zip 2.4.2", + "zip", ] [[package]] @@ -4917,17 +4728,6 @@ dependencies = [ "windows-link", ] -[[package]] -name = "password-hash" -version = "0.4.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7676374caaee8a325c9e7a2ae557f216c5563a171d6997b0ef8a65af35147700" -dependencies = [ - "base64ct", - "rand_core 0.6.4", - "subtle", -] - [[package]] name = "password-hash" version = "0.5.0" @@ -4946,9 +4746,6 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "83a0692ec44e4cf1ef28ca317f14f8f07da2d95ec3fa01f86e4467b725e60917" dependencies = [ "digest 0.10.7", - "hmac", - "password-hash 0.4.2", - "sha2 0.10.9", ] [[package]] @@ -4961,23 +4758,6 @@ dependencies = [ "hmac", ] -[[package]] -name = "pdf-extract" -version = "0.10.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1e28ba1758a3d3f361459645780e09570b573fc3c82637449e9963174c813a98" -dependencies = [ - "adobe-cmap-parser", - "cff-parser", - "encoding_rs", - "euclid 0.20.14", - "log", - "lopdf", - "postscript", - "type1-encoding-parser", - "unicode-normalization", -] - [[package]] name = "percent-encoding" version = "2.3.2" @@ -5201,7 +4981,7 @@ checksum = "740ebea15c5d1428f910cd1a5f52cebf8d25006245ed8ade92702f4943d91e07" dependencies = [ "base64 0.22.1", "indexmap", - "quick-xml 0.38.4", + "quick-xml", "serde", "time", ] @@ -5253,12 +5033,6 @@ dependencies = [ "universal-hash 0.6.1", ] -[[package]] -name = "pom" -version = "1.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "60f6ce597ecdcc9a098e7fddacb1065093a3d66446fa16c675e7e71d1b5c28e6" - [[package]] name = "portable-atomic" version = "1.13.1" @@ -5274,12 +5048,6 @@ dependencies = [ "portable-atomic", ] -[[package]] -name = "postscript" -version = "0.14.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "78451badbdaebaf17f053fd9152b3ffb33b516104eacb45e7864aaa9c712f306" - [[package]] name = "potential_utf" version = "0.1.5" @@ -5295,23 +5063,6 @@ version = "0.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "439ee305def115ba05938db6eb1644ff94165c5ab5e9420d1c1bcedbba909391" -[[package]] -name = "ppt-rs" -version = "0.2.14" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "eb145020aba8cd682d92b8d00174f4b81fc53feec7919abc15357b9f011c2df5" -dependencies = [ - "chrono", - "clap", - "pulldown-cmark", - "regex", - "syntect", - "thiserror 1.0.69", - "uuid 1.23.1", - "xml-rs", - "zip 0.6.6", -] - [[package]] name = "ppv-lite86" version = "0.2.21" @@ -5431,25 +5182,6 @@ dependencies = [ "prost", ] -[[package]] -name = "pulldown-cmark" -version = "0.10.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "76979bea66e7875e7509c4ec5300112b316af87fa7a252ca91c448b32dfe3993" -dependencies = [ - "bitflags 2.13.1", - "getopts", - "memchr", - "pulldown-cmark-escape", - "unicase", -] - -[[package]] -name = "pulldown-cmark-escape" -version = "0.10.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "bd348ff538bc9caeda7ee8cad2d1d48236a1f443c1fa3913c6a02fe0043b1dd3" - [[package]] name = "pxfm" version = "0.1.29" @@ -5468,16 +5200,6 @@ version = "2.0.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "a993555f31e5a609f617c12db6250dedcac1b0a85076912c436e6fc9b2c8e6a3" -[[package]] -name = "quick-xml" -version = "0.36.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f7649a7b4df05aed9ea7ec6f628c67c9953a43869b8bc50929569b2999d443fe" -dependencies = [ - "encoding_rs", - "memchr", -] - [[package]] name = "quick-xml" version = "0.38.4" @@ -5660,12 +5382,6 @@ dependencies = [ "rand_core 0.9.5", ] -[[package]] -name = "rangemap" -version = "1.7.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "973443cf09a9c8656b574a866ab68dfa19f0867d0340648c7d2f6a71b8a8ea68" - [[package]] name = "ratatui" version = "0.30.2" @@ -6832,17 +6548,6 @@ dependencies = [ "pin-project-lite", ] -[[package]] -name = "stringprep" -version = "0.1.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7b4df3d392d81bd458a8a621b8bffbd2302a12ffe288a9d931670948749463b1" -dependencies = [ - "unicode-bidi", - "unicode-normalization", - "unicode-properties", -] - [[package]] name = "strsim" version = "0.11.1" @@ -6946,24 +6651,6 @@ dependencies = [ "syn 2.0.117", ] -[[package]] -name = "syntect" -version = "5.3.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "656b45c05d95a5704399aeef6bd0ddec7b2b3531b7c9e900abbf7c4d2190c925" -dependencies = [ - "bincode", - "fancy-regex 0.16.2", - "flate2", - "fnv", - "once_cell", - "regex-syntax", - "serde", - "serde_derive", - "thiserror 2.0.18", - "walkdir", -] - [[package]] name = "sysinfo" version = "0.33.1" @@ -7050,7 +6737,7 @@ dependencies = [ "anyhow", "base64 0.22.1", "bitflags 2.13.1", - "fancy-regex 0.11.0", + "fancy-regex", "filedescriptor", "finl_unicode", "fixedbitset 0.4.2", @@ -7224,12 +6911,18 @@ name = "tinybus" version = "0.1.0" dependencies = [ "async-trait", + "flate2", "serde", "serde_json", + "tar", + "tempfile", "thiserror 2.0.18", "tinybus-macros", "tokio", + "toml 0.8.23", "tracing", + "ureq", + "zip", ] [[package]] @@ -7333,9 +7026,8 @@ dependencies = [ [[package]] name = "tinydocs" -version = "0.1.0" +version = "0.1.12" dependencies = [ - "docx-rs", "serde", "thiserror 2.0.18", ] @@ -7854,12 +7546,6 @@ version = "0.2.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "e421abadd41a4225275504ea4d6566923418b7f05506fbc9c0fe86ba7396114b" -[[package]] -name = "ttf-parser" -version = "0.25.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d2df906b07856748fa3f6e0ad0cbaa047052d4a7dd609e231c4f72cee8c36f31" - [[package]] name = "tungstenite" version = "0.24.0" @@ -7897,15 +7583,6 @@ dependencies = [ "thiserror 2.0.18", ] -[[package]] -name = "type1-encoding-parser" -version = "0.1.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "fa10c302f5a53b7ad27fd42a3996e23d096ba39b5b8dd6d9e683a05b01bee749" -dependencies = [ - "pom", -] - [[package]] name = "typed-arena" version = "2.0.2" @@ -7977,33 +7654,12 @@ version = "2.9.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "dbc4bc3a9f746d862c45cb89d705aa10f187bb96c76001afab07a0d35ce60142" -[[package]] -name = "unicode-bidi" -version = "0.3.18" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5c1cb5db39152898a79168971543b1cb5020dff7fe43c8dc468b0885f5e29df5" - [[package]] name = "unicode-ident" version = "1.0.24" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "e6e4313cd5fcd3dad5cafa179702e2b244f760991f45397d14d4ebf38247da75" -[[package]] -name = "unicode-normalization" -version = "0.1.25" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5fd4f6878c9cb28d874b009da9e8d183b5abc80117c40bbd187a1fde336be6e8" -dependencies = [ - "tinyvec", -] - -[[package]] -name = "unicode-properties" -version = "0.1.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7df058c713841ad818f1dc5d3fd88063241cc61f49f5fbea4b951e8cf5a8d71d" - [[package]] name = "unicode-segmentation" version = "1.13.2" @@ -8083,6 +7739,7 @@ checksum = "dea7109cdcd5864d4eeb1b58a1648dc9bf520360d7af16ec26d0a9354bafcfc0" dependencies = [ "base64 0.22.1", "cookie_store", + "flate2", "log", "percent-encoding", "rustls", @@ -8636,7 +8293,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7012add459f951456ec9d6c7e6fc340b1ce15d6fc9629f8c42853412c029e57e" dependencies = [ "bitflags 1.3.2", - "euclid 0.22.14", + "euclid", "lazy_static", "serde", "wezterm-dynamic", @@ -9390,12 +9047,6 @@ version = "0.2.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b9cc00251562a284751c9973bace760d86c0276c471b4be569fe6b068ee97a56" -[[package]] -name = "xml-rs" -version = "0.8.28" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3ae8337f8a065cfc972643663ea4279e04e7256de865aa66fe25cec5fb912d3f" - [[package]] name = "xz2" version = "0.1.7" @@ -9568,26 +9219,6 @@ dependencies = [ "syn 2.0.117", ] -[[package]] -name = "zip" -version = "0.6.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "760394e246e4c28189f19d488c058bf16f564016aefac5d32bb1f3b51d5e9261" -dependencies = [ - "aes", - "byteorder", - "bzip2", - "constant_time_eq", - "crc32fast", - "crossbeam-utils", - "flate2", - "hmac", - "pbkdf2 0.11.0", - "sha1", - "time", - "zstd", -] - [[package]] name = "zip" version = "2.4.2" @@ -9629,35 +9260,6 @@ dependencies = [ "simd-adler32", ] -[[package]] -name = "zstd" -version = "0.11.2+zstd.1.5.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "20cc960326ece64f010d2d2107537f26dc589a6573a316bd5b1dba685fa5fde4" -dependencies = [ - "zstd-safe", -] - -[[package]] -name = "zstd-safe" -version = "5.0.2+zstd.1.5.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1d2a5585e04f9eea4b2a3d1eca508c4dee9592a89ef6f450c11719da0726f4db" -dependencies = [ - "libc", - "zstd-sys", -] - -[[package]] -name = "zstd-sys" -version = "2.0.16+zstd.1.5.7" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "91e19ebc2adc8f83e43039e79776e3fda8ca919132d68a1fed6a5faca2683748" -dependencies = [ - "cc", - "pkg-config", -] - [[package]] name = "zune-core" version = "0.5.1" diff --git a/Cargo.toml b/Cargo.toml index a21f5ca0d6..4099f978ec 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -246,6 +246,13 @@ tinychannels = { version = "0.1", features = ["relay-websocket"] } # `uds` is on so the kernel can join a broker that out-of-process integrations # are already attached to (`core::bus::init_over_socket`). `cli` is not: the # `tinybus` binary is a developer tool, not something the kernel links. +# +# The dynamic module loader is NOT enabled here. It arrives through this +# crate's own default-ON `modules` feature, which forwards `tinybus/modules`. +# Enabling it unconditionally would put a `dlopen` loader in the kernel +# profile — `tinybus` is always-on surface, so `--no-default-features +# --features flows` would carry `ureq` and the archive stack for a host that +# never loads a module. The kernel-floor ratchet catches exactly that, and did. tinybus = { path = "vendor/tinybus/crates/tinybus", default-features = false, features = ["macros", "uds"] } serde = { version = "1", features = ["derive"] } serde_json = "1" @@ -399,11 +406,9 @@ coins-bip39 = { version = "0.8", optional = true } curve25519-dalek = { version = "4", default-features = false, features = ["alloc"], optional = true } fantoccini = { version = "0.22.0", optional = true, default-features = false, features = ["rustls-tls"] } -pdf-extract = { version = "0.10", optional = true } # The WhatsApp Web provider (and its `whatsapp-rust` / `wacore` / `serde-big-array` # stack) now lives in the tinychannels crate; the `whatsapp-web` feature forwards # to `tinychannels/whatsapp-web`. -ppt-rs = { version = "0.2.14", optional = true } # Terminal chat UI (`openhuman tui` / `chat`). Exclusive to the default-ON # `tui` feature (see `[features]` below): a slim / headless build without `tui` # drops both `ratatui` and `crossterm`. Kept in lockstep — ratatui 0.30 @@ -415,12 +420,20 @@ crossterm = { version = "0.29", optional = true } # Terminal column-width measurement for the `tui` chat renderer # (`src/tui/render.rs`); only compiled behind `#[cfg(feature = "tui")]`. unicode-width = { version = "0.2", optional = true } -# TinyDocs — host-agnostic document synthesis. Owns the `.docx` spec types, -# their size limits, the validation rules, and the OOXML writer (`docx-rs` -# lives behind it now, not here). OpenHuman keeps only the host policy the -# crate deliberately refuses to guess at: the artifact pipeline, the -# `spawn_blocking` hop, and the generation deadline — see -# `src/openhuman/tools/impl/document/`. +# TinyDocs — the document wire contract, and nothing else. +# +# `default-features = false` is load-bearing: it takes the spec types, their +# size limits, the validation rules and the PNG/JPEG header reader, and leaves +# every writer behind. Synthesis happens in the `tinydocs` TinyBus module +# (`src/openhuman/modules/`), so this build carries the shape of a document +# without carrying the code that produces one — which is why `docx-rs`, +# `ppt-rs` and `pdf-extract` are absent from the graph entirely rather than +# merely gated. +# +# Sharing the contract rather than re-declaring it is the point. The specs are +# what an LLM is shown as a JSON tool schema and what the module validates +# against; two definitions of that would drift, and the drift would be a tool +# description promising limits the module does not enforce. # # Vendored as a path dependency like `tinyhumans-sdk`: the crate is not # published to crates.io, so there is no `[patch.crates-io]` entry for it. @@ -617,15 +630,39 @@ http-server = ["dep:axum", "dep:socketioxide"] # nothing and would silently drop the microphone probe from the shipped app if # the forwarding list were not updated in lockstep. inference = ["dep:cpal"] -# Office-document tools: the `generate_presentation` (ppt-rs) and -# `generate_document` (docx-rs) agent tools, plus `pdf-extract` for PDF text -# extraction during multimodal file ingest. Default-ON. Slim / headless builds -# opt out via `--no-default-features --features ""`, which drops all three crates. Leaf gate (no stub facade): when -# off, the two tools are absent from the tool list rather than degraded to an -# error, and PDF ingest degrades the file to a reference instead of extracted -# text (`agent::multimodal::extract_pdf_text`). -documents = ["dep:pdf-extract", "dep:ppt-rs", "dep:tinydocs", "tinydocs/docx"] +# Office-document tools: the `generate_document` and `generate_presentation` +# agent tools, plus PDF text extraction during multimodal file ingest. +# Default-ON. +# +# The synthesis is NOT in this build. All three run in the `tinydocs` TinyBus +# module, downloaded and verified at first use (`src/openhuman/modules/`), so +# this gate turns on the tools and the host policy around them — the artifact +# pipeline, the deadlines, image resolution under the security policy — and +# `tinydocs` is here for the wire contract only. `docx-rs`, `ppt-rs` and +# `pdf-extract` are not in the dependency graph in any configuration. +# +# Leaf gate (no stub facade): when off, the two tools are absent from the tool +# list rather than degraded to an error, and PDF ingest degrades the file to a +# reference instead of extracted text +# (`agent::multimodal::extract_pdf_text`). Slim / headless builds opt out via +# `--no-default-features --features ""`. +documents = ["dep:tinydocs", "modules"] +# The dynamic module host (`openhuman::modules`): the loader that admits a +# compiled `cdylib` through tinybus's ABI descriptor, manifest, dependency and +# SHA-256 gates, plus the `modules` RPC namespace and the registry of modules +# this build trusts. Default-ON, and implied by every gate whose work happens in +# a module — today that is `documents`. +# +# Off ⇒ the `modules.*` controllers are unregistered (unknown-method over +# `/rpc`, absent from `/schema`) and nothing can load a native module. Forwarding +# `tinybus/modules` here rather than on the dependency itself keeps the loader +# out of the kernel profile, where `tinybus` is always-on: a workflow-only host +# should not carry `ureq` and an archive stack to support a loader it cannot use. +# +# A loaded module is trusted in-process code with this process's privileges, and +# tinybus never unloads one. See `src/openhuman/modules/` before adding an entry +# to the registry. +modules = ["tinybus/modules"] # Voice + audio_toolkit domains: STT/TTS providers, the standalone dictation # server, always-on listening, and podcast audio generation/email delivery. # Default-ON — the desktop app always ships with voice. Slim / headless builds diff --git a/app/src-tauri/Cargo.lock b/app/src-tauri/Cargo.lock index 34fed5bbc4..00b0fb977e 100644 --- a/app/src-tauri/Cargo.lock +++ b/app/src-tauri/Cargo.lock @@ -72,15 +72,6 @@ version = "2.0.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "320119579fcad9c21884f5c4861d16174d0e06250625266f50fe6898340abefa" -[[package]] -name = "adobe-cmap-parser" -version = "0.4.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ae8abfa9a4688de8fc9f42b3f013b6fffec18ed8a554f5f113577e0b9b3212a3" -dependencies = [ - "pom", -] - [[package]] name = "aead" version = "0.5.2" @@ -264,7 +255,7 @@ dependencies = [ "base64ct", "blake2", "cpufeatures 0.2.17", - "password-hash 0.5.0", + "password-hash", ] [[package]] @@ -633,6 +624,12 @@ version = "0.22.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "72b3254f16251a8381aa12e40e3c4d2f0199f8c6508fbecb9d91f575e0fbb8c6" +[[package]] +name = "base64" +version = "0.23.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ac07cdecf99051d9a5238b80f35af32cdeba5b336e55d957b318b50137e18da5" + [[package]] name = "base64ct" version = "1.8.3" @@ -865,12 +862,6 @@ version = "1.2.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7575182f7272186991736b70173b0ea045398f984bf5ebbb3804736ce1330c9d" -[[package]] -name = "bytecount" -version = "0.6.9" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "175812e0be2bccb6abe50bb8d566126198344f707e304f45c648fd8f2cc0365e" - [[package]] name = "bytemuck" version = "1.25.0" @@ -898,26 +889,6 @@ dependencies = [ "serde", ] -[[package]] -name = "bzip2" -version = "0.4.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "bdb116a6ef3f6c3698828873ad02c3014b3c85cadb88496095628e3ef1e347f8" -dependencies = [ - "bzip2-sys", - "libc", -] - -[[package]] -name = "bzip2-sys" -version = "0.1.13+1.0.8" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "225bff33b2141874fe80d71e07d6eec4f85c5c216453dd96388240f96e1acc14" -dependencies = [ - "cc", - "pkg-config", -] - [[package]] name = "cairo-rs" version = "0.18.5" @@ -1032,12 +1003,6 @@ dependencies = [ "uuid 1.23.1", ] -[[package]] -name = "cff-parser" -version = "0.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "31f5b6e9141c036f3ff4ce7b2f7e432b0f00dee416ddcd4f17741d189ddc2e9d" - [[package]] name = "cfg-expr" version = "0.15.8" @@ -1217,12 +1182,6 @@ dependencies = [ "thiserror 1.0.69", ] -[[package]] -name = "color_quant" -version = "1.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3d7b894f5411737b7867f4827955924d7c254fc9f4d91a6aad6b097804b1018b" - [[package]] name = "colorchoice" version = "1.0.5" @@ -1329,12 +1288,6 @@ dependencies = [ "unicode-xid", ] -[[package]] -name = "constant_time_eq" -version = "0.1.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "245097e9a4535ee1e3e3931fcfcd55a796a44c643e8596ff6566d68f09b87bbc" - [[package]] name = "cookie" version = "0.18.1" @@ -1958,21 +1911,6 @@ dependencies = [ "const-random", ] -[[package]] -name = "docx-rs" -version = "0.4.20" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ed73cbf5e1c37baa23f4132569ac1187829f03922c206bd68fe109e3001a343d" -dependencies = [ - "base64 0.22.1", - "image", - "quick-xml 0.36.2", - "serde", - "serde_json", - "thiserror 2.0.18", - "zip 0.6.6", -] - [[package]] name = "dom_query" version = "0.27.0" @@ -2051,15 +1989,6 @@ version = "1.0.20" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "d0881ea181b1df73ff77ffaaf9c7544ecc11e82fba9b5f27b262a3c73a332555" -[[package]] -name = "ecb" -version = "0.1.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1a8bfa975b1aec2145850fcaa1c6fe269a16578c44705a532ae3edc92b8881c7" -dependencies = [ - "cipher", -] - [[package]] name = "ecdsa" version = "0.16.9" @@ -2160,15 +2089,6 @@ version = "1.2.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "4ef6b89e5b37196644d8796de5268852ff179b44e96276cf4290264843743bb7" -[[package]] -name = "encoding_rs" -version = "0.8.35" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "75030f3c4f45dafd7586dd6780965a8c7e8e285a5ecb86713e63a79c5b2766f3" -dependencies = [ - "cfg-if", -] - [[package]] name = "endi" version = "1.1.1" @@ -2414,15 +2334,6 @@ dependencies = [ "tracing", ] -[[package]] -name = "euclid" -version = "0.20.14" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2bb7ef65b3777a325d1eeefefab5b6d4959da54747e33bd6258e789640f307ad" -dependencies = [ - "num-traits", -] - [[package]] name = "euclid" version = "0.22.14" @@ -2932,16 +2843,6 @@ dependencies = [ "polyval", ] -[[package]] -name = "gif" -version = "0.14.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ee8cfcc411d9adbbaba82fb72661cc1bcca13e8bba98b364e62b2dba8f960159" -dependencies = [ - "color_quant", - "weezl", -] - [[package]] name = "gimli" version = "0.32.3" @@ -3602,14 +3503,10 @@ checksum = "85ab80394333c02fe689eaf900ab500fbd0c2213da414687ebf995a65d5a6104" dependencies = [ "bytemuck", "byteorder-lite", - "color_quant", - "gif", "moxcms", "num-traits", "png 0.18.1", "tiff", - "zune-core", - "zune-jpeg", ] [[package]] @@ -4033,7 +3930,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "c62026ae44756f8a599ba21140f350303d4f08dcdcc71b5ad9c9bb8128c13c62" dependencies = [ "arrayvec", - "euclid 0.22.14", + "euclid", "smallvec", ] @@ -4224,34 +4121,6 @@ version = "0.4.29" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "5e5032e24019045c762d3c0f28f5b6b8bbf38563a65908389bf7978758920897" -[[package]] -name = "lopdf" -version = "0.38.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c7184fdea2bc3cd272a1acec4030c321a8f9875e877b3f92a53f2f6033fdc289" -dependencies = [ - "aes", - "bitflags 2.11.1", - "cbc", - "ecb", - "encoding_rs", - "flate2", - "getrandom 0.3.4", - "indexmap 2.14.0", - "itoa", - "log", - "md-5", - "nom 8.0.0", - "nom_locate", - "rand 0.9.4", - "rangemap", - "sha2 0.10.9", - "stringprep", - "thiserror 2.0.18", - "ttf-parser", - "weezl", -] - [[package]] name = "lru-slab" version = "0.1.2" @@ -4343,16 +4212,6 @@ version = "0.8.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "47e1ffaa40ddd1f3ed91f717a33c8c0ee23fff369e3aa8772b9605cc1d22f4c3" -[[package]] -name = "md-5" -version = "0.10.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d89e7ee0cfbedfc4da3340218492196241d89eefb6dab27de5df917a6d2e78cf" -dependencies = [ - "cfg-if", - "digest 0.10.7", -] - [[package]] name = "memchr" version = "2.8.0" @@ -4599,17 +4458,6 @@ dependencies = [ "memchr", ] -[[package]] -name = "nom_locate" -version = "5.0.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0b577e2d69827c4740cba2b52efaad1c4cc7c73042860b199710b3575c68438d" -dependencies = [ - "bytecount", - "memchr", - "nom 8.0.0", -] - [[package]] name = "notify-rust" version = "4.17.0" @@ -5232,8 +5080,6 @@ dependencies = [ "objc2-foundation 0.3.2", "once_cell", "parking_lot", - "pdf-extract", - "ppt-rs", "rand 0.10.1", "rdev", "regex", @@ -5476,17 +5322,6 @@ dependencies = [ "windows-link 0.2.1", ] -[[package]] -name = "password-hash" -version = "0.4.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7676374caaee8a325c9e7a2ae557f216c5563a171d6997b0ef8a65af35147700" -dependencies = [ - "base64ct", - "rand_core 0.6.4", - "subtle", -] - [[package]] name = "password-hash" version = "0.5.0" @@ -5511,9 +5346,6 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "83a0692ec44e4cf1ef28ca317f14f8f07da2d95ec3fa01f86e4467b725e60917" dependencies = [ "digest 0.10.7", - "hmac", - "password-hash 0.4.2", - "sha2 0.10.9", ] [[package]] @@ -5526,23 +5358,6 @@ dependencies = [ "hmac", ] -[[package]] -name = "pdf-extract" -version = "0.10.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1e28ba1758a3d3f361459645780e09570b573fc3c82637449e9963174c813a98" -dependencies = [ - "adobe-cmap-parser", - "cff-parser", - "encoding_rs", - "euclid 0.20.14", - "log", - "lopdf", - "postscript", - "type1-encoding-parser", - "unicode-normalization", -] - [[package]] name = "percent-encoding" version = "2.3.2" @@ -5767,12 +5582,6 @@ dependencies = [ "universal-hash", ] -[[package]] -name = "pom" -version = "1.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "60f6ce597ecdcc9a098e7fddacb1065093a3d66446fa16c675e7e71d1b5c28e6" - [[package]] name = "portable-atomic" version = "1.13.1" @@ -5788,12 +5597,6 @@ dependencies = [ "portable-atomic", ] -[[package]] -name = "postscript" -version = "0.14.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "78451badbdaebaf17f053fd9152b3ffb33b516104eacb45e7864aaa9c712f306" - [[package]] name = "potential_utf" version = "0.1.5" @@ -5809,18 +5612,6 @@ version = "0.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "439ee305def115ba05938db6eb1644ff94165c5ab5e9420d1c1bcedbba909391" -[[package]] -name = "ppt-rs" -version = "0.2.16" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9a7c5e7639749a6ad0c5ea5a31636b88f8a6cefb188c99af0486a93ab50be3cf" -dependencies = [ - "thiserror 1.0.69", - "uuid 1.23.1", - "xml-rs", - "zip 0.6.6", -] - [[package]] name = "ppv-lite86" version = "0.2.21" @@ -5972,16 +5763,6 @@ version = "2.0.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "a993555f31e5a609f617c12db6250dedcac1b0a85076912c436e6fc9b2c8e6a3" -[[package]] -name = "quick-xml" -version = "0.36.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f7649a7b4df05aed9ea7ec6f628c67c9953a43869b8bc50929569b2999d443fe" -dependencies = [ - "encoding_rs", - "memchr", -] - [[package]] name = "quick-xml" version = "0.37.5" @@ -6182,12 +5963,6 @@ dependencies = [ "rand_core 0.9.5", ] -[[package]] -name = "rangemap" -version = "1.7.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "973443cf09a9c8656b574a866ab68dfa19f0867d0340648c7d2f6a71b8a8ea68" - [[package]] name = "raw-window-handle" version = "0.6.2" @@ -7583,17 +7358,6 @@ dependencies = [ "quote", ] -[[package]] -name = "stringprep" -version = "0.1.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7b4df3d392d81bd458a8a621b8bffbd2302a12ffe288a9d931670948749463b1" -dependencies = [ - "unicode-bidi", - "unicode-normalization", - "unicode-properties", -] - [[package]] name = "strsim" version = "0.11.1" @@ -8343,12 +8107,18 @@ name = "tinybus" version = "0.1.0" dependencies = [ "async-trait", + "flate2", "serde", "serde_json", + "tar", + "tempfile", "thiserror 2.0.18", "tinybus-macros", "tokio", + "toml 0.8.2", "tracing", + "ureq", + "zip 2.4.2", ] [[package]] @@ -8447,9 +8217,8 @@ dependencies = [ [[package]] name = "tinydocs" -version = "0.1.0" +version = "0.1.12" dependencies = [ - "docx-rs", "serde", "thiserror 2.0.18", ] @@ -9035,15 +8804,6 @@ dependencies = [ "thiserror 2.0.18", ] -[[package]] -name = "type1-encoding-parser" -version = "0.1.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "fa10c302f5a53b7ad27fd42a3996e23d096ba39b5b8dd6d9e683a05b01bee749" -dependencies = [ - "pom", -] - [[package]] name = "typed-arena" version = "2.0.2" @@ -9171,15 +8931,6 @@ version = "1.0.24" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "e6e4313cd5fcd3dad5cafa179702e2b244f760991f45397d14d4ebf38247da75" -[[package]] -name = "unicode-normalization" -version = "0.1.25" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5fd4f6878c9cb28d874b009da9e8d183b5abc80117c40bbd187a1fde336be6e8" -dependencies = [ - "tinyvec", -] - [[package]] name = "unicode-properties" version = "0.1.4" @@ -9248,6 +8999,35 @@ dependencies = [ "typenum", ] +[[package]] +name = "ureq" +version = "3.4.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "972d7902c8735f2695410b8aed7df6ed12a47394aa1c8d7af49f0497b731a94d" +dependencies = [ + "base64 0.23.1", + "flate2", + "log", + "percent-encoding", + "rustls", + "rustls-pki-types", + "ureq-proto", + "utf8-zero", + "webpki-roots 1.0.7", +] + +[[package]] +name = "ureq-proto" +version = "0.6.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "da5f78b09e6941e1a0f2e30e695e4b120377b54d5e0aec11b594bb57b3971613" +dependencies = [ + "base64 0.23.1", + "http", + "httparse", + "log", +] + [[package]] name = "url" version = "2.5.8" @@ -9312,6 +9092,12 @@ version = "0.7.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "09cc8ee72d2a9becf2f2febe0205bbed8fc6615b7cb429ad062dc7b7ddd036a9" +[[package]] +name = "utf8-zero" +version = "0.8.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b8c0a043c9540bae7c578c88f91dda8bd82e59ae27c21baca69c8b191aaf5a6e" + [[package]] name = "utf8_iter" version = "1.0.4" @@ -10617,12 +10403,6 @@ version = "0.2.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b9cc00251562a284751c9973bace760d86c0276c471b4be569fe6b068ee97a56" -[[package]] -name = "xml-rs" -version = "0.8.28" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3ae8337f8a065cfc972643663ea4279e04e7256de865aa66fe25cec5fb912d3f" - [[package]] name = "xmlwriter" version = "0.1.0" @@ -10817,26 +10597,6 @@ dependencies = [ "syn 2.0.117", ] -[[package]] -name = "zip" -version = "0.6.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "760394e246e4c28189f19d488c058bf16f564016aefac5d32bb1f3b51d5e9261" -dependencies = [ - "aes", - "byteorder", - "bzip2", - "constant_time_eq", - "crc32fast", - "crossbeam-utils", - "flate2", - "hmac", - "pbkdf2 0.11.0", - "sha1", - "time", - "zstd", -] - [[package]] name = "zip" version = "2.4.2" @@ -10884,35 +10644,6 @@ dependencies = [ "simd-adler32", ] -[[package]] -name = "zstd" -version = "0.11.2+zstd.1.5.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "20cc960326ece64f010d2d2107537f26dc589a6573a316bd5b1dba685fa5fde4" -dependencies = [ - "zstd-safe", -] - -[[package]] -name = "zstd-safe" -version = "5.0.2+zstd.1.5.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1d2a5585e04f9eea4b2a3d1eca508c4dee9592a89ef6f450c11719da0726f4db" -dependencies = [ - "libc", - "zstd-sys", -] - -[[package]] -name = "zstd-sys" -version = "2.0.16+zstd.1.5.7" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "91e19ebc2adc8f83e43039e79776e3fda8ca919132d68a1fed6a5faca2683748" -dependencies = [ - "cc", - "pkg-config", -] - [[package]] name = "zune-core" version = "0.5.1" diff --git a/app/src-tauri/Cargo.toml b/app/src-tauri/Cargo.toml index d072f869aa..c7227fb080 100644 --- a/app/src-tauri/Cargo.toml +++ b/app/src-tauri/Cargo.toml @@ -156,6 +156,7 @@ openhuman_core = { path = "../..", package = "openhuman", default-features = fal "voice", "web3", "documents", + "modules", "flows", "meet", "skills", diff --git a/docs/specs/2026-08-11-loadable-native-modules.md b/docs/specs/2026-08-11-loadable-native-modules.md new file mode 100644 index 0000000000..3c39ff4d63 --- /dev/null +++ b/docs/specs/2026-08-11-loadable-native-modules.md @@ -0,0 +1,130 @@ +# Loadable native modules + +Status: Implemented (`documents` is the first consumer) + +## Problem + +A codec is not kernel work, and each one drags a tree of parsers into a binary +that mostly does something else. The document surface was the clearest case: the +`documents` gate carried `docx-rs`, `ppt-rs` and `pdf-extract` plus their font, +PostScript and XML tails — 39 crates, including four native C builds — to support +three agent tools. + +Gating them helped the builds that turned them off and did nothing for the build +that ships, which turns them on. What was needed was a boundary that survives +compilation: the capability present, the dependencies absent. + +## Goals + +- Run a capability as a compiled artifact loaded at runtime, outside this binary. +- Verify what gets loaded, from a decision made at build time rather than by + whatever a release page serves. +- Keep the tool surface, the tool schemas and the agent-facing errors unchanged. +- Leave the kernel profile untouched: a host embedding workflows must not acquire + a dynamic-library loader. + +## Non-goals + +- Running untrusted third-party code. A module is first-party code that ships + separately; anything untrusted belongs in a process. +- A module marketplace, or any path by which a server or a config file can name + an artifact to load. +- Unloading. tinybus never unloads a library, and nothing here pretends otherwise. + +## Behaviour + +`openhuman::modules` owns a private in-process broker, a tinybus `ModuleHost`, and +a compiled-in registry. A registry entry names a module's id, the interfaces it +claims, its release tag, and one SHA-256 per published artifact. + +On first use of a capability a module provides, `ops::ensure_loaded` resolves it — +already serving, then a configured local artifact, then the install directory, +then the tinybus module search path, then a verified download — and caches the +outcome for the process. Loaded, the module is an ordinary bus peer, and the core +calls it over a proxy. + +Inbound payloads ride tinybus streams alongside the call. Outbound payloads are +held by the module and pulled in chunks, because a served object cannot open a +stream back to its caller. + +Config (`[modules]`) controls whether modules load, whether this host may fetch +them, where they install, and whether a local build stands in for a release. It +cannot add a module. + +## Invariants and constraints + +- **A loaded module is trusted in-process code.** It shares the address space, the + privileges and the crash domain. Deadlines, bounded queues and caught panics + contain ordinary misbehaviour, not a segfault. The ABI, manifest and digest + gates decide what is *admitted*, never what is *safe*. +- **The set of loadable modules is compiled in.** A registry that config or RPC + could extend would be remote code execution with a download step. +- **Digests are the host's half of a two-sided check.** tinybus fetches the + release's own checksum manifest, compares it with the pinned digest, hashes the + download, and extracts only after. A release re-cut under the same tag stops + matching rather than silently replacing what runs in-process. +- **Failure is terminal for the process.** tinybus never unloads, so a refused or + faulted module cannot reach a different outcome without a restart. Failures are + cached and say so. +- **Admission is permissive, not strict.** Strict mode also refuses a module whose + rustc version differs from the host's, and released artifacts are built on + whatever toolchain CI had while this crate pins its own — the real published + artifact is refused that way. Everything protecting the address space is still + enforced; only the toolchain string is relaxed. +- **A target triple is not enough.** A library built against glibc 2.39 fails to + `dlopen` on a 2.35 host. Artifact selection is an ordered list of candidates, + glibc-aware, and empty on a host no published artifact targets. +- **Modules cannot publish `DomainEvent`s.** `OnceBus::init_in_process` owns its + broker privately, so modules run on a second one. +- **The loader stays out of the kernel profile.** `tinybus/modules` is forwarded + from this crate's `modules` feature, never enabled on the dependency, because + `tinybus` is always-on surface. + +## Acceptance criteria + +- The `documents` gate carries no codec: `docx-rs`, `ppt-rs`, `pdf-extract`, + `lopdf`, `syntect`, `pulldown-cmark` and `xml-rs` are absent from both the + `documents` and the product profiles, proven with `scripts/assert-shed.sh`. +- The kernel profile is unchanged against upstream: `scripts/kernel-floor.sh + flows` reports the same packages, names and native builds. +- `modules.*` is unknown-method with the feature off, and both halves are pinned + by tests. +- The three document tools produce openable artifacts through a loaded module, + including a deck whose images cross as a stream. + +## Testing notes + +**The module bus belongs to whichever runtime creates it.** The core has one +runtime and never notices. Two `#[tokio::test]` functions each build their own, +and the second to call a loaded module finds a broker whose tasks died with the +first — the call hangs until a deadline above it fires, rather than failing. + +Any test that drives a real module must therefore be the only one in its process. +The module-backed tool tests are `#[ignore]`d for that reason, not merely because +they need an artifact. Run them one at a time: + +Linux, where the artifact is a `.so`. On macOS the built library is +`libtinydocs_module.dylib` and on Windows `tinydocs_module.dll`; substitute the +filename, and use the platform's own temporary directory rather than `/tmp`. + +```sh +cargo build --release --package tinydocs-module \ + --manifest-path vendor/tinydocs/Cargo.toml +mkdir -p /tmp/oh-modules && chmod 700 /tmp/oh-modules +cp vendor/tinydocs/target/release/libtinydocs_module.so /tmp/oh-modules/ + +OPENHUMAN_MODULE_PATH=/tmp/oh-modules \ + cargo test -p openhuman --lib --features documents -- --ignored \ + implementations::document::tests::execute_happy_path +``` + +## Open questions + +**A reply-stream seam in tinybus would delete the module's output store.** The +only reason a produced document is held at all is that `Interface::call` receives +no caller identity and no connection, so a served object cannot stream a reply. + +**Per-interface method lists in `module_export!`** would let one module serve a +transfer interface and a format interface separately. Today the macro attaches its +method list to the first entry in `provides` and leaves the rest empty, so a +second fully-declared interface is not expressible. diff --git a/scripts/ci/product-features.txt b/scripts/ci/product-features.txt index 03fc5dad3e..1cc60cfd8e 100644 --- a/scripts/ci/product-features.txt +++ b/scripts/ci/product-features.txt @@ -42,9 +42,18 @@ voice # Wallet / web3 / x402 domains and their agent tools. web3 -# Document ingestion and conversion. +# Document ingestion and conversion. The synthesis itself is NOT in the binary: +# `.docx` / `.pptx` generation and PDF extraction run in the loaded `tinydocs` +# module, so this gate ships the tools and the host policy around them. Implies +# `modules`. documents +# The dynamic module host: the loader that admits a compiled cdylib through +# tinybus's ABI, manifest and SHA-256 gates, the compiled-in registry of modules +# this build trusts, and the `modules` RPC namespace. Required by `documents` — +# without it the document tools have nothing to call. +modules + # Saved automation graphs: create/run/schedule + the workflow_builder and # flow_discovery agents. flows diff --git a/scripts/kernel-floor.limits b/scripts/kernel-floor.limits index 025c8b7cf4..7a3200b92d 100644 --- a/scripts/kernel-floor.limits +++ b/scripts/kernel-floor.limits @@ -13,6 +13,34 @@ # Simulate with: scripts/dep-sim.py --cut # # History +# 305/282/2 2026-08-11 documents moved into the tinydocs TinyBus module. +# NO CHANGE to this profile, and that is the point of the +# entry: `documents` was never in the `flows` profile, so +# the codecs it carried were invisible to this ratchet. +# What the ratchet DID catch is the other half of the same +# change — enabling `tinybus/modules` on the dependency +# itself put a `dlopen` loader plus `ureq` and an archive +# stack into the kernel profile (305 -> 308 packages, +# 282 -> 285 names), because tinybus is always-on surface. +# Fixed by forwarding `tinybus/modules` from this crate's +# own default-ON `modules` feature instead, so a +# workflow-only host carries no loader. Back to 305/282/2, +# identical to upstream/main. +# The shed lands on the PRODUCT profile, which is what +# ships: 505 -> 448 unique crate names, and 39 crates leave +# Cargo.lock entirely — docx-rs, ppt-rs, pdf-extract and +# their font/PostScript/XML tails (lopdf, syntect, +# pulldown-cmark, xml-rs, quick-xml, zip 0.6, zstd, bzip2, +# encoding_rs, euclid, ttf-parser, …). Verified with +# `scripts/assert-shed.sh documents docx-rs ppt-rs +# pdf-extract syntect pulldown-cmark lopdf xml-rs` and the +# same against the product feature set. +# NOTE: this profile already exceeded its limit on +# upstream/main (305/282 vs 302/279) before this branch; +# `scripts/check-kernel-floor.sh` fails identically there. +# Not raised here — the number is unchanged, and raising a +# limit to paper over inherited growth is what this file +# exists to prevent. # 302/279/2 2026-08-09 `memory-git` gate (-3 packages / -3 names / -2 NATIVE), # measured on top of the runtime-node entry below. # diff --git a/src/core/all.rs b/src/core/all.rs index 5e6029b0a6..afd370e653 100644 --- a/src/core/all.rs +++ b/src/core/all.rs @@ -141,13 +141,16 @@ pub enum DomainGroup { Hosted, /// The multi-agent relay surface (`tinyplace/`). Relay, + /// Loadable native modules: the module host, its registry, and the `modules` + /// RPC surface (`modules/`). + Modules, // Everything not in a named family — always on in `full()`, off otherwise. Platform, } impl DomainGroup { /// Number of variants. Kept in sync by `domain_group_all_lists_every_variant`. - pub const COUNT: usize = 22; + pub const COUNT: usize = 23; /// Every variant, for exhaustive iteration in drift guards. /// @@ -180,6 +183,7 @@ impl DomainGroup { DomainGroup::Desktop, DomainGroup::Hosted, DomainGroup::Relay, + DomainGroup::Modules, DomainGroup::Platform, ]; @@ -209,7 +213,8 @@ impl DomainGroup { DomainGroup::Desktop => 18, DomainGroup::Hosted => 19, DomainGroup::Relay => 20, - DomainGroup::Platform => 21, + DomainGroup::Modules => 21, + DomainGroup::Platform => 22, } } } @@ -1093,6 +1098,16 @@ fn build_internal_only_controllers() -> Vec { DomainGroup::Mcp, crate::openhuman::mcp::audit::all_mcp_audit_internal_controllers(), ); + // Loadable native modules: list/status and an explicit load. Read-only apart + // from that load, and it cannot name an artifact — the loadable set is + // compiled into `modules::registry`, so this namespace can start a module + // the build already trusts and nothing else. + #[cfg(feature = "modules")] + push( + &mut controllers, + DomainGroup::Modules, + crate::openhuman::modules::all_registered_controllers(), + ); // tiny.place A2A social-network integration: renderer-callable via core_rpc_relay // but NOT advertised to agents in tool listings or schema discovery. push( diff --git a/src/core/all_tests.rs b/src/core/all_tests.rs index e6c00e0fc5..364dea8c69 100644 --- a/src/core/all_tests.rs +++ b/src/core/all_tests.rs @@ -1125,6 +1125,32 @@ fn meet_controllers_registered_when_feature_on() { } } +/// The `modules` namespace registers when the `modules` feature is on. +#[cfg(feature = "modules")] +#[test] +fn modules_controllers_registered_when_feature_on() { + assert_eq!( + group_for_namespace("modules"), + Some(DomainGroup::Modules), + "`modules` must register under DomainGroup::Modules when the feature is on" + ); +} + +/// The `modules` namespace is absent when the `modules` feature is off. +/// +/// The half that proves the gate. It matters more than the usual both-ways pair, +/// because what this feature compiles in is a `dlopen` loader: a build that opted +/// out must have no way to reach one, not a loader that merely refuses. +#[cfg(not(feature = "modules"))] +#[test] +fn modules_controllers_absent_when_feature_off() { + assert_eq!( + group_for_namespace("modules"), + None, + "`modules` must leave no trace in the registry when the feature is off" + ); +} + /// No Meet namespace registers when the `meet` feature is off (#4800). /// /// This is the half that proves the gate: with `meet` compiled out the three @@ -1507,6 +1533,10 @@ fn every_domain_group_is_accounted_for_in_store_init_plan() { DomainGroup::Desktop, DomainGroup::Hosted, DomainGroup::Relay, + // The registry is a compiled-in `const` table and the loaded-module set + // lives in tinybus's own `ModuleHost`, so there is nothing for + // `init_stores` to stand up. + DomainGroup::Modules, DomainGroup::Platform, ]; @@ -1565,6 +1595,9 @@ fn every_domain_group_is_accounted_for_in_subscriber_plan() { DomainGroup::Runtimes, DomainGroup::Hosted, DomainGroup::Relay, + // Modules run on their own in-process broker, so they cannot publish a + // `DomainEvent` and there is nothing on the core bus to subscribe to. + DomainGroup::Modules, ]; for g in DomainGroup::ALL { diff --git a/src/core/runtime/builder.rs b/src/core/runtime/builder.rs index 312af93c5f..6a31a189b6 100644 --- a/src/core/runtime/builder.rs +++ b/src/core/runtime/builder.rs @@ -222,6 +222,8 @@ pub struct DomainSet { pub hosted: bool, /// The multi-agent relay surface (tinyplace). pub relay: bool, + /// Loadable native modules: the module host, registry and `modules` RPC. + pub modules: bool, /// Everything not in a named family — always on in `full()`. pub platform: bool, } @@ -252,6 +254,7 @@ impl DomainSet { desktop: true, hosted: true, relay: true, + modules: true, platform: true, } } @@ -282,6 +285,7 @@ impl DomainSet { desktop: false, hosted: false, relay: false, + modules: false, platform: false, } } @@ -329,6 +333,7 @@ impl DomainSet { desktop: false, hosted: false, relay: false, + modules: false, platform: true, } } @@ -365,6 +370,7 @@ impl DomainSet { desktop: false, hosted: false, relay: false, + modules: false, platform: false, } } @@ -393,6 +399,7 @@ impl DomainSet { desktop: false, hosted: false, relay: false, + modules: false, platform: false, } } @@ -421,6 +428,7 @@ impl DomainSet { DomainGroup::Desktop => self.desktop, DomainGroup::Hosted => self.hosted, DomainGroup::Relay => self.relay, + DomainGroup::Modules => self.modules, DomainGroup::Platform => self.platform, } } diff --git a/src/openhuman/agent/multimodal.rs b/src/openhuman/agent/multimodal.rs index ddbf15cb9e..f906d42b64 100644 --- a/src/openhuman/agent/multimodal.rs +++ b/src/openhuman/agent/multimodal.rs @@ -1457,7 +1457,7 @@ async fn file_payload_from_bytes( } if mime == "application/pdf" { - match extract_pdf_text(bytes.clone()).await { + match extract_pdf_text(bytes.to_vec()).await { Ok(raw) => { let (text, truncated_chars) = truncate_chars(raw, max_extracted_text_chars); if truncated_chars > 0 { @@ -1619,21 +1619,38 @@ fn extract_utf8_text(bytes: &[u8]) -> Result { } } -/// Run `pdf-extract` on a copy of `bytes` inside a `spawn_blocking` -/// worker, bounded by [`PDF_EXTRACTION_TIMEOUT`]. Returns the raw -/// extracted text on success; on timeout / panic / parse error the -/// caller degrades the file to [`FilePayload::Reference`] rather than -/// surface the failure to the user (avoids Sentry noise on broken PDFs). +/// Extract a PDF's text layer through the `tinydocs` module, bounded by +/// [`PDF_EXTRACTION_TIMEOUT`]. +/// +/// The parsing runs in the module, which owns its own blocking pool, so there is +/// no `spawn_blocking` here — but the deadline stays, because the cost is set by +/// the input rather than by anything this side validated, and only the host +/// knows how long an attachment is worth waiting for. +/// +/// The bytes ride a bus stream rather than a JSON frame: a `.pdf` is bounded +/// only by what the multimodal config accepted, which is far past what a frame +/// holds. +/// +/// Every failure — an unavailable module, a damaged document, the deadline — is +/// reported the same way it always was, and the caller degrades the file to +/// [`FilePayload::Reference`] rather than surfacing it (which avoids Sentry +/// noise on broken PDFs). #[cfg(feature = "documents")] async fn extract_pdf_text(bytes: Vec) -> Result { - let extraction = tokio::task::spawn_blocking(move || { - pdf_extract::extract_text_from_mem(&bytes).map_err(|error| error.to_string()) - }); - - match tokio::time::timeout(PDF_EXTRACTION_TIMEOUT, extraction).await { - Ok(Ok(Ok(text))) => Ok(text), - Ok(Ok(Err(reason))) => Err(reason), - Ok(Err(join_error)) => Err(format!("pdf extraction worker panicked: {join_error}")), + use crate::openhuman::modules::documents; + + let config = crate::openhuman::config::Config::load_or_init() + .await + .map_err(|error| format!("config unavailable for pdf extraction: {error}"))?; + + match tokio::time::timeout( + PDF_EXTRACTION_TIMEOUT, + documents::extract_text(&config, &bytes), + ) + .await + { + Ok(Ok(text)) => Ok(text), + Ok(Err(error)) => Err(error.to_string()), Err(_) => Err(format!( "pdf extraction exceeded {}s timeout", PDF_EXTRACTION_TIMEOUT.as_secs() diff --git a/src/openhuman/config/schema/mod.rs b/src/openhuman/config/schema/mod.rs index d0273b84b2..e9d902d7c4 100644 --- a/src/openhuman/config/schema/mod.rs +++ b/src/openhuman/config/schema/mod.rs @@ -42,6 +42,7 @@ pub mod claude_agent_sdk; pub use claude_agent_sdk::ClaudeAgentSdkConfig; mod local_ai; mod meet; +mod modules; mod node; mod observability; mod orchestration; @@ -79,6 +80,7 @@ pub use identity_cost::{CostConfig, ModelPricing}; pub use learning::{LearningConfig, ReflectionSource}; pub use local_ai::{LocalAiConfig, LocalAiUsage}; pub use meet::{AutoJoinPolicy, AutoSummarizePolicy, CalendarProvider, MeetConfig}; +pub use modules::{ModuleOverride, ModulesConfig}; pub use node::NodeConfig; pub use observability::{AgentTracingBackend, AgentTracingConfig, ObservabilityConfig}; pub use orchestration::{ diff --git a/src/openhuman/config/schema/modules.rs b/src/openhuman/config/schema/modules.rs new file mode 100644 index 0000000000..e71f8e2fb2 --- /dev/null +++ b/src/openhuman/config/schema/modules.rs @@ -0,0 +1,68 @@ +//! `[modules]` — loadable native modules. +//! +//! A module is trusted in-process code, so this block deliberately cannot add +//! one. The set of loadable modules is compiled into +//! `openhuman::modules::registry`; what config controls is whether they load at +//! all, whether this host may fetch them, and where a developer's own build +//! lives. A config file that could name a new artifact to `dlopen` would be a +//! remote-code-execution surface with a download step. + +use schemars::JsonSchema; +use serde::{Deserialize, Serialize}; + +use super::defaults; + +/// Configuration for the module host. +#[derive(Debug, Clone, Serialize, Deserialize, JsonSchema)] +#[serde(default)] +pub struct ModulesConfig { + /// Whether modules may be loaded at all. + /// + /// Off means the features they provide are unavailable and say so, rather + /// than failing at the point of use. + pub enabled: bool, + + /// Whether a missing module may be downloaded from its pinned release. + /// + /// Off pins this host to artifacts that are already installed, which is what + /// an air-gapped or strictly-reproducible deployment wants. Nothing is + /// downloaded silently either way: the digest comes from the compiled-in + /// registry, not from the release. + pub allow_download: bool, + + /// Where downloaded artifacts are installed. + /// + /// Defaults to the user cache directory, falling back to the workspace when + /// the host has none. + #[serde(skip_serializing_if = "Option::is_none")] + pub install_dir: Option, + + /// Local artifacts to load instead of the pinned release. + /// + /// For developing a module against a live core. An override bypasses the + /// digest check — the artifact is whatever is at that path — which is + /// acceptable only because it is the operator's own file, named by the + /// operator's own config. + pub overrides: Vec, +} + +impl Default for ModulesConfig { + fn default() -> Self { + Self { + enabled: defaults::default_true(), + allow_download: defaults::default_true(), + install_dir: None, + overrides: Vec::new(), + } + } +} + +/// A local artifact standing in for one registry entry. +#[derive(Debug, Clone, Serialize, Deserialize, JsonSchema)] +#[serde(deny_unknown_fields)] +pub struct ModuleOverride { + /// Registry id this override replaces, e.g. `tinydocs`. + pub id: String, + /// Absolute path to a platform library (`.so` / `.dylib` / `.dll`). + pub path: String, +} diff --git a/src/openhuman/config/schema/types.rs b/src/openhuman/config/schema/types.rs index 6a9b9d64f7..12adc9553a 100644 --- a/src/openhuman/config/schema/types.rs +++ b/src/openhuman/config/schema/types.rs @@ -292,6 +292,12 @@ pub struct Config { #[serde(default)] pub mcp_client: McpClientConfig, + /// Loadable native modules — whether they load, whether this host may fetch + /// them, and where a developer's own build lives. The loadable *set* is + /// compiled in, not configured: see `openhuman::modules::registry`. + #[serde(default)] + pub modules: super::ModulesConfig, + /// Trust metadata for external capability providers. Empty by default so /// existing installations keep the same tool-discovery behavior. #[serde(default)] @@ -809,6 +815,7 @@ impl Default for Config { curl: CurlConfig::default(), gitbooks: GitbooksConfig::default(), mcp_client: McpClientConfig::default(), + modules: super::ModulesConfig::default(), capability_providers: Vec::new(), multimodal: MultimodalConfig::default(), multimodal_files: MultimodalFileConfig::default(), diff --git a/src/openhuman/memory/diff/mod.rs b/src/openhuman/memory/diff/mod.rs index 30e8357766..fecf0954af 100644 --- a/src/openhuman/memory/diff/mod.rs +++ b/src/openhuman/memory/diff/mod.rs @@ -76,4 +76,5 @@ pub use tinycortex::memory::diff::types::{ ChangeKind, Checkpoint, CrossSourceDiff, DiffResult, DiffSummary, ItemChange, Snapshot, SnapshotTrigger, }; +#[cfg(feature = "memory-git")] pub use tools::MemoryDiffTool; diff --git a/src/openhuman/memory/diff/stub.rs b/src/openhuman/memory/diff/stub.rs index ed0e1428e2..1fec6adff5 100644 --- a/src/openhuman/memory/diff/stub.rs +++ b/src/openhuman/memory/diff/stub.rs @@ -26,7 +26,7 @@ use crate::openhuman::config::Config; use crate::openhuman::memory::sources::types::MemorySourceEntry; -use super::types::{Checkpoint, CrossSourceDiff, Snapshot}; +use super::{Checkpoint, CrossSourceDiff, Snapshot}; /// The message every disabled entry point returns. /// diff --git a/src/openhuman/mod.rs b/src/openhuman/mod.rs index faa46e1be5..fdbfa3abee 100644 --- a/src/openhuman/mod.rs +++ b/src/openhuman/mod.rs @@ -51,6 +51,8 @@ pub mod medulla; // submodule carries its own `#[cfg(feature = "meet")]` (see `meet/mod.rs`). pub mod meet; pub mod memory; +#[cfg(feature = "modules")] +pub mod modules; pub mod platform; pub mod runtime; pub mod sandbox; diff --git a/src/openhuman/modules/boot.rs b/src/openhuman/modules/boot.rs new file mode 100644 index 0000000000..e5b9159941 --- /dev/null +++ b/src/openhuman/modules/boot.rs @@ -0,0 +1,91 @@ +//! What happens to modules at startup. +//! +//! Two things, and deliberately not a third. +//! +//! Modules marked [`LoadPolicy::Eager`] are loaded, because their absence would +//! change what the core offers rather than merely delay it. Modules on the search +//! path are loaded, because that is how an operator or a developer puts an +//! artifact in front of this host without editing config — tinybus honours +//! `OPENHUMAN_MODULE_PATH` first, then the platform data directories. +//! +//! What does not happen is downloading every registry entry. A [`LoadPolicy::Lazy`] +//! module stays unloaded until something asks for it, so a user who never +//! produces a document never pays a download, a `dlopen`, or the resident cost of +//! a library that is never unloaded. Boot is also the worst moment to spend +//! network on something nobody has asked for yet. + +use super::types::LoadPolicy; +use super::{host, ops, registry}; +use crate::openhuman::config::Config; + +/// Load the modules that should be serving before the first request. +/// +/// Never fails the boot: a module that cannot load leaves its feature +/// unavailable, and the feature says so at the point of use. Taking the core +/// down because an optional codec is missing would be a worse trade. +pub async fn load_declared_modules(config: &Config) { + if !config.modules.enabled { + log::debug!("[modules] boot load skipped: modules are disabled in configuration"); + return; + } + + let runtime = match host::runtime().await { + Ok(runtime) => runtime, + Err(err) => { + log::warn!("[modules] boot load skipped: the module bus could not start: {err}"); + return; + } + }; + + // Search paths first: an artifact an operator has placed deliberately should + // win over a download, and `ensure_loaded` below then finds it already + // serving rather than fetching a second copy. + for outcome in runtime.host().load_search_paths() { + match outcome { + Ok(info) => log::info!( + "[modules] loaded '{}' {} from the module search path", + info.name, + info.manifest.module.version + ), + // Expected and not worth a warning: a search directory usually holds + // nothing, and tinybus reports each refusal with a sanitised reason. + Err(err) => log::debug!("[modules] search-path artifact not admitted: {err}"), + } + } + + for record in registry::ALL { + if record.load != LoadPolicy::Eager { + continue; + } + if let Err(reason) = ops::ensure_loaded(config, record.id).await { + log::warn!( + "[modules] eager module '{}' did not load: {reason}", + record.id + ); + } + } +} + +#[cfg(test)] +mod tests { + use super::load_declared_modules; + use crate::openhuman::config::Config; + + #[tokio::test] + async fn boot_is_a_no_op_when_modules_are_disabled() { + // Must not start a broker as a side effect of being switched off. + let mut config = Config::default(); + config.modules.enabled = false; + load_declared_modules(&config).await; + } + + #[tokio::test] + async fn boot_tolerates_an_empty_search_path() { + // The ordinary case on a fresh machine: nothing installed, nothing eager, + // and boot must complete rather than warn or fail. + let mut config = Config::default(); + config.modules.enabled = true; + config.modules.allow_download = false; + load_declared_modules(&config).await; + } +} diff --git a/src/openhuman/modules/documents.rs b/src/openhuman/modules/documents.rs new file mode 100644 index 0000000000..fede77ec0b --- /dev/null +++ b/src/openhuman/modules/documents.rs @@ -0,0 +1,340 @@ +//! Calling the `tinydocs` module: the three document operations, over the bus. +//! +//! Each function here is the host half of one method on +//! `ai.tinyhumans.tinydocs.Documents`. They exist so the three tools that need a +//! document do not each have to know about streams, held outputs, or wire error +//! names — a tool asks for bytes and gets bytes or a reason. +//! +//! # The shape of a call +//! +//! Inbound payloads ride a `TinyBus` stream opened alongside the call, so a +//! `.pdf` or a set of slide images is never squeezed through a 16 MiB JSON +//! frame. Outbound payloads cannot use a stream — a served object has no way to +//! open one back to its caller — so a produced document is held by the module +//! and pulled here with `ReadOutput`, then released. +//! +//! The release is in a `defer`-shaped position on purpose: the module bounds what +//! it holds and expires it, but leaving a document to time out costs its budget +//! in the meantime, and the next caller sees a full store rather than a slot. +//! +//! # Deadlines belong to the caller +//! +//! Nothing here imposes one. The document tools already wrap their calls in a +//! `tokio::time::timeout` chosen for what the tool is doing, and a second +//! deadline underneath would make the effective limit the smaller of two numbers +//! nobody picked together. + +use base64::engine::general_purpose::STANDARD as BASE64; +use base64::Engine as _; +use serde::Deserialize; +use tinybus::stream::StreamRef; +use tinydocs::spec::{DocumentSpec, WirePresentationSpec}; + +use super::{host, ops, registry}; +use crate::openhuman::config::Config; + +/// Registry id of the module these calls go to. +const MODULE_ID: &str = "tinydocs"; + +/// How much of a held document to pull per `ReadOutput`. +/// +/// Below the module's own per-chunk cap with room for the base64 expansion and +/// the JSON envelope around it. +const READ_CHUNK: u64 = 1024 * 1024; + +/// Largest document this host will assemble from a module. +/// +/// Matches the module's own per-output cap, and is enforced here anyway. A +/// declared length is a number a module sent us: trusting it for a +/// `Vec::with_capacity` turns a wrong or hostile value into an allocation +/// failure, which aborts the process rather than returning an error. The rest of +/// `read_all` already treats the declared length as possibly wrong; this applies +/// the same distrust to the allocation. +const MAX_DOCUMENT_BYTES: u64 = 64 * 1024 * 1024; + +/// Why a document call did not produce bytes. +/// +/// Three variants rather than one string because the three tools that call this +/// map them onto three different agent-facing shapes: a spec the model can fix, +/// a failure it cannot, and a capability that is not present at all. +#[derive(Debug, Clone, PartialEq, Eq)] +pub enum DocumentCallError { + /// The module is not loaded and cannot be: unsupported host, downloads off, + /// disabled in config, or a load that already failed in this process. + Unavailable(String), + /// The spec was rejected. A model can act on this. + InvalidInput(String), + /// Synthesis, extraction, or the transfer itself failed. + Failed(String), +} + +impl std::fmt::Display for DocumentCallError { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + match self { + Self::Unavailable(message) | Self::InvalidInput(message) | Self::Failed(message) => { + f.write_str(message) + } + } + } +} + +/// A handle to a document the module is holding for us. +#[derive(Debug, Deserialize)] +struct OutputRef { + output_id: String, + total_bytes: u64, + sha256: String, +} + +/// Generate a `.docx` from `spec`. +/// +/// # Errors +/// +/// [`DocumentCallError`] describing whether the spec, the module, or the +/// synthesis was at fault. +pub async fn generate_docx( + config: &Config, + spec: &DocumentSpec, +) -> Result, DocumentCallError> { + let (runtime, record) = ready(config).await?; + let proxy = proxy(runtime, record)?; + let handle: OutputRef = proxy + .call("GenerateDocx", (spec,)) + .await + .map_err(|error| classify(&error))?; + collect(&proxy, handle).await +} + +/// Generate a `.pptx` from `deck`, streaming `images` alongside the call. +/// +/// `images` is every slide image concatenated in slide order; the `byte_len` on +/// each entry in `deck` says where one ends and the next begins. +/// +/// # Errors +/// +/// [`DocumentCallError`], including an `InvalidInput` if `images` does not add +/// up to the lengths `deck` declares. +pub async fn generate_pptx( + config: &Config, + deck: &WirePresentationSpec, + images: &[u8], +) -> Result, DocumentCallError> { + let (runtime, record) = ready(config).await?; + let proxy = proxy(runtime, record)?; + + let handle: OutputRef = if images.is_empty() { + // A text-only deck opens no stream: there is nothing to send, and an + // empty stream is a round trip for nothing. + proxy + .call("GeneratePptx", (deck, Option::::None)) + .await + .map_err(|error| classify(&error))? + } else { + let (destination, path, interface) = address(record)?; + runtime + .connection() + .call_with_stream( + destination, + path, + interface, + member("GeneratePptx")?, + |stream| serde_json::json!([deck, stream]), + images, + ) + .await + .map_err(|error| classify(&error))? + }; + collect(&proxy, handle).await +} + +/// Extract the text layer of the `.pdf` in `document`. +/// +/// # Errors +/// +/// [`DocumentCallError`]. A document that parses but carries no text layer — a +/// scan — is not an error: it yields an empty string. +pub async fn extract_text(config: &Config, document: &[u8]) -> Result { + let (runtime, record) = ready(config).await?; + let proxy = proxy(runtime, record)?; + let (destination, path, interface) = address(record)?; + + let handle: OutputRef = runtime + .connection() + .call_with_stream( + destination, + path, + interface, + member("ExtractText")?, + |stream| serde_json::json!([stream]), + document, + ) + .await + .map_err(|error| classify(&error))?; + + let bytes = collect(&proxy, handle).await?; + String::from_utf8(bytes) + .map_err(|_| DocumentCallError::Failed("extracted text was not valid UTF-8".to_string())) +} + +/// Load the document module if it is not already serving. +/// +/// Callers do not have to invoke this — every operation below does it — but a +/// caller that wraps its work in a deadline should, *outside* that deadline. +/// A first use may download and verify an artifact, and charging that against a +/// generation timeout means the first document a user ever asks for is the one +/// that fails. Every later call finds it cached and returns immediately. +/// +/// # Errors +/// +/// The same [`DocumentCallError::Unavailable`] the operations return. +pub async fn ensure_ready(config: &Config) -> Result<(), DocumentCallError> { + ops::ensure_loaded(config, MODULE_ID) + .await + .map_err(DocumentCallError::Unavailable) +} + +/// Ensure the module is serving and hand back what a call needs. +async fn ready( + config: &Config, +) -> Result<(&'static host::ModuleRuntime, &'static super::ModuleRecord), DocumentCallError> { + ops::ensure_loaded(config, MODULE_ID) + .await + .map_err(DocumentCallError::Unavailable)?; + let record = registry::find(MODULE_ID) + .ok_or_else(|| DocumentCallError::Unavailable(format!("unknown module '{MODULE_ID}'")))?; + let runtime = host::runtime() + .await + .map_err(|_| DocumentCallError::Unavailable("the module bus is not running".to_string()))?; + Ok((runtime, record)) +} + +/// A proxy for the module's object. +fn proxy( + runtime: &'static host::ModuleRuntime, + record: &super::ModuleRecord, +) -> Result { + runtime + .proxy(record.bus_name, record.object_path) + .map_err(|error| DocumentCallError::Failed(error.to_string())) +} + +/// The destination triple a streaming call needs. +fn address( + record: &super::ModuleRecord, +) -> Result< + ( + tinybus::BusName, + tinybus::ObjectPath, + tinybus::InterfaceName, + ), + DocumentCallError, +> { + let bad = |error: tinybus::Error| DocumentCallError::Failed(error.to_string()); + Ok(( + tinybus::BusName::new(record.bus_name).map_err(bad)?, + tinybus::ObjectPath::new(record.object_path).map_err(bad)?, + tinybus::InterfaceName::new(record.bus_name).map_err(bad)?, + )) +} + +/// A member name, which is a constant in every caller here. +fn member(name: &str) -> Result { + tinybus::MemberName::new(name).map_err(|error| DocumentCallError::Failed(error.to_string())) +} + +/// Pull a held document, verify it, and release it. +/// +/// The release runs whether or not the read succeeded: a document left behind +/// costs the module's budget until its TTL expires, and the next caller sees a +/// full store rather than a slot. +async fn collect(proxy: &tinybus::Proxy, handle: OutputRef) -> Result, DocumentCallError> { + let result = read_all(proxy, &handle).await; + if let Err(error) = proxy + .call::<()>("ReleaseOutput", (handle.output_id.clone(),)) + .await + { + // Not fatal: the module expires what nobody reads. Worth a line, because + // a pattern of these means documents are sitting in the module's budget + // until they time out. + log::debug!("[modules] releasing a read document failed: {error}"); + } + result +} + +/// Read a held document in chunks and check it against its digest. +async fn read_all( + proxy: &tinybus::Proxy, + handle: &OutputRef, +) -> Result, DocumentCallError> { + if handle.total_bytes > MAX_DOCUMENT_BYTES { + return Err(DocumentCallError::Failed(format!( + "the module declared a {}-byte document, over the {MAX_DOCUMENT_BYTES}-byte limit", + handle.total_bytes + ))); + } + // Reserve against the declared length only once it is known to be sane, and + // still only up to one chunk beyond what has arrived — the loop grows the + // buffer as real bytes land rather than trusting the number up front. + let capacity = usize::try_from(handle.total_bytes).unwrap_or(0); + let mut out = Vec::with_capacity(capacity.min(MAX_DOCUMENT_BYTES as usize)); + while (out.len() as u64) < handle.total_bytes { + let encoded: String = proxy + .call( + "ReadOutput", + (handle.output_id.clone(), out.len() as u64, READ_CHUNK), + ) + .await + .map_err(|error| classify(&error))?; + let chunk = BASE64.decode(encoded).map_err(|_| { + DocumentCallError::Failed("a document chunk was not base64".to_string()) + })?; + if chunk.is_empty() { + // The module clamps a read to what is left, so an empty chunk before + // the declared length means the two sides disagree about the size. + // Looping would spin forever. + return Err(DocumentCallError::Failed( + "the document ended before its declared length".to_string(), + )); + } + out.extend_from_slice(&chunk); + } + + let digest = sha256_hex(&out); + if digest != handle.sha256 { + return Err(DocumentCallError::Failed( + "the assembled document did not match its declared digest".to_string(), + )); + } + Ok(out) +} + +/// Lowercase hex SHA-256, matching what the module declares. +fn sha256_hex(bytes: &[u8]) -> String { + use sha2::{Digest, Sha256}; + let digest = Sha256::digest(bytes); + let mut out = String::with_capacity(64); + for byte in digest { + use std::fmt::Write as _; + let _ = write!(out, "{byte:02x}"); + } + out +} + +/// Map a bus failure onto the shape a tool can act on. +/// +/// The wire name is the contract; the message is for a human. An unrecognised +/// name is `Failed` rather than `InvalidInput`, because telling a model its input +/// was wrong when it was not sends it into a rewrite loop. +fn classify(error: &tinybus::Error) -> DocumentCallError { + let message = error.to_string(); + match error.wire_name() { + "ai.tinyhumans.tinydocs.Error.InvalidInput" => DocumentCallError::InvalidInput(message), + // The module is loaded but not answering: refused, faulted, or gone. + name if name.contains("ModuleUnavailable") => DocumentCallError::Unavailable(message), + _ => DocumentCallError::Failed(message), + } +} + +#[cfg(test)] +#[path = "documents_tests.rs"] +mod tests; diff --git a/src/openhuman/modules/documents_tests.rs b/src/openhuman/modules/documents_tests.rs new file mode 100644 index 0000000000..7e97c5eeb6 --- /dev/null +++ b/src/openhuman/modules/documents_tests.rs @@ -0,0 +1,129 @@ +//! Tests for the document call client. +//! +//! Nothing here loads a module. What is testable without one is the part that +//! decides what a tool does next: how a bus failure is classified, and that the +//! unavailable path is reached without a broker. The round trips themselves are +//! covered where they can be honest — `tinydocs`' own loader E2E, which drives a +//! real module over a real broker. + +use super::{classify, sha256_hex, DocumentCallError}; +use crate::openhuman::config::Config; +use tinydocs::spec::{DocumentSpec, WirePresentationSpec}; + +/// A config with modules enabled but nothing fetchable. +fn offline_config() -> Config { + let mut config = Config::default(); + config.modules.enabled = true; + config.modules.allow_download = false; + config +} + +/// A bus failure carrying `name`. +fn failure(name: &str) -> tinybus::Error { + tinybus::Error::MethodFailed { + name: name.to_string(), + message: "something went wrong".to_string(), + } +} + +#[test] +fn an_invalid_input_is_reported_as_something_a_model_can_fix() { + assert!(matches!( + classify(&failure("ai.tinyhumans.tinydocs.Error.InvalidInput")), + DocumentCallError::InvalidInput(_) + )); +} + +#[test] +fn generation_and_extraction_failures_are_not_input_errors() { + // Telling a model its spec was wrong when the writer broke sends it into a + // rewrite loop over a spec that was fine. + for name in [ + "ai.tinyhumans.tinydocs.Error.GenerationFailed", + "ai.tinyhumans.tinydocs.Error.ExtractionFailed", + "ai.tinyhumans.tinydocs.Error.ModuleFailed", + "ai.tinyhumans.tinydocs.Error.TransferFailed", + "ai.tinyhumans.tinydocs.Error.OutputRefused", + "ai.tinyhumans.tinydocs.Error.UnknownOutput", + ] { + assert!( + matches!(classify(&failure(name)), DocumentCallError::Failed(_)), + "{name} should not be reported as an input error" + ); + } +} + +#[test] +fn an_unrecognised_wire_name_is_a_failure_not_an_input_error() { + // The conservative direction: a name this build does not know about is more + // likely a newer module than a bad spec. + assert!(matches!( + classify(&failure("ai.tinyhumans.tinydocs.Error.SomethingNewer")), + DocumentCallError::Failed(_) + )); +} + +#[test] +fn a_missing_module_reads_as_unavailable() { + assert!(matches!( + classify(&failure("ai.tinyhumans.tinybus.Error.ModuleUnavailable")), + DocumentCallError::Unavailable(_) + )); +} + +#[test] +fn every_error_renders_as_its_message() { + for error in [ + DocumentCallError::Unavailable("gone".to_string()), + DocumentCallError::InvalidInput("bad title".to_string()), + DocumentCallError::Failed("writer stopped".to_string()), + ] { + assert!(!error.to_string().is_empty()); + } + assert_eq!( + DocumentCallError::InvalidInput("bad title".to_string()).to_string(), + "bad title" + ); +} + +#[test] +fn the_digest_matches_the_modules_own_vector() { + // Both sides compute this independently; if they disagree every document + // round trip fails its integrity check. + assert_eq!( + sha256_hex(b""), + "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" + ); +} + +#[tokio::test] +async fn a_disabled_host_reports_unavailable_without_starting_a_broker() { + let mut config = offline_config(); + config.modules.enabled = false; + + let spec = DocumentSpec { + title: "Charter".to_string(), + author: None, + sections: vec![], + }; + assert!(matches!( + super::generate_docx(&config, &spec).await, + Err(DocumentCallError::Unavailable(_)) + )); + + let deck = WirePresentationSpec { + title: "Deck".to_string(), + author: None, + theme: None, + slides: vec![], + }; + assert!(matches!( + super::generate_pptx(&config, &deck, &[]).await, + Err(DocumentCallError::Unavailable(_)) + )); + + assert!(matches!( + super::extract_text(&config, b"%PDF-1.4\n").await, + Err(DocumentCallError::Unavailable(_)) + )); +} diff --git a/src/openhuman/modules/host.rs b/src/openhuman/modules/host.rs new file mode 100644 index 0000000000..7edbb6cd40 --- /dev/null +++ b/src/openhuman/modules/host.rs @@ -0,0 +1,182 @@ +//! The module host: a broker, a connection, and the loaded-module table. +//! +//! # Why modules get their own bus +//! +//! The core already runs a bus — `core::bus::BUS`, a `OnceBus` — but +//! it cannot be reused here. `OnceBus::init_in_process` constructs its `Broker` +//! internally and never hands it out, and `ModuleHost::new` needs a `Broker` to +//! attach loaded modules to. Rather than change tinybus to expose it, modules run +//! on a second in-process broker of their own. +//! +//! That is a real limitation and worth naming: a module on this bus cannot +//! publish a `DomainEvent`, so it can serve requests but cannot participate in +//! the core's event flow. For a codec that is exactly right — a document writer +//! has nothing to say to the subconscious. A module that did need to emit events +//! would need `OnceBus` to share its broker first. +//! +//! # What loading a module means +//! +//! `dlopen` runs code before anything can inspect it. tinybus's ABI descriptor, +//! manifest and digest gates decide whether a module is *admitted*, not whether +//! it is *safe*: once loaded it can read and write this process's memory, and a +//! native fault in it takes the core down. It is first-party code that happens to +//! ship separately. tinybus also never unloads a library, so a module that fails +//! is failed until restart — which is why [`super::ops`] caches failures instead +//! of retrying. +//! +//! Everything here is created once and lives for the process. There is no +//! shutdown path because there is nothing a shutdown could reclaim. +//! +//! # The runtime that gets here first owns the bus +//! +//! The broker and the connection are tokio tasks, so they belong to whichever +//! runtime calls [`runtime`] first. In the core that is the one runtime the +//! process has, and the question never arises. +//! +//! It arises in tests. Two `#[tokio::test]` functions each build their own +//! runtime, and the second one to call a loaded module finds a broker whose tasks +//! died with the first runtime — the call does not fail, it hangs until whatever +//! deadline is above it fires. Any test that drives a real module therefore has +//! to be the only one in its process, which is why the module-backed tool tests +//! are `#[ignore]`d rather than merely gated on an artifact being present. + +use std::sync::OnceLock; + +use tinybus::broker::Broker; +use tinybus::module::ModuleHost; +use tinybus::transport::memory::MemoryBus; +use tinybus::{Connection, Proxy}; + +/// The module bus, built once on first use. +static RUNTIME: OnceLock = OnceLock::new(); + +/// The broker, the host's own connection to it, and the module loader. +pub struct ModuleRuntime { + /// The loader. Owns every admitted module for the process lifetime. + host: ModuleHost, + /// This process's client connection, used to call into loaded modules. + connection: Connection, +} + +impl ModuleRuntime { + /// The loader. + #[must_use] + pub fn host(&self) -> &ModuleHost { + &self.host + } + + /// The connection modules are called over. + #[must_use] + pub fn connection(&self) -> &Connection { + &self.connection + } + + /// A proxy for one object on a loaded module. + /// + /// # Errors + /// + /// Returns an error if `bus_name` or `object_path` is not well formed, which + /// for a registry entry means the table is wrong rather than the module. + pub fn proxy(&self, bus_name: &str, object_path: &str) -> tinybus::Result { + self.connection.proxy(bus_name, object_path, bus_name) + } +} + +/// The process-wide module runtime, standing it up on first use. +/// +/// # Errors +/// +/// Returns an error if the broker's in-memory transport cannot be connected, +/// which in practice means the tokio runtime is shutting down. +/// +/// # Panics +/// +/// Does not panic: a lost initialisation race reuses the winner's runtime. +pub async fn runtime() -> tinybus::Result<&'static ModuleRuntime> { + if let Some(existing) = RUNTIME.get() { + return Ok(existing); + } + + let transport = MemoryBus::new(); + let broker = Broker::new(); + // The broker task is deliberately not retained. It lives as long as the + // process, and holding the handle would only offer an abort that must never + // be called: a module whose transport disappears faults, and a faulted + // module cannot be reloaded without a restart. + broker.spawn(transport.clone()); + + // Permissive admission, which is tinybus's default, and the choice is + // deliberate rather than inherited. + // + // Strict mode additionally refuses a module whose rustc version string + // differs from the host's. That sounds like the safer setting and is the + // wrong one here: released artifacts are built by CI on whatever toolchain + // that runner had, and this crate pins its own. Turning strict on was tried + // first and refused the real published artifact outright — `module + // libtinydocs_module.so refused: rustc version does not match in strict + // mode` — which would have meant the feature never worked in the field + // while every local build looked fine. + // + // Everything that protects the address space is still enforced in permissive + // mode: the ABI revision, the descriptor layout, the target triple, pointer + // width, endianness, feature bits, and the refusal of a panic=abort module. + // Only the toolchain string is relaxed, and it is reported rather than + // ignored. If a rustc mismatch ever does break a module, the failure is a + // refusal or a fault at load time, not silent corruption. + let host = ModuleHost::new(broker); + let connection = Connection::connect(transport.connect().await?).await?; + + let runtime = ModuleRuntime { host, connection }; + // A concurrent caller may have won the race. Its runtime is equivalent, so + // take the winner's and let ours drop. + Ok(RUNTIME.get_or_init(|| runtime)) +} + +/// Whether the module runtime has been stood up. +/// +/// Lets status reporting answer without starting a broker as a side effect of +/// being asked a question. +#[must_use] +pub fn is_started() -> bool { + RUNTIME.get().is_some() +} + +#[cfg(test)] +mod tests { + use super::{is_started, runtime}; + + /// Everything that touches the process-global module runtime, in one test. + /// + /// One test and not several, on purpose: `runtime()` is a process-global + /// started by whichever tokio runtime reaches it first, and each + /// `#[tokio::test]` builds its own. A second test function would find a + /// broker whose tasks died with the runtime that spawned it, and its call + /// would hang until something above it timed out rather than failing — the + /// same affinity hazard the module spec documents for the module-backed tool + /// tests. Splitting these up would reintroduce it. + #[tokio::test] + async fn the_module_bus_is_a_singleton_and_serves_proxies() { + let first = runtime().await.expect("runtime should start"); + assert!(is_started()); + let second = runtime().await.expect("runtime should be reused"); + assert!( + std::ptr::eq(first, second), + "runtime() handed out two different runtimes" + ); + + // Building a proxy is a local operation — it validates names and nothing + // else. Nothing has claimed the name, so the call fails rather than + // hanging, which is what makes `ensure_loaded` worth having. + let proxy = first + .proxy( + "ai.tinyhumans.tinydocs.Documents", + "/ai/tinyhumans/tinydocs/Documents", + ) + .expect("registry names should be well formed"); + let result: tinybus::Result = proxy.call("GenerateDocx", ()).await; + assert!(result.is_err(), "an unloaded module should not answer"); + + // A name that cannot be a bus name is refused without reaching the bus. + assert!(first.proxy("not a bus name", "/nope").is_err()); + } +} diff --git a/src/openhuman/modules/mod.rs b/src/openhuman/modules/mod.rs new file mode 100644 index 0000000000..96b30ad2ad --- /dev/null +++ b/src/openhuman/modules/mod.rs @@ -0,0 +1,48 @@ +//! Loadable native modules: capabilities that live outside this binary. +//! +//! A module is a compiled `cdylib` that speaks the tinybus module ABI. It is +//! downloaded from a pinned release, verified against a digest compiled into +//! [`registry`], admitted through tinybus's ABI and manifest gates, and attached +//! to a private in-process broker as an ordinary bus peer. The core then calls it +//! over that bus like any other service. +//! +//! # What this buys, and what it costs +//! +//! It buys a dependency boundary that survives compilation. A document writer, a +//! codec, an integration SDK — none of these are kernel work, and each one drags +//! a tree of parsers into a binary that mostly does something else. Moving one +//! out removes its dependencies from the build rather than merely gating them. +//! +//! It costs process isolation, and that is not a small thing. A loaded module +//! shares this address space, these privileges and this crash domain: tinybus's +//! deadlines, bounded queues and caught panics contain ordinary misbehaviour, not +//! a segfault. `dlopen` also runs code before any symbol can be inspected, so the +//! ABI, manifest and digest gates decide what is *admitted*, never what is +//! *safe*. Modules are first-party code that happens to ship separately. Anything +//! untrusted belongs in a process, not here. +//! +//! And tinybus never unloads a library. A module that fails is failed until the +//! process restarts, which is why [`ops`] caches failures rather than retrying. +//! +//! # Layout +//! +//! - [`registry`] — the compiled-in set of loadable modules and their digests. +//! - [`documents`] — the host half of the `tinydocs` module's three operations. +//! - [`platform`] — which published artifact belongs to this host. +//! - [`host`] — the module broker, connection and loader. +//! - [`ops`] — resolving, loading, and reporting status. +//! - [`schemas`] — the `modules` RPC surface. +//! - [`boot`] — what happens at startup. + +pub mod boot; +pub mod documents; +pub mod host; +pub mod ops; +pub mod platform; +pub mod registry; +pub mod schemas; +pub mod types; + +pub use ops::ensure_loaded; +pub use schemas::{all_controller_schemas, all_registered_controllers}; +pub use types::{LoadPolicy, ModuleRecord, ModuleState, ModuleStatus, PlatformAsset}; diff --git a/src/openhuman/modules/ops.rs b/src/openhuman/modules/ops.rs new file mode 100644 index 0000000000..b4ae108e65 --- /dev/null +++ b/src/openhuman/modules/ops.rs @@ -0,0 +1,346 @@ +//! Loading modules, and deciding when not to. +//! +//! [`ensure_loaded`] is the entry point every caller uses. It resolves a module +//! once per process and remembers the outcome — including failure, which is the +//! part worth explaining. +//! +//! tinybus never unloads a library. A module that was refused, faulted, or failed +//! to initialise keeps whatever it mapped, and loading it again cannot reach a +//! different outcome without a restart. Retrying would therefore mean paying a +//! download and a `dlopen` on every tool call to arrive at the same error, so a +//! failure is cached and returned directly. The user-visible consequence is that +//! fixing a module means restarting the core, which is stated in the error. + +use std::collections::HashMap; +use std::path::{Path, PathBuf}; +use std::sync::{Mutex, OnceLock}; + +use super::types::{ModuleRecord, ModuleState, ModuleStatus}; +use super::{host, platform, registry}; +use crate::openhuman::config::Config; + +/// Outcome of resolving one module, remembered for the process lifetime. +#[derive(Debug, Clone)] +enum Resolution { + Ready, + /// Terminal. Carries the sanitised reason, never a path or URL. + Failed(String), +} + +/// Per-module resolution results. +fn resolutions() -> &'static Mutex> { + static RESOLUTIONS: OnceLock>> = OnceLock::new(); + RESOLUTIONS.get_or_init(|| Mutex::new(HashMap::new())) +} + +/// Serialises resolution so two concurrent tool calls do not both download. +/// +/// Coarse on purpose — one lock for all modules rather than one per module. +/// Resolution happens at most once per module per process, so contention is +/// bounded by the number of modules, and a per-module lock table would be more +/// machinery than the problem justifies. +fn resolve_gate() -> &'static tokio::sync::Mutex<()> { + static GATE: OnceLock> = OnceLock::new(); + GATE.get_or_init(|| tokio::sync::Mutex::new(())) +} + +/// Ensure `id` is loaded and serving, loading it if this is the first ask. +/// +/// Resolution order is cheapest-first: already loaded, then an artifact already +/// on disk, then the module search path, then a verified download. +/// +/// # Errors +/// +/// Returns a message suitable for surfacing to a user or a model when the module +/// cannot be loaded: no artifact for this host, downloads disabled, a refused +/// artifact, or a previous failure in this process. +pub async fn ensure_loaded(config: &Config, id: &str) -> Result<(), String> { + if !config.modules.enabled { + return Err(format!( + "module '{id}' is unavailable: modules are disabled in configuration" + )); + } + let record = registry::find(id).ok_or_else(|| format!("unknown module '{id}'"))?; + + if let Some(cached) = cached_resolution(id) { + return match cached { + Resolution::Ready => Ok(()), + Resolution::Failed(reason) => Err(reason), + }; + } + + // One resolver at a time. Two chat turns asking for a document at once must + // not both download the same archive. + let _guard = resolve_gate().lock().await; + // Re-check: the winner of the race may have resolved it while we waited. + if let Some(cached) = cached_resolution(id) { + return match cached { + Resolution::Ready => Ok(()), + Resolution::Failed(reason) => Err(reason), + }; + } + + let outcome = resolve(config, record).await; + let resolution = match &outcome { + Ok(()) => Resolution::Ready, + Err(reason) => Resolution::Failed(reason.clone()), + }; + if let Ok(mut cache) = resolutions().lock() { + cache.insert(id.to_string(), resolution); + } + outcome +} + +/// The remembered outcome for `id`, if it has been resolved. +fn cached_resolution(id: &str) -> Option { + resolutions().lock().ok()?.get(id).cloned() +} + +/// Do the actual work of getting `record` serving. +async fn resolve(config: &Config, record: &'static ModuleRecord) -> Result<(), String> { + let runtime = host::runtime().await.map_err(|_| { + format!( + "module '{}' is unavailable: the module bus could not start", + record.id + ) + })?; + + // Already serving — a module loaded from the search path at boot, or by an + // earlier explicit `modules.load_local`. + if runtime + .host() + .list() + .iter() + .any(|info| info.manifest.bus_name.as_str() == record.bus_name) + { + return Ok(()); + } + + // An override points at a developer's own build. Checked before the pinned + // release so a module can be iterated on against a live core. + if let Some(path) = local_override(config, record.id) { + return blocking(move || load_local(runtime, &path, record.id)).await; + } + + // An artifact already extracted into the install directory by an earlier run. + if let Some(path) = installed_artifact(config, record) { + return blocking(move || load_local(runtime, &path, record.id)).await; + } + + // The search path tinybus itself honours, including OPENHUMAN_MODULE_PATH. + // A refused search-path artifact is ordinary — most directories hold + // nothing, and tinybus reports each refusal with a sanitised reason — so the + // errors are dropped here and only a match on the bus name counts. + let bus_name = record.bus_name; + let found_on_search_path = tokio::task::spawn_blocking(move || { + runtime + .host() + .load_search_paths() + .into_iter() + .flatten() + .any(|info| info.manifest.bus_name.as_str() == bus_name) + }) + .await + .unwrap_or(false); + if found_on_search_path { + return Ok(()); + } + + if !config.modules.allow_download { + return Err(format!( + "module '{}' is unavailable: no local artifact is installed and downloads are \ + disabled in configuration", + record.id + )); + } + + // Off the runtime worker. `load_github_release` fetches over the network, + // hashes the archive, extracts it and `dlopen`s the result — all + // synchronously. Left inline it would stall every other task sharing this + // worker for the length of a download on whatever link the user has. + blocking(move || download(runtime, record)).await +} + +/// Run a blocking module operation on the blocking pool. +/// +/// A panic in the loader is reported rather than propagated: it would otherwise +/// take down whichever task happened to be awaiting the load. +async fn blocking(work: F) -> Result<(), String> +where + F: FnOnce() -> Result<(), String> + Send + 'static, +{ + match tokio::task::spawn_blocking(work).await { + Ok(result) => result, + Err(err) => Err(format!( + "the module loader did not finish: {err}. This is terminal for the running \ + process; restart the app to try again" + )), + } +} + +/// Download, verify, and load the pinned release artifact for this host. +fn download( + runtime: &'static host::ModuleRuntime, + record: &'static ModuleRecord, +) -> Result<(), String> { + let candidates = platform::host_candidates(); + let assets: Vec<_> = candidates + .iter() + .filter_map(|key| record.asset_for(key)) + .collect(); + if assets.is_empty() { + return Err(format!( + "module '{}' is not available for this platform, so the feature it provides is \ + unavailable in this build", + record.id + )); + } + + // Try the preferred artifact first and fall through on admission failure — + // a host newer than the newest published build runs that build, and one + // whose toolchain the newest artifact does not match falls back. + let mut last_error = String::new(); + for asset in assets { + match runtime.host().load_github_release( + record.release_url, + asset.archive, + Some(asset.sha256), + serde_json::json!({}), + ) { + Ok(_) => { + log::info!( + "[modules] loaded '{}' {} from the pinned release ({})", + record.id, + record.version, + asset.host_key + ); + return Ok(()); + } + Err(err) => { + // Sanitised: tinybus's own errors carry only a basename and a + // fixed reason, and nothing here adds a path or a URL. + last_error = err.to_string(); + log::warn!( + "[modules] '{}' artifact for {} was not admitted: {last_error}", + record.id, + asset.host_key + ); + } + } + } + Err(format!( + "module '{}' could not be loaded: {last_error}. This is terminal for the running \ + process; restart the app to try again", + record.id + )) +} + +/// Load a platform library from `path`. +pub(super) fn load_local( + runtime: &host::ModuleRuntime, + path: &Path, + id: &str, +) -> Result<(), String> { + match runtime.host().load_file(path) { + Ok(_) => { + log::info!("[modules] loaded '{id}' from a local artifact"); + Ok(()) + } + Err(err) => Err(format!( + "module '{id}' could not be loaded from its local artifact: {err}. This is terminal \ + for the running process; restart the app to try again" + )), + } +} + +/// A configured local artifact for `id`, if one is set. +fn local_override(config: &Config, id: &str) -> Option { + config + .modules + .overrides + .iter() + .find_map(|entry| (entry.id == id).then(|| PathBuf::from(entry.path.clone()))) +} + +/// The artifact an earlier run extracted, if it is still there. +fn installed_artifact(config: &Config, record: &ModuleRecord) -> Option { + let dir = install_dir(config)?.join(record.id).join(record.version); + let candidate = dir.join(platform_library_name(record.id)); + candidate.is_file().then_some(candidate) +} + +/// Where downloaded artifacts are kept. +/// +/// The user cache directory, falling back to the workspace when there is none — +/// the same shape the Node and Python runtime installers use, for the same +/// reason: a headless container often has no `XDG_CACHE_HOME`, and failing to +/// install because of that would be worse than writing beside the workspace. +#[must_use] +pub fn install_dir(config: &Config) -> Option { + if let Some(configured) = &config.modules.install_dir { + return Some(PathBuf::from(configured)); + } + if let Some(cache) = dirs::cache_dir() { + return Some(cache.join("openhuman").join("modules")); + } + log::warn!("[modules] no cache directory; installing modules under the workspace instead"); + Some(config.workspace_dir.join("modules")) +} + +/// The platform library file name for a module id. +fn platform_library_name(id: &str) -> String { + let stem = id.replace('-', "_"); + if cfg!(target_os = "windows") { + format!("{stem}_module.dll") + } else if cfg!(target_os = "macos") { + format!("lib{stem}_module.dylib") + } else { + format!("lib{stem}_module.so") + } +} + +/// Status of every module this build knows about. +#[must_use] +pub fn list(config: &Config) -> Vec { + registry::ALL + .iter() + .map(|record| status_of(config, record)) + .collect() +} + +/// Status of one module. +fn status_of(config: &Config, record: &ModuleRecord) -> ModuleStatus { + let (state, detail) = match cached_resolution(record.id) { + Some(Resolution::Ready) => (ModuleState::Ready, None), + Some(Resolution::Failed(reason)) => (ModuleState::Failed, Some(reason)), + None if !config.modules.enabled => ( + ModuleState::Unsupported, + Some("modules are disabled in configuration".to_string()), + ), + None => { + let supported = platform::host_candidates() + .iter() + .any(|key| record.asset_for(key).is_some()); + if supported { + (ModuleState::Available, None) + } else { + ( + ModuleState::Unsupported, + Some("no artifact is published for this platform".to_string()), + ) + } + } + }; + ModuleStatus { + id: record.id.to_string(), + description: record.description.to_string(), + version: record.version.to_string(), + bus_name: record.bus_name.to_string(), + state, + detail, + } +} + +#[cfg(test)] +#[path = "ops_tests.rs"] +mod tests; diff --git a/src/openhuman/modules/ops_tests.rs b/src/openhuman/modules/ops_tests.rs new file mode 100644 index 0000000000..2727a987a8 --- /dev/null +++ b/src/openhuman/modules/ops_tests.rs @@ -0,0 +1,146 @@ +//! Tests for module resolution and status reporting. +//! +//! Nothing here downloads. The paths that matter for correctness are the +//! refusals — disabled, unknown, unsupported, downloads-off — and each one is +//! reachable without touching the network, which is what keeps them in the unit +//! suite instead of behind an ignore. + +use crate::openhuman::config::Config; +use crate::openhuman::modules::ops::{self, install_dir, list}; +use crate::openhuman::modules::registry; +use crate::openhuman::modules::types::ModuleState; + +/// A config with modules on but downloads off, so nothing reaches the network. +fn offline_config() -> Config { + let mut config = Config::default(); + config.modules.enabled = true; + config.modules.allow_download = false; + config +} + +#[test] +fn the_default_config_enables_modules_and_downloads() { + let config = Config::default(); + assert!(config.modules.enabled); + assert!(config.modules.allow_download); + assert!(config.modules.install_dir.is_none()); + assert!(config.modules.overrides.is_empty()); +} + +#[test] +fn list_reports_every_registry_entry() { + let statuses = list(&offline_config()); + assert_eq!(statuses.len(), registry::ALL.len()); + assert!(statuses.iter().any(|status| status.id == "tinydocs")); + for status in &statuses { + assert!(!status.version.is_empty()); + assert!(!status.bus_name.is_empty()); + } +} + +#[test] +fn a_module_is_available_or_unsupported_before_anything_is_loaded() { + // Which one depends on whether this host has a published artifact, and both + // are correct answers — what must not happen is `Ready` for something that + // has never been loaded. + for status in list(&offline_config()) { + assert!( + matches!( + status.state, + ModuleState::Available | ModuleState::Unsupported + ), + "{} reported {:?} before any load", + status.id, + status.state + ); + if status.state == ModuleState::Unsupported { + assert!( + status.detail.is_some(), + "an unsupported module must say why" + ); + } + } +} + +#[test] +fn disabling_modules_marks_everything_unsupported_with_a_reason() { + let mut config = offline_config(); + config.modules.enabled = false; + for status in list(&config) { + assert_eq!(status.state, ModuleState::Unsupported); + assert!( + status + .detail + .as_deref() + .is_some_and(|detail| detail.contains("disabled")), + "the reason should name the configuration, got {:?}", + status.detail + ); + } +} + +#[tokio::test] +async fn a_disabled_host_refuses_before_starting_a_broker() { + let mut config = offline_config(); + config.modules.enabled = false; + let err = ops::ensure_loaded(&config, "tinydocs") + .await + .expect_err("modules are disabled"); + assert!(err.contains("disabled"), "unhelpful message: {err}"); +} + +#[tokio::test] +async fn an_unknown_module_is_refused_by_name() { + let err = ops::ensure_loaded(&offline_config(), "not-a-module") + .await + .expect_err("unknown module"); + assert!(err.contains("not-a-module"), "unhelpful message: {err}"); +} + +#[test] +fn the_install_directory_is_namespaced_under_openhuman() { + // Two arms where the first implies the second would make this vacuous, so + // assert the components: artifacts land under an `openhuman` directory, in a + // `modules` subdirectory, and never at the root of a shared cache. + let dir = install_dir(&offline_config()).expect("an install directory is always resolvable"); + assert!( + dir.ends_with("modules"), + "install directory does not end in `modules`: {}", + dir.display() + ); + assert!( + dir.parent() + .and_then(|parent| parent.file_name()) + .is_some_and(|name| name == "openhuman"), + "install directory is not namespaced under `openhuman`: {}", + dir.display() + ); +} + +#[test] +fn a_configured_install_directory_is_honoured() { + let mut config = offline_config(); + config.modules.install_dir = Some("/tmp/openhuman-modules-test".to_string()); + assert_eq!( + install_dir(&config).expect("configured"), + std::path::PathBuf::from("/tmp/openhuman-modules-test") + ); +} + +#[test] +fn errors_never_leak_a_path_or_a_url() { + // Status details are rendered into a UI and pasted into bug reports. + let mut config = offline_config(); + config.modules.enabled = false; + for status in list(&config) { + let detail = status.detail.unwrap_or_default(); + assert!( + !detail.contains('/'), + "a path leaked into a status: {detail}" + ); + assert!( + !detail.contains("http"), + "a URL leaked into a status: {detail}" + ); + } +} diff --git a/src/openhuman/modules/platform.rs b/src/openhuman/modules/platform.rs new file mode 100644 index 0000000000..c37205ac5c --- /dev/null +++ b/src/openhuman/modules/platform.rs @@ -0,0 +1,236 @@ +//! Choosing which published artifact belongs to this host. +//! +//! A target triple is not enough. tinybus admits a module only if its target, +//! pointer width, endianness and toolchain all match, and a `.so` built against +//! glibc 2.39 does not load on a host with glibc 2.35 — the failure is a +//! `dlopen` error about a missing symbol version, not something the ABI gate can +//! phrase helpfully. Releases therefore publish per-distro artifacts +//! (`ubuntu-22.04-x86_64` beside `ubuntu-24.04-x86_64`), and this module decides +//! which one to reach for. +//! +//! It returns an ordered list rather than a single answer. The newest build that +//! could work is tried first, and an admission failure falls through to the next +//! candidate: a host newer than every published artifact runs the newest one, +//! which is the case that works, while a host older than all of them gets the +//! oldest, which is the case most likely to. +//! +//! Every path here is pure — `std::env::consts` and, on Linux, the glibc version +//! string. That is what makes the whole table unit-testable on one machine. + +/// Ordered artifact keys for this host, newest first. +/// +/// Empty when no published artifact can run here, which is the honest answer for +/// a musl or BSD host: the alternative is downloading something that cannot +/// possibly `dlopen`. +#[must_use] +pub fn host_candidates() -> Vec { + candidates_for( + std::env::consts::OS, + std::env::consts::ARCH, + glibc_version(), + ) +} + +/// The pure half of [`host_candidates`], with the host facts injected. +/// +/// `glibc` is `None` off Linux, and `None` on a Linux host whose libc is not +/// glibc — which is the distinction that decides whether any Linux artifact is +/// usable at all. +#[must_use] +pub fn candidates_for(os: &str, arch: &str, glibc: Option<(u32, u32)>) -> Vec { + let arch_key = match arch { + "x86_64" => "x86_64", + "aarch64" => "arm64", + _ => return Vec::new(), + }; + + match os { + // Releases are cut on the two maintained LTS images. A host at or above + // the newer one takes the newer build; anything older takes the older + // build, which is the one whose glibc floor it can satisfy. + "linux" => { + let Some((major, minor)) = glibc else { + // musl, or a libc we could not identify. No published artifact + // targets it, and guessing produces a dlopen failure at first + // use rather than a clear answer now. + return Vec::new(); + }; + if (major, minor) >= (2, 39) { + vec![ + format!("ubuntu-24.04-{arch_key}"), + format!("ubuntu-22.04-{arch_key}"), + ] + } else if (major, minor) >= (2, 35) { + vec![format!("ubuntu-22.04-{arch_key}")] + } else { + // Older than every published floor. Offering the oldest build + // anyway would fail at dlopen with a symbol-version error. + Vec::new() + } + } + // macOS artifacts are forward-compatible, so the ordering is newest + // first and an older host falls through to the older build. + "macos" => vec![ + format!("macos-26-{arch_key}"), + format!("macos-15-{arch_key}"), + ], + "windows" => match arch_key { + // The arm64 build is published only against Windows 11. + "arm64" => vec!["windows-11-arm64".to_string()], + _ => vec![ + "windows-2025-x86_64".to_string(), + "windows-2022-x86_64".to_string(), + ], + }, + _ => Vec::new(), + } +} + +/// The host's glibc version, or `None` if it is not glibc. +/// +/// Gated on `target_env = "gnu"`, not merely on Linux: musl does not provide +/// this symbol, so a musl build would fail to *link* rather than fall through to +/// the `None` below. That fallback is what makes a musl host report "no +/// artifact for this platform" instead of downloading one it cannot load. +#[cfg(all(target_os = "linux", target_env = "gnu"))] +fn glibc_version() -> Option<(u32, u32)> { + // `gnu_get_libc_version` is the only reliable answer — parsing `ldd + // --version` means spawning a process and reading localised output, and + // reading a symlink target guesses. On musl the symbol is absent, which is + // exactly the distinction that matters. + unsafe extern "C" { + fn gnu_get_libc_version() -> *const std::os::raw::c_char; + } + + // SAFETY: `gnu_get_libc_version` takes no arguments and returns a pointer to + // a static NUL-terminated string owned by libc, valid for the process + // lifetime. On a non-glibc libc the symbol does not resolve and this code is + // not reached, because the binary would fail to link against it — which is + // why the musl case is handled by the returned `None` below rather than here. + let raw = unsafe { gnu_get_libc_version() }; + if raw.is_null() { + return None; + } + // SAFETY: non-null, NUL-terminated, static for the process lifetime. + let version = unsafe { std::ffi::CStr::from_ptr(raw) }.to_str().ok()?; + parse_glibc_version(version) +} + +/// Every other target: not glibc, so there is no version to report. +/// +/// Covers musl and every non-Linux host. Both want the same answer — `None`, +/// which `candidates_for` turns into an empty candidate list. +#[cfg(not(all(target_os = "linux", target_env = "gnu")))] +fn glibc_version() -> Option<(u32, u32)> { + None +} + +/// Parse a `major.minor` glibc version string, ignoring any suffix. +fn parse_glibc_version(raw: &str) -> Option<(u32, u32)> { + let mut parts = raw.trim().split('.'); + let major = parts.next()?.parse().ok()?; + // A version like "2.39" has no third component; one like "2.39.1" does, and + // the patch level never affects which artifact is usable. + let minor_raw = parts.next()?; + let minor = minor_raw + .chars() + .take_while(char::is_ascii_digit) + .collect::() + .parse() + .ok()?; + Some((major, minor)) +} + +#[cfg(test)] +mod tests { + use super::{candidates_for, host_candidates, parse_glibc_version}; + + #[test] + fn a_modern_linux_host_prefers_the_newer_build_but_can_fall_back() { + assert_eq!( + candidates_for("linux", "x86_64", Some((2, 39))), + vec!["ubuntu-24.04-x86_64", "ubuntu-22.04-x86_64"] + ); + assert_eq!( + candidates_for("linux", "aarch64", Some((2, 41))), + vec!["ubuntu-24.04-arm64", "ubuntu-22.04-arm64"] + ); + } + + #[test] + fn an_older_linux_host_is_offered_only_what_its_glibc_can_load() { + // Offering the 24.04 build here would fail at dlopen with a symbol + // version error, which is a worse outcome than not offering it. + assert_eq!( + candidates_for("linux", "x86_64", Some((2, 35))), + vec!["ubuntu-22.04-x86_64"] + ); + } + + #[test] + fn a_linux_host_below_every_published_floor_gets_nothing() { + assert!(candidates_for("linux", "x86_64", Some((2, 31))).is_empty()); + } + + #[test] + fn a_non_glibc_linux_host_gets_nothing() { + // musl. No published artifact targets it; guessing produces a dlopen + // failure at first use rather than a clear answer now. + assert!(candidates_for("linux", "x86_64", None).is_empty()); + } + + #[test] + fn macos_prefers_the_newer_build_and_falls_back() { + assert_eq!( + candidates_for("macos", "aarch64", None), + vec!["macos-26-arm64", "macos-15-arm64"] + ); + assert_eq!( + candidates_for("macos", "x86_64", None), + vec!["macos-26-x86_64", "macos-15-x86_64"] + ); + } + + #[test] + fn windows_arm64_has_only_the_windows_11_build() { + assert_eq!( + candidates_for("windows", "aarch64", None), + vec!["windows-11-arm64"] + ); + assert_eq!( + candidates_for("windows", "x86_64", None), + vec!["windows-2025-x86_64", "windows-2022-x86_64"] + ); + } + + #[test] + fn an_unsupported_os_or_architecture_gets_nothing() { + assert!(candidates_for("freebsd", "x86_64", Some((2, 39))).is_empty()); + assert!(candidates_for("linux", "riscv64", Some((2, 39))).is_empty()); + assert!(candidates_for("macos", "powerpc", None).is_empty()); + } + + #[test] + fn glibc_versions_parse_with_and_without_a_patch_level() { + assert_eq!(parse_glibc_version("2.39"), Some((2, 39))); + assert_eq!(parse_glibc_version("2.39.1"), Some((2, 39))); + assert_eq!(parse_glibc_version(" 2.35 "), Some((2, 35))); + // Ubuntu ships versions like "2.39-0ubuntu8.3". + assert_eq!(parse_glibc_version("2.39-0ubuntu8.3"), Some((2, 39))); + assert_eq!(parse_glibc_version("garbage"), None); + assert_eq!(parse_glibc_version("2"), None); + } + + #[test] + fn the_running_host_resolves_without_panicking() { + // Whatever this machine is, asking must be safe and must agree with the + // pure table for its own os/arch. + let live = host_candidates(); + let expected = candidates_for( + std::env::consts::OS, + std::env::consts::ARCH, + super::glibc_version(), + ); + assert_eq!(live, expected); + } +} diff --git a/src/openhuman/modules/registry.rs b/src/openhuman/modules/registry.rs new file mode 100644 index 0000000000..0b025330d1 --- /dev/null +++ b/src/openhuman/modules/registry.rs @@ -0,0 +1,273 @@ +//! The set of modules this build knows how to load. +//! +//! # Why a compiled-in table +//! +//! A loaded module is trusted native code in this process: it shares the address +//! space, the privileges and the crash domain, and tinybus never unloads it. +//! Which modules may be loaded, and which bytes count as legitimate, are +//! therefore build-time decisions rather than runtime discovery. There is no +//! "module marketplace" here on purpose — a registry a server could add entries +//! to would be a remote-code-execution surface with a download step. +//! +//! # The digests are a second gate, not the only one +//! +//! tinybus fetches the release's own `checksum.toml`, compares it with the digest +//! the host supplies, hashes the downloaded archive, and only then extracts and +//! loads. The digests below are the host's half of that agreement. Pinning them +//! in the source is what makes the check auditable offline: a reviewer can read +//! this file against the release page, and a release re-cut under the same tag +//! stops matching rather than silently replacing what runs in-process. +//! +//! # Adding an entry +//! +//! Take the values verbatim from the release's `checksum.toml`. Do not compute +//! them from a local build — the point is to pin what the release publishes, and +//! a locally recomputed digest would agree with itself no matter what was served. + +use super::types::{LoadPolicy, ModuleRecord, PlatformAsset}; + +/// The `tinydocs` module: `.docx` / `.pptx` synthesis and `.pdf` extraction. +/// +/// Lazy, because a user who never asks for a document should not pay a download, +/// a `dlopen`, and the resident cost of a library that is never unloaded. +const TINYDOCS: ModuleRecord = ModuleRecord { + id: "tinydocs", + description: "Document synthesis (.docx, .pptx) and PDF text extraction", + bus_name: "ai.tinyhumans.tinydocs.Documents", + object_path: "/ai/tinyhumans/tinydocs/Documents", + version: "0.1.12", + release_url: "https://github.com/tinyhumansai/tinydocs/releases/tag/v0.1.12", + assets: &[ + PlatformAsset { + host_key: "ubuntu-24.04-x86_64", + archive: "tinydocs-module-0.1.12-ubuntu-24.04-x86_64.tar.gz", + sha256: "89a1c6f3ff386a2190bfa4efbef75d564651f75cd8136c8940ec4de950f69a05", + }, + PlatformAsset { + host_key: "ubuntu-24.04-arm64", + archive: "tinydocs-module-0.1.12-ubuntu-24.04-arm64.tar.gz", + sha256: "685b38dbb9b5beba0105b2991212882ba2d4cb74fa1f3613c9eb9b75de023f0b", + }, + PlatformAsset { + host_key: "ubuntu-22.04-x86_64", + archive: "tinydocs-module-0.1.12-ubuntu-22.04-x86_64.tar.gz", + sha256: "35ac3d05202dfcb425c3d6448f1740656b5df3e6276ecd97ded973f92c356591", + }, + PlatformAsset { + host_key: "ubuntu-22.04-arm64", + archive: "tinydocs-module-0.1.12-ubuntu-22.04-arm64.tar.gz", + sha256: "3870486bd42fc729cc56b7dae9343aaa854de2b24d30f4a6386a8083db6ef32e", + }, + PlatformAsset { + host_key: "macos-26-arm64", + archive: "tinydocs-module-0.1.12-macos-26-arm64.tar.gz", + sha256: "18ab086bd58d8fec2ac407981f2013d7284a8d7e0c07cdc51ee6fdde4535f431", + }, + PlatformAsset { + host_key: "macos-26-x86_64", + archive: "tinydocs-module-0.1.12-macos-26-x86_64.tar.gz", + sha256: "426711799118bae95a691d6a61920c4bc93b76e930cdbce4209e730aa8b9efa2", + }, + PlatformAsset { + host_key: "macos-15-arm64", + archive: "tinydocs-module-0.1.12-macos-15-arm64.tar.gz", + sha256: "f0aa5d7076a1ce3cdf4c0cf4dd15e274bfbd7d4ccfced6793e55651a7499f3d4", + }, + PlatformAsset { + host_key: "macos-15-x86_64", + archive: "tinydocs-module-0.1.12-macos-15-x86_64.tar.gz", + sha256: "9fbc1aa2dfabe35e492aa6abea90515ab83ea13a191c87306401a499d432e5e3", + }, + PlatformAsset { + host_key: "windows-2025-x86_64", + archive: "tinydocs-module-0.1.12-windows-2025-x86_64.zip", + sha256: "4870bb1084ad0435b44d1ec845c5d2f398e27e430bec00ec0f58e0664e5bfc3f", + }, + PlatformAsset { + host_key: "windows-2022-x86_64", + archive: "tinydocs-module-0.1.12-windows-2022-x86_64.zip", + sha256: "f1fc72690dd59890d7a629002ab2ade0547b2a3ca23c5cc54f5d40ed0e8b24af", + }, + PlatformAsset { + host_key: "windows-11-arm64", + archive: "tinydocs-module-0.1.12-windows-11-arm64.zip", + sha256: "c4f7bda63c17a5bbdb10d8e5bab04c9b0113fbd420f83465208b64a286f2127a", + }, + ], + load: LoadPolicy::Lazy, +}; + +/// Every module this build can load. +pub const ALL: &[ModuleRecord] = &[TINYDOCS]; + +/// The record for `id`, if this build knows it. +#[must_use] +pub fn find(id: &str) -> Option<&'static ModuleRecord> { + ALL.iter().find(|record| record.id == id) +} + +#[cfg(test)] +mod tests { + use super::{find, ALL}; + use crate::openhuman::modules::platform::candidates_for; + + #[test] + fn ids_and_bus_names_are_unique() { + // Two records claiming one bus name is a conflict tinybus would only + // surface at load time, on whichever one happened to be second. + for (i, record) in ALL.iter().enumerate() { + for other in &ALL[i + 1..] { + assert_ne!(record.id, other.id, "duplicate module id"); + assert_ne!(record.bus_name, other.bus_name, "duplicate bus name"); + } + } + } + + #[test] + fn every_object_path_matches_its_bus_name() { + // tinybus derives a module's object path from its bus name by replacing + // dots with slashes, and admission compares the two. A mismatch here is + // a module that downloads and then refuses to load. + for record in ALL { + assert_eq!( + record.object_path, + format!("/{}", record.bus_name.replace('.', "/")), + "{} object path does not match its bus name", + record.id + ); + } + } + + #[test] + fn every_digest_is_a_lowercase_sha256() { + // An uppercase or truncated digest is refused by tinybus at download + // time, which is a slow way to find a typo in this file. + for record in ALL { + for asset in record.assets { + assert_eq!( + asset.sha256.len(), + 64, + "{} / {} digest is not 64 characters", + record.id, + asset.host_key + ); + assert!( + asset + .sha256 + .bytes() + .all(|b| b.is_ascii_digit() || (b'a'..=b'f').contains(&b)), + "{} / {} digest is not lowercase hex", + record.id, + asset.host_key + ); + } + } + } + + #[test] + fn every_asset_name_carries_its_host_key_and_a_known_extension() { + // tinybus selects the asset by exact name and requires a `.tar.gz` or + // `.zip` archive, so a name that does not match its key is a module that + // loads the wrong platform's library. + for record in ALL { + for asset in record.assets { + assert!( + asset.archive.contains(asset.host_key), + "{} asset {} does not name its host key {}", + record.id, + asset.archive, + asset.host_key + ); + let windows = asset.host_key.starts_with("windows"); + assert_eq!( + windows, + asset.archive.ends_with(".zip"), + "{} asset {} has the wrong archive format for its host", + record.id, + asset.archive + ); + if !windows { + assert!(asset.archive.ends_with(".tar.gz")); + } + } + } + } + + #[test] + fn every_asset_name_carries_the_pinned_version() { + // The digests and the version have to describe one release; an asset + // left behind at an older version would download bytes the digest + // beside it never matched. + for record in ALL { + for asset in record.assets { + assert!( + asset.archive.contains(record.version), + "{} asset {} is not from version {}", + record.id, + asset.archive, + record.version + ); + } + } + } + + #[test] + fn the_release_url_is_a_tag_on_github() { + // tinybus refuses a URL that is not a tag, because a branch URL names + // bytes that can change under a digest that was checked once. + for record in ALL { + assert!( + record + .release_url + .starts_with("https://github.com/tinyhumansai/"), + "{} release url is not an upstream GitHub URL", + record.id + ); + assert!( + record.release_url.contains("/releases/tag/"), + "{} release url is not a tag", + record.id + ); + assert!( + record.release_url.ends_with(record.version), + "{} release url does not name version {}", + record.id, + record.version + ); + } + } + + #[test] + fn every_host_the_platform_table_can_produce_has_an_asset() { + // The two tables are written independently and would drift silently: + // `platform` offering a key no release publishes turns a supported host + // into an "unsupported host" at first use. + let hosts = [ + ("linux", "x86_64", Some((2, 39))), + ("linux", "aarch64", Some((2, 39))), + ("linux", "x86_64", Some((2, 35))), + ("linux", "aarch64", Some((2, 35))), + ("macos", "x86_64", None), + ("macos", "aarch64", None), + ("windows", "x86_64", None), + ("windows", "aarch64", None), + ]; + for record in ALL { + for (os, arch, glibc) in hosts { + for key in candidates_for(os, arch, glibc) { + assert!( + record.asset_for(&key).is_some(), + "{} publishes no asset for {key}, which {os}/{arch} would ask for", + record.id + ); + } + } + } + } + + #[test] + fn find_resolves_known_ids_only() { + assert!(find("tinydocs").is_some()); + assert!(find("not-a-module").is_none()); + } +} diff --git a/src/openhuman/modules/schemas.rs b/src/openhuman/modules/schemas.rs new file mode 100644 index 0000000000..b52f30085d --- /dev/null +++ b/src/openhuman/modules/schemas.rs @@ -0,0 +1,192 @@ +//! The `modules` RPC namespace. +//! +//! Read-only plus one deliberate action. `list` and `status` report what this +//! build knows and what it has loaded; `load` forces a lazy module to resolve +//! now, which is what a settings screen offering "install this now" needs. +//! +//! There is no `unload`, and there cannot be: tinybus never unloads a library. +//! There is also no way to name an artifact over RPC — the loadable set is +//! compiled into [`super::registry`], and a method that could point the loader at +//! an arbitrary path would turn this namespace into remote code execution. + +use serde_json::{Map, Value}; + +use super::ops; +use crate::core::all::{ControllerFuture, RegisteredController}; +use crate::core::{ControllerSchema, FieldSchema, TypeSchema}; +use crate::openhuman::config::rpc as config_rpc; + +pub fn all_controller_schemas() -> Vec { + vec![schemas("list"), schemas("status"), schemas("load")] +} + +pub fn all_registered_controllers() -> Vec { + vec![ + RegisteredController { + schema: schemas("list"), + handler: handle_list, + }, + RegisteredController { + schema: schemas("status"), + handler: handle_status, + }, + RegisteredController { + schema: schemas("load"), + handler: handle_load, + }, + ] +} + +pub fn schemas(function: &str) -> ControllerSchema { + match function { + "list" => ControllerSchema { + namespace: "modules", + function: "list", + description: "List every loadable module this build knows, with its state.", + inputs: vec![], + outputs: vec![FieldSchema { + name: "modules", + ty: TypeSchema::Array(Box::new(TypeSchema::Ref("ModuleStatus"))), + comment: "Status of each known module.", + required: true, + }], + }, + "status" => ControllerSchema { + namespace: "modules", + function: "status", + description: "Report the state of one module by id.", + inputs: vec![FieldSchema { + name: "id", + ty: TypeSchema::String, + comment: "Registry identifier, e.g. `tinydocs`.", + required: true, + }], + outputs: vec![FieldSchema { + name: "module", + ty: TypeSchema::Ref("ModuleStatus"), + comment: "Status of the requested module.", + required: true, + }], + }, + "load" => ControllerSchema { + namespace: "modules", + function: "load", + description: "Resolve and load a module now instead of on first use.", + inputs: vec![FieldSchema { + name: "id", + ty: TypeSchema::String, + comment: "Registry identifier, e.g. `tinydocs`.", + required: true, + }], + outputs: vec![FieldSchema { + name: "module", + ty: TypeSchema::Ref("ModuleStatus"), + comment: "Status after the load attempt.", + required: true, + }], + }, + _ => ControllerSchema { + namespace: "modules", + function: "unknown", + description: "Unknown modules controller function.", + inputs: vec![], + outputs: vec![FieldSchema { + name: "error", + ty: TypeSchema::String, + comment: "Lookup error details.", + required: true, + }], + }, + } +} + +fn handle_list(_params: Map) -> ControllerFuture { + Box::pin(async move { + let config = config_rpc::load_config_with_timeout().await?; + Ok(serde_json::json!({ "modules": ops::list(&config) })) + }) +} + +fn handle_status(params: Map) -> ControllerFuture { + Box::pin(async move { + let id = string_param(¶ms, "id").ok_or("`id` is required")?; + let config = config_rpc::load_config_with_timeout().await?; + match ops::list(&config).into_iter().find(|m| m.id == id) { + Some(module) => Ok(serde_json::json!({ "module": module })), + None => Err(format!("unknown module '{id}'")), + } + }) +} + +fn handle_load(params: Map) -> ControllerFuture { + Box::pin(async move { + let id = string_param(¶ms, "id").ok_or("`id` is required")?; + let config = config_rpc::load_config_with_timeout().await?; + // A failed load is reported through the returned status rather than as + // an RPC error: the caller asked "what happened", and the answer is a + // state plus a reason, not a transport failure. + if let Err(reason) = ops::ensure_loaded(&config, &id).await { + log::warn!("[modules] explicit load of '{id}' failed: {reason}"); + } + match ops::list(&config).into_iter().find(|m| m.id == id) { + Some(module) => Ok(serde_json::json!({ "module": module })), + None => Err(format!("unknown module '{id}'")), + } + }) +} + +/// A required string parameter, rejecting a blank one. +fn string_param(params: &Map, key: &str) -> Option { + params + .get(key) + .and_then(Value::as_str) + .map(str::trim) + .filter(|value| !value.is_empty()) + .map(str::to_string) +} + +#[cfg(test)] +mod tests { + use super::{all_controller_schemas, all_registered_controllers, schemas, string_param}; + use serde_json::{Map, Value}; + + #[test] + fn every_schema_is_in_the_modules_namespace() { + for schema in all_controller_schemas() { + assert_eq!(schema.namespace, "modules"); + assert_ne!( + schema.function, "unknown", + "an advertised function fell through to the unknown arm" + ); + } + } + + #[test] + fn registered_controllers_match_the_advertised_schemas() { + // Two lists that must agree: one drives `/schema`, the other dispatch. + let advertised: Vec<&str> = all_controller_schemas() + .iter() + .map(|s| s.function) + .collect(); + let registered: Vec<&str> = all_registered_controllers() + .iter() + .map(|c| c.schema.function) + .collect(); + assert_eq!(advertised, registered); + } + + #[test] + fn an_unknown_function_falls_through_to_the_unknown_arm() { + assert_eq!(schemas("nope").function, "unknown"); + } + + #[test] + fn a_blank_or_missing_id_is_not_a_parameter() { + let mut params = Map::new(); + assert_eq!(string_param(¶ms, "id"), None); + params.insert("id".to_string(), Value::String(" ".to_string())); + assert_eq!(string_param(¶ms, "id"), None); + params.insert("id".to_string(), Value::String(" tinydocs ".to_string())); + assert_eq!(string_param(¶ms, "id"), Some("tinydocs".to_string())); + } +} diff --git a/src/openhuman/modules/types.rs b/src/openhuman/modules/types.rs new file mode 100644 index 0000000000..d0fd2d54cf --- /dev/null +++ b/src/openhuman/modules/types.rs @@ -0,0 +1,123 @@ +//! Types describing a loadable module and where its artifact comes from. + +use serde::{Deserialize, Serialize}; + +/// A first-party module this build knows how to load. +/// +/// Every field is compiled in rather than discovered: which modules exist, which +/// interfaces they claim, and which bytes are legitimate are decisions that +/// belong to the build, not to whatever a release page happens to serve today. +/// See [`crate::openhuman::modules::registry`] for why that matters. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct ModuleRecord { + /// Stable identifier used by config, RPC, and `ensure_loaded`. + pub id: &'static str, + /// Human-readable summary for `modules.list`. + pub description: &'static str, + /// Well-known bus name the module claims once it is serving. + pub bus_name: &'static str, + /// Object path the module serves its interface at. + pub object_path: &'static str, + /// Release version, matching the tag the assets come from. + pub version: &'static str, + /// GitHub release tag URL the artifacts are published under. + pub release_url: &'static str, + /// Per-host artifacts, in the order [`super::platform`] prefers them. + pub assets: &'static [PlatformAsset], + /// When the module is loaded. + pub load: LoadPolicy, +} + +impl ModuleRecord { + /// The asset for `host_key`, if this release publishes one. + #[must_use] + pub fn asset_for(&self, host_key: &str) -> Option<&'static PlatformAsset> { + self.assets.iter().find(|asset| asset.host_key == host_key) + } +} + +/// One published artifact and the digest that makes it legitimate. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct PlatformAsset { + /// Host identifier this artifact targets, e.g. `ubuntu-24.04-x86_64`. + pub host_key: &'static str, + /// Exact release asset name to download. + pub archive: &'static str, + /// Lowercase hex SHA-256 of the archive, taken from the release manifest. + pub sha256: &'static str, +} + +/// When a module is loaded into the process. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum LoadPolicy { + /// On first use. + /// + /// The right default for a codec: a user who never asks for a document + /// should not pay a download, a `dlopen`, or the resident cost of a library + /// that is never unloaded. + Lazy, + /// At boot, before the first request. + /// + /// For a module whose absence would change behaviour rather than just delay + /// it — one that has to be serving before something else decides what it can + /// offer. + Eager, +} + +/// Where a module's artifact should come from. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +#[serde(tag = "kind", rename_all = "snake_case")] +pub enum ModuleSource { + /// The registry-pinned GitHub release for this build. + Pinned, + /// A local artifact, for developing a module against a live core. + LocalFile { + /// Absolute path to the platform library. + path: String, + }, +} + +/// Lifecycle of a module as this host sees it. +/// +/// Deliberately coarser than tinybus's own lifecycle: a caller of `modules.list` +/// wants to know whether it can use the thing, not which admission step it is on. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "snake_case")] +pub enum ModuleState { + /// Known to this build, not loaded yet. + Available, + /// Being downloaded, verified, or initialised. + Loading, + /// Serving its interface. + Ready, + /// Cannot be loaded in this process, and will not be retried. + /// + /// Terminal by tinybus's design: a refused or faulted module keeps its + /// library mapped, so retrying cannot produce a different outcome without a + /// restart. + Failed, + /// No artifact is published for this host. + Unsupported, +} + +/// A module's status as reported over RPC. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +pub struct ModuleStatus { + /// Registry identifier. + pub id: String, + /// Human-readable summary. + pub description: String, + /// Pinned release version. + pub version: String, + /// Well-known bus name. + pub bus_name: String, + /// Current lifecycle state. + pub state: ModuleState, + /// Why the module is in [`ModuleState::Failed`] or + /// [`ModuleState::Unsupported`], if it is. + /// + /// Never carries a path, a URL with credentials, or any payload: a status + /// reply is rendered into a UI and copied into bug reports. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub detail: Option, +} diff --git a/src/openhuman/tools/impl/document/engine.rs b/src/openhuman/tools/impl/document/engine.rs index 7f5df7a932..44d75d3879 100644 --- a/src/openhuman/tools/impl/document/engine.rs +++ b/src/openhuman/tools/impl/document/engine.rs @@ -20,10 +20,10 @@ use std::time::Duration; -use tokio::task::JoinError; -use tokio::time::{error::Elapsed, timeout}; +use tokio::time::timeout; use super::types::{DocumentError, GenerateDocumentInput}; +use crate::openhuman::modules::documents; /// Generate the `.docx` bytes for `input`, giving up after `deadline`. /// @@ -33,6 +33,30 @@ use super::types::{DocumentError, GenerateDocumentInput}; pub(super) async fn generate( input: &GenerateDocumentInput, deadline: Duration, +) -> Result, DocumentError> { + let config = match crate::openhuman::config::Config::load_or_init().await { + Ok(config) => config, + Err(error) => { + return Err(DocumentError::GenerationFailed { + stderr_truncated: DocumentError::truncate_stderr(&format!( + "config unavailable: {error}" + )), + }); + } + }; + generate_with(&config, input, deadline).await +} + +/// [`generate`], against a caller-supplied config. +/// +/// Split out so a test can drive the whole path without `load_or_init`, which +/// reads — and on a fresh machine writes — the real user config directory. A +/// unit test that touches it depends on whatever is on the developer's box and +/// can leave a config file behind. +async fn generate_with( + config: &crate::openhuman::config::Config, + input: &GenerateDocumentInput, + deadline: Duration, ) -> Result, DocumentError> { // Clone across the blocking boundary — cheap relative to the synthesis, // and it keeps the blocking closure `'static`. @@ -50,14 +74,18 @@ pub(super) async fn generate( "[document:engine] generate:start" ); - let join: Result, tinydocs::Error>, _>, Elapsed> = timeout( - deadline, - tokio::task::spawn_blocking(move || tinydocs::docx::generate(&owned)), - ) - .await; + // Loaded before the clock starts. A first use may download and verify the + // artifact, and a deadline meant for generation should not be spent on that + // — otherwise the first document a user ever asks for is the one that times + // out. Cached after the first call, so this is free from then on. + if let Err(error) = documents::ensure_ready(config).await { + return Err(DocumentError::from(error)); + } + + let call = timeout(deadline, documents::generate_docx(config, &owned)).await; let elapsed_ms = started.elapsed().as_millis() as u64; - match join { + match call { Err(_elapsed) => { tracing::warn!( target: "document", @@ -70,29 +98,18 @@ pub(super) async fn generate( timeout_secs: deadline_secs, }) } - Ok(Err(join_err)) => { - let err = map_join_error(join_err); - tracing::warn!( - target: "document", - elapsed_ms, - kind = "join_error", - err = %err, - "[document:engine] generate:failure" - ); - Err(err) - } - Ok(Ok(Err(crate_err))) => { - let err = DocumentError::from(crate_err); + Ok(Err(call_err)) => { + let err = DocumentError::from(call_err); tracing::warn!( target: "document", elapsed_ms, - kind = "engine_failure", + kind = "module_failure", err = %err, "[document:engine] generate:failure" ); Err(err) } - Ok(Ok(Ok(bytes))) => { + Ok(Ok(bytes)) => { tracing::debug!( target: "document", elapsed_ms, @@ -105,25 +122,6 @@ pub(super) async fn generate( } } -fn map_join_error(err: JoinError) -> DocumentError { - // The outer `tokio::time::timeout` already routes the timeout case, so a - // `JoinError` here is a panic (library bug / OOM on the blocking pool) or - // a cancellation (runtime shutdown / explicit abort). Both surface as - // `GenerationFailed` with context preserved — mirrors the presentation - // engine so a "0s timeout" message is never fabricated. - if err.is_panic() { - DocumentError::GenerationFailed { - stderr_truncated: DocumentError::truncate_stderr("document engine panicked"), - } - } else { - DocumentError::GenerationFailed { - stderr_truncated: DocumentError::truncate_stderr(&format!( - "document engine task cancelled: {err}" - )), - } - } -} - #[cfg(test)] mod tests { use super::*; @@ -167,141 +165,93 @@ mod tests { body } - #[tokio::test] - async fn generate_round_trips_to_valid_docx() { - // End-to-end through the wrapper: build → tinydocs → byte buffer → - // re-open as zip → confirm the OOXML skeleton + that our text reached - // document.xml. - let bytes = generate(&sample_input(), Duration::from_secs(30)) - .await - .expect("generate should succeed"); - - // A `.docx` is a zip: the magic bytes are the local-file-header - // signature `PK\x03\x04`. This is the acceptance-criteria check - // that any OOXML reader can open the file. - assert!( - bytes.len() > 200, - "docx unexpectedly small ({} bytes)", - bytes.len() - ); - assert_eq!(&bytes[0..2], b"PK", "docx must start with the zip magic PK"); - - let names = docx_entry_names(&bytes); - for required in ["[Content_Types].xml", "_rels/.rels", "word/document.xml"] { - assert!( - names.iter().any(|n| n == required), - "missing OOXML entry: {required} (got: {names:?})" - ); - } - - // Numbering was used → the numbering part must materialise. - assert!( - names.iter().any(|n| n == "word/numbering.xml"), - "bullet list should emit word/numbering.xml (got: {names:?})" - ); - - // Our title, heading, paragraph, and bullet text all reach the - // rendered document body — none dropped on the floor. - let doc = docx_entry_body(&bytes, "word/document.xml"); - for needle in [ - "Project Charter", - "Overview", - "This document describes the plan.", - "Goals", - "Ship v1", - "Delight users", - ] { - assert!( - doc.contains(needle), - "document.xml missing text: {needle:?}" - ); - } - } - - #[tokio::test] - async fn generate_drops_blank_paragraphs_and_bullets() { - // Whitespace-only entries must not blow up generation and must not - // emit empty runs — the engine trims + drops them. - let input = GenerateDocumentInput { - title: "Trimmed".to_string(), - author: Some(" ".to_string()), - sections: vec![DocumentSection { - heading: Some("Kept".to_string()), - paragraphs: vec!["real".to_string(), " ".to_string(), String::new()], - bullets: vec!["item".to_string(), "\t\n".to_string()], - }], - }; - let bytes = generate(&input, Duration::from_secs(30)) - .await - .expect("generate should succeed on whitespace-only entries"); - let doc = docx_entry_body(&bytes, "word/document.xml"); - assert!(doc.contains("real")); - assert!(doc.contains("item")); + /// A config with modules turned off, so nothing leaves this process. + /// + /// The test drives `generate_with` rather than `generate` deliberately: + /// `generate` calls `Config::load_or_init`, which reads — and on a fresh + /// machine writes — the real user config directory, so its behaviour would + /// depend on whatever is on the box running the test and it could leave a + /// file behind. Supplying the config also makes the outcome deterministic: + /// with modules disabled the only reachable error is `ModuleUnavailable`. + fn isolated_config() -> crate::openhuman::config::Config { + let mut config = crate::openhuman::config::Config::default(); + config.modules.enabled = false; + config.modules.allow_download = false; + config } #[tokio::test] - async fn generate_surfaces_a_crate_validation_failure() { - // `tinydocs` re-validates inside `generate`, so a spec that never went - // through `validate_input` still fails structurally rather than - // producing a blank document. - let input = GenerateDocumentInput { - title: String::new(), - author: None, - sections: vec![], - }; - match generate(&input, Duration::from_secs(30)).await { - Err(DocumentError::InvalidInput { field, .. }) => assert_eq!(field, "title"), - other => panic!("expected InvalidInput(title), got {other:?}"), + async fn generate_reports_a_structured_outcome_when_no_module_is_available() { + // Contract: with no module reachable, `generate` surfaces a clean, + // structured outcome — never a panic, never a half-written buffer, and + // never a hang waiting on a bus nobody is serving. + match generate_with(&isolated_config(), &sample_input(), Duration::from_secs(30)).await { + Err(DocumentError::ModuleUnavailable { .. }) => {} + other => panic!("expected ModuleUnavailable with modules disabled, got {other:?}"), } } #[tokio::test] - async fn generate_yields_clean_structured_result_under_zero_deadline() { - // Contract under an impossibly-short deadline: `generate` must surface a - // clean, structured outcome — never a panic or a half-written buffer. - // - // Which outcome we get is inherently racy and must NOT be pinned: a - // near-zero `timeout` wrapping `spawn_blocking` usually elapses first - // (GenerationTimeout), but the runtime can instead cancel the blocking - // task, which `map_join_error` maps to GenerationFailed, and a trivial - // input can even finish before the timer fires (Ok). Asserting one exact - // variant made this flake under coverage instrumentation. We assert the - // real invariant: any Ok is a non-empty buffer, any Err is one of the - // two documented structured variants, and nothing panics. - match generate(&sample_input(), Duration::ZERO).await { + async fn a_zero_deadline_never_panics_or_yields_a_partial_buffer() { + // Which outcome arrives is inherently racy and must NOT be pinned: the + // near-zero timeout usually elapses first, but the disabled-module check + // runs ahead of the clock. Assert the invariant instead — any Ok is a + // non-empty buffer, any Err is one of the documented variants. + match generate_with(&isolated_config(), &sample_input(), Duration::ZERO).await { Ok(bytes) => assert!(!bytes.is_empty(), "a completed docx must be non-empty"), Err(DocumentError::GenerationTimeout { timeout_secs }) => { assert_eq!(timeout_secs, 0, "zero deadline reports 0 seconds"); } - Err(DocumentError::GenerationFailed { .. }) => { - // Blocking task cancelled before the timer fired — still clean. + Err( + DocumentError::GenerationFailed { .. } | DocumentError::ModuleUnavailable { .. }, + ) => { + // Failed or refused before the timer fired — still clean. } Err(other) => panic!("unexpected error variant under a zero deadline: {other:?}"), } } - #[tokio::test] - async fn map_join_error_cancellation_becomes_generation_failed() { - // A non-panic JoinError (cancellation via abort) surfaces as - // GenerationFailed with the cancellation context preserved — never - // a fabricated "0s timeout". - let handle = tokio::spawn(async { - tokio::time::sleep(Duration::from_secs(3600)).await; - }); - handle.abort(); - let join_err = handle.await.expect_err("aborted task yields JoinError"); - assert!( - !join_err.is_panic(), - "abort() yields a cancellation, not a panic" - ); - match map_join_error(join_err) { - DocumentError::GenerationFailed { stderr_truncated } => { - assert!( - stderr_truncated.contains("document engine task cancelled"), - "cancellation context missing: {stderr_truncated:?}" - ); + // The OOXML round trips that used to live here — container shape, which + // text reaches document.xml, blank filtering — moved with the writer into + // `tinydocs::docx`, which tests them against the bytes it produces. Asserting + // them again through a bus call would test the same behaviour twice and + // drift the moment one copy changed. + + #[test] + fn a_module_failure_maps_onto_the_agent_facing_shape() { + // There is no blocking task to join any more — the module owns its own + // pool — so the failure this file has to classify is a call failure. + // The three outcomes drive three different agent behaviours, which is + // why the client distinguishes them at all. + use crate::openhuman::modules::documents::DocumentCallError; + + assert!(matches!( + DocumentError::from(DocumentCallError::InvalidInput("blank title".to_string())), + DocumentError::InvalidInput { .. } + )); + assert!(matches!( + DocumentError::from(DocumentCallError::Failed("writer stopped".to_string())), + DocumentError::GenerationFailed { .. } + )); + assert!(matches!( + DocumentError::from(DocumentCallError::Unavailable("no artifact".to_string())), + DocumentError::ModuleUnavailable { .. } + )); + } + + #[test] + fn a_module_reported_invalid_input_names_the_spec() { + // The structured field/reason pair does not survive the wire, so the + // conversion has to supply a field rather than leave the agent without + // one. Local validation runs first, so this path is the rare one. + use crate::openhuman::modules::documents::DocumentCallError; + + match DocumentError::from(DocumentCallError::InvalidInput("too long".to_string())) { + DocumentError::InvalidInput { field, reason } => { + assert_eq!(field, "spec"); + assert_eq!(reason, "too long"); } - other => panic!("expected GenerationFailed, got {other:?}"), + other => panic!("expected InvalidInput, got {other:?}"), } } } diff --git a/src/openhuman/tools/impl/document/tests.rs b/src/openhuman/tools/impl/document/tests.rs index 074b0c2556..9f9b81336e 100644 --- a/src/openhuman/tools/impl/document/tests.rs +++ b/src/openhuman/tools/impl/document/tests.rs @@ -160,6 +160,8 @@ async fn execute_rejects_unknown_field() { } #[tokio::test] +#[ignore = "needs a built tinydocs module (OPENHUMAN_MODULE_PATH) and its own process: \ +the module bus belongs to the runtime that creates it, so run this test alone"] async fn execute_happy_path_returns_artifact_metadata() { // End-to-end: drives the real docx-rs engine + artifact pipeline. // Asserts the success contract — the artifact is finalised on disk and diff --git a/src/openhuman/tools/impl/document/types.rs b/src/openhuman/tools/impl/document/types.rs index 8255b7e0aa..fd7eefb9c4 100644 --- a/src/openhuman/tools/impl/document/types.rs +++ b/src/openhuman/tools/impl/document/types.rs @@ -20,7 +20,13 @@ use serde::{Deserialize, Serialize}; -pub use tinydocs::docx::{ +use crate::openhuman::modules::documents::DocumentCallError; + +// Reached through `spec`, not `docx`: this build carries the contract and not +// the writer, so the gated `docx` module is not compiled here at all. The types +// are the same ones the module validates against, which is the whole reason +// `spec` is separable. +pub use tinydocs::spec::document::{ DocumentSpec as GenerateDocumentInput, MAX_BULLETS_PER_SECTION, MAX_PARAGRAPHS_PER_SECTION, MAX_PARAGRAPH_CHARS, MAX_SECTIONS, MAX_TEXT_CHARS, }; @@ -31,7 +37,7 @@ pub use tinydocs::docx::{ // is private, so the re-export reads as unused to the compiler — hence the // explicit allow rather than dropping a name callers legitimately need. #[allow(unused_imports)] -pub use tinydocs::docx::DocumentSection; +pub use tinydocs::spec::DocumentSection; /// Tool output returned via [`crate::openhuman::tools::traits::ToolResult`] /// as the JSON `data` field. @@ -61,6 +67,13 @@ pub enum DocumentError { #[error("document generation exceeded {timeout_secs}s timeout")] GenerationTimeout { timeout_secs: u64 }, + + /// The document module could not be loaded on this host. + /// + /// Terminal for the running process: the loader never retries a module that + /// failed, so the message says what would change the outcome. + #[error("document generation is unavailable: {reason}")] + ModuleUnavailable { reason: String }, } impl DocumentError { @@ -73,22 +86,23 @@ impl DocumentError { } impl From for DocumentError { - /// Map a `tinydocs` failure onto the agent-facing shape. + /// Map a spec-validation failure onto the agent-facing shape. /// - /// The mapping is total and lossless in both directions that matter: the - /// structured `field` / `reason` pair the agent self-corrects on survives - /// intact, and the already-truncated generation detail is carried across - /// without re-truncating. `GenerationTimeout` is deliberately absent here - /// — `tinydocs` is synchronous and has no deadline, so only - /// [`engine`](super::engine) can produce that variant. + /// This is the *local* path: `validate_input` below checks the spec against + /// the same limits the module would, before a call is made, so the agent + /// gets a structured `field` / `reason` pair it can self-correct on without + /// paying for a round trip. Errors that come back from the module take the + /// `DocumentCallError` conversion instead, where that structure has already + /// been flattened by the wire. + /// + /// `GenerationTimeout` is deliberately absent: validation has no deadline, + /// so only [`engine`](super::engine) produces that variant. /// /// `tinydocs::Error` is `#[non_exhaustive]`, so the catch-all arm is - /// required by the compiler rather than chosen. It degrades a variant - /// added by a future `tinydocs` release to `GenerationFailed` carrying - /// that variant's own `Display` text: the agent still sees a real reason - /// instead of a swallowed error, and the fallback is deliberately *not* - /// silent so a crate bump that introduces a case worth handling - /// structurally shows up in the logs. + /// required by the compiler rather than chosen. It degrades a variant added + /// by a future release to `GenerationFailed` carrying that variant's own + /// `Display` text, and logs, so a crate bump that introduces a case worth + /// handling structurally shows up rather than being swallowed. fn from(err: tinydocs::Error) -> Self { match err { tinydocs::Error::InvalidInput { field, reason } => Self::InvalidInput { field, reason }, @@ -110,6 +124,43 @@ impl From for DocumentError { } } +/// Reported when the document module cannot be loaded on this host. +/// +/// A distinct variant rather than a `GenerationFailed`, because the two mean +/// opposite things to whoever reads them: generation failing is a document that +/// might work on a retry, whereas an unavailable module means the capability is +/// not present in this build or on this machine and no amount of rephrasing the +/// request will produce one. +impl From for DocumentError { + /// Map a module-call failure onto the agent-facing shape. + /// + /// The three call outcomes map onto three different agent behaviours, which + /// is the whole reason the client distinguishes them: `InvalidInput` is + /// something a model can fix by rewriting its spec, `Failed` is not, and + /// `Unavailable` means it should stop asking. + /// + /// The structured `field` / `reason` pair does not survive the bus — an + /// error crosses as a name plus a message — so an `InvalidInput` from the + /// module names `spec` and carries the message as its reason. In practice + /// this is rare: [`validate_input`] runs against the same limits before the + /// call is made, so a spec that reaches the module has already passed the + /// checks the module would apply. + fn from(err: DocumentCallError) -> Self { + match err { + DocumentCallError::InvalidInput(reason) => Self::InvalidInput { + field: "spec".to_string(), + reason: Self::truncate_stderr(&reason), + }, + DocumentCallError::Unavailable(reason) => Self::ModuleUnavailable { + reason: Self::truncate_stderr(&reason), + }, + DocumentCallError::Failed(reason) => Self::GenerationFailed { + stderr_truncated: Self::truncate_stderr(&reason), + }, + } + } +} + /// Validate the input early — before the blocking engine hop — so the agent /// gets a structured `InvalidInput` it can self-correct on instead of a /// generic engine error. @@ -150,6 +201,17 @@ pub(super) fn validate_input(input: &GenerateDocumentInput) -> Result<(), Docume error_kind = "generation_failed", "[document:types] input rejected" ), + // Neither of the last two can come out of validation, which has no + // deadline and does not touch the module. Enumerated rather than caught + // by a wildcard so a new variant is a compile error here — this match is + // the one place every failure shape is named. + Err(DocumentError::ModuleUnavailable { .. }) => tracing::debug!( + target: "document", + title_chars, + section_count, + error_kind = "module_unavailable", + "[document:types] input rejected" + ), Err(DocumentError::GenerationTimeout { .. }) => tracing::debug!( target: "document", title_chars, diff --git a/src/openhuman/tools/impl/presentation/engine.rs b/src/openhuman/tools/impl/presentation/engine.rs index a71321c4b2..047ce8ab99 100644 --- a/src/openhuman/tools/impl/presentation/engine.rs +++ b/src/openhuman/tools/impl/presentation/engine.rs @@ -1,104 +1,84 @@ -//! Native Rust `.pptx` generator (replaces the python-pptx subprocess -//! path shipped in #2778). +//! Async wrapper around the `tinydocs` module's `.pptx` writer. //! -//! Backed by the [`ppt-rs`](https://crates.io/crates/ppt-rs) crate -//! (Apache-2.0). Pure CPU, no subprocess, no managed runtime, no -//! first-call venv-setup latency. Output is a byte buffer the caller -//! writes to the artifact's `output_path`. +//! The synthesis itself — the slide mapping, the single-column image layout, the +//! EMU geometry — lives in `tinydocs::pptx` and runs inside the loaded module. +//! What is left here is the policy only a host can supply: //! -//! ## Mapping `SlideSpec` → `ppt-rs` +//! 1. a deadline, because the module holds no opinion about how long a caller +//! is willing to wait, and +//! 2. the mapping from a module-call failure or an elapsed deadline onto the +//! agent-facing [`PresentationError`]. //! -//! `ppt-rs::SlideContent` does not expose a separate "body paragraph" -//! slot today; everything below the title is a bullet. We collapse -//! [`SlideSpec::body`] into a leading bullet so the body text still -//! reaches the rendered slide: +//! There is no `spawn_blocking` hop any more: the module owns its own blocking +//! pool, so the CPU-bound pack never runs on this executor to begin with. //! -//! ```text -//! SlideSpec { title, body: Some(body), bullets: [b1, b2], speaker_notes: Some(n) } -//! → SlideContent::new(title).add_bullet(body).add_bullet(b1).add_bullet(b2).notes(n) -//! ``` +//! # Images cross as one stream //! -//! Empty / whitespace-only entries are filtered out so a trailing -//! blank `body` does not produce an empty bullet marker. +//! A deck's images are concatenated in slide order and sent on a single bus +//! stream, with each image's length declared in the wire spec. Images cannot +//! ride inside the call: a frame is a 16 MiB JSON document and a deck may +//! legally carry 40 MiB of pictures. //! -//! ## Title slide -//! -//! `ppt-rs::create_pptx_with_content(title, slides)` treats `title` -//! as deck metadata only (lands in `docProps/core.xml`) — it does NOT -//! emit a separate title-slide. To preserve the python-pptx -//! contract — title slide first, content slides after, with -//! [`GeneratePresentationOutput::slide_count`] excluding the title -//! slide — we prepend a synthetic title slide built from -//! [`GeneratePresentationInput::title`] (+ optional `author` byline). -//! -//! ## Runtime -//! -//! `ppt-rs::create_pptx_with_content` is synchronous, CPU-bound, and -//! typically completes in <100 ms even for the 64-slide cap. We still -//! drive it through `spawn_blocking` so the async executor is not -//! blocked, and wrap the whole call in a `tokio::time::timeout` so a -//! runaway generation cannot wedge the agent loop. +//! Resolution stays on this side — reading an artifact, checking a path against +//! the security policy — because it is host policy the module must not hold. use std::time::Duration; -use ppt_rs::generator::{create_pptx_with_content, Image, SlideContent}; -use tokio::task::JoinError; -use tokio::time::{error::Elapsed, timeout}; +use tinydocs::spec::{WirePresentationSpec, WireSlideImage, WireSlideSpec}; +use tokio::time::timeout; use super::types::{GeneratePresentationInput, PresentationError, ResolvedSlideImage}; +use crate::openhuman::modules::documents; -/// Slide geometry (matches `ppt-rs`'s default 4:3 deck: 10in × 7.5in). -const SLIDE_WIDTH_EMU: u32 = 9_144_000; -const SLIDE_HEIGHT_EMU: u32 = 6_858_000; -/// 1 inch side margins → usable content column width. -const SIDE_MARGIN_EMU: u32 = 914_400; -/// Images live in the lower band of the slide, beneath the title/body -/// placeholder. Top of that band ≈ slide midpoint; bottom keeps a 0.5in -/// margin. v1 single-column: images stack vertically inside this band. -const IMAGE_BAND_TOP_EMU: u32 = 3_429_000; -const IMAGE_BAND_BOTTOM_MARGIN_EMU: u32 = 457_200; -/// Vertical gap between stacked images. -const IMAGE_STACK_GAP_EMU: u32 = 91_440; -/// EMU per pixel at 96 DPI (matches `ppt-rs`'s own px→EMU convention). -const EMU_PER_PX: u32 = 9_525; - -/// Run the synthesis. Returns the serialised `.pptx` bytes ready to -/// be written to the artifact path. +/// Run the synthesis. Returns the serialised `.pptx` bytes ready to be written +/// to the artifact path. /// -/// The `deadline` covers the entire blocking call (including the -/// `spawn_blocking` thread acquisition). Hitting it surfaces as -/// [`PresentationError::GenerationTimeout`]. +/// The `deadline` covers the whole call, including the image transfer. Hitting +/// it surfaces as [`PresentationError::GenerationTimeout`]. pub(super) async fn generate( input: &GeneratePresentationInput, images: &[Vec], deadline: Duration, ) -> Result, PresentationError> { - // Build the SlideContent vector on the async thread — cheap allocation - // work, no need to send the original `input` across the blocking - // boundary as a borrow. - let slides = build_slides(input, images); - let deck_title = input.title.clone(); + let (deck, payload) = build_request(input, images); let started = std::time::Instant::now(); - let slide_count = slides.len(); + let slide_count = deck.slides.len(); let deadline_secs = deadline.as_secs(); - let title_chars = deck_title.chars().count(); + let image_bytes = payload.len(); tracing::debug!( target: "presentation", deadline_secs, slide_count, - title_chars, + image_bytes, + title_chars = input.title.chars().count(), "[presentation:engine] generate:start" ); - let join: Result, EngineFailure>, _>, Elapsed> = timeout( - deadline, - tokio::task::spawn_blocking(move || generate_blocking(&deck_title, slides)), - ) - .await; + let config = match crate::openhuman::config::Config::load_or_init().await { + Ok(config) => config, + Err(error) => { + return Err(PresentationError::GenerationFailed { + exit_code: -1, + stderr_truncated: PresentationError::truncate_stderr(&format!( + "config unavailable: {error}" + )), + }); + } + }; + + // Loaded before the clock starts. A first use may download and verify the + // artifact, and a deadline meant for generation should not be spent on that + // — otherwise the first document a user ever asks for is the one that times + // out. Cached after the first call, so this is free from then on. + if let Err(error) = documents::ensure_ready(&config).await { + return Err(PresentationError::from(error)); + } + + let call = timeout(deadline, documents::generate_pptx(&config, &deck, &payload)).await; let elapsed_ms = started.elapsed().as_millis() as u64; - match join { + match call { Err(_elapsed) => { tracing::warn!( target: "presentation", @@ -111,29 +91,18 @@ pub(super) async fn generate( timeout_secs: deadline_secs, }) } - Ok(Err(join_err)) => { - let err = map_join_error(join_err); + Ok(Err(call_err)) => { + let err = PresentationError::from(call_err); tracing::warn!( target: "presentation", elapsed_ms, - kind = "join_error", + kind = "module_failure", err = %err, "[presentation:engine] generate:failure" ); Err(err) } - Ok(Ok(Err(engine_err))) => { - let err = map_engine_failure(engine_err); - tracing::warn!( - target: "presentation", - elapsed_ms, - kind = "engine_failure", - err = %err, - "[presentation:engine] generate:failure" - ); - Err(err) - } - Ok(Ok(Ok(bytes))) => { + Ok(Ok(bytes)) => { tracing::debug!( target: "presentation", elapsed_ms, @@ -146,451 +115,188 @@ pub(super) async fn generate( } } -/// Pure transformation from our schema to `ppt-rs`'s. Pulled out of -/// `generate` for unit-testability — the slide ordering + empty -/// filtering rules are load-bearing for the rendered deck shape. -fn build_slides( +/// Turn the tool's input and its resolved images into the wire deck plus the +/// concatenated image payload. +/// +/// The two have to agree: every `byte_len` in the deck is the length of the +/// corresponding slice in `payload`, in the same order, and the module refuses +/// the call if they do not add up. Building both here, in one pass, is what +/// keeps them consistent. +fn build_request( input: &GeneratePresentationInput, images: &[Vec], -) -> Vec { - let mut out = Vec::with_capacity(input.slides.len() + 1); - - // Synthetic title slide — preserves the python-pptx behaviour where - // the first rendered slide carries the deck title (+ optional author - // byline). Without this prepend, the deck would open straight onto - // the first content slide and the `title` argument would only land - // in core.xml metadata. The title slide carries no images. - let mut title_slide = SlideContent::new(&input.title); - if let Some(author) = input.author.as_deref().filter(|a| !a.trim().is_empty()) { - title_slide = title_slide.add_bullet(author); - } - out.push(title_slide); - - for (idx, spec) in input.slides.iter().enumerate() { - let mut slide = SlideContent::new(&spec.title); - if let Some(body) = spec.body.as_deref().filter(|b| !b.trim().is_empty()) { - slide = slide.add_bullet(body); - } - for bullet in &spec.bullets { - if !bullet.trim().is_empty() { - slide = slide.add_bullet(bullet); - } - } - // Attach resolved images AFTER the text, single-column in the - // lower band of the slide. Each image's caption (if any) is - // rendered as a trailing bullet so the label is not lost. - let slide_images = images.get(idx).map(Vec::as_slice).unwrap_or(&[]); - for placed in place_single_column(slide_images) { - slide = slide.add_image(placed.image); - if let Some(caption) = placed.caption { - slide = slide.add_bullet(&caption); - } - } - if let Some(notes) = spec - .speaker_notes - .as_deref() - .filter(|n| !n.trim().is_empty()) - { - slide = slide.notes(notes); +) -> (WirePresentationSpec, Vec) { + let mut payload = Vec::new(); + let mut slides = Vec::with_capacity(input.slides.len()); + + for (index, slide) in input.slides.iter().enumerate() { + let resolved = images.get(index).map(Vec::as_slice).unwrap_or(&[]); + let mut wire_images = Vec::with_capacity(resolved.len()); + for image in resolved { + payload.extend_from_slice(&image.bytes); + wire_images.push(WireSlideImage { + byte_len: image.bytes.len() as u64, + caption: image.caption.clone(), + }); } - out.push(slide); - } - - out -} - -/// A positioned `ppt-rs` image plus its (optional) caption text. -struct PlacedImage { - image: Image, - caption: Option, -} - -/// Lay resolved images out in a single vertical column inside the lower -/// band of the slide. Each image is scaled to fit its slot while -/// preserving aspect ratio and centred horizontally + vertically within -/// the slot. v1 layout — a multi-image grid is deferred. -fn place_single_column(images: &[ResolvedSlideImage]) -> Vec { - let n = images.len() as u32; - if n == 0 { - return Vec::new(); - } - - let content_left = SIDE_MARGIN_EMU; - let content_width = SLIDE_WIDTH_EMU.saturating_sub(2 * SIDE_MARGIN_EMU); - let band_height = SLIDE_HEIGHT_EMU - .saturating_sub(IMAGE_BAND_TOP_EMU) - .saturating_sub(IMAGE_BAND_BOTTOM_MARGIN_EMU); - let total_gap = IMAGE_STACK_GAP_EMU.saturating_mul(n.saturating_sub(1)); - let slot_height = band_height.saturating_sub(total_gap) / n; - - images - .iter() - .enumerate() - .map(|(i, img)| { - let slot_top = IMAGE_BAND_TOP_EMU + (i as u32) * (slot_height + IMAGE_STACK_GAP_EMU); - let (w, h) = fit_within( - img.width_px.saturating_mul(EMU_PER_PX), - img.height_px.saturating_mul(EMU_PER_PX), - content_width, - slot_height, - ); - let x = content_left + content_width.saturating_sub(w) / 2; - let y = slot_top + slot_height.saturating_sub(h) / 2; - let image = Image::from_bytes(img.bytes.clone(), w, h, img.format).position(x, y); - PlacedImage { - image, - caption: img.caption.clone(), - } - }) - .collect() -} - -/// Scale `(w, h)` (EMU) to fit within `(max_w, max_h)` preserving aspect -/// ratio. Never upscales beyond the bounding box; only shrinks when the -/// source overflows. Falls back to the bounding box if the source has a -/// degenerate zero dimension. -fn fit_within(w: u32, h: u32, max_w: u32, max_h: u32) -> (u32, u32) { - if w == 0 || h == 0 { - return (max_w, max_h); + slides.push(WireSlideSpec { + title: slide.title.clone(), + body: slide.body.clone(), + bullets: slide.bullets.clone(), + speaker_notes: slide.speaker_notes.clone(), + images: wire_images, + }); } - // Scale factor in fixed-point-ish f64; min of the two axis ratios. - let ratio_w = max_w as f64 / w as f64; - let ratio_h = max_h as f64 / h as f64; - let scale = ratio_w.min(ratio_h); - let fit_w = ((w as f64) * scale).round().max(1.0) as u32; - let fit_h = ((h as f64) * scale).round().max(1.0) as u32; - (fit_w.min(max_w).max(1), fit_h.min(max_h).max(1)) -} - -/// Blocking inner — runs on the `spawn_blocking` pool. Returns a -/// dedicated `EngineFailure` so the async wrapper can distinguish -/// "library returned an error" from "the blocking task itself panicked -/// or was cancelled". -fn generate_blocking( - deck_title: &str, - slides: Vec, -) -> Result, EngineFailure> { - create_pptx_with_content(deck_title, slides) - .map_err(|err| EngineFailure::Library(format!("{err}"))) -} - -/// Internal failure shape used to keep the blocking-thread surface -/// `Send`-clean (the `ppt-rs` error type is not guaranteed to be -/// `Send + Sync + 'static`). -#[derive(Debug)] -enum EngineFailure { - Library(String), -} -fn map_engine_failure(failure: EngineFailure) -> PresentationError { - match failure { - EngineFailure::Library(msg) => PresentationError::GenerationFailed { - exit_code: -1, - stderr_truncated: PresentationError::truncate_stderr(&msg), + ( + WirePresentationSpec { + title: input.title.clone(), + author: input.author.clone(), + theme: input.theme.clone(), + slides, }, - } -} - -fn map_join_error(err: JoinError) -> PresentationError { - // A bare panic indicates a `ppt-rs` bug or an OOM on the blocking - // pool; surface as `GenerationFailed` so the user sees a structured - // error and the agent can retry with a smaller deck. - // - // Cancellation (non-panic `JoinError`) is a distinct shape: the - // outer `tokio::time::timeout` already routes the timeout case - // before us, so a cancellation that reaches `map_join_error` is - // something else — runtime shutdown, an explicit abort, or the - // runtime cancelling the blocking task for unrelated reasons. - // Reporting it as `GenerationTimeout { timeout_secs: 0 }` produced - // a misleading "exceeded 0s timeout" message and discarded the - // underlying `JoinError` detail that's valuable for triage. We - // surface it as `GenerationFailed` and preserve the cancellation - // context in `stderr_truncated`. - if err.is_panic() { - PresentationError::GenerationFailed { - exit_code: -1, - stderr_truncated: PresentationError::truncate_stderr("presentation engine panicked"), - } - } else { - PresentationError::GenerationFailed { - exit_code: -1, - stderr_truncated: PresentationError::truncate_stderr(&format!( - "presentation engine task cancelled: {err}" - )), - } - } + payload, + ) } #[cfg(test)] mod tests { + //! What is left to test on this side of the bus. + //! + //! The deck shape, the image layout and the OOXML container are tested in + //! `tinydocs::pptx`, where the code now lives — reproducing them here would + //! assert the same behaviour twice and drift the moment one copy changed. + //! + //! What only exists here is [`build_request`]: the deck and the concatenated + //! payload have to agree byte for byte, in order, or the module refuses the + //! call. That agreement is this file's job. + use super::*; use crate::openhuman::tools::implementations::presentation::types::SlideSpec; - fn input_with_one_slide() -> GeneratePresentationInput { - GeneratePresentationInput { - title: "Quarterly review".to_string(), - author: Some("Alice".to_string()), - theme: None, - slides: vec![SlideSpec { - title: "Highlights".to_string(), - body: Some("Revenue up 12% QoQ.".to_string()), - bullets: vec![ - "Closed two key deals".to_string(), - "Hired 3 engineers".to_string(), - ], - speaker_notes: Some("Emphasise headcount efficiency.".to_string()), - images: vec![], - }], + fn slide(title: &str) -> SlideSpec { + SlideSpec { + title: title.to_string(), + body: Some("Body".to_string()), + bullets: vec!["Bullet".to_string()], + speaker_notes: Some("Notes".to_string()), + images: vec![], } } - #[test] - fn build_slides_prepends_title_slide_with_author_byline() { - let slides = build_slides(&input_with_one_slide(), &[]); - // Title slide + 1 content slide. - assert_eq!(slides.len(), 2); - // ppt-rs SlideContent fields are pub but private to this crate - // boundary; downstream `create_pptx_with_content` is the only - // semantically meaningful assertion — covered by the - // `generate_round_trips_to_valid_pptx` test below. Here we only - // assert the *count* invariant (title-slide prepended), since - // the public API of SlideContent does not expose its bullets. - } - - #[test] - fn build_slides_drops_blank_body_and_bullet_entries() { - let mut input = input_with_one_slide(); - input.author = Some(" ".to_string()); - input.slides[0].body = Some("".to_string()); - input.slides[0].bullets = vec!["real".to_string(), " ".to_string(), "".to_string()]; - input.slides[0].speaker_notes = Some("\n\t ".to_string()); - - let slides = build_slides(&input, &[]); - // 2 slides regardless — empty filtering happens INSIDE the slide, - // not at the slide-list level. Behaviour assertion: the call - // does not panic on whitespace-only fields and downstream - // ppt-rs generation succeeds (cross-checked by the round-trip - // test below). - assert_eq!(slides.len(), 2); + fn input(slides: Vec) -> GeneratePresentationInput { + GeneratePresentationInput { + title: "Quarterly".to_string(), + author: Some("Alice".to_string()), + theme: Some("plain".to_string()), + slides, + } } - #[tokio::test] - async fn generate_round_trips_to_valid_pptx() { - // End-to-end: build → ppt-rs → byte buffer → re-open as zip → - // confirm OOXML skeleton entries. This is the load-bearing - // assertion that the engine swap produces a deck that any - // OOXML reader (PowerPoint, Keynote, LibreOffice, Google - // Slides) can open. - let input = input_with_one_slide(); - let bytes = generate(&input, &[], Duration::from_secs(30)) - .await - .expect("generate should succeed on a 1-slide deck"); - - assert!( - bytes.len() > 1000, - "deck unexpectedly small ({} bytes)", - bytes.len() - ); - - let cursor = std::io::Cursor::new(&bytes); - let mut zip = zip::ZipArchive::new(cursor).expect("output is a valid zip archive"); - - let names: Vec = (0..zip.len()) - .map(|i| zip.by_index(i).unwrap().name().to_string()) - .collect(); - - // OOXML spec-required entries — without these PowerPoint will - // refuse to open the file with "PowerPoint found a problem". - for required in [ - "[Content_Types].xml", - "_rels/.rels", - "ppt/presentation.xml", - "ppt/_rels/presentation.xml.rels", - "ppt/theme/theme1.xml", - "ppt/slideMasters/slideMaster1.xml", - "ppt/slideLayouts/slideLayout1.xml", - "docProps/core.xml", - "docProps/app.xml", - ] { - assert!( - names.iter().any(|n| n == required), - "missing OOXML entry: {required} (got: {names:?})" - ); + fn resolved(bytes: &[u8], caption: Option<&str>) -> ResolvedSlideImage { + ResolvedSlideImage { + bytes: bytes.to_vec(), + format: tinydocs::spec::ImageFormat::Png, + width_px: 4, + height_px: 4, + caption: caption.map(str::to_string), } - - // Title slide (slide1) + 1 content slide (slide2) = 2. - assert!(names.iter().any(|n| n == "ppt/slides/slide1.xml")); - assert!(names.iter().any(|n| n == "ppt/slides/slide2.xml")); - // No slide3 — we only had one SlideSpec. - assert!(!names.iter().any(|n| n == "ppt/slides/slide3.xml")); - - // Speaker notes were set on the content slide → notesSlide - // must materialise. Without this, the notes pane in PowerPoint - // / Keynote stays empty even though the agent populated it. - assert!(names - .iter() - .any(|n| n.starts_with("ppt/notesSlides/notesSlide"))); - - // Sanity: the title text shows up somewhere in the generated - // slide XML. We do not assert exact placement (the placeholder - // structure is owned by ppt-rs's slide layout) — only that the - // string was not dropped on the floor. - let mut slide1 = zip.by_name("ppt/slides/slide1.xml").unwrap(); - let mut slide1_body = String::new(); - std::io::Read::read_to_string(&mut slide1, &mut slide1_body).unwrap(); - assert!( - slide1_body.contains("Quarterly review"), - "deck title missing from rendered slide1.xml" - ); } - #[tokio::test] - async fn map_join_error_cancellation_becomes_generation_failed() { - // A non-panic JoinError (cancellation via abort) MUST NOT surface - // as GenerationTimeout { timeout_secs: 0 } — that produces a - // misleading "exceeded 0s timeout" message and loses the - // JoinError detail useful for triage. Cancellation belongs in - // GenerationFailed with the cancellation context preserved. - let handle = tokio::spawn(async { - // Park forever; we abort before this returns. - tokio::time::sleep(Duration::from_secs(3600)).await; - }); - handle.abort(); - let join_err = handle.await.expect_err("aborted task yields JoinError"); - assert!( - !join_err.is_panic(), - "abort() should produce a cancellation JoinError, not a panic" - ); - - match map_join_error(join_err) { - PresentationError::GenerationFailed { - exit_code, - stderr_truncated, - } => { - assert_eq!(exit_code, -1, "cancellation maps to exit_code -1"); - assert!( - stderr_truncated.contains("presentation engine task cancelled"), - "cancellation context missing from stderr_truncated: {stderr_truncated:?}" - ); - } - other => panic!("expected GenerationFailed for cancellation, got {other:?}"), - } + #[test] + fn the_wire_deck_carries_every_text_field() { + let (deck, payload) = build_request(&input(vec![slide("First")]), &[]); + assert_eq!(deck.title, "Quarterly"); + assert_eq!(deck.author.as_deref(), Some("Alice")); + assert_eq!(deck.theme.as_deref(), Some("plain")); + assert_eq!(deck.slides.len(), 1); + assert_eq!(deck.slides[0].title, "First"); + assert_eq!(deck.slides[0].body.as_deref(), Some("Body")); + assert_eq!(deck.slides[0].bullets, vec!["Bullet".to_string()]); + assert_eq!(deck.slides[0].speaker_notes.as_deref(), Some("Notes")); + assert!(payload.is_empty(), "a text-only deck sends no image bytes"); } - #[tokio::test] - async fn generate_surfaces_timeout_under_tiny_deadline() { - // A 1 ns deadline cannot complete any real work — we expect a - // structured Timeout, not a panic or a half-written buffer. - let input = input_with_one_slide(); - let err = generate(&input, &[], Duration::from_nanos(1)) - .await - .expect_err("1 ns deadline should never satisfy the timeout"); - match err { - PresentationError::GenerationTimeout { timeout_secs } => { - assert_eq!( - timeout_secs, 0, - "nanosecond timeout rounds down to 0 seconds" - ); + #[test] + fn declared_lengths_slice_the_payload_back_into_the_original_images() { + // The property the module relies on: walking the deck's byte_lens in + // order must reproduce exactly the images that went in. If this drifts, + // a deck renders with pictures assembled from two different images. + let first = vec![1u8; 10]; + let second = vec![2u8; 25]; + let third = vec![3u8; 7]; + let images = vec![ + vec![resolved(&first, Some("one")), resolved(&second, None)], + vec![resolved(&third, Some("three"))], + ]; + let (deck, payload) = build_request(&input(vec![slide("A"), slide("B")]), &images); + + assert_eq!(payload.len(), first.len() + second.len() + third.len()); + let mut cursor = 0usize; + let mut seen = Vec::new(); + for wire_slide in &deck.slides { + for image in &wire_slide.images { + let len = image.byte_len as usize; + seen.push(payload[cursor..cursor + len].to_vec()); + cursor += len; } - other => panic!("expected GenerationTimeout, got {other:?}"), } + assert_eq!( + cursor, + payload.len(), + "the lengths must consume the payload" + ); + assert_eq!(seen, vec![first, second, third]); } - /// Canonical 1×1 PNG used to exercise the embed path. - fn png_1x1() -> Vec { - use base64::{engine::general_purpose::STANDARD, Engine as _}; - STANDARD - .decode("iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mNk+M9QDwADhgGAWjR9awAAAABJRU5ErkJggg==") - .unwrap() - } - - fn resolved_png(caption: Option<&str>) -> ResolvedSlideImage { - ResolvedSlideImage { - bytes: png_1x1(), - format: "PNG", - width_px: 320, - height_px: 240, - caption: caption.map(ToString::to_string), - } + #[test] + fn captions_survive_onto_the_wire_images() { + let images = vec![vec![ + resolved(&[9u8; 3], Some("A chart")), + resolved(&[8u8; 3], None), + ]]; + let (deck, _) = build_request(&input(vec![slide("A")]), &images); + assert_eq!(deck.slides[0].images[0].caption.as_deref(), Some("A chart")); + assert_eq!(deck.slides[0].images[1].caption, None); } #[test] - fn fit_within_preserves_aspect_and_clamps() { - // 2:1 source into a square box → width-limited, half-height. - let (w, h) = fit_within(2000, 1000, 1000, 1000); - assert_eq!(w, 1000); - assert_eq!(h, 500); - // Never exceeds the bounding box on either axis. - let (w, h) = fit_within(10, 10, 4000, 800); - assert!(w <= 4000 && h <= 800); - // Degenerate zero dimension falls back to the box. - assert_eq!(fit_within(0, 10, 500, 600), (500, 600)); + fn a_slide_with_no_resolved_images_declares_none() { + // `resolve_images` skips an unreadable image with a warning rather than + // failing the deck, so a slide can arrive here with fewer images than it + // asked for — and the deck must declare what is actually being sent. + let images = vec![vec![]]; + let (deck, payload) = build_request(&input(vec![slide("A")]), &images); + assert!(deck.slides[0].images.is_empty()); + assert!(payload.is_empty()); } #[test] - fn place_single_column_stacks_within_slide_bounds() { - let imgs = vec![resolved_png(None), resolved_png(None), resolved_png(None)]; - let placed = place_single_column(&imgs); - assert_eq!(placed.len(), 3); - let mut prev_bottom = 0u32; - for p in &placed { - // Horizontally inside the content column. - assert!(p.image.x >= SIDE_MARGIN_EMU); - assert!(p.image.x + p.image.width <= SLIDE_WIDTH_EMU - SIDE_MARGIN_EMU + 1); - // Vertically inside the lower band and monotonically stacked. - assert!(p.image.y >= IMAGE_BAND_TOP_EMU); - assert!(p.image.y + p.image.height <= SLIDE_HEIGHT_EMU); - assert!( - p.image.y >= prev_bottom, - "images must not overlap vertically" - ); - prev_bottom = p.image.y + p.image.height; - } + fn a_short_images_argument_leaves_later_slides_imageless() { + // Defensive: `images` is indexed by slide, and a caller that passes a + // shorter vector must not panic or shift images onto the wrong slide. + let images = vec![vec![resolved(&[5u8; 4], None)]]; + let (deck, payload) = build_request(&input(vec![slide("A"), slide("B")]), &images); + assert_eq!(deck.slides[0].images.len(), 1); + assert!(deck.slides[1].images.is_empty()); + assert_eq!(payload.len(), 4); } - #[tokio::test] - async fn generate_embeds_image_into_media_and_content_types() { - // Images are supplied via the resolved arg, not the input spec. - let input = input_with_one_slide(); - let images = vec![vec![resolved_png(Some("Figure 1"))]]; - - let bytes = generate(&input, &images, Duration::from_secs(30)) - .await - .expect("generate with one image should succeed"); - - let cursor = std::io::Cursor::new(&bytes); - let mut zip = zip::ZipArchive::new(cursor).expect("valid zip"); - let names: Vec = (0..zip.len()) - .map(|i| zip.by_index(i).unwrap().name().to_string()) - .collect(); - - assert!( - names.iter().any(|n| n == "ppt/media/image1.png"), - "embedded PNG missing from ppt/media (got: {names:?})" - ); - - let ct_body = { - let mut ct = zip.by_name("[Content_Types].xml").unwrap(); - let mut body = String::new(); - std::io::Read::read_to_string(&mut ct, &mut body).unwrap(); - body - }; - assert!( - ct_body.contains("Extension=\"png\""), - "[Content_Types].xml missing png default extension" - ); - - // Caption rendered as a bullet → its text lands in slide2.xml - // (slide1 is the synthetic title slide). - let slide2_body = { - let mut slide2 = zip.by_name("ppt/slides/slide2.xml").unwrap(); - let mut body = String::new(); - std::io::Read::read_to_string(&mut slide2, &mut body).unwrap(); - body - }; - assert!( - slide2_body.contains("Figure 1"), - "caption text missing from rendered slide2.xml" - ); + #[test] + fn a_module_failure_maps_onto_the_agent_facing_shape() { + use crate::openhuman::modules::documents::DocumentCallError; + + assert!(matches!( + PresentationError::from(DocumentCallError::InvalidInput("bad".to_string())), + PresentationError::InvalidInput { .. } + )); + assert!(matches!( + PresentationError::from(DocumentCallError::Failed("writer stopped".to_string())), + PresentationError::GenerationFailed { exit_code: -1, .. } + )); + assert!(matches!( + PresentationError::from(DocumentCallError::Unavailable("no artifact".to_string())), + PresentationError::ModuleUnavailable { .. } + )); } } diff --git a/src/openhuman/tools/impl/presentation/image_util.rs b/src/openhuman/tools/impl/presentation/image_util.rs deleted file mode 100644 index 9000083d3e..0000000000 --- a/src/openhuman/tools/impl/presentation/image_util.rs +++ /dev/null @@ -1,167 +0,0 @@ -//! Self-contained PNG / JPEG sniffing + pixel-dimension extraction for -//! the presentation image pipeline. -//! -//! This deliberately does **not** reuse `agent::multimodal`'s private -//! magic-byte helpers, for three reasons: -//! -//! 1. Those helpers live outside this feature's edit boundary. -//! 2. They accept `webp` / `gif` / `bmp`, none of which `ppt-rs` 0.2.14 -//! can embed safely — there is no `webp` default in the generated -//! `[Content_Types].xml`, and `ImageBuilder::auto` misclassifies -//! `webp` as `PNG`, producing a part PowerPoint refuses to render. -//! v1 therefore restricts embeddable images to PNG + JPEG. -//! 3. They do not expose pixel dimensions, which we need to place -//! images aspect-correctly in the single-column layout. - -/// Return the `ppt-rs` format token (`"PNG"` / `"JPEG"`) for `bytes`, -/// or `None` if the bytes are not one of the two embeddable formats. -pub(super) fn sniff_format(bytes: &[u8]) -> Option<&'static str> { - if bytes.len() >= 8 && bytes.starts_with(&[0x89, 0x50, 0x4E, 0x47, 0x0D, 0x0A, 0x1A, 0x0A]) { - Some("PNG") - } else if bytes.len() >= 3 && bytes.starts_with(&[0xFF, 0xD8, 0xFF]) { - Some("JPEG") - } else { - None - } -} - -/// Native `(width, height)` in pixels for a PNG or JPEG. Returns `None` -/// when the header is truncated / malformed or the format is unsupported. -pub(super) fn pixel_dimensions(bytes: &[u8], format: &str) -> Option<(u32, u32)> { - match format { - "PNG" => png_dimensions(bytes), - "JPEG" => jpeg_dimensions(bytes), - _ => None, - } -} - -/// PNG: 8-byte signature, then an `IHDR` chunk whose width / height are -/// big-endian `u32`s at byte offsets 16 and 20. -fn png_dimensions(bytes: &[u8]) -> Option<(u32, u32)> { - if bytes.len() < 24 || &bytes[12..16] != b"IHDR" { - return None; - } - let w = u32::from_be_bytes([bytes[16], bytes[17], bytes[18], bytes[19]]); - let h = u32::from_be_bytes([bytes[20], bytes[21], bytes[22], bytes[23]]); - if w == 0 || h == 0 { - return None; - } - Some((w, h)) -} - -/// JPEG: walk the marker segments until a Start-Of-Frame (`SOF0`/`SOF2`, -/// and the other non-differential / progressive SOF markers) is hit; its -/// payload carries height then width as big-endian `u16`s. -fn jpeg_dimensions(bytes: &[u8]) -> Option<(u32, u32)> { - let mut i = 2; // skip the leading FF D8 SOI - while i + 3 < bytes.len() { - if bytes[i] != 0xFF { - i += 1; - continue; - } - let marker = bytes[i + 1]; - i += 2; - // Standalone markers (no length field): padding fill bytes and - // RSTn / SOI / EOI. Skip without consuming a segment length. - if marker == 0xFF || marker == 0xD8 || marker == 0xD9 || (0xD0..=0xD7).contains(&marker) { - continue; - } - if i + 1 >= bytes.len() { - return None; - } - let seg_len = u16::from_be_bytes([bytes[i], bytes[i + 1]]) as usize; - if seg_len < 2 { - return None; - } - // SOF markers carrying frame dimensions. Excludes 0xC4 (DHT), - // 0xC8 (JPG), 0xCC (DAC), which share the 0xCn range but are not - // frame headers. - let is_sof = matches!( - marker, - 0xC0 | 0xC1 - | 0xC2 - | 0xC3 - | 0xC5 - | 0xC6 - | 0xC7 - | 0xC9 - | 0xCA - | 0xCB - | 0xCD - | 0xCE - | 0xCF - ); - if is_sof { - // segment: [len_hi len_lo precision h_hi h_lo w_hi w_lo ...] - if i + 6 >= bytes.len() { - return None; - } - let h = u16::from_be_bytes([bytes[i + 3], bytes[i + 4]]) as u32; - let w = u16::from_be_bytes([bytes[i + 5], bytes[i + 6]]) as u32; - if w == 0 || h == 0 { - return None; - } - return Some((w, h)); - } - i += seg_len; - } - None -} - -#[cfg(test)] -mod tests { - use super::*; - - /// Canonical 1×1 PNG (full IHDR + IDAT + IEND). - fn png_1x1() -> Vec { - use base64::{engine::general_purpose::STANDARD, Engine as _}; - STANDARD - .decode("iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mNk+M9QDwADhgGAWjR9awAAAABJRU5ErkJggg==") - .unwrap() - } - - /// Minimal JPEG: SOI + APP0 stub + SOF0 declaring 7×5. - fn jpeg_7x5() -> Vec { - vec![ - 0xFF, 0xD8, // SOI - 0xFF, 0xE0, 0x00, 0x04, 0x00, 0x00, // APP0, len=4, 2 payload bytes - 0xFF, 0xC0, 0x00, 0x0B, // SOF0, len=11 - 0x08, // precision - 0x00, 0x05, // height = 5 - 0x00, 0x07, // width = 7 - 0x03, 0x00, 0x00, 0x00, // components (filler) - 0xFF, 0xD9, // EOI - ] - } - - #[test] - fn sniffs_png_and_jpeg() { - assert_eq!(sniff_format(&png_1x1()), Some("PNG")); - assert_eq!(sniff_format(&jpeg_7x5()), Some("JPEG")); - } - - #[test] - fn rejects_non_image_and_unsupported() { - assert_eq!(sniff_format(b"not an image"), None); - // GIF magic — recognised by multimodal but NOT embeddable here. - assert_eq!(sniff_format(b"GIF89a....."), None); - // WebP magic — same story. - assert_eq!(sniff_format(b"RIFF\0\0\0\0WEBP"), None); - } - - #[test] - fn reads_png_dimensions() { - assert_eq!(pixel_dimensions(&png_1x1(), "PNG"), Some((1, 1))); - } - - #[test] - fn reads_jpeg_dimensions() { - assert_eq!(pixel_dimensions(&jpeg_7x5(), "JPEG"), Some((7, 5))); - } - - #[test] - fn truncated_headers_yield_none() { - assert_eq!(png_dimensions(&[0x89, 0x50, 0x4E, 0x47]), None); - assert_eq!(jpeg_dimensions(&[0xFF, 0xD8]), None); - } -} diff --git a/src/openhuman/tools/impl/presentation/mod.rs b/src/openhuman/tools/impl/presentation/mod.rs index a7827a664e..689ef869f1 100644 --- a/src/openhuman/tools/impl/presentation/mod.rs +++ b/src/openhuman/tools/impl/presentation/mod.rs @@ -30,6 +30,7 @@ use std::path::PathBuf; use std::sync::Arc; use std::time::Duration; +use tinydocs::spec::ImageFormat; use async_trait::async_trait; use serde_json::{json, Value}; @@ -41,7 +42,6 @@ use crate::openhuman::security::SecurityPolicy; use crate::openhuman::tools::traits::{PermissionLevel, Tool, ToolResult}; mod engine; -mod image_util; mod types; #[cfg(test)] @@ -429,11 +429,17 @@ impl PresentationTool { )); } - let format = image_util::sniff_format(&bytes).ok_or_else(|| { + // Identification and measurement live in `tinydocs::spec::image`, which + // is ungated: a host resolving image bytes has to do this to build a + // spec, and it must not need the writer to do it. One implementation + // also means the host and the module cannot disagree about what is + // embeddable. + let format = ImageFormat::sniff(&bytes).ok_or_else(|| { "unsupported image type (only PNG and JPEG are embeddable)".to_string() })?; - let (width_px, height_px) = image_util::pixel_dimensions(&bytes, format) + let (width_px, height_px) = format + .dimensions(&bytes) .ok_or_else(|| format!("could not read {format} dimensions (corrupt header?)"))?; Ok(ResolvedSlideImage { diff --git a/src/openhuman/tools/impl/presentation/tests.rs b/src/openhuman/tools/impl/presentation/tests.rs index bb60795e59..1dbcd49f95 100644 --- a/src/openhuman/tools/impl/presentation/tests.rs +++ b/src/openhuman/tools/impl/presentation/tests.rs @@ -12,6 +12,7 @@ //! swap continues to produce a valid `.pptx` from this tool's //! perspective. +use super::types::GeneratePresentationInput; use super::types::{PresentationError, MAX_BULLETS_PER_SLIDE, MAX_SLIDES, MAX_TEXT_CHARS}; use super::*; @@ -145,6 +146,8 @@ async fn execute_rejects_too_many_slides() { } #[tokio::test] +#[ignore = "needs a built tinydocs module (OPENHUMAN_MODULE_PATH) and its own process: \ +the module bus belongs to the runtime that creates it, so run this test alone"] async fn execute_happy_path_returns_artifact_metadata() { // End-to-end: drives the real ppt-rs engine and the artifact // pipeline. Asserts the tool's success contract — `slide_count` @@ -157,9 +160,12 @@ async fn execute_happy_path_returns_artifact_metadata() { .await .expect("execute returns Ok"); + // `text()` is empty for a Json-only result, so it would report nothing on + // failure. Render the blocks themselves. assert!( !result.is_error, - "happy path should not be flagged as error" + "happy path should not be flagged as error: {:?}", + result.content ); let payload = match result.content.first().expect("at least one content block") { @@ -222,6 +228,8 @@ fn pptx_entry_names(artifact_path: &str) -> Vec { } #[tokio::test] +#[ignore = "needs a built tinydocs module (OPENHUMAN_MODULE_PATH) and its own process: \ +the module bus belongs to the runtime that creates it, so run this test alone"] async fn execute_embeds_file_image_into_deck() { let ws = workspace(); let img_path = ws.path().join("chart.png"); @@ -274,19 +282,16 @@ async fn execute_skips_unsupported_mime_image_with_warning() { "images": [{ "source": { "type": "file", "path": txt_path.to_string_lossy() } }] }] }); - let result = tool.execute(args).await.expect("execute returns Ok"); - // Partial success: deck still produced, but the bad image is reported. - assert!(!result.is_error, "bad image must not fail the whole deck"); - let payload = payload_of(&result); - let warnings = payload["image_warnings"] - .as_array() - .expect("warnings array"); + let input: GeneratePresentationInput = serde_json::from_value(args).expect("args parse"); + let (resolved, warnings) = tool.resolve_images(&input).await; + // Partial success: the deck still renders, the bad image is reported. + assert!( + resolved.iter().all(Vec::is_empty), + "a rejected image must not reach the deck" + ); assert_eq!(warnings.len(), 1, "exactly one image warning expected"); assert!( - warnings[0] - .as_str() - .unwrap() - .contains("unsupported image type"), + warnings[0].contains("unsupported image type"), "warning should name the MIME problem: {:?}", warnings[0] ); @@ -309,15 +314,15 @@ async fn execute_skips_oversize_image_with_warning() { "images": [{ "source": { "type": "file", "path": big_path.to_string_lossy() } }] }] }); - let result = tool.execute(args).await.expect("execute returns Ok"); - assert!(!result.is_error); - let payload = payload_of(&result); - let warnings = payload["image_warnings"] - .as_array() - .expect("warnings array"); + let input: GeneratePresentationInput = serde_json::from_value(args).expect("args parse"); + let (resolved, warnings) = tool.resolve_images(&input).await; + assert!( + resolved.iter().all(Vec::is_empty), + "a rejected image must not reach the deck" + ); assert_eq!(warnings.len(), 1); assert!( - warnings[0].as_str().unwrap().contains("cap"), + warnings[0].contains("cap"), "warning should mention the size cap: {:?}", warnings[0] ); @@ -335,14 +340,14 @@ async fn execute_skips_missing_artifact_with_warning() { "images": [{ "source": { "type": "artifact", "artifact_id": "does-not-exist" } }] }] }); - let result = tool.execute(args).await.expect("execute returns Ok"); - assert!(!result.is_error); - let payload = payload_of(&result); - let warnings = payload["image_warnings"] - .as_array() - .expect("warnings array"); + let input: GeneratePresentationInput = serde_json::from_value(args).expect("args parse"); + let (resolved, warnings) = tool.resolve_images(&input).await; + assert!( + resolved.iter().all(Vec::is_empty), + "a rejected image must not reach the deck" + ); assert_eq!(warnings.len(), 1); - assert!(warnings[0].as_str().unwrap().contains("unreadable")); + assert!(warnings[0].contains("unreadable")); } #[tokio::test] diff --git a/src/openhuman/tools/impl/presentation/types.rs b/src/openhuman/tools/impl/presentation/types.rs index ce8440e1c9..051cbb0c1d 100644 --- a/src/openhuman/tools/impl/presentation/types.rs +++ b/src/openhuman/tools/impl/presentation/types.rs @@ -1,6 +1,9 @@ //! Typed input / output / error contracts for the `generate_presentation` tool. use serde::{Deserialize, Serialize}; +use tinydocs::spec::ImageFormat; + +use crate::openhuman::modules::documents::DocumentCallError; /// Maximum number of slides a single `generate_presentation` call may /// produce. Hard cap to bound generation time and output size; the @@ -95,7 +98,7 @@ pub struct SlideSpec { #[derive(Debug, Clone)] pub(super) struct ResolvedSlideImage { pub bytes: Vec, - pub format: &'static str, + pub format: ImageFormat, pub width_px: u32, pub height_px: u32, pub caption: Option, @@ -160,6 +163,14 @@ pub enum PresentationError { #[error("presentation generation exceeded {timeout_secs}s timeout")] GenerationTimeout { timeout_secs: u64 }, + /// The document module could not be loaded on this host. + /// + /// Distinct from `GenerationFailed`: that is a deck that might work on a + /// retry, this is a capability that is not present and will not become + /// present without a restart. + #[error("presentation generation is unavailable: {reason}")] + ModuleUnavailable { reason: String }, + /// Reserved for the planned `format` selector that will let callers /// request alternative deck formats (`.pdf` / `.key` / image /// strips). Today the tool only emits `.pptx`, so this variant is @@ -174,6 +185,40 @@ pub enum PresentationError { }, } +impl From for PresentationError { + /// Map a module-call failure onto the agent-facing shape. + /// + /// The three call outcomes mean three different things to an agent: + /// `InvalidInput` is a spec it can rewrite, `Failed` is a deck that might + /// work on a retry, and `Unavailable` means the capability is not present + /// and it should stop asking. + /// + /// The structured `field` / `reason` pair does not survive the bus — an + /// error crosses as a name plus a message — so an `InvalidInput` from the + /// module names `spec`. In practice this is rare: `validate_input` checks + /// the same limits before the call, so a deck that reaches the module has + /// already passed them. + /// + /// `exit_code` is `-1` because it always is: the field is a vestige of the + /// python-pptx subprocess this path replaced in #2778, kept so the agent's + /// error shape did not churn. + fn from(err: DocumentCallError) -> Self { + match err { + DocumentCallError::InvalidInput(reason) => Self::InvalidInput { + field: "spec".to_string(), + reason: Self::truncate_stderr(&reason), + }, + DocumentCallError::Unavailable(reason) => Self::ModuleUnavailable { + reason: Self::truncate_stderr(&reason), + }, + DocumentCallError::Failed(reason) => Self::GenerationFailed { + exit_code: -1, + stderr_truncated: Self::truncate_stderr(&reason), + }, + } + } +} + impl PresentationError { /// Truncate a stderr string to the per-#2780 cap of 500 chars /// (UTF-8-safe). Used when wrapping a non-zero exit into diff --git a/src/openhuman/tools/ops_tests.rs b/src/openhuman/tools/ops_tests.rs index f373edb41c..6c43d870f6 100644 --- a/src/openhuman/tools/ops_tests.rs +++ b/src/openhuman/tools/ops_tests.rs @@ -2766,7 +2766,10 @@ const REPRESENTATIVE: &[(&str, crate::core::all::DomainGroup)] = { /// Families with no agent tools of their own. const TOOL_LESS: &[crate::core::all::DomainGroup] = { use crate::core::all::DomainGroup as G; - &[G::Config, G::Security, G::Meet, G::Medulla] + // `Modules` is the loader, not a capability: a loaded module's own surface + // is reached through whichever domain calls it (documents go through the + // document tools), so the family itself owns no agent tool. + &[G::Config, G::Security, G::Meet, G::Medulla, G::Modules] }; // ---- tool_capability() drift guard (M5.3) ---------------------------------- diff --git a/src/openhuman/web3/wallet/chains/btc.rs b/src/openhuman/web3/wallet/chains/btc.rs index b44524ccb4..71024c6f05 100644 --- a/src/openhuman/web3/wallet/chains/btc.rs +++ b/src/openhuman/web3/wallet/chains/btc.rs @@ -464,9 +464,7 @@ mod tests { #[test] fn validate_btc_address_rejects_testnet() { - let err = - validate_btc_address("tb1qw508d6qejxtdg4y5r3zarvary0c5xw7kxpjzsx") - .unwrap_err(); + let err = validate_btc_address("tb1qw508d6qejxtdg4y5r3zarvary0c5xw7kxpjzsx").unwrap_err(); // `tinywallet` reports a wrong-network address as a distinct condition // from a malformed one, so the message names the required network. assert!(err.contains("not on mainnet"), "got: {err}"); diff --git a/vendor/tinybus b/vendor/tinybus index dfcdd2c391..6ca0b0b673 160000 --- a/vendor/tinybus +++ b/vendor/tinybus @@ -1 +1 @@ -Subproject commit dfcdd2c391938a6f7885246634743d58ddb987e7 +Subproject commit 6ca0b0b6739a49396e36be21d450f07cf85b9de2 diff --git a/vendor/tinydocs b/vendor/tinydocs index 3b300d30c4..7c907265fb 160000 --- a/vendor/tinydocs +++ b/vendor/tinydocs @@ -1 +1 @@ -Subproject commit 3b300d30c4e474e87abf14c37c282917a3dcf49e +Subproject commit 7c907265fbc99c45397676202047ab2ac84e8643