Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,9 @@

### Fixed

- CUDA NVFP4 long-form splitting preserves paired delivery tags, bracketed cues and special tokens,
including topic-shift handling, and recognizes CJK sentence/clause punctuation. An indivisible
protected span may exceed the character target; existing prompt and frame limits still apply.
- Metal reference-prefill reuse now invalidates before incompatible/default prefill, compares the
complete invariant prefix plus tower lookahead, preserves original chunk boundaries, and reuses
one GPU-owned snapshot buffer pair per lane instead of allocating on every voice switch.
Expand All @@ -19,6 +22,13 @@

### Added

- Per-request HTTP speech `temperature`, `top_p`, `top_k` and exact unsigned 64-bit
`seed` on all pinned profiles, with strict validation and propagation through
streaming, references and redraws. Omitted/null values retain existing defaults.
- A checksummed CUDA NVFP4 engine patch series without changing its pinned upstream or weights.
Model-free build tests exercise real patch application, selection on Linux/macOS, corruption
refusal, repeat builds and preservation of modified checkouts. Manifests must cover each
selected patch exactly once; extra, missing and duplicate entries fail before checkout.
- Native Apple-silicon support for the pinned NVFP4-pruned checkpoint: packed experts, compact
original-ID routing and reference-conditioned semantic CFG, with independent backbone states.
The Mac engine uses `engine/nvfp4/macos/UPSTREAM` and its checksum-verified patches; the existing
Expand Down
39 changes: 32 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@

[TensorFold](https://github.com/ashhart/TensorFold) runs Drama 3 natively since v1.1.0. This repository pins
upstream TensorFold and adds a small series of opt-in patches for the full model. NVFP4 retains its
measured CUDA engine pin and uses a separate upstream pin plus Metal patches on Mac. It builds the engine
measured CUDA engine pin with a checksummed patch series and uses a separate upstream pin plus Metal patches on Mac. It builds the engine
and starts a local speech server with an OpenAI-style `/v1/audio/speech` route, whole or streamed.

**Built with Fish Audio.** The model weights are not in this repository; the launcher downloads them from
Expand Down Expand Up @@ -64,7 +64,8 @@ For the full model, write `--weights full` in both commands.
The first start downloads the weights at the revision pinned in [`recipe/sources.json`](recipe/sources.json),
into `drama-3-NVFP4-Pruned/` or `drama-3/`. Every start checks them against [`recipe/checksums`](recipe/checksums)
and refuses missing or changed files; for the 21 GiB NVFP4 weights that takes about 11 seconds on a GB10.
On Linux, `--weights nvfp4` builds [`engine/nvfp4/UPSTREAM`](engine/nvfp4/UPSTREAM) and serves with
On Linux, `--weights nvfp4` builds [`engine/nvfp4/UPSTREAM`](engine/nvfp4/UPSTREAM) plus
[`engine/nvfp4/patches`](engine/nvfp4/patches) and serves with
guidance γ 1.5 and batched guidance, as measured; a `--cfg-gamma` of your own after `--` replaces both.
On Mac it builds [`engine/nvfp4/macos/UPSTREAM`](engine/nvfp4/macos/UPSTREAM) plus its patch series.
Metal's `--cfg-batched` pairs independent states in one command buffer, not a vectorized GEMM.
Expand Down Expand Up @@ -122,7 +123,9 @@ the license's Section III(c), which needs a separate license from Fish Audio.

`POST /v1/audio/speech` takes `model` (`fishaudio/drama-3`), `input` (at most 4096 characters), `voice`
(`default`, or a name loaded with `--voices`), `response_format` (`wav` or `pcm`) and `stream` (`true` needs
`pcm`). Request-supplied references and other synthesis modifiers are refused. `GET /v1/voices` lists the voices.
`pcm`). Optional `temperature`, `top_p`, `top_k` and exact integer `seed` select
request-local sampling; see [ranges, defaults and redraw behavior](recipe/README-sampling.md).
Other synthesis modifiers are refused. `GET /v1/voices` lists the voices.
The Linux CUDA NVFP4 engine accepts its encoder's `drama3-voice` catalogs and request `references`
as codec codes (not audio). The Mac NVFP4 engine instead uses attested `drama3-http-voice/1`
catalogs and refuses request-supplied references; see the [Mac voice commands](recipe/README-macos.md#attested-reference-voices).
Expand All @@ -139,11 +142,31 @@ Guidance-enabled replies carry the configured γ in `X-Cfg-Gamma`; builtin voice

## The patches

The patches build the full-model engine (`--weights full`). Upstream TensorFold already serves Drama 3. Each patch
### CUDA NVFP4 long speech

The CUDA NVFP4 patch series preserves delivery markup when `--long-form` splits a request.
Paired tags such as `<whisper>…</whisper>`, bracketed cues and special tokens stay whole;
Chinese/Japanese sentence and clause punctuation are recognized. A topic-shift mark inside a
protected span does not create a cut. Plain English without these spans retains its existing cuts.

`--long-form-chars` remains a code-point target, with one exception: an indivisible protected span
may exceed it. The engine's prompt, segment-count and generated-frame limits still apply.
The full-model and Metal NVFP4 engines do not expose this legacy long-form route.

The behavioral reference is [Sogni-AI/drama3-mlx's chunking](https://github.com/Sogni-AI/drama3-mlx/blob/ecc235039ccd5f982f66a6982184f57432354ab4/drama3_mlx/chunking.py).
The native patch extends TensorFold's own implementation; it does not copy that Fish-licensed source.
See the [CUDA patch index](engine/nvfp4/patches/README.md) for scope and tests.

### Full-model engine

These patches build the full-model engine (`--weights full`). Upstream TensorFold already serves Drama 3. Each patch
below adds one feature that stays off until you pass its flag, so a server started without them answers as
upstream does. The exceptions are the first two: batched prefill is on by default (`TF_DRAMA_PREFILL_ROWS=1`
restores upstream's single-row prefill) and the replay fix is a bug fix. The patches stack and apply in order;
each one's notes are in the engine tree's `CHANGELOG.md` or `docs/`.
upstream does. The exceptions are the first two and one request-level feature: batched prefill is on by default
(`TF_DRAMA_PREFILL_ROWS=1` restores upstream's single-row prefill), the replay fix is a bug fix, and per-request
`temperature`, `top_p`, `top_k` and `seed` (0013/0014) need no launch flag. A request without those fields is
answered as upstream answers it; upstream refuses them with a 400, and these patches serve them. The CUDA NVFP4
series (0002) and the Metal NVFP4 series (0004, 0005) carry the same request-level feature. The patches stack and
apply in order; each one's notes are in the engine tree's `CHANGELOG.md` or `docs/`.

| Patch | Adds | Opt in with | Upstream |
|---|---|---|---|
Expand All @@ -159,6 +182,8 @@ each one's notes are in the engine tree's `CHANGELOG.md` or `docs/`.
| 0010 (K6) | Whole replies that stopped short of their text are drawn again | `--short-reply-ratio`, `--short-reply-chars-per-s` | [#1161](https://github.com/ashhart/TensorFold/pull/1161) |
| 0011 (K8) | A whole reply is drawn once more when a phrase opens far above the others | `--register-redraw-st`, `--register-redraw-high`, `--register-high-hz` | [#1162](https://github.com/ashhart/TensorFold/pull/1162) |
| 0012 | One bounded reference-prefix snapshot per Metal lane, with exact identity and safe invalidation | none (automatic) | [#1237](https://github.com/ashhart/TensorFold/pull/1237) |
| 0013 | Request-local speech sampling and exact integer seeds | HTTP `temperature`, `top_p`, `top_k`, `seed` | [#1277](https://github.com/ashhart/TensorFold/pull/1277) |
| 0014 | Parsed-JSON floating boundary documentation and regressions | none (tests/docs) | [#1277](https://github.com/ashhart/TensorFold/pull/1277) |

`drama3-voice`, a tool that writes attested voice files for `--voices`, is proposed upstream in
[#1153](https://github.com/ashhart/TensorFold/pull/1153); it is not part of this series.
Expand Down
4 changes: 2 additions & 2 deletions engine/nvfp4/UPSTREAM
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
# The engine `scripts/build-engine.sh --weights nvfp4` builds, unpatched: the only engine that loads the NVFP4
# pruned weights (plotarmordev/drama-3-NVFP4-Pruned). Commit b1f4b9b4 on branch drama-nvfp4 of the TensorFold fork;
# The CUDA engine base for `scripts/build-engine.sh --weights nvfp4`, followed by patches/.
# Commit b1f4b9b4 on branch drama-nvfp4 of the TensorFold fork loads the NVFP4 pruned weights;
# results/full-vs-quant-01 measured the same source (v1.0.11-193-g70dec469; the two differ in comments and test
# wording only).
TENSORFOLD_REPOSITORY=https://github.com/plotarmordev/TensorFold.git
Expand Down
Loading