PoC Proposal: Applying BitState/HSDR to Bonsai 2 Ternary GEMV — Direct 8-Phase Reduction and Reusable Activation State Sets

[nohara-makoto]

While experimenting with Bonsai 2 ternary inference, I have been exploring a way to keep activations and products in small logarithmic states and reduce a large number of terms into eight phase aggregates before numerical materialization.

I have been calling this experimental idea BitState/HSDR.

Later, I came across Ye Qiao's CurveFP v2 and realized that many parts of my approach overlap substantially with existing work. In particular, Sections 3.2 and 5 describe not only product generation through integer-index addition and sign XOR, but also signed histogram/POPCOUNT reduction and direct shifting into wide per-phase accumulators.

Because of this overlap, I do not intend to claim novelty for these mathematical ideas themselves.

My professional background is mainly in IT and systems infrastructure. I am not a researcher specializing in numerical formats or CPU/GPU kernel design; I have been exploring this topic as a personal side project and hobby.

My current interest is therefore much more implementation-oriented:

• how these ideas can be applied to an existing ternary model such as Bonsai 2,
• which representation and reduction method fit naturally into the llama.cpp runtime,
• how far reduction should remain in the state domain before returning to conventional integer arithmetic,
• and where direct reduction, reusable state sets, POPCOUNT, and INT32 fixed-point arithmetic should be combined.

I also do not assume that the complete computation must remain in LNS or in the state domain until the end.

A practical implementation may first reduce many lanes in the state domain, then materialize the reduced result into an INT32 fixed-point representation such as Q15.16, and continue the upper part of the reduction using conventional integer arithmetic.

For the first PoC, I would like to focus on a single batch-1 CPU ternary GEMV path. Performance improvement and real-model quality impact have not yet been measured.

The current materials are published on Hugging Face.

The English Bonsai 2 demo Excel workbook may be the easiest starting point. It shows the reduction from 128 elements into eight phase aggregates and the position at which the group scale is applied.

The Excel workbooks and test vectors are not intended to represent a finished research result. Their main purpose is to provide implementation reference material so that someone writing a scalar reference, SIMD kernel, or GPU kernel can compare intermediate states and expected values.

The baseline activation format currently under consideration is 1 sign bit plus a 7-bit magnitude code:

|x| = 2^((k-74)/8), k=1..127
k=0 is exact zero
every 8 codes correspond exactly to a factor of 2

This uses the same binary-log grid spacing as CurveFP, but it does not assume the same storage format, bias, or scale contract.

1. Start with a direct 8-phase path

Let the ternary weight be

w_i = d_g * t_i
t_i ∈ {-1, 0, +1}

where d_g is the existing FP16 scale shared by a 128-element group.

If d_g remains outside the inner group reduction, the inner operation only needs ternary zero selection and sign inversion. No LNS magnitude addition with the weight is required.

For each non-zero activation code, the following integer accumulation can be used:

p = k & 7
q = k >> 3

A[p] += activation_sign * t_i * (1 << q)

group_sum = d_g/512 * sum_p A[p] * 2^((p-2)/8)

The result of the inner reduction is only eight integer aggregates.

POPCOUNT is not mandatory for this path. A phase mask can be used as a routing selector while preserving the E4 weighting as ±(1 << q) during masked or grouped reduction.

For real GEMV dimensions such as K=4096 or K=5120, the important cost is therefore not the final eight-value reconstruction, but the cost of routing and reducing the large number of terms inside each scale group into those eight aggregates.

Because Bonsai 2 uses different scales for different groups, each group result must incorporate its own scale before reduction across groups. I do not assume one shared scale across the entire K dimension.

For a more general LNS product path, or for a path in which scale is included in the product coordinate, the required offset must be applied before routing, and an internal coordinate wider than the stored 7-bit code may be necessary.

I would like to evaluate this separately from the factorized ternary path.

The direct 8-phase result also does not necessarily need to return immediately to floating point.

For example, after state-domain reduction within a group, the result could be materialized into an INT32 fixed-point representation such as Q15.16, followed by group-level or higher-level reduction using ordinary integer SIMD.

In this interpretation, the purpose of the state domain is not to replace the entire numerical system.

Instead, it is to delay numerical materialization and move as much dense lane-level reduction as practical into a compact state-domain representation before returning to conventional arithmetic.

2. Compare against reusable activation state sets

Another candidate is to build lane masks for activation magnitude states once for a 128-element activation tile and reuse them across multiple output rows that share the same activation input.

Each row's ternary weights can be represented using a non-zero mask and a negative mask.

M    = activation_state_mask & weight_nonzero_mask
Pneg = activation_negative_mask XOR weight_negative_mask
C    = popcount(M) - 2*popcount(M & Pneg)

From this signed population,

A[p] = sum_q C[8*q+p] * 2^q

produces the same eight aggregates as the direct path.

The key questions are whether the state-mask construction cost can be amortized across enough output rows, whether real activations contain a sufficiently small number of active states, and how weight decoding, cache footprint, and register pressure affect the result.

For histogram final reduction, I would also like to compare the conventional dyadic shift/add path against a fractional-LNS candidate.

For example,

lambda = k + 8*log2(abs(C))

can be generated using a small count LUT, followed by a balanced reduction tree using fractional-LNS κ± correction LUTs.

The final tree across eight phase values is only three stages, but this does not mean that phase-internal reduction, group reduction, or state construction also become three stages.

Cancellation error and internal precision therefore need to be evaluated independently.

The public Bonsai2_LNS8_128 Excel sheet also contains an example of moving the shared scale into the final fractional coordinate:

delta = 8*log2(d_g)

which is added once to the reduced 128-term result.

The Excel LOG and POWER functions are only explanatory references. An actual kernel would use LUTs, preconverted parameters, or another low-cost representation.

I do not infer runtime performance or real-model quality from algebraic equality checks on synthetic examples.

I also do not consider the pure fractional-LNS path to be a required final design.

If reducing in the state domain and then returning to INT32 fixed-point arithmetic is simpler and faster on real CPUs or GPUs, I would prefer that hybrid path.

3. Implementation and measurement scope

For correctness, I would first start with PQ2_0 or predecoded ternary weights and compare:

• AVX2 direct masked/grouped ADD,
• Tile64 bitslice + POPCOUNT,
• reusable activation state sets,
• and INT32 fixed-point materialization after state-domain reduction.

After correctness is established, I would integrate PTQ1_0 dense decoding and compare against the current kernel including its real memory footprint and decode cost.

I would like to measure the following components separately:

• load/decode,
• activation encoding,
• routing/state-set construction,
• row/group reduction,
• scale processing,
• INT32 materialization,
• critical-path depth,
• and effective weight bandwidth.

The final comparison should include decode latency and TG128, including conversion overhead and any model-quality difference.

The current kernels already use SIMD and multiple accumulators, so I do not want to judge performance by comparing against a naive serial reduction.

If the CPU path is promising, I would then like to try Vulkan Compute or CUDA implementations using direct subgroup reduction and ballot-based state-set construction.

A possible GPU structure is:

packed input
→ product/state generation
→ subgroup-local reduction into A[0..7]
→ partial aggregates
→ workgroup merge
→ numerical materialization

The goal is to reduce the computational and reduction overhead of decode until it approaches the time required to fetch the weights, and to identify regions where the kernel can become closer to memory-bandwidth-bound.

BitState/HSDR does not necessarily need to replace the entire GEMV computation.

If even part of the overall reduction can be moved into the state domain, and the remaining computation can then continue efficiently using ordinary INT32 arithmetic, that would already be a useful result.

If the approach is slower, I would still like to identify which component dominates:

• routing,
• transpose,
• mask construction,
• POPCOUNT,
• LUT access,
• register usage,
• decode,
• fixed-point materialization,
• or memory bandwidth.

That information would still be useful for deciding which directions are worth pursuing.

4. Why I am publishing this now

There is also a practical reason why I am publishing these materials at this stage.

Starting in October 2026, my regular work is expected to become considerably busier, so the amount of time I can spend on this PoC may temporarily become quite limited.

Rather than keeping the work private until I can finish a complete implementation, I would prefer to publish the specifications, Excel workbooks, test vectors, and experiment ideas that are already usable.

I may not be able to complete the implementation quickly, and I may sometimes be slow to respond to questions or suggestions.

Even so, feedback such as:

• “this part looks worth testing,”
• “this would probably be slower than the existing kernel,”
• “this layout could be implemented more simply,”
• “there is already a similar implementation,”
• or “this reference vector would be useful for implementation testing”

would be extremely valuable to me.

My goal is not to claim ownership of the BitState name or the underlying ideas.

Where the work overlaps with CurveFP or other existing research, I would prefer to treat that prior work as the foundation and focus on producing implementation notes, reproducible examples, and test material that may be useful when applying the ideas to Bonsai 2 and llama.cpp.

5. Language note

English is not my primary working language.

Most of my technical thinking, notes, and spreadsheet development are done in Japanese.

I provide English versions so that the material can be discussed more widely, but some wording may be awkward or may accidentally sound stronger than I intended.

If an English sentence appears to make an excessive novelty claim, or if the technical meaning is unclear, that is not intentional.

Please feel free to point out both wording problems and technical mistakes.

At this stage, my goal is simply:

to leave useful implementation information, numerical examples, test vectors, and experiment ideas for people interested in running existing low-bit and ternary models such as Bonsai 2 efficiently on ordinary CPUs and GPUs.

If even a small part of these materials becomes useful for a future llama.cpp or related implementation, I would consider that a worthwhile outcome.

Public reference materials

I suggest starting with the following materials:

Material	Suggested focus
English Excel: 128-element Bonsai 2 ternary and scale application	Direct_8Phase and Bonsai2_LNS8_128. Reduction into eight aggregates and scale-first/scale-late handling of the FP16 group scale.
Japanese Excel: same Bonsai 2 demo	推奨_8Phase直集計 and Bonsai2_LNS8_128. Japanese explanation of the same processing path.
BitState/HSDR v0.5 English specification	Direct 8-phase reduction, signed population, pure fractional-LNS, and WHT/RMSNorm assumptions.
BitState/HSDR v0.5 Japanese specification	Japanese explanation, including the product-completion rule and scale factorization.
LNS8 base74 v0.5 numerical specification	Zero handling, bias, 8 codes/octave, phase/octave decomposition, and wide intermediate coordinates.
PoC Experiment Matrix v0.5	Kernel variants, K/reuse sweeps, internal precision, correctness checks, and performance metrics.
Test Vectors v3 JSON	Six test cases covering direct decoding, direct A[8], and integer equality against histogram reduction.
Performance expectations and implementation targets v0.5	Routing, state-set reuse, ballot-based GPU paths, and bandwidth-oriented measurement targets for ordinary CPUs/GPUs.

A shorter overview is available in the dataset README / Technical Brief v0.5.

Additional materials in the same public folder include:

• Basic English LNS8/RMSNorm demo Excel: baseline 8-phase/histogram/pure-LNS workbook without the Bonsai 2-specific sheets.
• Basic Japanese LNS8/RMSNorm demo Excel: Japanese version of the baseline workbook.
• PoC Review Notes v0.5: notes distinguishing the direct path, dependency reduction, and pure-LNS path.
• LNS8/LNS16 Five-Profile Design Guide v0.2: future comparisons of precision, range, phase count, and shared exponent formats.
• Five-Profile Numerical Guide Excel v0.2: codebooks, ranges, mixed-format products, LNS16 correction LUTs, and guard-bit examples.
• GitHub Proposal Author Notes v0.5: notes on the intended PoC scope and questions for implementers.

The five-profile and LNS16 ideas are future exploration material.

For now, I would like to keep the discussion focused on the base74 / S1E4L3 Bonsai 2 PoC.

Questions for implementers

1. Assuming post-Hadamard activations are reused across multiple output rows, where would direct routing or reusable activation-state construction fit most naturally into the current CPU scheduling/layout?

2. Does it make sense to begin correctness testing with PQ2_0 or predecoded ternary weights and integrate PTQ1_0 decoding only after the reduction paths are validated?

3. For AVX2 8-way weighted reduction, or for GPU subgroup-local A[8] plus workgroup merge, are there layouts or measurement methods that would be particularly useful to try first?

4. For a hybrid path that returns from state-domain reduction to an INT32 fixed-point representation such as Q15.16, are there obvious CPU/GPU disadvantages or more appropriate fixed-point placements that I may be overlooking?

5. When activation state sets are reused across multiple output rows, what would be a useful llama.cpp-like way to measure the break-even point between state-mask construction cost, register pressure, cache footprint, and row reuse?

References and verification scope

• Ye Qiao, CurveFP v2: arXiv, HTML. I reviewed Sections 3.1–3.3, Section 5, and Appendices A.1/C, including both direct per-phase accumulation and histogram-based paths.
• PrismML llama.cpp fork and Bonsai-demo / MODEL-FORMATS: references for the target runtime, numerical basis, and model packing. Pinning the current kernel commit and performing actual performance measurements remain part of the PoC work.
• The BitState reference files are linked above. The current proposal is intended to track the published versions of those materials.
• For a fixed snapshot, see Bitstate revision 16c936b6. Links to main are intended to follow future updates.