New third-party implementation: vortex-java

[dfa1]

Hi Vortex team and community 👋

I've been building a pure-Java implementation of the Vortex format: dfa1/vortex-java.

What it is:

• 100% Java, no JNI, no sun.misc.Unsafe — uses the FFM API (MemorySegment/Arena) for zero-copy mmap
  reads
• Reads and writes all 35 encodings in the v0.72.0 S3 fixture set, verified against the Rust JNI reader
  as a test oracle
• The vortex-jni JAR is 258 MB; this is <1 MB pure Java — embeds
  anywhere, debugs with standard JVM tooling
• The goal is a fully functional interoperable library with a GitHub organisation around it
• When the Vector API ships, the plan is to use SIMD to speed up the hot encode/decode paths

A few questions:

1. Is there a detailed format specification, or plans to write one? Right now the Rust source is the
   ground truth — encoding metadata (e.g. RunEndMetadata, ZstdMetadata) has no upstream .proto file, and
   some wire-format invariants are implicit in the implementation. A spec (even a minimal one) would make
   third-party implementations much more robust.
2. Any undocumented invariants worth knowing — things the Rust implementation assumes but doesn't
   validate?
3. I noticed the Go implementation (vortex-go) as well — good to see the ecosystem growing. Do you also have plans for non-JNI version for Java?

Happy to contribute back anything useful.
Thanks for building such a well-designed format!

[robert3005]

We do not plan a specification right now. We treat the rust implementation as the canonical source and produce language specific bindings for them. There's still much to cover and harden before we can produce a format spec.

You might need to watch out for runtime pluggability. Encodings, DTypes, Compute and Layouts can be registered at runtime

[dfa1]

Thanks, that's helpful context. Treating the Rust impl as canonical works for me — CLAUDE.md in
vortex-java already points there for wire-format questions, and I've been reading the vtable
serialize/deserialize methods rather than probing bytes.

On pluggability: vortex-java currently has runtime registration for Encodings only (ServiceLoader +
programmatic EncodingRegistry.register()). DTypes and Layouts are sealed hierarchies, and there's no Compute layer yet.

Quick sanity check: are user-registered Encodings expected to round-trip through the file format
(i.e. unknown encoding IDs in a .vortex file should fail loudly rather than silently degrade), or is
the expectation that readers without the encoding just can't read the column?

[robert3005]

There's option on the session whether you should fail loudly or should you pass through the value from disk. Rust by default fails loudly.

[dfa1]

thanks @robert3005 for the idea:

EncodingRegistry registry = EncodingRegistry.loadAll().allowUnknown();
try (VortexReader vf = VortexReader.open(path, registry)) {
    // columns with unknown encodings are returned as UnknownArray
}

otherwise by default is throwing error

[dfa1]

Progress update — three weeks in 🚀

Quick followup to my original post. A lot
has happened since June 4, and @robert3005's reply shaped some of it — thanks for that.

It's on Maven Central. vortex-java now publishes as small modules (vortex-reader,
vortex-writer, vortex-core) — still well under 1 MB, pure Java, embeds anywhere. Released
through 0.8.x, with 0.9.0 in flight.

Zero third-party wire-format runtime deps. The biggest change: I replaced both
flatbuffers-java and protobuf-java with in-house code generators that compile the
.fbs/.proto schemas straight to MemorySegment-native Java — no flatc, no protoc, no
codegen runtime on the classpath. So the whole read/write path is FFM end-to-end with nothing
between the mmap and the decoder. Bonus: dropping the last automatic-module dependency clears the
way for a proper JPMS module-info.

On the runtime-pluggability warning — this was the most useful note, so here's where I landed:

• Encodings and extension DTypes are runtime-pluggable: custom
  EncodingDecoder/EncodingEncoder and ExtensionDecoder/ExtensionEncoder register on the
  reader/writer registries (via ServiceLoader + an explicit builder), immutable after construction.
• Layouts are still a fixed set (flat / chunked / zoned / struct / dict) — no SPI yet. Open
  question for you: is runtime-registered layout pluggability something third-party readers are
  expected to support for interop, or is the fixed set safe to rely on for now?

Still oracle-tested against vortex-jni. Every encoding round-trip and file-format boundary is
verified by an integration suite that diffs against the Rust reference. Reads and writes all
encodings in the fixture set; writer now emits vortex.stats zone maps (MIN/MAX/NULL_COUNT/SUM)
matching the reference.

One coverage gap I can't close on my own: across every published fixture set
(s3://vortex-compat-fixtures/, through the current v0.75.0/arrays/) there are no files for
masked, patched, or variant. Those are the only registered encodings I implement with no
Rust-written oracle to diff against, so they're verified by self-round-trip only. Any plans to add
them to the fixture set — or a recommended way to generate canonical fixtures for them — so
third-party readers can validate against the reference?

Thanks again for building such a clean format — it's been a genuine pleasure to implement against.