feat(rest-server): add FileSystemCatalog-backed REST catalog server#371
Merged
Conversation
Member
Author
|
@JingsongLi Hello, I'm not sure if the community needs this ability, so I'd like to ask for your opinion first. Internally, we mainly use rest catalogs. However, when conducting rest catalog tests on rust, it is often necessary to start the real rest catalog before we can test the behavior of real rest clients on the rust side. Therefore, I introduced a Rust version of the Rest Catalog that is directly based on the FileSystem wrapper. There is similar code in Java as well. Do you think this contribution to the community is reasonable? Or do I need to start it directly into the Test module instead of using it as a complete rest catalog server? |
TheR1sing3un
force-pushed
the
pr/rest-server
branch
2 times, most recently
from
c4d4a4d to
c66c374
Compare
JingsongLi
force-pushed
the
pr/rest-server
branch
from
c66c374 to
55f4dc2
Compare
JingsongLi
reviewed
Jun 19, 2026
Add a paimon-rest-server crate (library + binary, publish = false) that maps the Paimon REST protocol onto a real FileSystemCatalog, enabling end-to-end testing of the REST catalog without a Java server: config, database/table metadata CRUD, append write + commit (persisting the posted snapshot via SnapshotManager) + read back, and column-level alter table. The wire format mirrors Java Paimon so the existing RESTCatalog client can drive it and warehouses round-trip with Java. Add a CI job to run the crate's e2e tests, and a README describing usage and endpoints.
The REST client's list_partitions hits GET /v1/.../partitions, but the router did not serve it: the request fell through to Axum's 404, mapped to NoSuchResource, and the client's filesystem fallback (which only triggers on 501 NotImplemented) never ran, so partition listing failed. Add the partitions route, served from the latest on-disk snapshot via list_partitions_from_file_system (same warehouse the client wrote to), returning a ListPartitionsResponse. Cover it with an e2e test that writes across two partitions, commits, and asserts the listed partitions and their record counts.
TheR1sing3un
force-pushed
the
pr/rest-server
branch
from
55f4dc2 to
85a305a
Compare
The e2e suite round-trips through a real FileSystemCatalog, whose path handling has known Windows incompatibilities (drive-letter handling in the fs storage relative-path logic). The paimon crate already gates its own filesystem catalog tests with #[cfg(not(windows))] for the same reason, so mirror that by skipping this CI step on Windows runners.
JingsongLi
reviewed
Jun 20, 2026
The client (`ResourcePaths`) builds path segments with `RESTUtil::encode_string` (`application/x-www-form-urlencoded`), so a space becomes `+` and a literal `+` becomes `%2B`. Axum's `Path`/`RawPathParams` extractors percent-decode `%xx` but leave `+` untouched, so catalog names containing spaces became unaddressable through `RESTCatalog` (the request 404'd instead of resolving). Decode the raw, still-percent-encoded URI segments with `RESTUtil::decode_string` via a small `RestPath` extractor, mirroring Java's `RESTCatalogServer`. Decoding the raw segment (rather than post-processing Axum's already percent-decoded value) is the only way to recover names correctly for all inputs, since a real space (encoded `+`) and a literal `+` (encoded `%2B`) are indistinguishable once `%xx` has been decoded. Add an e2e case covering database/table names with a space and a literal `+`.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Linked issue: close #369
Note
Stacked on #370 (column-level alter table). This branch currently also
contains #370's commit, so the diff includes it until #370 merges. Kept as a
draft for now; once #370 is merged I will rebase and only the
paimon-rest-servercommit will remain.Add a real REST catalog server backed by
FileSystemCatalog, so theRESTCatalogclient can be tested end to end without a Java server.Brief change log
paimon-rest-servercrate (library + binary,publish = false) that maps the Paimon REST protocol onto a realFileSystemCatalog: config, database/table metadata CRUD, append write + commit (the commit endpoint persists the posted snapshot viaSnapshotManager) + read back, and column-level alter table.RESTCatalogclient drives it and warehouses round-trip with Java.README.Tests
tests/e2e.rs: metadata CRUD, write/commit/read round trip, and alter-table over REST. Wired into CI via a newcargo test -p paimon-rest-serverstep.API and Format
No change to
paimon's public API or storage format. The new crate introducesaxum/tower(MIT) dependencies;cargo deny check licensespasses without changes todeny.toml.Documentation
Crate
README.md(purpose, usage, endpoints).