Track Iceberg v3 Feature Support (Roadmap for 0.4.0)

[zhjwpku]

We already have partial support for Iceberg v3 in the codebase (e.g., Row Lineage and partition statistics). However, a number of key v3 features are still missing.

This issue serves as a parent tracking issue for the remaining v3 work we plan to (partially) include in the 0.3.0 release.

If you are interested in working on any of the items below, please open a dedicated sub-issue and link it back to this one.

V3 related

1. Schema-Level Support for New v3 Types

Add support for all newly introduced v3 types at the schema level, including:

• timestamp_ns, timestamptz_ns @zhjwpku
• unknown @manuzhang
• variant @zhjwpku
• geometry, geography @wgtmac

2. Column Default Values (Schema Evolution)

Enable schema evolution with default values:

• Support initial-default and write-default @huan233usc
• Apply defaults correctly during read and write paths

3. Multi-Argument Transforms (Not planned, as they are not supported by the Java implementation)

4. Binary Deletion Vectors

Add support for deletion vectors (DV):

• Puffin-based DV parsing @zhaoxuan1994
• Scan-time application of DV
• Writer-side support for generating DV @zhaoxuan1994

5. Table-Level Encryption Metadata (Not planned for 0.4.0) @shangxinli

Support encryption-related metadata defined in v3:

• Parse and preserve encryption metadata

6. Puffin-Based Statistics (Not planned for 0.4.0)

Support table and partition statistics stored in Puffin files:

• Metadata parsing
• Optional integration into scan planning

Table update and maintenance: #364

• OverwriteFiles @lishuxu
• RowDelta @manuzhang
• MergeAppend @zhjwpku
• DeleteFiles @manuzhang
• RewriteFiles @WZhuo
• ReplacePartitions @shangxinli
• RewriteManifests @zhjwpku
• CherryPickOperation @shangxinli

Miscellaneous

• Metadata table inspection @All-less @WZhuo
• LRU cache @HuaHuaY
• Logging interface and spdlog integration @kamcheungting-db

If anything is missing or you have suggestions, feel free to comment or open additional sub-issues.

[manuzhang]

@zhjwpku Can you help to link #665 here as sub-issue? I don't think I have the permission to do so.

[manuzhang]

@zhjwpku Meanwhile, it might be valuable to create a milestone 0.4.0 to track all issues?

[zhjwpku]

@zhjwpku Meanwhile, it might be valuable to create a milestone 0.4.0 to track all issues?

Good suggestion. We haven't used milestones for previous releases, but I think we should start doing that for future releases. Let me try that.

[zhjwpku]

@zhjwpku Meanwhile, it might be valuable to create a milestone 0.4.0 to track all issues?

Good suggestion. We haven't used milestones for previous releases, but I think we should start doing that for future releases. Let me try that.

Oh, I don't have permission to create milestones. Let's wait for @wgtmac to handle this. I'm not sure if we should do this after the 0.3.0 release.

[wgtmac]

Created: https://github.com/apache/iceberg-cpp/milestone/1

[wgtmac]

I think we don't need to support 5. Table-Level Encryption Metadata and 6. Puffin-Based Statistics at the moment.

[zhjwpku]

I think we don't need to support 5. Table-Level Encryption Metadata and 6. Puffin-Based Statistics at the moment.

Added a comment noting that this is not planned for 0.4.0.

[zhjwpku]

I've moved the unfinished items from Roadmap for 0.3.0 (#523) into this issue. Feel free to ping me if I missed anything.

I've also linked the iceberg-go V3 tracking issue [1] here so we can keep track of any V3-related work we might otherwise miss.

[1] apache/iceberg-go#589