Feature: lance support

[aheev]

We'll follow duckDB lance extension for implementation
Data types: https://lance.org/guide/data_types/#data-type-mapping-for-integrations

INSTALL lance;
LOAD lance;
CREATE NODE TABLE node_table1 ... WITH (storage = '<path-to-lance-table>', format = 'lance');
CREATE REL TABLE rel_table ... WITH (storage = '<path-to-lance-table>', format = 'lance');

Supported features:

• All cypher queries currently supported in ladybug
• lance_hybrid_search func
• lance_fts func
• lance_vector_search func
• COPY FROM
• LOAD FROM

Not supported

• mutability
• lance namespaces
• mixed table types i.e NODE TABLE lance and REL TABLE non-lance (OR) NODE TABLE non-lance and REL TABLE lance

References

• duckDB's lance extension

[aheev]

@adsharma how does this look?

[aheev]

Notes to the implementor:

• vector, blob types need extensive testing
• Apache 2.0 License
• update docs

[adsharma]

Shouldn't this issue be on the extension repo?
icebug-format extension is another topic to consider.

[aheev]

Shouldn't this issue be on the extension repo?

oops!! Let's keep it here for now. I have already started working on it

icebug-format extension is another topic to consider.

will create one. I believe it should not be urgent 🤔 . we have EXPORT DATABASE which exports in icebug-disk format w/o the indptr, but can be easily induced and query_as_arrow and .csr for icebug-memory

[adsharma]

Once lance-format/lance#654 is fixed, it should be trivial to implement lance on icebug-format.

[aheev]

Once lance-format/lance#654 is fixed, it should be trivial to implement lance on icebug-format.

How does this help on the implementation side? We don't have lance-csr read support in ladybug yet 🤔

Also, lance-duckdb ext has write support already

[adsharma]

Edit: I see your point now. We don't use EXPORT DATABASE. We use COPY which is already supported by the lance extension.

It helps as in it moves the ball forward, but doesn't complete the task. We still have to implement Columnar{Node,Rel}TableBase subclasses for lance. These subclasses would live in the extension with a well thought out narrow interface so changes to base don't break the lance extension because someone added a new field to a class.

[aheev]

We still have to implement Columnar{Node,Rel}TableBase subclasses for lance. These subclasses would live in the extension with a well thought out narrow interface so changes to base don't break the lance extension because someone added a new field to a class.

In that case we would have to bring a new catalogEntryType because FOREIGN_TABLE_ENTRY accepts a single scanFunc and doesn't differentiate between rel and node tables

Bigger question is how do we want to support lance in ladybug? independent table URIs(similar to parquet, arrow) or via catalog/namespace(duckDB, postgres)

Independent Table URIs

Shadow table creation could be supported via EITHER of the following

• Create a new catalog entry type for extension tables without a catalog and new shadow classes in ladybug: ExtensionNodeTable class extending NodeTable class and ExtensionRelTable class extending RelTable. Then in the extension, create respective nodeTable and RelTable classes extending these shadow classes
• Using the current FOREIGN_TABLE_ENTRY flow would require decoupling attachedDatabase and foreign_table_catalog_entry and adding checks for catalog vs no-catalog wherever these tables(or catalogEntries) are instantiated
• making lance a first class citizen

Catalog: The duckDB way

ATTACH 'namespace-dir' AS uw (dbtype lance);
CREATE NODE TABLE Employee (id INT64, name STRING, PRIMARY KEY (id))
WITH (storage='uw.employee', format = 'lance');

This would seamlessly fit into current foreign table paradigm

Note: If tables share a comon root directory, then this would still work for independent table URIs. Just pass root to ATTACH

However COPY and LOAD can be supported on Table URIs

COPY Person FROM "<path>/person.lance";
LOAD Person FROM "<path>/person.lance";

References

• https://github.com/lance-format/lance-duckdb/blob/main/docs/sql.md

[adsharma]

Similar to parquet/arrow.