Types: Name unresolvable reflected types, and resolve geo, ip, character and arrays

[florinutz]

When SQLAlchemy reads a table's structure from crate, it gets each column's type as a name such as geo_point or integer_array, and the dialect looks that name up in its list of known types. A name missing from the list became an abstract type. Reading still worked, but creating the table again crashed without naming the column or the type. sys.summits hits this through its geo_point column. Closes #295.

The fix comes first: a name the dialect cannot represent now reflects as UnresolvedType, a new type that keeps the crate name. A SELECT on the column still works, and compiling DDL fails with that name. SQLAlchemy's NullType, which the issue suggests, would fail with "did you forget to specify a type" and name nothing.

Then the list grows, so common tables stop reaching that fallback. geo_point, geo_shape, ip (through a new IP type) and character resolve, and every <element>_array is derived from its element instead of a fixed list. Mapping character needs its length, because a bare CHAR holds one character and a reflected CHAR(5) would reject 'abcde'. So reflection reads the length from information_schema.columns. Of the other types the issue lists, numeric resolves since #300, and uuid moves to #311 together with the fix that lets UUID columns be written. bit stays unresolved, because SQLAlchemy has no generic BIT. date can be stored on crate nightly, but it stays unresolved too, because the dialect compiles DATE to TIMESTAMP and a copied table would get a timestamp column. crate refuses to store interval, time with time zone, regclass, regproc and regtype, and has no time or row column type.

Reflecting and compiling three tables:

main    all three         AttributeError: 'UserDefinedType' object has no attribute 'get_col_spec'
branch  sys.summits       CREATE TABLE sys.summits (classification VARCHAR, coordinates GEO_POINT, ...)
branch  pg_catalog.pg_am  CompileError: (in table 'pg_am', column 'amhandler'): Unable to represent CrateDB type 'regproc' in SQLAlchemy
branch  doc.codes         CREATE TABLE doc.codes (code CHAR(5), host IP)

For review:

1. Arrays of CHAR stay unresolved, because crate reports no length for array elements and ARRAY(CHAR) holds one character per element. Downside: such columns cannot be recreated from reflection.
2. The length also reaches VARCHAR: a reflected VARCHAR(10) compiles to VARCHAR(10) instead of an unbounded VARCHAR. Downside: DDL generated from reflected tables changes for every bounded VARCHAR column.
3. Arrays of arrays stay unresolved, although crate stores them. SQLAlchemy refuses to nest ARRAY, and this dialect refuses the dimensions form it offers instead. Downside: they wait on Nested array columns cannot be created #310, which proposes lifting the dialect's refusal.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Advanced

Run ID: 9bc4a85d-3dd8-4b41-9663-a12e6f4bd124

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[bgunebakan]

Thank you! I've reviewed that with NUMERIC PR because there is some places that touches both implementation. I've added some comments.

[florinutz]

I initially assumed "SQLAlchemy's ARRAY cannot hold an array", but that turned out to be not fully true. Crate can store nested arrays and sqlalchemy can carry them. It's our dialect's compiler that refuses them. I added this as an issue: #310

Then one of this pr's comments was about dealing with every type named in the ticket (makes sense, ofc). When I got to UUID, I mapped it and the test exposed that sa.UUID() can't be sent to crate. Fixing that with #311 (pr #312)

cc @bgunebakan