Skip to content

Commit 1bed15e

Browse files
magnuspmagnusp
authored
fix(jdbc): treat non-ASCII characters as identifier characters (#137)
Fixes #103. Contract: Hrana 3 is the source of truth for transport, values, metadata, batching, transactions, and error reporting. `java.sql` is the source of truth for the JDBC surface. xerial/sqlite-jdbc is a behavioral reference and conformance foundation, never a copy target. ## Change `isWordChar` in `SqlTableReference` and `LibsqlResultSetMetaData` now treats every character at or above `0x80` as an identifier character, matching SQLite. `SqlTableReference.UNQUOTED_IDENTIFIER` accepts the same range, because `identifierOf` re-validates word tokens against that pattern; without it the tokenizer would still split `användare`. ASCII-only behavior is unchanged. Base `aad9c7e456fad5c874e8efdc6f6c2d368244b654`, head `c47144e6935ed9b145cf2e58eca7dfd39387d526`. ## Red and green - Red at trunk code with the new tests: 7 failures, `Tests run: 66, Failures: 7` (`/tmp/xerial-parity-i103/lane-1-regression.log`). `SqlTableReferenceTest.resolvesNonAsciiTableAndColumn` got `[null]` instead of `användare`; `LibsqlResultSetMetaDataTest.resolvesNonAsciiTableName` got an empty table name. - Green at head: `SqlTableReferenceTest` 34/34, `LibsqlResultSetMetaDataTest` 32/32, BUILD SUCCESS. ## Lanes | Lane | Command | Result | | --- | --- | --- | | 1 | targeted tests at trunk vs head | red 7 failures, green 66/66 | | 2 | `LibsqlResultSetMetaDataTest` | 32/32; `getTableName` returns `användare` | | 3 | non-ASCII table + `anv` shadow (`LibsqlResultSetMetaDataTest`, prefix case) | 32/32; reports `användare`, not `anv` | | 4 | `./mvnw -pl libsql-jdbc test` | 245/245, BUILD SUCCESS | | 5 | `./mvnw -pl libsql-conformance -am verify` | 123 tests, 0 failures, 1 skipped, BUILD SUCCESS | | 6 | `./mvnw -pl libsql-conformance -am -Pxerial-control verify` | 100 tests, 0 failures, 1 skipped, BUILD SUCCESS | | 7 | `./mvnw spotless:check` | BUILD SUCCESS | | 8 | `./mvnw test` | BUILD SUCCESS | | 9 | parser/result-set suite (98 tests) | BUILD SUCCESS; no ASCII case changed outcome | | 10 | `./mvnw -pl libsql-jdbc -am package -DskipTests` | BUILD SUCCESS; jar and shaded jar built | Perf: `./mvnw -pl libsql-jdbc test` medians, three interleaved runs per side: trunk 2.08 s, head 2.08 s (within 10%). Logs in `/tmp/xerial-parity-i103/`. ## Divergences - `UNQUOTED_IDENTIFIER` gained the `\x80-\uFFFF` range. The plan's Build box names only `isWordChar`, but `identifierOf` rejects any word token outside that pattern, so the tokenizer fix alone cannot make `användare` resolve. - `LibsqlResultSetMetaDataTest` is not in the Files block; lane 2 names it explicitly, so the two metadata cases live there. - `resolvesQuotedNonAsciiNames` mixes quoted names with an unquoted non-ASCII column, so it fails at trunk as lane 1 requires. The fully quoted assertion (`SELECT "namn_ö" FROM "användare"`) already passed at trunk and is kept as regression coverage only. Tests added: non-ASCII tables, columns, table/column aliases, quoted and bracketed names, `SELECT *` expansion, a table named `anv` shadowing `användare`, and the metadata path.
1 parent aad9c7e commit 1bed15e

4 files changed

Lines changed: 122 additions & 4 deletions

File tree

‎libsql-jdbc/src/main/java/com/github/magnusp/libsql/jdbc/LibsqlResultSetMetaData.java‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -762,7 +762,12 @@ private static String unescape(String content, char quote) {
762762
}
763763

764764
private static boolean isWordChar(char c) {
765-
return (c >= 'a' && c <= 'z') || (c >= 'A' && c <= 'Z') || (c >= '0' && c <= '9') || c == '_' || c == '$';
765+
return c >= 0x80
766+
|| (c >= 'a' && c <= 'z')
767+
|| (c >= 'A' && c <= 'Z')
768+
|| (c >= '0' && c <= '9')
769+
|| c == '_'
770+
|| c == '$';
766771
}
767772

768773
private static String text(Value value) {

‎libsql-jdbc/src/main/java/com/github/magnusp/libsql/jdbc/SqlTableReference.java‎

Lines changed: 8 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -121,7 +121,8 @@ static FromClause of(List<Source> sources) {
121121
}
122122
}
123123

124-
private static final Pattern UNQUOTED_IDENTIFIER = Pattern.compile("[A-Za-z_][A-Za-z0-9_$]*");
124+
private static final Pattern UNQUOTED_IDENTIFIER =
125+
Pattern.compile("[A-Za-z_\\x80-\\uFFFF][A-Za-z0-9_$\\x80-\\uFFFF]*");
125126

126127
private static final Set<String> FROM_END = Set.of(
127128
"WHERE", "GROUP", "HAVING", "ORDER", "LIMIT", "OFFSET", "WINDOW", "RETURNING",
@@ -1273,6 +1274,11 @@ private static String unescape(String content, char quote) {
12731274
}
12741275

12751276
private static boolean isWordChar(char c) {
1276-
return (c >= 'a' && c <= 'z') || (c >= 'A' && c <= 'Z') || (c >= '0' && c <= '9') || c == '_' || c == '$';
1277+
return c >= 0x80
1278+
|| (c >= 'a' && c <= 'z')
1279+
|| (c >= 'A' && c <= 'Z')
1280+
|| (c >= '0' && c <= '9')
1281+
|| c == '_'
1282+
|| c == '$';
12771283
}
12781284
}

‎libsql-jdbc/src/test/java/com/github/magnusp/libsql/jdbc/LibsqlResultSetMetaDataTest.java‎

Lines changed: 51 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -54,6 +54,30 @@ void resolvesTableNameNullabilityAndAutoIncrement() throws SQLException {
5454
assertThat(metadata.isAutoIncrement(2)).isFalse();
5555
}
5656

57+
@Test
58+
void resolvesNonAsciiTableName() throws SQLException {
59+
LibsqlResultSetMetaData metadata = new LibsqlResultSetMetaData(
60+
List.of(new Col("namn_ö", "TEXT")),
61+
List.of(),
62+
"SELECT namn_ö FROM användare",
63+
connectionForSwedishTables());
64+
65+
assertThat(metadata.getTableName(1)).isEqualTo("användare");
66+
assertThat(metadata.isNullable(1)).isEqualTo(ResultSetMetaData.columnNullable);
67+
}
68+
69+
@Test
70+
void prefersNonAsciiTableOverItsAsciiPrefix() throws SQLException {
71+
LibsqlResultSetMetaData metadata = new LibsqlResultSetMetaData(
72+
List.of(new Col("namn_ö", "TEXT")),
73+
List.of(),
74+
"SELECT namn_ö FROM användare",
75+
connectionForSwedishTables());
76+
77+
assertThat(metadata.getTableName(1)).isEqualTo("användare");
78+
assertThat(metadata.getColumnName(1)).isEqualTo("namn_ö");
79+
}
80+
5781
@Test
5882
void ignoresAutoIncrementInsideSqlComments() throws SQLException {
5983
String createSql = "CREATE TABLE People (id INTEGER PRIMARY KEY /* AUTOINCREMENT */, name TEXT)";
@@ -644,6 +668,33 @@ private static LibsqlResultSetMetaData metadataForTable(String createSql, List<C
644668
return new LibsqlResultSetMetaData(cols, List.of(), "SELECT * FROM T", connectionReturning(result));
645669
}
646670

671+
private static LibsqlConnection connectionForSwedishTables() {
672+
Map<String, String> tables = Map.of(
673+
"anv", "CREATE TABLE anv (id INTEGER)",
674+
"användare", "CREATE TABLE användare (id INTEGER, namn_ö TEXT)");
675+
Map<String, List<String>> columns = Map.of("anv", List.of("id"), "användare", List.of("id", "namn_ö"));
676+
return new LibsqlConnection(null, null) {
677+
@Override
678+
public StmtResult executeStatement(Stmt stmt) throws SQLException {
679+
String table = argument(stmt, 0);
680+
String createSql = tables.get(table);
681+
if (createSql == null) {
682+
return new StmtResult(List.of(), List.of(), 0, null);
683+
}
684+
List<Row> rows = new ArrayList<>();
685+
for (String column : columns.get(table)) {
686+
rows.add(new Row(List.of(
687+
Value.of(table),
688+
Value.of(createSql),
689+
Value.of(column),
690+
Value.of(0L),
691+
Value.of(0L))));
692+
}
693+
return new StmtResult(List.of(), rows, 0, null);
694+
}
695+
};
696+
}
697+
647698
private static LibsqlConnection connectionReturning(StmtResult result) {
648699
return new LibsqlConnection(null, null) {
649700
@Override

‎libsql-jdbc/src/test/java/com/github/magnusp/libsql/jdbc/SqlTableReferenceTest.java‎

Lines changed: 57 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -29,7 +29,12 @@ class SqlTableReferenceTest {
2929
new SqlTableReference.TableDefinition("CREATE TABLE c (z TEXT)", List.of("z")),
3030
"g",
3131
new SqlTableReference.TableDefinition(
32-
"CREATE TABLE g (p INTEGER, q AS (p+1), r TEXT NOT NULL)", List.of("p", "q", "r")));
32+
"CREATE TABLE g (p INTEGER, q AS (p+1), r TEXT NOT NULL)", List.of("p", "q", "r")),
33+
"anv",
34+
new SqlTableReference.TableDefinition("CREATE TABLE anv (id INTEGER)", List.of("id")),
35+
"användare",
36+
new SqlTableReference.TableDefinition(
37+
"CREATE TABLE användare (id INTEGER, namn_ö TEXT)", List.of("id", "namn_ö")));
3338

3439
private static final SqlTableReference.Schema SCHEMA = reference -> {
3540
if (reference.table() == null) {
@@ -58,6 +63,57 @@ void resolvesQuotedAndBracketedNames() throws SQLException {
5863
.containsExactly(new SqlTableReference.TableRef(null, "my table"));
5964
}
6065

66+
@Test
67+
void resolvesNonAsciiTableAndColumn() throws SQLException {
68+
SqlTableReference.Resolution resolution = resolve("SELECT namn_ö FROM användare");
69+
70+
assertThat(resolution.labels()).containsExactly("namn_ö");
71+
assertThat(resolution.tables())
72+
.containsExactly(new SqlTableReference.TableRef(null, "användare"));
73+
}
74+
75+
@Test
76+
void resolvesNonAsciiColumnThroughTableAlias() throws SQLException {
77+
assertThat(resolve("SELECT a.namn_ö FROM användare a").tables())
78+
.containsExactly(new SqlTableReference.TableRef(null, "användare"));
79+
assertThat(resolve("SELECT * FROM användare a").tables())
80+
.containsExactly(
81+
new SqlTableReference.TableRef(null, "användare"),
82+
new SqlTableReference.TableRef(null, "användare"));
83+
}
84+
85+
@Test
86+
void resolvesNonAsciiAliases() throws SQLException {
87+
assertThat(resolve("SELECT namn_ö AS namn FROM användare").labels()).containsExactly("namn");
88+
assertThat(resolve("SELECT namn_ö namn FROM användare").labels()).containsExactly("namn");
89+
assertThat(resolve("SELECT namn_ö AS namn FROM användare").tables())
90+
.containsExactly(new SqlTableReference.TableRef(null, "användare"));
91+
assertThat(resolve("SELECT å.namn_ö FROM användare AS å").tables())
92+
.containsExactly(new SqlTableReference.TableRef(null, "användare"));
93+
}
94+
95+
@Test
96+
void resolvesQuotedNonAsciiNames() throws SQLException {
97+
assertThat(resolve("SELECT namn_ö FROM \"användare\"").tables())
98+
.containsExactly(new SqlTableReference.TableRef(null, "användare"));
99+
assertThat(resolve("SELECT namn_ö FROM [användare]").tables())
100+
.containsExactly(new SqlTableReference.TableRef(null, "användare"));
101+
assertThat(resolve("SELECT \"namn_ö\" FROM \"användare\"").labels()).containsExactly("namn_ö");
102+
}
103+
104+
@Test
105+
void keepsNonAsciiTableWholeWhenAnAsciiPrefixTableExists() throws SQLException {
106+
SqlTableReference.Resolution resolution = resolve("SELECT * FROM användare");
107+
108+
assertThat(resolution.labels()).containsExactly("id", "namn_ö");
109+
assertThat(resolution.tables())
110+
.containsExactly(
111+
new SqlTableReference.TableRef(null, "användare"),
112+
new SqlTableReference.TableRef(null, "användare"));
113+
assertThat(resolve("SELECT id FROM anv").tables())
114+
.containsExactly(new SqlTableReference.TableRef(null, "anv"));
115+
}
116+
61117
@Test
62118
void resolvesAliasesWithAndWithoutAs() throws SQLException {
63119
assertThat(resolve("SELECT a AS z FROM w").labels()).containsExactly("z");

0 commit comments

Comments
 (0)