Upgrade Databricks SQL Alchemy version to avoid issues with metadata operations

[msrathore-db]

Bug description

Hi there,

Some of the users are hitting an issue on Superset v6 using the databricks python connector

The metadata discovery (table browser) is currently broken when using catalogs and/or schemas containing hyphens (which is very common in Databricks Unity Catalog). Superset fails to fetch the table list because the generated SQL is not properly quoted.

For example, it currently generates:

SHOW TABLES FROM my-staging-catalog.my-poc-schema

Instead of:

SHOW TABLES FROM `my-staging-catalog`.`my-poc-schema`
This triggers a [INVALID_IDENTIFIER] error because the hyphen is interpreted as a subtraction operator rather than part of the identifier.

[INVALID_IDENTIFIER] The unquoted identifier my-staging-catalog is invalid and must be back quoted as: `my-staging-catalog`.
Unquoted identifiers can only contain ASCII letters ('a' - 'z', 'A' - 'Z'), digits ('0' - '9'), and underbar ('_').

This was fixed in SQL Alchemy in v2.0.4 but the repo has pinned v1.0.5
Can we please update to point to the recent version of Databricks SQL Alchemy?

Screenshots/recordings

No response

Superset version

master / latest-dev

Python version

3.9

Node version

Not applicable

Browser

Chrome

Additional context

No response

Checklist

• I have searched Superset docs and Slack and didn't find a solution to my problem.
• I have searched the GitHub issue tracker and didn't find a similar bug report.
• I have checked Superset's logs for errors and if I found a relevant Python stacktrace, I included it here as text in the "additional context" section.

[dosubot]

Hi @msrathore-db! I'm Dosu and I’m helping the superset team.

Thanks for the detailed report! I found some relevant context that may help clarify the situation.

The current state: Superset uses sqlalchemy-databricks v1.0.5 (which is SQLAlchemy 1.x compatible), not databricks-sqlalchemy v2.x. These are different packages—databricks-sqlalchemy v2.x requires SQLAlchemy 2.x, which Superset hasn't migrated to yet.

Existing support for hyphenated identifiers: Superset's Databricks engine spec already has code to handle hyphenated identifiers by backtick-quoting them in the get_prequeries method, and tests verify this works for names like with-hyphen.

Possible related regression: There's a known regression in Superset 6.0.0rc1+ affecting identifier quoting for databases with supports_cross_catalog_queries enabled. Instead of quoting each component separately (e.g., `my-catalog`.`my-schema`), it incorrectly quotes the full path as one string. This affects Databricks and other databases. Could this be what you're hitting?

If the cross-catalog quoting regression isn't the cause, it would help to know:

• Which specific operation is failing (browsing tables in SQL Lab, dataset creation, etc.)?
• Can you share the full stacktrace from the Superset logs?

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}

   

[msrathore-db]

A few corrections to the Dosu bot's response:

1. "These are different packages" — Incorrect. Superset's pyproject.toml pins databricks-sqlalchemy==1.0.5, which is the same package from databricks/databricks-sqlalchemy. sqlalchemy-databricks (which the bot seems to be referring to) is a completely different, abandoned package (last version 0.2.0 on PyPI).

2. "Existing support for hyphenated identifiers via get_prequeries" — Unrelated. The get_prequeries backtick quoting only applies to USE CATALOG / USE SCHEMA session setup during query execution (SQL Lab, charts). It is never called during metadata discovery. The metadata path goes through get_inspector() → inspector.get_table_names(schema) → dialect's get_table_names(), which bypasses get_prequeries entirely.

3. The #35116 supports_cross_catalog_queries regression — Also unrelated. That issue affects physical dataset chart rendering (SELECT FROM queries) for engines with supports_cross_catalog_queries=True. The bug reported here is about metadata discovery (SHOW TABLES FROM), which is a completely different code path. Additionally, DatabricksPythonConnectorEngineSpec (the recommended connector) doesn't even set supports_cross_catalog_queries=True — only the legacy DatabricksNativeEngineSpec does.

Root cause: In databricks-sqlalchemy==1.0.5, the dialect's get_table_names() generates unquoted SQL:

sql_str = "SHOW TABLES FROM {}".format(".".join([self.catalog, schema or self.schema]))
# → SHOW TABLES FROM my-staging-catalog.my-poc-schema  ← BROKEN

This was fixed in databricks-sqlalchemy>=2.0.4 with unconditional backtick quoting:

_target = f"`{_target_catalog}`.`{_target_schema}`"
# → SHOW TABLES FROM `my-staging-catalog`.`my-poc-schema`  ← CORRECT

The same applies to get_view_names() and has_table().

The blocker for upgrading is that Superset pins sqlalchemy>=1.4, <2 while databricks-sqlalchemy>=2.0.4 requires sqlalchemy>=2.0.21.

[fowlerp397-tech]

Thank you for the clarification

…

On Sun, Mar 15, 2026, 2:53 PM Madhavendra Rathore ***@***.***> wrote: *msrathore-db* left a comment (apache/superset#38658) <#38658 (comment)> A few corrections to the Dosu bot's response: *1. "These are different packages" — Incorrect.* Superset's pyproject.toml pins databricks-sqlalchemy==1.0.5, which is the same package from databricks/databricks-sqlalchemy <https://github.com/databricks/databricks-sqlalchemy>. sqlalchemy-databricks (which the bot seems to be referring to) is a completely different, abandoned package (last version 0.2.0 on PyPI). *2. "Existing support for hyphenated identifiers via get_prequeries" — Unrelated.* The get_prequeries backtick quoting only applies to USE CATALOG / USE SCHEMA session setup during *query execution* (SQL Lab, charts). It is never called during *metadata discovery*. The metadata path goes through get_inspector() → inspector.get_table_names(schema) → dialect's get_table_names(), which bypasses get_prequeries entirely. *3. The #35116 <#35116> supports_cross_catalog_queries regression — Also unrelated.* That issue affects physical dataset *chart rendering* (SELECT FROM queries) for engines with supports_cross_catalog_queries=True. The bug reported here is about *metadata discovery* (SHOW TABLES FROM), which is a completely different code path. Additionally, DatabricksPythonConnectorEngineSpec (the recommended connector) doesn't even set supports_cross_catalog_queries=True — only the legacy DatabricksNativeEngineSpec does. *Root cause:* In databricks-sqlalchemy==1.0.5, the dialect's get_table_names() generates unquoted SQL: sql_str = "SHOW TABLES FROM {}".format(".".join([self.catalog, schema or self.schema]))# → SHOW TABLES FROM my-staging-catalog.my-poc-schema ← BROKEN This was fixed in databricks-sqlalchemy>=2.0.4 with unconditional backtick quoting: _target = f"`{_target_catalog}`.`{_target_schema}`"# → SHOW TABLES FROM `my-staging-catalog`.`my-poc-schema` ← CORRECT The same applies to get_view_names() and has_table(). The blocker for upgrading is that Superset pins sqlalchemy>=1.4, <2 while databricks-sqlalchemy>=2.0.4 requires sqlalchemy>=2.0.21. — Reply to this email directly, view it on GitHub <#38658 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/B72HASNYZN5EMVOU6YM5BX34Q4C5DAVCNFSM6AAAAACWSUK3ZKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHM2DANRTG43DSOJQGE> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>