Deprecated sort_by method

[sushantvema]

Hi. This adapter for harlequin is great. As a Databricks power user, I've been looking forward to a terminal-based SQL interface for DBSQL.

It seems that during the initial catalog indexing, there is a sort_by method which is being called in the adapter. However I think that method has been deprecated, so I'm getting this error:

Catalog Error: Could not update data catalog 'ColumnTable' has no attribute 'sort_by'

Unless the analagous method has been renamed in the DB sql client, it should probably be removed from the harlequin adapter.

[alexmalins]

thanks for the report @sushantvema - this has been caused by a recent update to the Databricks SQL Connector
will investigate today and try and come up with a solution

If your Databricks instance is using unity catalog and you don't need any of the legacy hive metastores indexed in the catalog pane, run harlequin with the --skip-legacy-indexing flag as the problem is confined to when the adapter is trying to index the legacy catalogs.

[sushantvema]

Yes our Databricks instance is only using unity catalog without any legacy hive metastores. The flag fixed the problem. I'm sure people request this like a broken record, but if only some vim keybindings were implemented, this would be an excellent replacement to DBeaver with the databricks adapter :)

I will check out the recent capabilities to modify keybindings via config files.

[alexmalins]

Thanks @sushantvema 🎉

I think in a future release I will invert --skip-legacy-indexing to become --index-legacy-metastores, so the default behaviour if no flag is supplied is to not index the legacy metastores. This is preferable as it means the catalog indexing is much faster, and I guess most Databricks users don't care about legacy metastores any more as they will have migrated to unity catalog.

I'm going to reopen this issue though, as the bug still exists for anyone who does want to index legacy metastores and it's going to trip up others who don't know about --skip-legacy-indexing.

Another solution is to downgrade the databricks-sql-connector to <=3.4.0, as >=3.5.0 introduced the bug. I just made a bug report over here: databricks/databricks-sql-python#550

I can fix the issue in this package by executing "SHOW CATALOGS" "SHOW SCHEMAS" SQL queries instead of the cursor methods that have the >=3.5.0 bug of not returning a pyarrow table. Or by switching from fetchall_pyarrow() to fetchall() instead.

vim keybindings were implemented, this would be an excellent replacement to DBeaver with the databricks adapter :)

Checkout tconbeer/harlequin#371 as vim bindings would happen upstream of this adapter in the main harlequin package 🙌

[alexmalins]

Databricks have merged a fix for the upstream issue.. After they cut the next databricks-sql-connector release, I'll block out the problematic databricks-sql-connector versions 3.5.0 to 4.0.3 from the allowed dependencies of this project. 👍