Databricks

mehdi1 · ‎05-09-2022

Hi,

I could create spark tables based on SQL server (Azure SQL actually), and query the tables from a notebook.

On another hand, I can see the tables listed in the hive metastore in Databricks SQL. However when I try to query the tables there, it says only CSV, json, parquet, etc. (meaning file based) tables are supported.

Is there anyway to overcome that ? It looks pretty inconsistent to me that we can query the tables in Data Science and Engineering mode but not in SQL mode.

Kaniz · ‎05-19-2022

Hi @Mehdi Aziz Boumedine , For Databricks SQL, see SQL reference for Databricks SQL.

Prabakar · ‎06-07-2022

@Mehdi Aziz Boumedine

When you query Hive tables on Databricks SQL, you get an error "Reading Hive table <table-name> is not supported on Databricks SQL. Only csv, json, avro, delta, parquet, orc, text, unity_catalog data sources are supported". This is because Hive based tables are not supported on Databricks SQL

The solution is to add few records/overwrite records either in parquet/orc or any desired format. Adding rows in parquet for instance will change the relation from "HiveTableRelation" to Parquet relation, which makes the table queryable in Databricks SQL

Anonymous · ‎06-20-2022

Hey @Mehdi Aziz Boumedine

Hope you are well. Just wanted to see if you were able to find an answer to your question and would you like to mark an answer as best? It would be really helpful for the other members too.

Cheers!