Hi there,

We have a pretty large hive-partitioned parquet table on S3, we followed the document to recreate the table with partition metadata logging on Unity Catalog.

We're using Databricks Runtime 16.4 LTS, but despite the release note mentioned that partition metadata logging setting would be anchored to the table, we noticed that all query sessions must set

SET spark.databricks.nonDelta.partitionLog.enabled = true;

so the query wouldn't scan all directories.

With DBR clusters we can make this part of the cluster's spark config, but when using Databricks SQL, it doesn't allow us to set this config, and it doesn't seem to honor the table settings automatically either. It would simply scan all directories and cause queries being extremely slow.

We tried both current and preview channels, but the behaviors are the same. Is there anyway we can make Databricks SQL honor the partition metadata logging settings?