Significant Delay in Deploying View on Unity Catalog Compared to Hive Metastore

[dnz]

Hi everyone,

I'm experiencing a significant delay when deploying a view in the Unity Catalog compared to the Hive Metastore. Specifically, the deployment on the Unity Catalog takes 20 to 30 minutes, whereas the same deployment on the Hive Metastore completes within seconds.

Details:

• Deployment Time in Unity Catalog: 20-30 minutes
• Deployment Time in Hive Metastore: Seconds
• Underlying Table: External table in S3 (not a Delta table)

My Questions:

1. Does Databricks internally scan the entire data before deploying a view in the Unity Catalog? if it does, then is this a new feature of UC and is there anyway to skip this step ?
2. Are there any specific configurations or optimizations needed for deploying views on Unity Catalog that could reduce the deployment time?

Any insights or suggestions would be greatly appreciated. Thank you!

Regards,
Dains

[Kaniz_Fatma]

Hi @dnz, Unity Catalog might perform extra checks or operations, especially with external tables in S3, though there's no explicit documentation confirming a full data scan. Ensure that your Unity Catalog's metadata operations are optimized and that your cluster is properly sized and configured for the task. This might involve checking network connectivity and avoiding unnecessary operations. If you're not already using Delta tables, consider converting your external tables. Delta tables are optimized for performance in Databricks, potentially reducing deployment time.