Hey I need some help / suggestions troubleshooting this, I have two DataBricks Workspaces Common and Lakehouse.
There difference between them is:
Major Differences:
- Lakehouse is using Unity Catalog
- Lakehouse is using External Locations whereas credentials are set using service principal for Common.
- Listing directories in common is at least 4-8 times faster than Lakehouse environment.
- Lakehouse is in VNET and is accessed using company VPN.
Configuration:
- Both DLT pipelines are configured the same except for catalog location and since common isn't UC it is saved to a dbfs location.
- Both DLT pipelines are reading from the same storage container. Other than that nothing else is different.
DLT Comparisons:
- Common (17m 21s) + 5 minutes because cluster is already running.
- 0m - Task Starts (Cluster already running) Initialising
- 0m 30s Setting Up Tables (7m 30s)
- 8m Graph Initialised and tables are being populated (9m 30s)
- 17m 30s All tables complete
- Lakehouse (1h 17m 23s)
- 0m - Task Starts (Spinning up Cluster)
- 4m 30s Initialising (15m 20s)
- 19m 50s Setting Up tables (47m)
- 1h 6m 50 Graph Initialised and tables are being populated (11m)
- 1h 17 23s All tables complete
I am assuming there is bottleneck somewhere but having a hard time troubleshooting it. I think it could be one of the following Unity Catalog overhead or VPN performance between Databricks and Storage Container.
