Performance issue with Spark SQL when working with data from Unity Catalog
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
04-01-2025 07:30 AM
We're encountering a performance issue with Spark SQL when working with data from Unity Catalog. Specifically, when I use Spark to read data from a Unity Catalog partition created by DLT and then create a view, the executor retrieval is very slow. However, if I clone the data outside of the Unity Catalog partition (still managed by DLT), the performance improves dramatically.
Has anyone seen similar behavior or can shed some light on what might be causing the discrepancy? Could it be related to metadata overhead, permission handling, or something else specific to how Unity Catalog manages partitions? Any insights or troubleshooting tips would be greatly appreciate.
Hung Nguyen
- Labels:
-
Workflows