Cluster Reuse for delta live tables
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-21-2022 09:40 AM
I have several delta live table notebooks that are tied to different delta live table jobs so that I can use multiple target schema names. I know it's possible to reuse a cluster for job segments but is it possible for these delta live table jobs (which are run in sequence) to reuse the cluster that was created by the first job. I'm running into quota issues in the customer azure environment and this would help a lot.
It seems like if I run an individual job (lets call it sliver) multiple times, it will pickup the cluster that it used before, but if I run another job (lets call it gold), it tries to start its own cluster even though they are configured with the same cluster configuration
- Labels:
-
Delta Live Tables