Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-13-2023 05:59 AM
I see 2 easy ways:
- use databricks workflows. this creates job clusters for you
- use data factory and the Databricks notebook activity: this does the same
Basically you tell databricks to create short-lived clusters which are alive during the execution of a spark program. When the program is finished, the cluster is terminated.
There is also something like cluster pools, which keeps instances warm.