- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-07-2026 08:16 PM
Hi @Ajay-Pandey
only solution for you
1. Create an all-purpose cluster called for example:
continuous-job-cluster and Disable auto-termination or set it to a large value.
2. Configure job to use existing_cluster_id
In Jobs UI or DAB YAML:
existing_cluster_id: <cluster-id-of-continuous-job-cluster>
Now:
a. The cluster stays alive
b. Your continuous trigger reuses the same compute
c. No cold starts
d. No cluster recreation
3. For streaming workloads
Instead of continuous jobs, write your notebook as a streaming query:
spark.readStream → writeStream.start() → awaitTermination()
Run it on the existing cluster and let Spark manage the lifecycle.
This is how Databricks expects continuous pipelines to run.