mukul1409
Contributor II

Hi @Ajay-Pandey 
only solution for you 

1. Create an all-purpose cluster called for example:
     continuous-job-cluster and Disable auto-termination or set it to a large value.

2. Configure job to use existing_cluster_id
   
In Jobs UI or DAB YAML:
   existing_cluster_id: <cluster-id-of-continuous-job-cluster>
   Now:
    a. The cluster stays alive
    b. Your continuous trigger reuses the same compute
    c. No cold starts
    d. No cluster recreation

3. For streaming workloads 
Instead of continuous jobs, write your notebook as a streaming query:
spark.readStream → writeStream.start() → awaitTermination()
Run it on the existing cluster and let Spark manage the lifecycle.
This is how Databricks expects continuous pipelines to run.

Mukul Chauhan