@AnkurMittal008
There is no specific requirement to have Serverless to enable Predictive Optimization. You must fulfill the following requirements to enable predictive optimization:
Your Databricks workspace must be on the Premium plan or above in ...
@PKD28
The error indicates that the driver memory is not enough to handle the load.Please refer to this doc for more info and on how to fix thishttps://kb.databricks.com/en_US/jobs/driver-unavailable
@abduldjafar Use this general doc to optimize your workload based on your job analysis
https://www.databricks.com/discover/pages/optimize-data-workloads-guide
@skumarrm Please try the below:
Set Up Task Parameters:
In the job configuration, you can set up task parameters to pass values from one task to another.For TASK1 (DLT), ensure it outputs the PipelineID or PipelineName.
Use Task Parameters in TASK2:...
@theanhdo
To schedule a job to run at 8 AM every day, you should use the Scheduled trigger type rather than the Continuous trigger type. The Continuous trigger type is designed to keep a job running continuously, which is not suitable for running a ...