cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

Delta Live Tables Development Mode Resets Cluster On Each Trigger

lurban
New Contributor

I believe this is a bug identified, but in the last few days, each time I trigger a test Delta Live Tables run in Development mode, the associated cluster will take 5-7 minutes to spin up each time. The cluster does stay on as anticipated in the compute -> job clusters window (shown in the image), but when I click "start" on my pipeline, it resets the cluster and starts spinning up again for another few minutes and DLT stays on the "Waiting for Resources" state for several minutes. I have also specified the ClusterShutdown.Delay configuration settings in DLT on Production mode and get the same issue.

I have encountered this issue in both the current and preview channels.

Has anyone come across this issue before? Would appreciate any insight. Thanks in advance!

1 REPLY 1

jose_gonzalez
Moderator
Moderator

Hi,

Can you share your cluster JSON settings? it will help us to undertand the settings and VMs you are using.

Welcome to Databricks Community: Lets learn, network and celebrate together

Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections. 

Click here to register and join today! 

Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.