cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

Is there a way to control the cluster runtime version for DLT

Brad
New Contributor III

Hi team, 

When I create a DLT job, is there a way to control the cluster runtime version somewhere? E.g. I want to use 14.3 LTS. I tried to add `"spark_version": "14.3.x-scala2.12",` inside cluster default label but not work.

Thanks

5 REPLIES 5

ezhil
New Contributor III

If you are running as a notebook you can change it while creating the cluster under 
Performance -> Databricks Runtime Version  you can select the needed DBR version.

 

Brad
New Contributor III

Thanks. I mean running DLT, not run a cell from notebook sourced for DLT.

BamBam
New Contributor II

Hi Brad,

I don't think you can because if you look at a recent DBRS article here (that apparently it won't let me link) it says in the "DLT Compute" section: "DLT will manage and optimize the node type and DBR selection, ensuring the best choice of nodes and the latest DBR runtime are selected, reducing the management overhead for users. "

Thanks.

MinThuraZaw
New Contributor III

Hello @Brad ,

This page listed cluster attributes that you cannot change and why, including spark_version. 

https://docs.databricks.com/en/delta-live-tables/properties.html#cluster-attributes-that-are-not-use...

 

Brad
New Contributor III

Thanks. Got it.

And the cluster has to be share mode. Can different DLT jobs share clusters or when DLT job is running, can other people use the cluster? Seems each DLT job running will start a new cluster. If it is not be able to shared, why it has to be share mode?

Welcome to Databricks Community: Lets learn, network and celebrate together

Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections. 

Click here to register and join today! 

Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.