cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Is there a way to control the cluster runtime version for DLT

Brad
Contributor

Hi team, 

When I create a DLT job, is there a way to control the cluster runtime version somewhere? E.g. I want to use 14.3 LTS. I tried to add `"spark_version": "14.3.x-scala2.12",` inside cluster default label but not work.

Thanks

5 REPLIES 5

ezhil
New Contributor III

If you are running as a notebook you can change it while creating the cluster under 
Performance -> Databricks Runtime Version  you can select the needed DBR version.

 

Brad
Contributor

Thanks. I mean running DLT, not run a cell from notebook sourced for DLT.

BamBam
New Contributor II

Hi Brad,

I don't think you can because if you look at a recent DBRS article here (that apparently it won't let me link) it says in the "DLT Compute" section: "DLT will manage and optimize the node type and DBR selection, ensuring the best choice of nodes and the latest DBR runtime are selected, reducing the management overhead for users. "

Thanks.

MinThuraZaw
New Contributor III

Hello @Brad ,

This page listed cluster attributes that you cannot change and why, including spark_version. 

https://docs.databricks.com/en/delta-live-tables/properties.html#cluster-attributes-that-are-not-use...

 

Brad
Contributor

Thanks. Got it.

And the cluster has to be share mode. Can different DLT jobs share clusters or when DLT job is running, can other people use the cluster? Seems each DLT job running will start a new cluster. If it is not be able to shared, why it has to be share mode?

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group