topic Re: Is there a way to control the cluster runtime version for DLT in Data Engineering

Is there a way to control the cluster runtime version for DLT

MikeGo — Fri, 23 Feb 2024 07:31:17 GMT

Hi team,

When I create a DLT job, is there a way to control the cluster runtime version somewhere? E.g. I want to use 14.3 LTS. I tried to add `"spark_version": "14.3.x-scala2.12",` inside cluster default label but not work.

Thanks

Re: Is there a way to control the cluster runtime version for DLT

ezhil — Fri, 23 Feb 2024 07:54:46 GMT

If you are running as a notebook you can change it while creating the cluster under
Performance -> Databricks Runtime Version you can select the needed DBR version.

Re: Is there a way to control the cluster runtime version for DLT

MikeGo — Fri, 23 Feb 2024 18:26:08 GMT

Thanks. I mean running DLT, not run a cell from notebook sourced for DLT.

Re: Is there a way to control the cluster runtime version for DLT

BamBam — Tue, 27 Feb 2024 01:48:38 GMT

Hi Brad,

I don't think you can because if you look at a recent DBRS article here (that apparently it won't let me link) it says in the "DLT Compute" section: "DLT will manage and optimize the node type and DBR selection, ensuring the best choice of nodes and the latest DBR runtime are selected, reducing the management overhead for users. "

Thanks.

Re: Is there a way to control the cluster runtime version for DLT

MinThuraZaw — Tue, 27 Feb 2024 03:17:33 GMT

Hello @MikeGo ,

This page listed cluster attributes that you cannot change and why, including spark_version.

https://docs.databricks.com/en/delta-live-tables/properties.html#cluster-attributes-that-are-not-user-settable

Re: Is there a way to control the cluster runtime version for DLT

MikeGo — Tue, 27 Feb 2024 23:20:31 GMT

Thanks. Got it.

And the cluster has to be share mode. Can different DLT jobs share clusters or when DLT job is running, can other people use the cluster? Seems each DLT job running will start a new cluster. If it is not be able to shared, why it has to be share mode?