topic Re: how to specify the runtime version for serverless job in Data Engineering

how to specify the runtime version for serverless job

GeKo — Tue, 17 Jun 2025 15:16:48 GMT

Hello,

if I understood correctly.... using a serverless cluster comes always with the latest runtime version, by default.

Now I need to stick to e.g. runtime version 15.4 for a certain job, which gets deployed via asset bundles. How do I specify/configure the job, so that the serverless cluster provides runtime 15.4 ?

any help highly apprecited 😁

#serverless #assetbundles

Re: how to specify the runtime version for serverless job

lingareddy_Alva — Tue, 17 Jun 2025 16:27:09 GMT

Hi @GeKo

You're correct that serverless clusters typically default to the latest runtime version, but you can specify a particular runtime version for your jobs.
The exact method depends on your platform, but here are the common approaches:

Databricks (Most Common for Asset Bundles)
In your bundle configuration (databricks.yml or job definition):

For serverless compute specifically:

Re: how to specify the runtime version for serverless job

jesseryoung — Tue, 17 Jun 2025 18:25:39 GMT

HI @GeKo,

Serverless versions behave a bit differently than classic runtime versions. With serverless you no longer have control over which cluster runtime is used as it's continuously updated, you do however have control over which API to target and what your base environment looks like by setting the environment version. Take a look at the release notes for serverless versions here.

For notebooks, you need to define this environment directly in the notebook even when they're scheduled to run in jobs. You can see how to do that here.

For non notebook tasks, here's an example of specifying the environment for a task:

resources: jobs: example_job: name: example job tasks: - task_key: example_task ... environment_key: some_environment_key environments: - environment_key: some_environment_key spec: client: "2"

Re: how to specify the runtime version for serverless job

jesseryoung — Tue, 17 Jun 2025 18:39:49 GMT

Hi @lingareddy_Alva - The second DAB example you provided is not valid. I believe the LLM you used to generate this code may have hallucinated.

Re: how to specify the runtime version for serverless job

GeKo — Wed, 18 Jun 2025 10:03:29 GMT

Hi @lingareddy_Alva ,

many thanks for answering!

The serverless "solution" you provided is unfortunately just a plain response from chatgpt/gemini/etc .... I tried that as well and the AI response is pure nonsense .... as also commented by jesseryoung

Re: how to specify the runtime version for serverless job

GeKo — Wed, 18 Jun 2025 10:05:34 GMT

Many thanks @jesseryoung ,

that's exactly what I was looking for 👌

Re: how to specify the runtime version for serverless job

GeKo — Wed, 18 Jun 2025 13:29:39 GMT

Hi @jesseryoung ,

one quick followup question. I am playing aroung with the "environment" property in job config, and I have the following:

resources: jobs: serverless_testjob: name: serverless_testjob tasks: - task_key: sample_script_15_4 spark_python_task: python_file: /Workspace/Users/gerd/get_version.py environment_key: serverless_15_4 - task_key: sample_script_14_3 spark_python_task: python_file: /Workspace/Users/gerd/get_version.py environment_key: serverless_14_3 queue: enabled: true environments: - environment_key: serverless_14_3 spec: client: "1" - environment_key: serverless_15_4 spec: client: "2" performance_target: STANDARD

both tasks are executing the same python script, but with different environments. The script itself is pretty simple and looks like:

import sys from pyspark.sql import SparkSession python_version = sys.version print(f"The current Python version is: {python_version}") spark = SparkSession.builder.getOrCreate() spark_version = spark.version print(f"Apache Spark Version: {spark_version}") current_version_info = spark.sql("SELECT current_version()").collect()[0][0] dbr_version = current_version_info['dbr_version'] print(f"Databricks Runtime Version: {dbr_version}")

Now I am wondering why both tasks output for "Databricks Runtime Version" is => "Databricks Runtime Version: 16.4.x-photon-scala2.12" ?!?!

According the the serverless release notes regarding environments (LINK), I expected a different runtime for each environment. The output of both tasks only show differences in the used python version :

output task sample_script_15_4

The current Python version is: 3.11.10 (main, Sep 7 2024, 18:35:41) [GCC 11.4.0] Apache Spark Version: 3.5.2 Databricks Runtime Version: 16.4.x-photon-scala2.12

output task sample_script_14_3

The current Python version is: 3.10.12 (main, Feb 4 2025, 14:57:36) [GCC 11.4.0] Apache Spark Version: 3.5.2 Databricks Runtime Version: 16.4.x-photon-scala2.12

This combination of versions in the outputs doesn't make any sense to me. I mean...I expected a different python version in the different environments, but runtime 16.4 is unexpected and should be contained only in environment version "3".

What is going wrong here ?

Re: how to specify the runtime version for serverless job

jesseryoung — Wed, 18 Jun 2025 15:15:25 GMT

With serverless, changing the "version" doesn't actually change the cluster runtime, it just makes sure that your code doesn't break over time while allowing Databricks to keep the cluster runtime up to date with the latest changes. Serverless makes use of Spark Connect to break up the cluster runtime dependencies (DBR) and client dependencies (serverless environments.)

Re: how to specify the runtime version for serverless job

GeKo — Wed, 18 Jun 2025 15:35:47 GMT

👍