Hi @jesseryoung ,

one quick followup question. I am playing aroung with the "environment" property in job config, and I have the following:

resources:
  jobs:
    serverless_testjob:
      name: serverless_testjob
      tasks:
        - task_key: sample_script_15_4
          spark_python_task:
            python_file: /Workspace/Users/gerd/get_version.py
          environment_key: serverless_15_4
        - task_key: sample_script_14_3
          spark_python_task:
            python_file: /Workspace/Users/gerd/get_version.py
          environment_key: serverless_14_3
      queue:
        enabled: true
      environments:
        - environment_key: serverless_14_3
          spec:
            client: "1"
        - environment_key: serverless_15_4
          spec:
            client: "2"
      performance_target: STANDARD

both tasks are executing the same python script, but with different environments. The script itself is pretty simple and looks like:

import sys
from pyspark.sql import SparkSession

python_version = sys.version
print(f"The current Python version is: {python_version}")

spark = SparkSession.builder.getOrCreate()
spark_version = spark.version
print(f"Apache Spark Version: {spark_version}")

current_version_info = spark.sql("SELECT current_version()").collect()[0][0]
dbr_version = current_version_info['dbr_version']
print(f"Databricks Runtime Version: {dbr_version}")

Now I am wondering why both tasks output for "Databricks Runtime Version" is => "Databricks Runtime Version: 16.4.x-photon-scala2.12" ?!?!

According the the serverless release notes regarding environments (LINK), I expected a different runtime for each environment. The output of both tasks only show differences in the used python version :

output task sample_script_15_4

The current Python version is: 3.11.10 (main, Sep  7 2024, 18:35:41) [GCC 11.4.0]
Apache Spark Version: 3.5.2
Databricks Runtime Version: 16.4.x-photon-scala2.12

output task sample_script_14_3

The current Python version is: 3.10.12 (main, Feb  4 2025, 14:57:36) [GCC 11.4.0]
Apache Spark Version: 3.5.2
Databricks Runtime Version: 16.4.x-photon-scala2.12

 This combination of versions in the outputs doesn't make any sense to me. I mean...I expected a different python version in the different environments, but runtime 16.4 is unexpected and should be contained only in environment version "3".

What is going wrong here ?