Cluster environment variables not visible to Java Dataset.foreachPartition / mapPartitions

[capoorhimanshu]

We run a Spark JAR task (Java) on a standard access mode cluster. Credentials come from cluster environment
variables set as secret references in the cluster's Environment variables field, e.g.
MY_PASSWORD={{secrets/<scope>/<key>}}.

The variables are visible to the JAR task's driver code, but not to code running inside Dataset.foreachPartition /
mapPartitions on the workers (we only check presence, never values):

- DBR 16.4 LTS (Scala 2.12), standard access mode: inside foreachPartition the code runs as process "3@sandbox" and sees only 27 variables (DBR_VERSION, DB_*, HOME, LANG, PATH, PWD, SHLVL, payload_format, stageId, taskAttemptId). None of the cluster-configured variables are present, including a plain non-secret one. On the driver all are present.

- DBR 19 (Scala 2.13, Spark 4.2), standard access mode: driver sees MY_PASSWORD, mapPartitions code does not. The cluster edit page says "Databricks Runtime 19 and above restricts environment variables. The following
variables will be available in user code, but not loaded into the Spark session", listing our variables.

The standard compute limitations page says variables outside the predefined set "remain available to your user code,including UDFs", which does not match what we see.


Minimal reproduction (Java JAR task, MY_PASSWORD set as a cluster environment variable):

SparkSession spark = SparkSession.builder().getOrCreate();
boolean onDriver = System.getenv("MY_PASSWORD") != null; // true
List<String> onWorkers = spark.range(0, 2, 1, 2)
.mapPartitions((MapPartitionsFunction<Long, String>) rows -> Collections.singletonList(
String.valueOf(System.getenv("MY_PASSWORD") != null)).iterator(), Encoders.STRING())
.collectAsList();


Questions:
1. Is this expected for Scala/Java Dataset.foreachPartition / mapPartitions code on standard access mode?
2. Is there a supported way to make specific cluster environment variables visible to that code?

[SumeshKashyap]

Hello @capoorhimanshu 

On standard access mode this is expected. Scala/Java closures (foreachPartition, mapPartitions, UDFs) don't run in the executor JVM where the cluster env vars are set. They run in an isolated per-user sandbox process with a fixed, minimal environment. That's the 3@sandbox process and the 27 vars you're seeing. On DBR 19 spark.executorEnv.* is also restricted on standard, so you can't push the vars that way. The supported fix is to read the value on the driver and pass it into the closure, or to run the job on dedicated access mode.

Option 1: capture on the driver. This works on standard mode, DBR 16.4 and 19:

final String password = System.getenv("MY_PASSWORD");   // driver: present

ds.foreachPartition((ForeachPartitionFunction<Row>) rows -> {

    // 'password' is serialized with the task closure

    try (Connection c = DriverManager.getConnection(url, user, password)) {

        while (rows.hasNext()) { /* ... */ }

    }

});

Keep it in a local final variable (not a static field), or the sandbox will just see null. It travels inside the serialized task, so don't log or toString() the closure. Spark's redaction won't touch it there.

Option 2: dedicated (single user) access mode. Executors there run your code in their own JVM, which gets the cluster's environment variables, so System.getenv("MY_PASSWORD") works inside mapPartitions unchanged. This is the cleaner choice if the job runs as a service principal anyway.

About the docs line saying non-predefined variables "remain available to your user code, including UDFs": your repro shows that doesn't hold for JVM closures on standard. It's worth sending docs feedback on https://docs.databricks.com/aws/en/compute/standard-limitations so it gets clarified.

If this helps you and solves your problem , kindly accept them as answer so that community can be benefited