topic Re: Restrictions on setting environment variables in Compute Policies in Administration & Architecture

Restrictions on setting environment variables in Compute Policies

tcmx — Mon, 14 Oct 2024 08:58:00 GMT

As recommended by Databricks, we are trying to use Compute Policies to set environment variables, which are used by our notebooks, across clusters.

However, when specifying a JSON string as env var, we are getting this error upon applying the policy to a shared cluster:

Cannot update compute configuration:
$$, \, ", ', | and backticks are not allowed in the environment variable value for Unity Catalog shared clusters. Please use single user clusters instead.

Where is this error coming from? It's not documented anywhere, and seems arbitrary.
What are possible ways around this?

Re: Restrictions on setting environment variables in Compute Policies

-werners- — Mon, 14 Oct 2024 15:14:51 GMT

This is because you use Shared access mode.
This enables multiple users to use the cluster simultaneously.
However, there are features that do not work on these Shared access mode clusters:
https://docs.databricks.com/en/compute/access-mode-limitations.html#shared-access-mode-limitations-on-unity-catalog
The limitation about backticks in env vars is not mentioned but the error message is clear: it is not supported.
To overcome this limitation, use Single user as access mode.