Valid Workspace Conf keys

[mathiaskvist]

Hi

I'm trying to automate the configuration of Admin Settings of our Databricks Workspace using Terraform. However identifying the correct config keys is very difficult.
Databricks exposes a Workspace Conf API (Enable/disable features | Workspace Conf API | REST API reference | Databricks on AWS) and a python client Workspace Conf — Databricks SDK for Python alpha documentation (databricks-sdk-py.readthedocs.io)

However, none of these document which keys are supported. The Terraform provider for Databricks has a few mentioned, but not the ones I'm looking for databricks_workspace_conf | Resources | databrickslabs/databricks | Terraform | Terraform Registry.

I'm trying to configure the Workspace Access Control and the Cluster, Pool and Job Access Control. 

Looking at the Databricks UI the config is fetched from https://adb-X.X.azuredatabricks.net/config and in the payload a few keys of interest are shown:

- enforceWorkspaceViewAcls
- enableWorkspaceAclsConfig
- enableClusterAclsConfig
- enableJobAclsConfig
- enableJobViewAcls

However, when I try to run the following:

curl -X PATCH \

    --header 'content-type:application/json' \

    --header 'authorization:Bearer XXX' \

    -d '{"enableJobViewAcls": "true", "enforceWorkspaceViewAcls": "true", "enableClusterAclsConfig": "true", "enableJobAclsConfig": "true", "enableWorkspaceAclsConfig": "true"}' \

    https://adb-XXXX.azuredatabricks.net/api/2.0/workspace-conf

it returns: 

{"error_code":"BAD_REQUEST","message":"Invalid keys: [\"enableJobAclsConfig\",\"enableClusterAclsConfig\",\"enableWorkspaceAclsConfig\"]"}

If I try to use these keys in Terraform I also get an error. Furthermore, if seems to accept some of the keys, but don't make any changes.

How can I identify the correct keys to use with Terraform or using the API if Terraform is not supported?

[Corbin · Accepted solution]

Hey Mathias!

Some of the configs are documented in tf here as well as an example. You can list multiple confs in the custom_config block.

I've raised the lack of documentation around this with the team and it is a known issue. We are actually hoping to deprecate many of these configs and just enforce the sensible default, hence why there is reticence to document it.

I believe the values you are trying to set these to is already the default, and will eventually be unconfigurable, so it may not be worth spending too many cycles on this. Have you confirmed that these values are not the defaults?

[mathiaskvist]

Thanks for investigating with the team. It seems to me that the default is not what I need. By default the access control is turned off. So this still leaves me without a way to enable it from Terraform. Is this what the team intends?

[Corbin]

enableJobAclsConfig, enableWorkspaceAclsConfig, enableClusterAclsConfig are the keys enabling ACL (and are default to ON for workspaces created after some time in 2020). Are you seeing something different on your end? What makes you think ACLs are turned off by default? (FYI, ACLs are only available in general in Premium and Enterprise tiers)

[mathiaskvist]

Just reprovisioned a premium workspace and you are right. The ACL's are enabled by default.

Before my workspace was a premium workspace upgraded from standard. So that could be why the settings were disabled.

[david-sanabria]

We follow an "infrastructure as code" model for Workspace deployment, but the limited documentation means that we end up configuring most settings by hand, which is a tedious and time-consuming process. This may be fine for small companies that are only using Databricks to handle non-sensitive data, but our organization is highly regulated and required to enforce NIST 800-53 standards.

It doesn't help that Databricks does not include default configuration maps that implement this compliance, but the absence of configurable settings through an API is honestly inexcusable and hard to justify to our oversight organizations because we cannot run automation to set the values to spec, nor can we run automation to monitor (i.e. get) ongoing compliance of existing workspaces by evaluating their current values.

Databricks must do better if it wants to stay competitive with entrenched cloud competitors' (i.e. AWS and Azure) data management and analysis features. I need Databricks to be better so my teams are not forced to stop using it.

[Corbin]

Hello David,

I certainly agree, and I am pushing internally to get this better documented. Can you please tell me which settings are not correctly set by default for your compliance and I can look to push for those sooner rather than later?

[Alexis_Chicoine]

I wanted to know the key for Store interactive notebook results in customer account.
It's not ideal but by using the browser dev tools you can find out what it is by looking at the network activity after toggling it in the UI.