topic Re: Databricks bundle custom variables feature does not work for workspace host in Administration & Architecture

Databricks bundle custom variables feature does not work for workspace host

AlbertWang — Tue, 17 Sep 2024 23:36:08 GMT

Hi all,

I am not sure whether this is a bug. I followed the document.

My databricks.yml file:

bundle: name: test variables: DATABRICKS_HOST: description: The host of the Databricks workspace. DATABRICKS_JOB_RUNNING_SERVICE_PRINCIPAL_CLIENT_ID: description: The Databricks workspace service principal for running jobs. targets: dev: mode: development default: true workspace: host: ${var.DATABRICKS_HOST} run_as: service_principal_name: ${var.DATABRICKS_JOB_RUNNING_SERVICE_PRINCIPAL_CLIENT_ID}

Then I run the following command:

databricks bundle validate --var="DATABRICKS_HOST=https://aaaaa" --var="DATABRICKS_JOB_RUNNING_SERVICE_PRINCIPAL_CLIENT_ID=asdasdasdaa"

Then I get the following error:

Name: test Target: dev Workspace: Host: ${var.DATABRICKS_HOST}

Looks like the `${var.DATABRICKS_HOST}` custom variable substitution does not work for `workspace.host`.

If I modify my databricks.yml file to use the real value instead variable for `workspace.host`:

bundle: name: test variables: DATABRICKS_HOST: description: The host of the Databricks workspace. DATABRICKS_JOB_RUNNING_SERVICE_PRINCIPAL_CLIENT_ID: description: The Databricks workspace service principal for running jobs. targets: dev: mode: development default: true workspace: host: https://aaaaa run_as: service_principal_name: ${var.DATABRICKS_JOB_RUNNING_SERVICE_PRINCIPAL_CLIENT_ID}

Everything works including `databricks bundle deploy`. That means the custom variable substitution works for other configurations, but not for `workspace.host`.

What do you think?

Thank you.

Regards,

Albert

Re: Databricks bundle custom variables feature does not work for workspace host

Witold — Wed, 18 Sep 2024 07:20:30 GMT

Hosts can't - and shouldn't be - variable. If you have multiple hosts, you need to define them properly in targets.

Re: Databricks bundle custom variables feature does not work for workspace host

filipniziol — Wed, 18 Sep 2024 07:21:40 GMT

Hi @AlbertWang ,

The documentation states you cannot map workspace host:

Re: Databricks bundle custom variables feature does not work for workspace host

AlbertWang — Wed, 18 Sep 2024 20:23:06 GMT

Thank you for your kind reply, @Witold. May I ask why host should not be variable? Is there any specific reason?

I use Azure DevOps pipelines as the CI/CD tool. So I define the host in variable groups. I don't want to define it again in a different location (like the Databricks bundle file).

Thanks again.

Regards,

Albert

Re: Databricks bundle custom variables feature does not work for workspace host

AlbertWang — Wed, 18 Sep 2024 20:25:43 GMT

Thank you, @filipniziol. Even though I don't know why, but since it is a documented feature, it is not a bug 🙂

Re: Databricks bundle custom variables feature does not work for workspace host

Witold — Thu, 19 Sep 2024 08:06:13 GMT

Try to see it from the perspective of DAB, and not Azure DevOps. Your DevOps pipeline is only the orchestrating tool for DAB. Just think of a workspace being an environment, like a production environment. Do you want "anyone from the outside" (outside of DAB) decide what your production is? Is your production environment supposed to change every time you deploy?

Your answers will most likely be no for obvious reasons. And if you really want to change our target, your production environment to another workspace, it's usually associated with other, quite high efforts.