Databricks bundle custom variables feature does not work for workspace host

[AlbertWang]

Hi all,

I am not sure whether this is a bug. I followed the document.

My databricks.yml file:

bundle:
  name: test

variables:
  DATABRICKS_HOST:
    description: The host of the Databricks workspace.
  DATABRICKS_JOB_RUNNING_SERVICE_PRINCIPAL_CLIENT_ID:
    description: The Databricks workspace service principal for running jobs.

targets:
  dev:
    mode: development
    default: true
    workspace:
      host: ${var.DATABRICKS_HOST}
    run_as:
      service_principal_name: ${var.DATABRICKS_JOB_RUNNING_SERVICE_PRINCIPAL_CLIENT_ID}

Then I run the following command:

databricks bundle validate --var="DATABRICKS_HOST=https://aaaaa" --var="DATABRICKS_JOB_RUNNING_SERVICE_PRINCIPAL_CLIENT_ID=asdasdasdaa"

Then I get the following error:

Name: test
Target: dev
Workspace:
  Host: ${var.DATABRICKS_HOST}

Looks like the `${var.DATABRICKS_HOST}` custom variable substitution does not work for `workspace.host`.

If I modify my databricks.yml file to use the real value instead variable for `workspace.host`:

bundle:
  name: test

variables:
  DATABRICKS_HOST:
    description: The host of the Databricks workspace.
  DATABRICKS_JOB_RUNNING_SERVICE_PRINCIPAL_CLIENT_ID:
    description: The Databricks workspace service principal for running jobs.

targets:
  dev:
    mode: development
    default: true
    workspace:
      host: https://aaaaa
    run_as:
      service_principal_name: ${var.DATABRICKS_JOB_RUNNING_SERVICE_PRINCIPAL_CLIENT_ID}

Everything works including `databricks bundle deploy`. That means the custom variable substitution works for other configurations, but not for `workspace.host`.

What do you think?

Thank you.

Regards,

Albert

[Witold]

Hosts can't - and shouldn't be - variable. If you have multiple hosts, you need to define them properly in targets.

[AlbertWang]

Thank you for your kind reply, @Witold. May I ask why host should not be variable? Is there any specific reason?

I use Azure DevOps pipelines as the CI/CD tool. So I define the host in variable groups. I don't want to define it again in a different location (like the Databricks bundle file).

Thanks again.

Regards,

Albert

[Witold]

Try to see it from the perspective of DAB, and not Azure DevOps. Your DevOps pipeline is only the orchestrating tool for DAB. Just think of a workspace being an environment, like a production environment. Do you want "anyone from the outside" (outside of DAB) decide what your production is? Is your production environment supposed to change every time you deploy?

Your answers will most likely be no for obvious reasons. And if you really want to change our target, your production environment to another workspace, it's usually associated with other, quite high efforts.

[filipniziol · Accepted solution]

Hi @AlbertWang ,

The documentation states you cannot map workspace host:

[AlbertWang]

Thank you, @filipniziol. Even though I don't know why, but since it is a documented feature, it is not a bug 🙂