Databricks Asset Bundle - Job Cluster - JDBC HTTP Path

[tonypiazza]

I am currently working on deploying dbt jobs using a Databricks Asset Bundle. In my existing job configuration, I am using an all-purpose cluster and the JDBC HTTP Path was manually copied from the web UI. Now that I am trying to switch to using a job cluster and deploying everything via DAB, I'm unsure how I would reference the HTTP Path using substitutions. Is there any variable which refers to the current resource? Something like "${this}"?

[Satyadeepak · Accepted solution]

To reference the HTTP Path using substitutions in Databricks Asset Bundles and job clusters, you can use the variables section in your databricks.yml configuration file

In your databricks.yml file, you can define a variable for the HTTP Path. For example:

bundle:
  name: my-databricks-bundle
  variables:
    http_path:
      description: "HTTP Path for the Databricks cluster"
      default: "your-http-path-here"

You can then reference this variable in other parts of your configuration file using the ${variables.<variable-name>} syntax. For example, if you need to use the HTTP Path in a job configuration, you can do it like this:

resources:
  jobs:
    my-job:
      name: "My Job"
      tasks:
        - task_key: my-task
          existing_cluster_id: "your-cluster-id"
          notebook_task:
            notebook_path: "/path/to/your/notebook"
            base_parameters:
              http_path: "${variables.http_path}"

When deploying the bundle, the variable ${variables.http_path} will be substituted with the value defined in the variables section.

https://docs.databricks.com/en/dev-tools/bundles/settings.html#variables