Databricks Asset Bundle - Job Cluster - JDBC HTTP Path

tonypiazza — Wed, 21 Aug 2024 20:16:00 GMT

I am currently working on deploying dbt jobs using a Databricks Asset Bundle. In my existing job configuration, I am using an all-purpose cluster and the JDBC HTTP Path was manually copied from the web UI. Now that I am trying to switch to using a job cluster and deploying everything via DAB, I'm unsure how I would reference the HTTP Path using substitutions. Is there any variable which refers to the current resource? Something like "${this}"?

Re: Databricks Asset Bundle - Job Cluster - JDBC HTTP Path

Satyadeepak — Thu, 16 Jan 2025 18:01:58 GMT

To reference the HTTP Path using substitutions in Databricks Asset Bundles and job clusters, you can use the variables section in your databricks.yml configuration file

In your databricks.yml file, you can define a variable for the HTTP Path. For example:

bundle:
  name: my-databricks-bundle
  variables:
    http_path:
      description: "HTTP Path for the Databricks cluster"
      default: "your-http-path-here"

You can then reference this variable in other parts of your configuration file using the ${variables.<variable-name>} syntax. For example, if you need to use the HTTP Path in a job configuration, you can do it like this:

resources:
  jobs:
    my-job:
      name: "My Job"
      tasks:
        - task_key: my-task
          existing_cluster_id: "your-cluster-id"
          notebook_task:
            notebook_path: "/path/to/your/notebook"
            base_parameters:
              http_path: "${variables.http_path}"

When deploying the bundle, the variable ${variables.http_path} will be substituted with the value defined in the variables section.

https://docs.databricks.com/en/dev-tools/bundles/settings.html#variables

topic Databricks Asset Bundle - Job Cluster - JDBC HTTP Path in Data Engineering

Databricks Asset Bundle - Job Cluster - JDBC HTTP Path

Re: Databricks Asset Bundle - Job Cluster - JDBC HTTP Path