twole
Databricks Employee
Databricks Employee

Hi,

This is generally how dbt profiles.yml works when using it with Databricks Lakeflow Jobs:

  • When you specify a warehouse, catalog etc, the "Profiles Directory" option is removed because Databricks automatically creates the "profiles.yml" required to run that job in the background, based on your configuration. That's why the option isn't there.
  • If you don't specify a warehouse, the UI gives you the option to input your profiles.yml directory. Databricks injects the DBT_ACCESS_TOKEN for the principal configured in the Run As field of the job. So your profiles.yml file could look like this:
jaffle_shop:
  target: dev
  outputs:
    dev:
      type: databricks
      method: http
      catalog: <CATALOG-NAME>
      schema: <SCHEMA-NAME>
      http_path: <WAREHOUSE-PATH>
      # The workspace host / token are provided by Databricks
      host: "{{ env_var('DBT_HOST') }}"
      token: "{{ env_var('DBT_ACCESS_TOKEN') }}"