BUG in Job Task of Type DBT

tom_1
New Contributor III

‎03-13-2025 06:02 AM

Hi, just wanted to let the Databricks Team know, that there is a bug in the task ui.
Currently it is not possible to save a task of "Type: dbt" if the "SQL Warehouse" is set to "None (Manual)".
Some weeks ago this was possible, also the "Profiles Directory" field is now missing (which is not a problem because it can be set using dbt run --profiles-dir, but still a change which confused me in the beginning).
Instead Warehouse Catalog and Schema must now be defined. Which makes no sense because the catalog and schema is actually defined in dbt's profiles.yml.

It would be great if this bug could be fixed, such that "SQL Warehouse: None (Manual)" is a valid option again.

UI Error:
Error updating Main Pipeline
No result returned

Edit as Yaml Error:
Could not update job
Catalog can only be defined if the warehouseId is defined

Old View
tom_1_0-1741870684542.png

New View

tom_1_1-1741870779606.png

 

0 Kudos

Alberto_Umana
Databricks Employee
Databricks Employee

‎03-13-2025 06:41 AM

Hello @tom_1,

Thanks for the detailed information provided, I will check and report this internally.

tom_1
New Contributor III

‎03-13-2025 09:16 AM

Its now working as before, thanks for fixing it so fast!

View solution in original post

0 Kudos

Gerard_Sola
New Contributor II

‎03-17-2025 03:31 AM

This bug has appeared again in Azure Databricks. Could you please review it?

Thanks!

0 Kudos

Aishu95
New Contributor II

‎07-01-2025 05:59 AM

I am facing this bug still. I don't want to select any SQL warehouse, what do I do? and from where can I pass the profiles directory

0 Kudos

twole
Databricks Employee
Databricks Employee
In response to Aishu95

‎09-29-2025 08:39 AM - edited ‎09-29-2025 08:42 AM

Hi,

This is generally how dbt profiles.yml works when using it with Databricks Lakeflow Jobs:

  • When you specify a warehouse, catalog etc, the "Profiles Directory" option is removed because Databricks automatically creates the "profiles.yml" required to run that job in the background, based on your configuration. That's why the option isn't there.
  • If you don't specify a warehouse, the UI gives you the option to input your profiles.yml directory. Databricks injects the DBT_ACCESS_TOKEN for the principal configured in the Run As field of the job. So your profiles.yml file could look like this:
jaffle_shop:
  target: dev
  outputs:
    dev:
      type: databricks
      method: http
      catalog: <CATALOG-NAME>
      schema: <SCHEMA-NAME>
      http_path: <WAREHOUSE-PATH>
      # The workspace host / token are provided by Databricks
      host: "{{ env_var('DBT_HOST') }}"
      token: "{{ env_var('DBT_ACCESS_TOKEN') }}"

 

0 Kudos