topic Re: BUG in Job Task of Type DBT in Data Engineering

BUG in Job Task of Type DBT

tom_1 — Thu, 13 Mar 2025 13:02:22 GMT

Hi, just wanted to let the Databricks Team know, that there is a bug in the task ui.
Currently it is not possible to save a task of "Type: dbt" if the "SQL Warehouse" is set to "None (Manual)".
Some weeks ago this was possible, also the "Profiles Directory" field is now missing (which is not a problem because it can be set using dbt run --profiles-dir, but still a change which confused me in the beginning).
Instead Warehouse Catalog and Schema must now be defined. Which makes no sense because the catalog and schema is actually defined in dbt's profiles.yml.

It would be great if this bug could be fixed, such that "SQL Warehouse: None (Manual)" is a valid option again.

UI Error:

Error updating Main Pipeline

No result returned

Edit as Yaml Error:

Could not update job

Catalog can only be defined if the warehouseId is defined

Old View

New View

Re: BUG in Job Task of Type DBT

Alberto_Umana — Thu, 13 Mar 2025 13:41:35 GMT

Hello @tom_1,

Thanks for the detailed information provided, I will check and report this internally.

Re: BUG in Job Task of Type DBT

tom_1 — Thu, 13 Mar 2025 16:16:19 GMT

Its now working as before, thanks for fixing it so fast!

Re: BUG in Job Task of Type DBT

Gerard_Sola — Mon, 17 Mar 2025 10:31:56 GMT

This bug has appeared again in Azure Databricks. Could you please review it?

Thanks!

Re: BUG in Job Task of Type DBT

Aishu95 — Tue, 01 Jul 2025 12:59:33 GMT

I am facing this bug still. I don't want to select any SQL warehouse, what do I do? and from where can I pass the profiles directory

Re: BUG in Job Task of Type DBT

twole — Mon, 29 Sep 2025 15:42:35 GMT

Hi,

This is generally how dbt profiles.yml works when using it with Databricks Lakeflow Jobs:

When you specify a warehouse, catalog etc, the "Profiles Directory" option is removed because Databricks automatically creates the "profiles.yml" required to run that job in the background, based on your configuration. That's why the option isn't there.
If you don't specify a warehouse, the UI gives you the option to input your profiles.yml directory. Databricks injects the DBT_ACCESS_TOKEN for the principal configured in the Run As field of the job. So your profiles.yml file could look like this:

jaffle_shop: target: dev outputs: dev: type: databricks method: http catalog: <CATALOG-NAME> schema: <SCHEMA-NAME> http_path: <WAREHOUSE-PATH> # The workspace host / token are provided by Databricks host: "{{ env_var('DBT_HOST') }}" token: "{{ env_var('DBT_ACCESS_TOKEN') }}"