BUG in Job Task of Type DBT

[tom_1]

Hi, just wanted to let the Databricks Team know, that there is a bug in the task ui.
Currently it is not possible to save a task of "Type: dbt" if the "SQL Warehouse" is set to "None (Manual)".
Some weeks ago this was possible, also the "Profiles Directory" field is now missing (which is not a problem because it can be set using dbt run --profiles-dir, but still a change which confused me in the beginning).
Instead Warehouse Catalog and Schema must now be defined. Which makes no sense because the catalog and schema is actually defined in dbt's profiles.yml.

It would be great if this bug could be fixed, such that "SQL Warehouse: None (Manual)" is a valid option again.

UI Error:

Error updating Main Pipeline

No result returned

Edit as Yaml Error:

Could not update job

Catalog can only be defined if the warehouseId is defined

Old View

New View

[Alberto_Umana]

Hello @tom_1,

Thanks for the detailed information provided, I will check and report this internally.

[tom_1 · Accepted solution]

Its now working as before, thanks for fixing it so fast!

[Gerard_Sola]

This bug has appeared again in Azure Databricks. Could you please review it?

Thanks!

[Aishu95]

I am facing this bug still. I don't want to select any SQL warehouse, what do I do? and from where can I pass the profiles directory

[twole]

Hi,

This is generally how dbt profiles.yml works when using it with Databricks Lakeflow Jobs:

• When you specify a warehouse, catalog etc, the "Profiles Directory" option is removed because Databricks automatically creates the "profiles.yml" required to run that job in the background, based on your configuration. That's why the option isn't there.
• If you don't specify a warehouse, the UI gives you the option to input your profiles.yml directory. Databricks injects the DBT_ACCESS_TOKEN for the principal configured in the Run As field of the job. So your profiles.yml file could look like this:

jaffle_shop:
  target: dev
  outputs:
    dev:
      type: databricks
      method: http
      catalog: <CATALOG-NAME>
      schema: <SCHEMA-NAME>
      http_path: <WAREHOUSE-PATH>
      # The workspace host / token are provided by Databricks
      host: "{{ env_var('DBT_HOST') }}"
      token: "{{ env_var('DBT_ACCESS_TOKEN') }}"