Running dbt macro inside Databricks notebook as a part of Workflow

JakubII — Tue, 01 Aug 2023 09:09:04 GMT

So.. I have Databricks and I'm using dbt-databricks adapter to run dbt. Everything works fine but for some reasons I need to run some stuff inside of notebook. That still worked fine for some time but after upgrading to dbt-databricks 1.5.5 something broke and I can't figure it what exactly. It works when downgrading but I would like to work with newest versions.

Workload always fails when I'm calling out dbf macro (run-operation ..). Weird think is that notebook independently works fine even with macro but inside of workflow it fails and always same error and nothing else:

" OSError: [Errno 22] Invalid argument: 'dbt_packages'

Command exited with return code 2 Execution of shell command failed "

I tried skipping macro and then it works fine but I need to run these macros. Tried to run just one basic macro and it still failed so I'm sure it's the execution of macro that is causing workflow to fail.

I'm expecting this to be some problem with how workflow Is execution notebooks but couldn't find anything that would help solve my issue.

topic Running dbt macro inside Databricks notebook as a part of Workflow in Data Engineering

Running dbt macro inside Databricks notebook as a part of Workflow