Running dbt macro inside Databricks notebook as a part of Workflow

[JakubII]

So.. I have Databricks and I'm using dbt-databricks adapter to run dbt. Everything works fine but for some reasons I need to run some stuff inside of notebook. That still worked fine for some time but after upgrading to dbt-databricks 1.5.5 something broke and I can't figure it what exactly. It works when downgrading but I would like to work with newest versions.

Workload always fails when I'm calling out dbf macro (run-operation ..). Weird think is that notebook independently works fine even with macro but inside of workflow it fails and always same error and nothing else:

" OSError: [Errno 22] Invalid argument: 'dbt_packages'

Command exited with return code 2 Execution of shell command failed "

I tried skipping macro and then it works fine but I need to run these macros. Tried to run just one basic macro and it still failed so I'm sure it's the execution of macro that is causing workflow to fail.

I'm expecting this to be some problem with how workflow Is execution notebooks but couldn't find anything that would help solve my issue.

[Kaniz_Fatma]

Hi @JakubII, Based on the provided information, it seems like there is an issue with running macros inside a notebook using dbt-databricks adapter version 1.5.5.

The error message "OSError: [Errno 22] Invalid argument:’ dbt_packages’" suggests that there might be an issue with the path to the dbt packages. 

One possible solution could be to downgrade to a previous version of dbt-databricks adapter that worked fine before.

Another possible explanation could be checking the path to the dbt packages and ensuring it is correctly configured. 

It is also possible that there might be an issue with how the workflow is executing the notebooks.

To troubleshoot this issue, you can try the following:•

• Check the logs to see if additional error messages or stack traces might provide more information about the issue.
•  Try running the notebook independently to see if it works without issues.
• Check if any changes in the configuration or environment might be causing the issue.

 Sources:
- https://docs.databricks.com/notebooks/notebook-workflows.html
- https://docs.databricks.com/dev-tools/databricks-utils.html