topic Re: dbt_project.yml won't load in databricks dbt job in Data Engineering

dbt_project.yml won't load in databricks dbt job

skooijman — Mon, 23 Jun 2025 15:10:07 GMT

We're running into issues with dbt jobs, which are not running anymore. The errors we receive suggest that the dbt_project.yml file cannot be found, while the profiles.yml can be found. We are running our dbt jobs with Databricks Workflows. We've tried to upgrade the clusters to a new LTS, the latest dbt-databricks version, change cluster policy (shared vs. single), manually inserting the project-dir in the dbt build command, but nothing seems to work. We've made no changes to the code-base. This happened to all our separate dbt projects at the same time.

Does anybody have any idea what is causing this?

This is our error message from the test environment, which was working yesterday:

Failed to load profile from dbt_project.yml, falling back to default profile `local_databricks_cluster` + dbt build --select +tag:daily +tag:monday +tag:23 +tag:working_day --vars '{"read_env":test}' --threads 4 08:56:42 Running with dbt=1.10.2 08:56:42 Encountered an error: Runtime Error No dbt_project.yml found at expected path /tmp/tmp-dbt-run-<job_id>/dbt_project.yml Verify that each entry within packages.yml (and their transitive dependencies) contains a file named dbt_project.yml

Re: dbt_project.yml won't load in databricks dbt job

analytics_eng — Mon, 23 Jun 2025 14:45:48 GMT

Yes I have the same problem. From today on dbt-databricks tasks fail and we didn't change anything. Is there something breaking in how dbt task are executed?

Re: dbt_project.yml won't load in databricks dbt job

Lahcen — Mon, 23 Jun 2025 17:24:09 GMT

We’re experiencing the same issue across all our environments (DEV, STAGING, PROD).

This began happening suddenly, without any changes to our code or environment.

All dbt jobs started failing with the same error:

Command 'dbt deps' returned non-zero exit status 2.
Failed to load profile from dbt_project.yml, falling back to default profile `local_databricks_cluster` + dbt deps 10:28:25 Running with dbt=1.10.2 10:28:25 Encountered an error: Runtime Error No dbt_project.yml found at expected path /tmp/tmp-dbt-run-1056643299291120/dbt_project.yml Verify that each entry within packages.yml (and their transitive dependencies) contains a file named dbt_project.yml

Workaround: We resolved it temporarily by explicitly setting the project path in our command:

dbt deps --project-dir /Workspace/Users/{{User}}/projectWorkspace

Would appreciate any updates from #Databricks or #dbt Labs on a permanent fix or root caus

Re: dbt_project.yml won't load in databricks dbt job

analytics_eng — Tue, 24 Jun 2025 07:41:23 GMT

Thanks for this. We also needed to change our profile settings in dbt_project.yml (and profiles.yml) from profile: "databricks_cluster" to profile: "local_databricks_cluster". It would be great if Databricks or dbt provided a response to this sudden and unexpected behavior

Re: dbt_project.yml won't load in databricks dbt job

LokmenChouaya — Wed, 25 Jun 2025 12:26:20 GMT

Hello is there any updates please regarding the issue? I'm having the same problem on my prod