We're running into issues with dbt jobs, which are not running anymore. The errors we receive suggest that the dbt_project.yml file cannot be found, while the profiles.yml can be found. We are running our dbt jobs with Databricks Workflows. We've tried to upgrade the clusters to a new LTS, the latest dbt-databricks version, change cluster policy (shared vs. single), manually inserting the project-dir in the dbt build command, but nothing seems to work. We've made no changes to the code-base. This happened to all our separate dbt projects at the same time.
Does anybody have any idea what is causing this?
This is our error message from the test environment, which was working yesterday:
Failed to load profile from dbt_project.yml, falling back to default profile `local_databricks_cluster` + dbt build --select +tag:daily +tag:monday +tag:23 +tag:working_day --vars '{"read_env":test}' --threads 4 08:56:42 Running with dbt=1.10.2 08:56:42 Encountered an error: Runtime Error No dbt_project.yml found at expected path /tmp/tmp-dbt-run-<job_id>/dbt_project.yml Verify that each entry within packages.yml (and their transitive dependencies) contains a file named dbt_project.yml
