cancel
Showing results for 
Search instead for 
Did you mean: 
Warehousing & Analytics
Engage in discussions on data warehousing, analytics, and BI solutions within the Databricks Community. Share insights, tips, and best practices for leveraging data for informed decision-making.
cancel
Showing results for 
Search instead for 
Did you mean: 

Warning in dbt task

martindlarsson
New Contributor III

We are using dbt core and running our transformations in a dbt task. Since a few months ago we have started to see a warning message in our runs. This  started happening after an update of dbt core and it seems that Databricks needs to update the definition of the generated prifiles file. Right now we are running version 1.8.7 of dbt core.

This is the warning:

+ dbt deps
12:48:49 Running with dbt=1.8.7
12:48:49 [WARNING]: Deprecated functionality User config should be moved from the 'config' key in profiles.yml to the 'flags' key in dbt_project.yml.
12:48:49 Installing brooklyn-data/dbt_artifacts

5 REPLIES 5

merry867
New Contributor

Hello,

The warning you’re encountering suggests that dbt has updated how user configurations should be structured. Specifically, some settings that were previously located under the config key in profiles.yml should now be moved to the flags key in dbt_project.yml.

Steps to Address the Warning
Understand the Change:

The config key in profiles.yml is deprecated for certain user-level configurations.
These configurations should now be placed in the flags key inside dbt_project.yml.
Inspect Your profiles.yml: Check your profiles.yml file for the config key. It might look something like this:

yaml
Copy code
your_profile_name:
target: dev
outputs:
dev:
type: databricks
host: your-databricks-host
token: your-token
schema: your-schema
config:
some_user_config: value
Move the Configuration:

Move any user-level configurations from the config section in profiles.yml to the flags section in dbt_project.yml.

Update your dbt_project.yml file like so:

yaml
Copy code
name: your_project_name
version: 1.0
config-version: 2

# Add the flags section
flags:
some_user_config: value
Verify Compatibility:

Ensure the moved configurations make sense in the context of dbt_project.yml.
Review the dbt 1.8.7 release notes or changelog for additional guidance on changes related to profiles.yml.
Test the Change:

Run dbt deps or a dbt task again to ensure the warning is resolved.
Additional Notes
If you rely on Databricks for your dbt transformations, ensure that:

The version of the Databricks adapter (dbt-databricks) is compatible with dbt-core 1.8.7.
Databricks may need to update their documentation or examples if they haven’t addressed these changes yet.
Best Regards
merry867

Looks like you had help from ChatGPT?

merry867
New Contributor

Hello,
I hope this info will helps you.
Best Regards
merry867
Digital E hall pass

bhanu_gautam
New Contributor III

@merry867 , Thanks for sharing , yes it looks like warning message indicates that user configuration should be moved from the config key in the profiles.yml file to the flags key in the dbt_project.yml file. 

Regards
Bhanu Gautam

Kudos are appreciated

martindlarsson
New Contributor III

Both of you missed the sentence "Databricks needs to update the definition of the generated profiles file."

The warning is regarding the structure of the profiles.yml which is generated by Databrick. This is not anything I as a user can change accept follow the guide to run with a custom profile which I do not have a need for.

So I wand to reach somebody working at Databricks and can give this information to the team that works on the dbt task.

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group